[Bugfix][V1] Use the Mamba cache group's block size for align-mode chunk splitting

[wickist]

Purpose

Hybrid KV cache groups may have different token block sizes. cache_config.block_size is the minimum over all groups (set in v1/engine/core.py once the KV config is built) and represents a generic allocation geometry — it is not necessarily the Mamba recurrent-state block geometry. Using it in _mamba_block_aligned_split schedules chunk ends on a grid where the worker can never materialize a Mamba state.

Reproduction

Heterogeneous layout derived from a production deployment (Qwen3.8-27B hybrid + DFlash drafter, mamba align + prefix caching):

• target FullAttention block 1648, drafter attention block 816 (page-size matching: 16 × 51), MambaSpec.block_size 1648, hash unit 16, cache_config.block_size = min(...) = 816.

Old scheduler stops chunks on the 816 grid; the worker (postprocess_mamba) checkpoints a state only where a chunk ends exactly on the 1648 grid. 816k == 1648m only at the scheduler LCM (84048), so mid-prefill:

• no state ever materializes,
• the mamba group publishes no full-block prefix hash — or one claiming a boundary the slot never materialized (state-poisoning class from [Bug]: Accuracy drops ~20% when --enable-prefix-caching is used together with MTP speculative decoding (Qwen3.6 35B-A3B) #43559) —
• and a request repeating the same prompt immediately reuses nothing (hybrid reconciliation min = 0).

The new tests in tests/v1/core/test_mamba_align_chunk_split.py reproduce this with real modules on current main (attention 816 + mamba 1648): they fail before this patch and pass after.

Root Cause

_mamba_block_aligned_split used the generic cache block size where the Mamba state checkpoint geometry is required. The function's own invariant ("slot p holds the state after exactly (p + 1) * block_size tokens; state is written at chunk ends, so chunk ends must be block aligned") is defined on the mamba grid, but the grid came from the group minimum, which a finer drafter/attention group (or an explicit --block-size) drags below MambaSpec.block_size.

Fix

Derive the state grid once at scheduler init from the mamba group's spec (self.mamba_state_block_size; fails closed with a clear assertion if mamba groups ever disagree) and use it for the split. No hardcoded sizes, no model-name special cases. Equal-geometry behavior is byte-identical (mamba_state_block_size == cache_config.block_size there).

This is the scheduler-side instance of the same invariant fixed on the worker side by #53798 / #53398 (issue #53142: state_idx seeded with cache_config.block_size instead of the mamba group's block size).

Safety

• No rounding of arbitrary pending positions: a state is only published where a chunk ended on the mamba grid, i.e. exactly where the worker commits one (the new tests assert every hashed slot holds the state its hash claims).
• Equal-geometry behavior unchanged (control test included).
• No BlockPool/hash-semantics change; no manager or worker change.

Tests

• Unpatched current main: tests/v1/core/test_mamba_align_chunk_split.py — 2 new heterogeneous-layout tests FAIL (2 failed, 34 passed), including detection of a misaligned (poisoned) mamba hash publication.
• Patched: 36 passed; tests/v1/core/prefix_cache/ + tests/v1/core/test_prefix_caching.py: 140 passed (4 existing stub-based tests updated to mirror the new init-derived attribute).
• ruff format --check / ruff check: clean on changed files.
• tests/v1/core/test_scheduler.py could not run standalone in the sandbox (requires full conftest fixtures); verified identical failures with and without this patch (environmental).

E2E

The production stack that exposed this (vLLM 0.27.1 + the scheduler-split portion of this fix, identical invariant) measured, for an immediate re-ask of a 16,764-token prompt: prefix-cache reuse 0 → 16,480/16,480 legal tokens (100%), TTFT 8.18 s → 0.41 s (−95%), with greedy output parity (fresh vs cached continuation identical), no preemptions/Xid/asserts, and KV pool size unchanged. External/secondary evidence — included here only as directional confirmation, not as CI-tested claims on this tree.

Related work

• [Bugfix] Keep mamba align prefill chunks block-aligned past last_cache_position #51113 — keeps mamba align chunks block-aligned past last_cache_position (this PR's tests extend that file; that fix assumed cache_config.block_size == MambaSpec.block_size, which the heterogeneous layouts here violate).
• [Bugfix] Register hybrid prefix-cache boundaries at hittable positions #53802 — same function, different defect: n-vs-n-1 flooring and the eagle hash-shift of reachable tail boundaries. Complementary; both are needed for full immediate-reuse coverage.
• [Bugfix][V1] Mamba align: materialize a state at every boundary and drop the speculative one-block back-off #53479 — materializes a state at every crossed boundary and drops the speculative one-block back-off; builds on the same (generic) grid source, so this fix composes with it.
• [Bugfix] Seed align-mode Mamba state_idx in Mamba blocks #53798 / Fix hybrid mamba running state block size seeding #53398 (issue [Bug] Hybrid mamba align precopy: illegal memory access on prefix-cache resume with explicit --block-size (state column seeded with wrong block size) #53142) — the worker-side state_idx seeding with the wrong block size; same invariant, different site.
• [Spec Decode][Core] Run an all-sliding DFlash drafter with prefix caching enabled #50457 — DFlash drafter prefix-caching booking; the production geometry above arises with such drafter groups (this fix is independently valid without it).

AI assistance

AI assistance was used for investigation, test/patch drafting and running the qualification matrix; all changes and results were reviewed and validated by the submitter.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment /ci run for upstream CI or /amd-ci run for AMD CI only whenever CI signals are needed.

Once the PR is approved or has the ready label, the PR author can also use the corresponding /ci run, /ci retry, and /ci cancel commands, or their /amd-ci variants. New commits do not start upstream CI automatically.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[wickist]

Reviewer context: this is a distinct heterogeneous-block-geometry correctness fix, not a model-specific optimization. Current main reproduces the issue with cache_config.block_size = 816 and MambaSpec.block_size = 1648; the wrong grid can lead to a Mamba state being published under a boundary it does not physically represent. The added real-module regressions fail before the patch and pass after, while equal-geometry behavior remains unchanged. The production deployment that exposed the invariant also saw immediate prefix reuse recover from 0 to 100% on the primary workload, but the upstream justification here is correctness of Mamba state geometry. Review would be appreciated.

[tomylin890]

Independent confirmation of both the defect and the fix direction, from a second deployment.

Setup: a hybrid attention+mamba model (48 layers: full-attention every 4th, GDN/linear-attention elsewhere plus one conv layer), mamba_cache_mode="align", prefix caching on, chunked prefill with max_num_batched_tokens=2048. The resolved MambaSpec.block_size is 1568; cache_config.block_size ended up 4 — dragged down by the min-over-groups assignment in engine/core.py because the model also registers a small circular-buffer KV group. So _mamba_block_aligned_split was aligning to a 4-token grid, i.e. effectively never clipping.

Observed consequences match this PR's description exactly:

1. Interior boundary slots stay null. The runner writes one state column per step, so with the split inert the stored-boundary cadence collapses to ceil(P / max_num_batched_tokens) − 1 — measured 7/10 boundaries materialized for a 15,680-token prompt and 127/167 for 262,144 tokens, with the missing boundaries exactly the ones a later prefix-reuse lookup asks for.
2. Positional hash poisoning. With the wrong grid a slot holding state@2048 is published under the hash of state@3136 (cache_full_blocks hashes positionally). Any sibling request sharing ≥2 aligned blocks of prefix can be served a truncated recurrent state with no error — reachable in production with shared system prompts, no KV connector involved.

We applied a fix identical in spirit to this PR (derive the grid from the resolved MambaSpec block size out of kv_cache_config.kv_cache_groups, with a fail-closed assert when align-mode groups disagree) and the prefix-cache hit quantization law became exact again — floor((P−1)/1568)×1568, verified bit-for-bit at prompt lengths from 1,568 to ~900K tokens, including the previously-failing exact-multiple lengths.

One addition worth folding in (or taking from #53479): the boundary stop should be unconditional. next_block_boundary if start % block_size != 0 else 0 permits a multi-block advance whenever the start is aligned and max_num_batched_tokens > 2 × block_size; since the allocator hands out at most one real mamba block per running step, a chunk spanning k blocks leaves k−1 slots permanently null even with the grid fixed. We ship both changes together and the null-slot class is gone at every max_num_batched_tokens we tested (512–16384).

[wickist]

Independent confirmation from a second deployment — thank you, this is exactly the kind of evidence that helps.

Your addition is correct and now folded in (2ee5edd): the boundary stop is now unconditional — next_block_boundary fires even from an aligned start, so a full-budget chunk can no longer span k blocks and leave the k−1 interior state slots null. One deliberate exemption: steps using internal prefill checkpointing (use_internal_checkpoint with an aligned start) keep the multi-block advance, because that mode materializes the interior states itself.

Verification on this branch: new regression test_aligned_start_does_not_span_multiple_state_blocks walks a 5-block prompt with the full remaining budget each step — before the change it materializes only [6592] (your failure mode exactly), after it all of [1648, 3296, 4944, 6592, 8240]. The expected-chunk-sequence tests that encoded the old conditional cadence were updated to the per-boundary cadence. Targeted suites: 177 passed (chunk-split + prefix-cache + prefix-caching); ruff clean.

Your min-over-groups report (cache_config.block_size = 4 next to MambaSpec 1568) is also a cleaner natural repro of the heterogeneous-grid reachability than the drafter-group case in the PR description — the mechanism is identical (engine core min() over group block sizes).

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @wickist.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[jschmied]

Measured on GB10 with MTP n=5: this patch alone takes healthy-acceptance turns from 44 % (12 starts) to 15/16; numbers and method in #53142 (comment).

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @wickist.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[wickist]

Rebased onto current main (post-#53614); the PR is mergeable again. Notes on the conflict resolution and one test update, for reviewers:

Conflict resolution (semantics preserved, one reconciliation). #53614 broadened the internal-checkpoint exemption so checkpoint-mode chunks are exempt from boundary stops from both aligned and mid-block starts (that mode materializes interior states itself). Reconciled the stops entry as 0 if use_internal_checkpoint else next_block_boundary: the unconditional boundary stop for non-checkpoint chunks — the fix in this PR — is unchanged, and checkpoint-mode chunks keep #53614's exemption. Side effect worth noting: mamba_state_block_size (derived here from the mamba group's spec) now also feeds is_mamba_prefill_checkpoint_valid(mamba_block_size=...), which is the correct grid for that validity check.

Test updates on rebase.

1. The heterogeneous-layout stub in _hetero_split gained use_eagle_block_drop / mamba_prefill_checkpoint_alignment — [Kimi K3] Support internal prefix checkpoints with partial prefix caching and spec-decoding #53614 reads the former unconditionally via get_mamba_prefill_checkpoint_position, and the stub predates that call site.
2. test_disabling_eagle_block_drop_keeps_the_trailing_cache_boundary is now test_split_is_boundary_locked_independently_of_eagle_block_drop. With unconditional boundary stops, last_cache_position can no longer move a chunk end: it is a block multiple, so it is either ≤ start (not a stop) or ≥ the next boundary (dominated). The split is therefore Eagle-drop-independent in non-checkpoint mode — and the old expectation (an aligned start spanning to the trailing cache boundary) was exactly the k−1-null-interior-slots case this PR prevents. The back-off still matters for the partial-tail and checkpoint interactions.

Revalidated on the rebased branch: tests/v1/core/test_mamba_align_chunk_split.py + tests/v1/core/prefix_cache/test_partial_prefix_cache_hits.py + test_hybrid_cache_mamba_align_shared_prefix_detection (97 passed), and the full tests/v1/core/test_prefix_caching.py (100 passed).

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Team

Run ID: f011af31-2f76-4d73-abd8-f7169e8e792a

📥 Commits

Reviewing files that changed from the base of the PR and between 6865e67 and b3a88b8.

📒 Files selected for processing (4)

• tests/v1/core/prefix_cache/test_partial_prefix_cache_hits.py
• tests/v1/core/test_mamba_align_chunk_split.py
• tests/v1/core/test_prefix_caching.py
• vllm/v1/core/sched/scheduler.py

Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review.

---

📝 Summary

Summary by CodeRabbit

• Bug Fixes
  • Improved Mamba prefill scheduling so chunks stop consistently at Mamba state-block boundaries.
  • Ensured Mamba states are materialized at each boundary for immediate reuse.
  • Improved scheduling behavior when cache blocks and Mamba state blocks use different sizes.
  • Updated alignment behavior to remain consistent regardless of Eagle block-drop settings.

Walkthrough

The scheduler now derives Mamba state block size from Mamba KV-cache groups. Mamba-aligned splits use that size and stop at each crossed boundary. Tests cover heterogeneous block sizes, state publication, prefix reuse, and updated scheduling expectations.

Changes

Mamba alignment and prefix caching

Layer / File(s)	Summary
Scheduler Mamba alignment vllm/v1/core/sched/scheduler.py	The scheduler derives a single Mamba state block size from Mamba KV-cache groups. _mamba_block_aligned_split uses it for alignment and stops at the next boundary unless internal checkpointing is active.
Heterogeneous block-size validation tests/v1/core/test_mamba_align_chunk_split.py	Tests cover separate attention and Mamba block sizes, boundary-locked splits, Mamba state publication, immediate prefix reuse, and equal-size control behavior.
Existing alignment expectations tests/v1/core/prefix_cache/test_partial_prefix_cache_hits.py, tests/v1/core/test_prefix_caching.py	Scheduler mocks now define mamba_state_block_size. Expected chunk lengths reflect stopping at each Mamba boundary.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Merge Risk: ⚪ Minimal · up to b3a88

Mamba prefill chunks now stop on Mamba state boundaries even when KV-cache groups use different block sizes, preserving reusable state publication and prefix-cache reuse. The covered scheduling and cache behaviors present no remaining merge-blocking risk.

Sequence Diagram(s)

sequenceDiagram
  participant Request
  participant Scheduler
  participant KVCacheManager
  participant MambaKVCache
  Request->>Scheduler: request prefill chunk
  Scheduler->>KVCacheManager: allocate Mamba state slot
  KVCacheManager->>MambaKVCache: publish full-block state hash
  Request->>KVCacheManager: repeat same prompt
  KVCacheManager-->>Request: reuse aligned Mamba prefix

Loading

Suggested reviewers: zeldahuang

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 77.78% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 18 functions across 4 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description check	✅ Passed	The description clearly explains the heterogeneous Mamba block-size bug, the fix, safety considerations, related work, and test results.
Title check	✅ Passed	The title clearly and concisely identifies the bug fix: using the Mamba cache group's block size for V1 align-mode chunk splitting.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @wickist.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[jschmied]

Closing out the correction I promised here on 2026-09-03, and the answer is that I am withdrawing the
number rather than restating it.

What I posted on 09-02 — "this patch alone takes healthy-acceptance turns from 44 % to 15/16" — used
a healthy/broken per-turn metric that I later found measures the harness, not the patch. Our agent loop
sent ignore_eos: true, max_tokens: 130 while the model's real answer was 30–40 tokens, so every turn
continued past its end-of-turn token into one of two near-tie continuations: a chat-template restart,
which the drafter predicts at ~100 %, or a wall of <|im_start|>, which it never predicts. "Healthy" and
"broken" turns were those two filler modes. The metric is a property of ignore_eos, and it does not
measure the align defect.

Why I am not supplying a replacement effect size. I re-ran the grid EOS-correctly and have a clean
unpatched baseline — acceptance by MTP n = 60 / 58 / 46 / 40 / 37 / 29 / 26 % for n = 2..8, three
starts agreeing, and I have confirmed that arm was genuinely unpatched from the runner's own preflight
gate (it aborts if mamba_state_block_size is present in the scheduler, and it logged OK). But I have
no EOS-correct patched arm to pair it with. So there is no corrected before/after, and "the direction
stands" from my 09-03 note is not something that data supports either. Both figures are withdrawn.

What is unaffected, and is why I still think this PR is right: the defect is a code fact, not a
benchmark result. The align-mode split used the QSA ring capacity as its unit instead of the mamba
state block, so on this configuration every prefix-cache resume continued from a GDN state up to one
mamba block stale — silently, with no error. In an agent loop every turn is a resume. That argument
stands on the code and does not depend on any number I posted.

Offer. This PR has been blocked on conflicts twice in three days with no reviewer, and I would rather
hand a reviewer a real measurement than an anecdote. We have a GB10 (sm_121, TP=1) and the EOS-correct
harness. If you name the cell you want — acceptance, TTFT, or prefix-cache hit rate, patched vs
unpatched, however many starts you consider enough — we will run it and post it with the raw data,
whichever way it comes out.

Apologies for the six-day gap on a correction I said would follow immediately.

AI assistance was used in preparing this comment; the measurements are ours and were reviewed before posting.

[jschmied]

@wickist — mechanical rather than substantive, and it may be why this has sat: DCO has been failing
since the PR opened, so CI never gets past it.

None of the three commits carries a Signed-off-by: line:

9548d605  [Test] Reproduce mamba align split with heterogeneous KV block sizes
10e28b6c  [Bugfix][V1] Use the mamba cache group's block size for align-mode chunk splitting
244edeed  [Bugfix][V1] Stop mamba align chunks at every crossed state boundary

DCO shows action_required and pre-run-check fails, which skips Check format and pre-commit
behind them. git rebase --signoff main && git push --force should clear it.

Worth flagging because DCO surfaces as a check rather than as a review comment, and it is easy to miss
while resolving merge conflicts — which you have now done twice. Fourteen comments here and no human
review submitted yet; a red DCO is a plausible reason reviewers have not picked it up.

AI assistance was used in preparing this comment.

[wickist]

Controlled A/B update from our side (RTX 3090 TP2, Qwen3.8 hybrid GDN + DFlash2, mamba align mode): on v0.29.0 this fix is a no-op for our configuration — the KV-config interface normalization (attention block size raised to the mamba page size, interface.py:918/942) already guarantees cache_config.block_size == MambaSpec.block_size, so the heterogeneous geometry never arises and a semantic port of this PR produced zero measurable delta (prefix-cache token-hit counts identical across arms). However, the scheduler invariant this PR adds remains valuable as defense-in-depth for configurations where heterogeneous block geometry CAN still occur (explicit --block-size, other page layouts) — the A/B now also documents precisely where the fix is and is not exercised. No changes requested; sharing the measurement for reviewer context.

[k3dani]

Independent reproduction on a single DGX Spark (GB10, sm_121, ARM64)

We hit a deterministic, reproducible logit divergence that looks like it sits exactly in the
align-mode Mamba state path this PR touches. Posting the measurement in case it is useful as an
extra test case — we are not claiming this PR fixes it, we have not been able to run a patched arm
yet (see the note at the end).

Reproduced on two independent stacks. We ran the same probe on two images that share the checkpoint, the machine and the recipe's
two out-of-tree patches (the PLE mmap loader and patch_mamba_block_size.py), but nothing of the
engine build, and the result is identical:

	arm A	arm B
base	pinned preview build (0.1.dev20073+g8e685d198)	official vllm/vllm-openai:v0.29.0 (VLLM_BUILD_COMMIT=98dff2a81d74)
model package	vllm/models/qwen3_8_flash_next	vllm/models/qwen4_exp
QSA top-k path	exact torch.topk fallback	deterministic CUDA kernel (#55122), QSADET active in the log
determinism probe (4 items × 10)	3/4 pass, one item diverges	3/4 pass, the same item, the same shape

Setup (all pinned, arm A figures unless noted):

hardware	NVIDIA DGX Spark, GB10 (torch.cuda.get_device_capability = (12, 1)), ARM64, 128 GB unified
driver / CUDA	580.173.02 / 13.0
vLLM	0.1.dev20073+g8e685d198 (preview image, model package vllm/models/qwen3_8_flash_next)
torch / flashinfer	2.13.0+cu130 / 0.6.17
model	RadixArk/Qwen3.8-Flash-Next-NVFP4, snapshot 7b719225242aacd3dbd3f9407468c2ee9a9d2594
serving	--max-model-len 262144 --max-num-seqs 4 --gpu-memory-utilization 0.78 --enable-prefix-caching --enable-chunked-prefill --max-num-batched-tokens 8192, cudagraph_mode=PIECEWISE, MTP=2, --kv-cache-dtype auto
sampling	temperature=0, top_p=1, max_tokens=48, logprobs=true, top_logprobs=20, strictly serial requests
MoE backend	FLASHINFER_CUTLASS
QSA top-k	exact torch.topk fallback enabled, so the known nondeterministic persistent_topk
path (#51782 / #55122) is not in play

The engine selects align mode automatically:

INFO [config.py:605] Mamba cache mode is set to 'align' for Qwen4ExpForConditionalGeneration
                     by default when prefix caching is enabled

The experiment. Two requests share a long identical prefix (the same document, ~3.2k prompt
tokens) and differ only in the task suffix. We compare a SHA-256 over the full per-token top-20
logprob list — text equality is not used as evidence.

arm (each on a fresh server start)	sequence	result
clean	B × 10	fdc948fbf8f7698f × 10 — stable
contaminated	A × 1, then B × 10	B#1 = fdc948fbf8f7698f, B#2..10 = dabe443eeea90df5

So:

1. Request B on its own is perfectly deterministic across 10 runs, including its own full
   prefix-cache hits (runs 2..10 of the clean arm).
2. In the contaminated arm, B#1 — which gets a partial hit on blocks written by A and
   prefills its own suffix — reproduces the clean-arm hash bit for bit.
3. B#2..10 — full prefix-cache hits on blocks that were first written by the differently
   sized request A — diverge, starting at token 0, and then stay stable on the new value.

In other words the divergence is not "cold vs. cached"; it is whose request wrote the shared
prefix blocks. That is what made this hard to spot: within a single client's repeated traffic the
output looks perfectly stable.

Control. Same contaminated sequence with --no-enable-prefix-caching on a fresh server:
c1dc6688b1004f2f × 10 — the divergence disappears. (The absolute hash necessarily differs there:
without prefix caching the engine does not switch to align Mamba mode at all, so a different
numeric path runs. The metric is stability, not the absolute value.)

We can produce it on demand — up to a point. The log reports the cache block size:

INFO [interface.py:915] Setting attention block size to 1600 tokens to ensure that
                        attention page size is >= mamba page size.

Across the pairs we measured, the failing one is the only one where the two requests seal a
different number of full blocks (3169 → 1, 3227 → 2; the passing pairs are 1/1 and 15/15). So we
took a document that had never been used on that server start and sized the two tails to land on
either side of a block boundary:

prefix	A prompt	B prompt	full blocks	result
synthetic filler, 1 680 tok	3 091	3 308	1 vs 2	stable
real document, 3 002 tok	3 086	3 267	1 vs 2	diverges (same shape)
same real document	3 307	3 371	2 vs 2	stable
a different real document, 3 696 tok — same server start as the row above	4 689	4 870	2 vs 3	stable
arm B (v0.29.0 + det kernel), real doc 3 002 tok	3 086	3 267	1 vs 2	diverges
arm B, control, same server start as the row above	3 264	3 346	2 vs 2	stable

So crossing a block boundary appears necessary but not sufficient: it reproduced the effect on a
fresh document, but a different document with the same block-count asymmetry stayed stable, and
synthetic low-entropy filler never triggered it. Content or length matters as well — the QSA sparse
block selection is our leading suspect for the missing factor, and we have not separated it out yet.

Why we care: our workload is Hungarian document extraction, where logit-level divergence has
already cost us once: in our first round, before the top-k kernel fix, run-to-run logit drift moved
extracted dates and amounts on 5 of 50 items. In the probe above the visible answer text stayed
identical (text variants: 1), which is exactly why we hash logprobs rather than text.

Important: the equivalent of both fixes is already active in the image we measured. The GB10
recipe we run (blazux/qwen3.8-Flash-DGX @ c578815)
carries a two-line out-of-tree patch that touches exactly the two files these PRs do:

mamba_hybrid.py   (new_req_data.num_computed_tokens - 1)
                  // (self.cache_config.mamba_block_size or self.cache_config.block_size)
scheduler.py      block_size = self.block_size  # scheduler block size (LCM of groups) == mamba block size

(The patch's own header describes the bug it fixes: EngineCore overwrites cache_config.block_size
with the minimum group block size — the QSA raw-key ring's 8/16 — while the align-mode state-slot
seed and the block-aligned split want the mamba block size, 1600; the result was an all-zero
restored state on every prefix hit.)

We verified both lines are present in the image under test. So what we are reporting is not the
already-known all-zero-state failure: that one is handled, and a subtler cross-request difference
remains. Your PRs do considerably more than those two lines (#54076 is +287/−17 across four files,
#53798 +64/−23 across six), so they may well cover the remaining case too — we simply cannot tell from here.

What this is not. We could not run an arm with this PR (in its full form) applied. Our image is the pinned
GB10 recipe (model package still named qwen3_8_flash_next), and vllm/vllm-openai:v0.29.0 — the
other realistic base for this hardware — also predates the current main layout. A patched arm
needs a full main build on ARM64, which for this NVFP4 checkpoint additionally depends on
#55334 and a PLE offload path (#54129 or an out-of-tree mmap loader).

Raw probe output, tooling, the exact launch commands and the full write-up:
https://github.com/k3net/docai-evals/tree/b1f14a36c5bcc02cf2cd65705031e676fa9cb73f/experiments/2026-09-12-qwen38-flash-next-prefix-cache-cross-request-gb10
(the arms in this comment are results/round3-A-t201-*.json and results/round3-B-*)

Long-form write-up: https://docai.hu/blog/prefix-cache-megvaltoztatja-a-valaszt

[k3dani]

Follow-up to my September 12 reproduction: we now have a GPU control/patched experiment on DGX Spark (GB10, ARM64). A narrow backport of this PR's boundary-stop condition removes the within-start logprob divergence on the measured cases.

This also corrects my earlier statement that the recipe already carried the equivalent of both fixes: it fixes the block-size selection and worker seed, but does not include the boundary stop from this PR.

Setup: the same recipe-built v0.29.0 image as our previous arm B, RadixArk/Qwen3.8-Flash-Next-NVFP4, deterministic QSA top-k active, MTP=2, align-mode block size 1600, chunked prefill budget 8192, PIECEWISE, serial greedy requests. Both arms add --enable-prompt-tokens-details. The experimental scheduler change is only:

-            next_block_boundary if start % block_size != 0 else 0,
+            0 if use_internal_checkpoint else next_block_boundary,

We have not tested the full four-file PR.

The source probe, extracted from the running image, predicts a threshold at 2 × block_size = 3200 under these MTP settings: below it, the old split can run across the 1600 boundary without stopping. Whether A reaches a checkpoint boundary explains all eight previously measured pairs, including the 4689/4870 pair that contradicted our earlier block-count heuristic.

GPU threshold matrix: A once, then B four times, with a shared document prefix and separate cache salts/filler seeds per cell. PASS means one digest over the per-generated-token top-20 logprob lists within that server start; canonicalized digests agree with the verdict.

A prompt tokens	B prompt tokens	Control	Boundary-stop backport	B cached tokens, both arms
3196	3259	FAIL	PASS	0, 1600, 1600, 1600
3205	3259	PASS	PASS	0, 1600, 1600, 1600
3097	3160	PASS	PASS	0, 0, 0, 0
3205	3160	PASS	PASS	0, 0, 0, 0

Two corrections/clarifications to the earlier report:

• B#1 is a zero hit, not the partial hit I previously inferred. The failing cell diverges when the reported hit becomes 1600 tokens on B#2.
• This is numerical, not tied-candidate reordering. For B#1 vs B#2 in the failing control cell, the first three positions have 19/20, 19/20 and 18/20 shared candidates; every shared candidate changes, with maximum absolute logprob deltas 0.74515, 0.66381 and 1.06244. The canonical digest also changes. Argmax and visible text remain unchanged in this probe.

The original, unchanged T3-01 → T2-01 reproducer (3169/3227 tokens) also passes on the patched arm: 10/10 identical logprob digests, after failing across our earlier rounds on two stacks.

Limits: the matrix uses different filler seeds per cell and arm, so it is not a literal “only nine tokens changed” A/B; the unchanged original reproducer is the stronger follow-up. Absolute hashes are not compared across server starts. Cache-hit counts remain unchanged after the patch, so the results do not by themselves prove cache provenance or tensor equality. Chunk-shape-dependent numerics are a plausible mechanism; the first differing operation still needs a tensor trace. We have not measured the backport's prefill cost, rerun the 50-item quality suite, or adopted it in production.

Scripts, raw results and experiment report.

Updated write-up: English · Hungarian.

This provides an independent numerical-stability case for the boundary-stop part of the PR, even where correcting the block-size selection alone is insufficient.