fix(hybrid): preserve cache reuse and decode under prefill pressure

[logprobz]

Purpose

Preserve hybrid prefix-cache reuse and active decode progress when many long prefills compete with decode. This draft consolidates the source behavior validated in the LP15 high-concurrency investigation onto the current dev/jovian-judgement base.

This is intentionally a draft integration PR. It keeps the existing focused PRs open while maintainers decide whether to review the combined behavior or split it by concern.

Root causes

The incident required fixes in two interacting paths:

1. A fixed mixed-prefill quantum still ran too frequently under active decode. It protected one scheduler step but could repeatedly fragment prefill and collapse job goodput.
2. Hybrid cache groups did not retain and reconcile the same replay boundary under DFlash, Mamba, EAGLE, and DCP. Aggregate counters hid which group limited the common hit, and repeated scheduler lookups could inflate new diagnostics before admission.

The r15 rebase exposed two more shared-path details. Full-attention cache lookup must use the coordinator DCP width, while replicated or non-full groups use width one. Sparse retention must align its primary mask to prefix_cache_alignment_tokens, not the larger scheduler page.

Changes

Decode and prefill scheduling

• Add zero-default adaptive mixed-prefill budgeting and bounded rotation.
• Limit active partial prefills without changing the disabled behavior.
• Add a zero-default decode-burst controller with an oldest-waiter fairness deadline.
• Preserve full prefill budget when no decode is eligible.

Hybrid prefix cache

• Retain DFlash and Mamba replay states reachable after EAGLE drops a block.
• Align sparse retention and lookup to the common prefix-cache boundary.
• Reconcile per-group hits with the correct DCP width.
• Record cacheable tokens, reconciled hits, per-group hits, registrations, and evictions only at admission.
• Keep metric labels bounded by configured cache groups and emit no request content or identifiers.

DCP correctness

• Mask ranks with no local sequences before DCP reduction.
• Gather sparse MLA queries when DCP is active without PCP.
• Add a CPU regression around the sparse MLA query-gather helper.

Related PRs

• Fix prefill cadence for non-DP engines #546 is merged and is not duplicated here.
• fix(dflash): restore aligned draft cache geometry #529 covers aligned DFlash cache geometry and replicated-DCP handling. The current base and r15 already contain newer equivalents, so this draft includes only missing common-boundary behavior.
• [BugFix] Mamba sparse retention keeps a state below each boundary under EAGLE/MTP #556 supplies the Mamba EAGLE retention work with original authorship preserved.
• fix(dcp): handle empty ranks and gather sparse MLA queries #560 supplies empty-rank masking and sparse MLA gathering with original authorship preserved.

This draft does not close or modify those PRs.

r15 baseline

Validation uses the immutable image:

voipmonitor/vllm@sha256:d9ca4c299fdb8e6176d70f36c8f98a687474eef807b7a71fe6fee9f154208fe0

Its source lock names dev/jovian-judgement@9c4dd05487629eccb26d7166459867a3db9b099f, which is also this PR's Git base. The image's exact composition commit is not published to either GitHub fork, so it is used as the installed-source and dependency baseline rather than as an unreproducible Git parent.

Exact draft head and tree:

commit 8dbf46366e1934d80fa6ff84c4abfda1c154ef1a
tree   7a7e08789b4c7211cdd650baf62cd63610165f3a

Validation

Pinned r15 CPU-only container, networking disabled, no GPU devices:

25 passed, 256 deselected, 1 xfailed

The xfail is the optional fine-grained resume checkpoint that this branch does not claim to implement. Added regressions cover decode-burst fairness, partial-prefill bounds, DFlash and Mamba replay boundaries, DCP1 and DCP4 group reconciliation, admission-only metrics, empty DCP ranks, and sparse MLA query gathering.

Changed-file pre-commit 4.3.0 passed Ruff, formatting, mypy, typos, SPDX, forbidden imports, configuration defaults, CUDA API guards, and all applicable repository hooks. git diff --check and the source-only scope check also passed.

A separate source-locked LP15 image completed a matched 20-agent DCP1 screen and reverse-order repeat against the prior LP8 runtime. It delivered 2.304x and 2.388x job goodput, with p99 ITL at 37.3% and 36.8% of baseline. That evidence motivated this port, but it is not an r15 GPU qualification claim.

Compatibility and gaps

• All new scheduler controls default to zero.
• No public default, sampling behavior, model output, KV dtype, or backend selection changes when the controls are disabled.
• The metrics have fixed names and bounded cache-group labels.
• TP1, TP2, TP8, ROCm, live DSpark, and r15 GPU serving were not tested here.
• DCP4 high-concurrency promotion remains unclaimed.
• The full repository suite was not run in the network-disabled image because unrelated tests require model downloads. The complete PR-added regression set passed.

AI assistance

OpenAI Codex assisted with investigation, source reconstruction, tests, and this draft description. The human submitter reviewed the source and accepts the recorded validation and limitations.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[voipmonitor]

#583 is superseded by focused, independently reviewable pull requests:

• feat(scheduler): bound mixed prefill work under active decode #596 contains the optional mixed-prefill budget, rotating prefill quanta,
  partial-prefill bound, decode-burst fairness policy, and its scheduler
  metrics. All scheduler controls default to zero.
• [BugFix] Mamba sparse retention keeps a state below each boundary under EAGLE/MTP #556 contains the local Mamba/GDN replay-state retention correction for
  EAGLE/MTP. Its full prefix-cache test module passes 95/95, and the commits
  apply cleanly on integration(glm53-flash): complete NVFP4 and prefix-cache stack #589.
• fix(dcp): handle empty ranks and gather sparse MLA queries #560 contains the empty-rank masking and sparse MLA query-gather corrections
  for DCP.
• integration(glm53-flash): complete NVFP4 and prefix-cache stack #589 contains the GLM-5.3 DFlash cache classification and cache geometry.
  DFlash and DSpark preserve target KV cache, so they must not receive the
  target-tail drop used by EAGLE/EAGLE3/MTP.

The prefix-hit correction in #556 is a general local vLLM automatic-prefix-
cache invariant; it is not specific to LMCache. A hybrid request can reuse only
the token boundary available in every cache group. Under EAGLE/MTP, the
full-attention lookup discards one volatile lookahead alignment unit. Sparse
Mamba/GDN retention previously kept only the state at the unadjusted boundary,
so no recurrent state existed at the lower boundary offered during group
reconciliation. Retaining the predecessor state restores a common local hit.
An external KV connector can restore such a state, but it does not cause the
local retention defect.

The hybrid-cache diagnostic code from #583 is not being published as a
replacement PR. It evaluates every cache group a second time on the admission
path and mixes diagnostic bookkeeping with replay-boundary ownership. That
overhead and coupling require a separate design before the metrics are suitable
for production.

The DFlash-specific replay commits from #583 are also excluded. They were based
on treating DFlash as if it polluted target KV with EAGLE lookahead; #589
provides the narrower capability check required by the actual DFlash contract.

Original authorship is retained in #556, #560, #589, and #596.