[Bugfix] Gate hybrid+connector divergent local-hit path on connector opt-in [necessary but NOT sufficient — predicted 0/38 refuted, see comment]

[malaiwah]

Closes #402.

Provenance

Derived directly from the reviewed, sha256-pinned patch file
patches/gg-vllm-hybrid-divergent-hit-gate.patch (sha256
5f9ad10bedff87c4f37ce61bfe9be5cc9ed84dce99d9d3685c54b0341529839f) in
malaiwah/qwen38-27b-exl3 — applied with git apply onto
dev/gilded-gnosis @ fa033bd4e, not retyped. That commit's scheduler.py is
byte-identical to the copy vendored in the pinned r34 serving image's rootfs
(verified by direct diff before this PR was opened), so this is exactly what
ships today.

Bug

See #402 for the full root cause and the four-arm ladder evidence
(receipts/lmcache-reuse-test.json): the hybrid per-group local prefix-cache
lookup in Scheduler.schedule() reports the full-attention group's hit as the
model-wide computed prefix whenever any KV connector is attached, and
relies on the connector to restore the lagging recurrent (Mamba/GDN) state at
that boundary. Only nixl's _apply_prefix_caching does that unconditionally.
LMCacheMPConnector does not, so generation resumes with an uninitialized or
stale recurrent state and is silently corrupted — HTTP 200, no crash — for
content inside the divergence window (measured: L0 no-connector control 0/38
failed vs L1cold/L2warm connector-attached 7/38, L3restart 38/38).

Fix

Gate the per-group hybrid lookup on the connector opting in:

if (
    self.connector is not None
    and getattr(self.connector, "supports_divergent_local_hybrid_hits", False)
    and self.has_mamba_layers
    and isinstance(self.kv_cache_manager.coordinator, HybridKVCacheCoordinator)
):
    ...

Connectors that don't set the flag (every connector on this branch today,
including LMCacheMPConnector) fall back to
KVCacheCoordinator.get_computed_blocks() — the reconciled min-across-groups
hit, which only resumes where every group's state, including the recurrent
one, actually exists. That is the same regime the clean L0 control ran under.

This mirrors upstream vLLM's own fix for this exact defect class —
vllm-project#48425 (commit 229e01e9e), which introduced
supports_divergent_local_hybrid_hits and this same gate. Our
dev/gilded-gnosis lineage predates that fix and carries neither the flag nor
the gate; a third party independently hit the identical bug on stock upstream
0.26.0 and cited vllm-project#48425 as their fix
(LMCache/LMCache#4247).

File touched: vllm/v1/core/sched/scheduler.py, one hunk at the
if request.num_computed_tokens == 0: block inside Scheduler.schedule()
(currently line 726 on dev/gilded-gnosis @ fa033bd4e; +23/-0, comment +
one getattr condition).

Status: CPU-verified only. GPU behavioral proof pending.

What is verified (CPU, this session):

• Patch applies cleanly to dev/gilded-gnosis @ fa033bd4e (git apply --check, then applied)
• Round-trips byte-identical against the reviewed patch file
• Patched vllm/v1/core/sched/scheduler.py py_compiles clean

What is NOT yet verified (explicitly, so this isn't mistaken for a proven fix):

• No GPU behavioral test has been run against this patch. The predicted result
  — the four-arm ladder in receipts/lmcache-reuse-test.json re-run with this
  gate applied turns L1cold/L2warm from 7/38 failing to 0/38 — is a prediction
  from the root-cause analysis, not a measurement.
• That GPU ladder re-run is planned separately (not part of this PR) and is
  the gate for treating this as behaviorally proven, not just CPU-clean.

Trigger condition / who this affects

Requires both: (a) a hybrid model (Mamba/GDN + full-attention layers) and
(b) an external KV connector attached. No currently running GG serving is
affected — the TB2.1 campaign's sr1 image and in-flight AIBoss passes use
native vLLM prefix caching only, no --kv-transfer-config, no connector
attached.

Tradeoff

A GG deployment running nixl P/D with a hybrid model loses the FA-hit
optimization until GG rebases the upstream capability-flag plumbing so nixl's
connector can set supports_divergent_local_hybrid_hits=True (nixl already
restores recurrent state unconditionally today, so it stays correct, just
un-opted-in). Not applicable to this campaign, which does not use nixl.

DCO

Single commit, signed off: Signed-off-by: Michel Belleau <michel.belleau@malaiwah.com>.

References

• Issue: [Bug] Hybrid (Mamba/GDN + attention) scheduler silently corrupts generation with any external KV connector attached; no connector = clean #402
• receipts/lmcache-fix.json, receipts/lmcache-reuse-test.json,
  patches/gg-vllm-hybrid-divergent-hit-gate.patch (sha256
  5f9ad10bedff87c4f37ce61bfe9be5cc9ed84dce99d9d3685c54b0341529839f), all in
  https://github.com/malaiwah/qwen38-27b-exl3
• Mirrors: [BugFix] Handle per-group prefix-hit divergence for hybrid models with KV connector vllm-project/vllm#48425

Summary by CodeRabbit

• Bug Fixes
  • Improved hybrid cache handling during request scheduling.
  • Prevents requests from resuming when required recurrent-state data is unavailable.
  • Uses safer, reconciled cache-hit behavior unless the connected cache explicitly supports independent group handling.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3e5ff00e-7adb-4528-b2d1-8d79422526f1

📥 Commits

Reviewing files that changed from the base of the PR and between fa033bd and 50be10c.

📒 Files selected for processing (1)

• vllm/v1/core/sched/scheduler.py

Included review availability: Your plan includes up to 1 review per rolling hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The scheduler now checks supports_divergent_local_hybrid_hits before using divergent per-group prefix-cache hits. Unsupported connectors use the reconciled minimum hit across KV groups.

Changes

Hybrid Prefix-Cache Scheduling

Layer / File(s)	Summary
Capability-gated cache-hit selection vllm/v1/core/sched/scheduler.py	The scheduler uses divergent per-group cache hits only when the KV connector advertises supports_divergent_local_hybrid_hits. Otherwise, it uses the reconciled minimum hit across KV groups.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 50be1

This localized scheduler change is merge-ready after normal checks and review; no actionable merge-blocking risk remains.

Possibly related issues

• local-inference-lab/vllm issue 402: The change directly gates divergent hybrid cache hits to address the reported scheduler corruption.

Possibly related PRs

• local-inference-lab/vllm#401: This PR extends the hybrid prefix-cache handling introduced there with explicit connector capability gating.

Suggested reviewers: njhill, voipmonitor

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main fix, but it includes extra commentary that reduces concision.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[malaiwah]

Correction from the author: this PR's predicted outcome is REFUTED by measurement

I predicted in this PR's body that the gate would take the LMCache corruption from 7/38 to 0/38.
It does not. On a 7-arm ladder (same frozen probe a96168f9, same prompts 9978404b, same
pre-registered thresholds, one variable per arm) it goes to 37/38 and 38/38:

arm	corrupted
L0 control — LMCache off, this overlay mounted	0/38
U1cold / U2warm — unpatched	7/38 / 7/38
L1cold / L2warm — this patch applied	37/38 / 38/38
L3restart over fresh L2 / over poisoned L2	38/38 / 38/38

Please do not read this PR as a corruption fix. It remains, I believe, correct scheduler code —
it removes a path that unconditionally treats the full-attention group's prefix-cache hit as the
model-wide computed prefix whenever any KV connector is attached — but it is necessary and
insufficient, and merging it alone makes the observable symptom worse.

The one-line reason, and why it is not "the patch broke it"

external_prefix_cache_hits delta is 0 across 88,760 queries with zero Retrieved lines on the
unpatched arms, versus 59,200 / 60,800 / 76,800 patched. Unpatched, LMCache was never actually
supplying bytes, so the bounded 7/38 was entirely scheduler-side. This gate closes that path, the
connector finally loads for the first time in the whole investigation — and the damage arrives from
what that exposes downstream.

What is downstream, now traced to a specific line rather than guessed

My first write-up blamed the retrieve path for not restoring Mamba/GDN state. That was wrong, and I
would rather correct it here than leave it standing: the build does carry the 48 GDN state pages in
every 1600-token chunk, and retrieve → preprocess_mamba → GDN is correct end to end in source. The
defect is store-side:

• GetStoreMetadata counts vLLM-APC-hit spans as storable (lmcache_mp_connector.py:372-374);
• the mamba block-table rows inside an APC span are the shared null block id 0 (zeroing covers
  attention specs only);
• nothing filters null ids — so when a request resumes from an APC hit that LMCache no longer holds
  (routine at the 600/300 s TTLs), the null page's stale bytes are stored as the boundary state under
  a valid key. Every later retrieve then restores garbage.

That also explains the otherwise puzzling ladder result that a poison-free L2 is unreachable: every
writer configuration is itself poisoning.

Second hole, and it is this repo's: acceptance is keyed to attention-chunk existence and never to state
validity (scheduler.py:745-753 assumes nixl by name).

Suggested disposition

Keep this PR as a scheduler-correctness change with the prediction struck, and pair it with a
store-side truncation at the first null mamba block id (~20 lines, no key/format/IPC change) before
either is presented as fixing corruption. A nixl-parity state-only push appears inexpressible without
per-group object keys (LMCache#3608).

Full evidence: 7 arms × 38 requests, all 266 scored rows, metric deltas and overlay provenance —
including the sentinel asserted in the loaded bytecode rather than the file, because the image ships
a stale .pyc — are published in
receipts/lmcache-l1-2x.json
and docs/46 §22. Our own operational verdict from the same data: we are not enabling LMCache on this
stack, with or without this patch.

[malaiwah]

Follow-up: my own mechanism attribution above was also wrong — measured, not argued

I posted earlier in this thread that the downstream defect was store-side (null mamba block ids stored
as boundary state under valid keys). A live three-arm reproduction on a single card has now superseded
that, and I would rather correct myself twice in one session than leave a wrong mechanism standing in a
thread maintainers may act on.

The dominant defect is fp8-KV transfer. With this PR applied, on one GPU, single writer, no
concurrency:

arm	KV dtype	outcome
cold store → APC-evict → retrieve	fp8	catastrophic corruption, garbage from token 1, divergence 3.52 — and no poisoning precondition required
same sequence	bf16	bit-clean, divergence 0.0000 — the 48 GDN state pages round-trip exactly
one partial-APC-hit request interleaved between store and retrieve	bf16	plausible-but-wrong text, divergence 0.71, reproduced twice

What that means for the three stories in play:

1. Retrieve-side GDN state restoration is fine — the bf16 arm is bitwise clean, which measures what
   my first comment merely asserted was broken.
2. The store-side null-block hole I described is source-true but did not fire here. The server ledger
   shows the store-under-miss precondition never triggered (a read TTL is not a since-write expiry), and a
   ~20-line store-side clamp I wrote did not fix the bf16 partial-hit case. I am archiving it as
   hygiene rather than proposing it as a fix.
3. A third, separate defect lives in the partial-retrieve path and is still unexplained.

Practical consequence worth stating plainly: any deployment serving fp8 KV — which is a common
choice, and is our qualified default — cannot get correct reuse from this connector today, independent of
this PR. Candidate root causes are the fake-view byte arithmetic for 1-byte dtypes and unregistered fp8
scale surfaces.

This PR's disposition is unchanged from my previous comment: a scheduler-correctness change whose
predicted 0/38 is refuted, necessary but not sufficient, and not a corruption fix. What changes is that
the remaining work is not the store-side patch I gestured at — it is fp8 transfer correctness first,
then the partial-retrieve path.

Three operational facts we hit that are not documented anywhere I can find, offered in case they save
someone a day: restarting the MP server kills the live engine with no reconnect; the connector
requires APC + mamba_cache_mode=align and refuses none, so the APC-hit regime is mandatory rather
than an edge case; and at bf16 the align block is 800, forcing --max-num-batched-tokens into
[800, 1599] (it is [1600, 3199] at the 1600-token block).

Five arms with server ledgers:
receipts/kernel-gap-lmcache-repro.json;
narrative and the sequence of withdrawn mechanisms in docs/46 §25.