Skip to content

[Core] Preserve connector-visible Mamba MTP boundaries - #1

Draft
roboalchemist wants to merge 1 commit into
mainfrom
fix/mamba-connector-mtp-boundary
Draft

roboalchemist wants to merge 1 commit into
mainfrom
fix/mamba-connector-mtp-boundary

Conversation

@roboalchemist

Copy link
Copy Markdown
Owner

Purpose

Keep the final complete Mamba/GDN boundary materialized when EAGLE-style MTP
prefill is connected to an external KV cache.

Without a connector, vLLM conservatively backs the last cache position off by
one block because full attention drops the last EAGLE match. With an external
connector, that backoff can let the final prefill step skip a complete recurrent
boundary. A connector may then persist the null table entry under a valid token
hash. A later external hit restores incorrect recurrent state.

This change retains the existing EAGLE backoff when no connector is present and
stops at the complete Mamba boundary when a connector is active.

Validation

  • tests/v1/core/test_mamba_align_chunk_split.py: 21 passed.
  • All pre-commit hooks applicable to the changed files passed, including Ruff,
    mypy, SPDX, forbidden-import, and configuration checks.
  • Live model gate on Qwen3.8-27B W4A16 AWQ with MTP3 and LMCache:
    • 9,600-token external prefix restore;
    • cold/hot output SHA-256 values identical;
    • three facts planted near tokens 400, 4,700, and 9,100 all recalled;
    • two native L1 restores: 3.916 and 4.118 seconds;
    • two raw NVMe L2 restores: 4.193 and 4.129 seconds.

Duplicate-work check

Related open work was reviewed before creating this personal-fork draft:

None is the exact connector-aware split used in the validated LMCache build.
However, vllm-project#50897 may provide the more general long-term capability model. Before
an upstream submission, this draft needs a human decision on whether the
condition should be an explicit connector capability rather than merely
connector is not None.

AI assistance and review status

AI assistance was used to investigate, implement, test, and draft this change.
This is intentionally a personal-fork draft. The required human line-by-line
review and ownership have not yet been recorded, so it must not be submitted
upstream as-is.

Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: Josh Fischer <jfischer@jfischer.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant