[Core] Preserve connector-visible Mamba MTP boundaries - #1
Draft
roboalchemist wants to merge 1 commit into
Draft
roboalchemist wants to merge 1 commit into
roboalchemist wants to merge 1 commit into
Conversation
Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Josh Fischer <jfischer@jfischer.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Keep the final complete Mamba/GDN boundary materialized when EAGLE-style MTP
prefill is connected to an external KV cache.
Without a connector, vLLM conservatively backs the last cache position off by
one block because full attention drops the last EAGLE match. With an external
connector, that backoff can let the final prefill step skip a complete recurrent
boundary. A connector may then persist the null table entry under a valid token
hash. A later external hit restores incorrect recurrent state.
This change retains the existing EAGLE backoff when no connector is present and
stops at the complete Mamba boundary when a connector is active.
Validation
tests/v1/core/test_mamba_align_chunk_split.py: 21 passed.mypy, SPDX, forbidden-import, and configuration checks.
Duplicate-work check
Related open work was reviewed before creating this personal-fork draft:
conservative behavior for connectors that do not opt in.
handoff and lifetime management.
exact-boundary backoff.
None is the exact connector-aware split used in the validated LMCache build.
However, vllm-project#50897 may provide the more general long-term capability model. Before
an upstream submission, this draft needs a human decision on whether the
condition should be an explicit connector capability rather than merely
connector is not None.AI assistance and review status
AI assistance was used to investigate, implement, test, and draft this change.
This is intentionally a personal-fork draft. The required human line-by-line
review and ownership have not yet been recorded, so it must not be submitted
upstream as-is.