Promote prefix_cache_retention_interval to an argument and change the default to 0 - #52216
Merged
Merged
Conversation
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
tlrmchlsmth
requested review from
ApostaC,
NickLucche,
WoosukKwon,
alexm-redhat,
heheda12345,
hmellor,
ivanium,
njhill,
orozery,
robertgshaw2-redhat,
xuechendi and
ywang96
as code owners
August 13, 2026 20:57
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
ivanium
approved these changes
Aug 13, 2026
ivanium
left a comment
ivanium
left a comment
Collaborator
There was a problem hiding this comment.
Briefly skimmed through and LGTM
Member
Author
|
/ci run |
|
✅ Triggered Buildkite CI #83809 for commit |
1 task
1 task
hmellor
approved these changes
Aug 14, 2026
Member
|
/ci retry |
|
✅ Queued 5 failed job(s) for retry in Buildkite CI #83809. |
Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Member
Author
|
/ci run |
|
✅ Triggered Buildkite CI #83959 for commit |
Member
Author
|
/ci run |
|
✅ Triggered Buildkite CI #84129 for commit |
tlrmchlsmth
enabled auto-merge (squash)
August 17, 2026 12:56
github-actions
Bot
added
the
ready
zyp2014
pushed a commit
to zyp2014/vllm
that referenced
this pull request
Aug 21, 2026
…he default to 0 (vllm-project#52216) Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Co-authored-by: OpenAI Codex <codex@openai.com>
wyettzeng
pushed a commit
to wyettzeng/vllm
that referenced
this pull request
Aug 21, 2026
…he default to 0 (vllm-project#52216) Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Wyett <wyettzeng@gmail.com>
zufangzhu
pushed a commit
to zufangzhu/vllm
that referenced
this pull request
Aug 24, 2026
…he default to 0 (vllm-project#52216) Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Zhu, Zufang <zufang.zhu@intel.com>
kamb-code
added a commit
to kamb-code/vllm
that referenced
this pull request
Aug 24, 2026
…GLE-reachable state `prefix_cache_retention_interval` defaults to 0 since vllm-project#52216: sparse Mamba retention hashes only the replay boundary and shared-prefix junctions (`MambaManager.reachable_block_mask`). Two consequences for the dense boundary stops introduced earlier in this PR, both found while @jschmied validated on a default install: 1. Stopping at every boundary splits the prefill for states the mask then discards. The split now stops only where a state will be hashed: every boundary under dense retention (`None`, or an interval at/below the block size), the next segment boundary under a positive interval, and only the kept boundaries under 0. 2. The retained replay boundary is `num_prompt_tokens - 1`, but under EAGLE/MTP the attention lookup drops its last matched block, so the kept state sat one block beyond what any speculative lookup can reach and an identical prompt only hit once a junction formed (third send). The hybrid coordinator now exposes that margin (`eagle_reach_margin`, the same rule its lookup applies: one hash block under fine-grained partial hits, otherwise one group block); `MambaManager` keeps the state at the eagle-reachable boundary too, and the split ends a chunk there so it exists. Measured on the scheduler+manager harness at retention 0 (main @ 8d6b183, 3-send identical protocol, 1,600-token align blocks): 7,292-token prompt carved [4800, 6400, 7292] instead of [1600, 3200, 4800, 6400, 7292], first hit on send 2 (4,800) instead of 3; 6,400-token prompt carved [3200, 4800, 6400], first hit on send 2 (3,200) instead of 3. Every dense-retention cell unchanged. Signed-off-by: Kam Basra <kameldipbasra@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jm6Q54FRX5t8uDLx2Jbrh5 Signed-off-by: Kam Basra <kameldipbasra@gmail.com>
This was referenced Aug 26, 2026
Closed
khushali9
pushed a commit
to khushali9/vllm
that referenced
this pull request
Aug 29, 2026
…he default to 0 (vllm-project#52216) Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: khushali9 <khushali.desai9@gmail.com>
1 task
Member
|
This let mamba prefix cache hit to 0 when using eagle |
ZJY0516
added a commit
to ZJY0516/vllm
that referenced
this pull request
Sep 7, 2026
…AGLE PR vllm-project#52216 changed the default of prefix_cache_retention_interval from None to 0. With sparse retention (0), only the latest replay boundary keeps a Mamba state checkpoint, and EAGLE additionally drops the tail block from prefix-cache hits, leaving Mamba checkpoints largely unreachable so prefix caching never hits for Mamba models with EAGLE-style speculative decoding. Keep the public argument unchanged (int | None, default 0) and only change how an unset value is resolved: EngineArgs now defaults the field to a private sentinel (or the deprecated VLLM_PREFIX_CACHE_RETENTION_INTERVAL env var when set) so that create_engine_config can tell an unset value apart from an explicit one and resolve it once the model and speculative-decoding configuration are known: dense (None) for Mamba models with EAGLE-style speculative decoding, 0 otherwise. Explicitly configured values (0 / N / None / env var) are always respected. Co-authored-by: Kimi Code Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
VLLM_PREFIX_CACHE_RETENTION_INTERVALwas added in #43447, and allows control over the frequency of caching blocks for SWA and SSM models.When setting this value to zero, vLLM retains only the latest replayable prompt boundary, and when setting it to a positive integer N, vLLM retains every N tokens.
This has become an extremely important setting especially for agentic workloads, so this PR upgrades it to an argument rather than an environment variable.
In addition, it now defaults to
0rather thanNone, so that we avoid caching useless blocks. Marconi-style cache retention guarantees that we still retain the system prompt.