[Hybrid] Block priorities enabling opportunistic APC hits with sparse caching

[s3woz]

Purpose

Recently prefix_cache_retention_interval has been introduced (#43447 #45845), affecting hybrid models such as DeepSeekv4, KIMI K3, Qwen 3.5, etc. When prefix_cache_retention_interval is not None, cache entries with low matching probabilities are masked, skipped hash assignment, and evicted in FIFO manner rather than LIFO (LRU). Cache hits can only occur then at designated blocks and via specific mechanisms, such as Marconi shared prefix detection (#37898) that provides cache hits on 3rd appearance of shared prefix.

In this PR, we notice that the resulting masked blocks:

• have valid data that could potentially result in APC hits
• compute effort has been spent on them

Rather than completely discarding them, we enable opportunistic APC hits. To achieve this the PR:

• introduces Block priorities (currently NORMAL and LOW), where masked blocks are assigned LOW priority and still considered in the APC matching
• the eviction mechanism extends the free_blocks queue in the following manner:
  [hash-less blocks | LOW PRIORITY | ... | NORMAL PRIORITY]
• thus it retains the LIFO logic for normal blocks, and FIFO logic for masked blocks
• but if there is no memory pressure, the LOW priority blocks have a chance to be hit

A synthetic test with truncated shared prefixes that simulates:

• 2nd appearance of shared prefix (not captured by Marconi shared prefix detection yet)
• user roll-back

demonstrates valid cache hits for masked blocks and 35% prefill time reduction for prefix_cache_retention_interval=0 with Qwen/Qwen3.5-9B

AI assistance disclosure: development of the code in this PR (all tests, some parts of core logic) was assisted with AI.

@tdoublep

Test Plan

if __name__ == "__main__":
    from vllm import LLM, SamplingParams
    from vllm.distributed import cleanup_dist_env_and_memory
    import time, string
    # Should be even more visible for DeepSeekV4 and KIMI K3
    MODEL = "Qwen/Qwen3.5-9B"
    sampling_params = SamplingParams(temperature=0.0, max_tokens=1)
    prefix1 = ( # examples/offline_inference/prefix_caching.py
        "You are an expert school principal, skilled in effectively managing "
        "faculty and staff. Draft 10-15 questions for a potential first grade "
        "Head Teacher for my K-12, all-girls', independent school that emphasizes "
        "community, joyful discovery, and life-long learning. The candidate is "
        "coming in for a first-round panel interview for a 8th grade Math "
        "teaching role. They have 5 years of previous teaching experience "
        "as an assistant teacher at a co-ed, public school with experience "
        "in middle school math teaching. ")
    prefix2 = ("Based on these information, fulfill "
                "the following paragraph: ")
    PROMPTS = []
    for i in range(10):
        PROMPTS.append(str(i) + prefix1 * 200 + prefix2)
        # Simulate partial match, e.g.:
        #  - user roll-back in conversation
        #  - 2nd system prompt appearance
        PROMPTS.append(str(i) + prefix1 * 190 + prefix2)

    for RET in [None, 0]:
        engine = LLM(prefix_cache_retention_interval=RET,
            model=MODEL, gpu_memory_utilization=0.4, disable_log_stats=False)
        print(f"Block size: {engine.llm_engine.vllm_config.cache_config.block_size}")
        # Measure:
        start_time = time.time()
        for prompt in PROMPTS:
            outputs = engine.generate(prompt, sampling_params)
            n_prefill = len(outputs[0].prompt_token_ids)
            print(f"Prefill tokens: {n_prefill} | Generated text: {outputs[0].outputs[0].text!r}")
        total_time = time.time() - start_time
        # Summary
        print('Execution with RET:', RET, "took --- %s seconds ---" % total_time)
        for m in engine.llm_engine.get_metrics():
            if 'vllm:prompt_tokens_cached' in m.name:
                print(m.name, m.value)
        del engine
        cleanup_dist_env_and_memory()

Test Result

Main:

[...]
Execution with RET: None took --- 20.84550404548645 seconds ---
vllm:prompt_tokens_cached 158400
[...]
Execution with RET: 0 took --- 32.47444725036621 seconds ---
vllm:prompt_tokens_cached 0

This PR:

[...]
Execution with RET: None took --- 20.97832942008972 seconds ---
vllm:prompt_tokens_cached 158400
[...]
Execution with RET: 0 took --- 21.07378101348877 seconds ---
vllm:prompt_tokens_cached 158400

Executed on 1x A100.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @s3woz.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork