[Dao007forever]

Overview

Start MooncakeStore's async prefix lookup from on_new_request(), when the scheduler adds a request, so the lookup can finish while the request waits in the scheduler's waiting queue. GPU KV allocation and transfer stay in the scheduler's existing path.

This uses the existing connector hook KVConnectorBase_V1.on_new_request(request). An earlier revision added a new hook that ran on the input thread. That needed a handle handoff to stay lock-free, and it bought little: the engine core thread and the input thread share the GIL, so the only thing the lookup needs to overlap is GPU execution. on_new_request runs on the engine core thread, so all lookup state stays on one thread and needs no locking.

on_new_request runs when the core thread handles the ADD:

• Today, that is in _process_input_queue between engine steps. With async scheduling (the default with the mp executor), the next batch is already running on the GPU at that point.
• With [Core] Handle new requests and aborts while waiting for model output #65466, ADDs are handled while the core thread waits for a step's output, so the lookup starts up to one step earlier.

Design

• MooncakeStoreConnector.on_new_request calls MooncakeStoreConnectorScheduler.start_lookup_prefetch. That calls LookupKeyClient.prefetch(), which submits the lookup to the existing single lookup executor and registers it in _lookups. The ZMQ socket is still used only by the executor thread.
• The scheduler's first get_num_new_matched_tokens probe goes through the existing lookup(), which reuses the registered lookup instead of submitting a second RPC.
• A completed async result is cached per request and reused by later probes until it is discarded or expires. Before this PR, the first read popped it.
• The lookup is discarded in update_state_after_alloc (consumed; a reschedule after a failed load looks up again), in request_finished / request_finished_all_groups, on preemption in build_connector_meta, and on reset() and close().
• Requests with skip_reading_external_kv, prompts shorter than the lookup alignment, or no block hashes are not prefetched.

Claims

• With lookup_async=True, enable_lookup=True, and a positive lookup_prefetch_limit, eligible requests submit their lookup when they are added. Off by default.
• lookup_prefetch_limit caps arrival-time lookups in flight. A slot is released when the RPC completes or is cancelled. Over the cap, the request is looked up when the scheduler first reaches it. Scheduler lookups are not capped.
• Arrival-time lookups create no LoadSpec or GPU allocation. The scheduler still computes the external tokens to load from its current local-prefix hit count.
• A completed async result older than lookup_prefetch_ttl_s (default 2 s, measured from when its RPC returned) is looked up again. So is a lookup whose token count or block-hash chain changed. A lookup still in flight never expires, so it is not resubmitted behind the backlog it is waiting on.
• MultiConnector already forwards on_new_request, so this also works when MooncakeStore is wrapped in one.

Validation

.venv/bin/python -m pytest --confcutdir=tests/v1/kv_connector/unit \
  tests/v1/kv_connector/unit/test_mooncake_store_scheduler.py \
  tests/v1/kv_connector/unit/test_mooncake_store_connector.py -q

141 passed (Linux aarch64, Python 3.12, at 86cc07f). New and updated cases cover:

• a lookup started by on_new_request and reused by the scheduler's probes with a single RPC, then dropped once allocated
• the in-flight cap, including slot release on completion and scheduler lookups bypassing the cap
• refreshing on a changed token count or hash chain, or an expired result, and not restarting a lookup that is still in flight
• skipping ineligible requests
• a finished request's queued lookup being cancelled before it runs, and discarding on preemption, reset and close

Not run: GPU serving, real Mooncake transfers, model evaluation, TTFT/throughput benchmarks.

Trade-offs

• An earlier lookup adds backend work for requests that later abort or hit entirely in the local cache, and the single lookup worker can queue that work ahead of later requests. The cap bounds this.
• The TTL can cause a second lookup after a long queue wait.
• Both defaults need measurement on real workloads.
• Cancelling a queued future works; an RPC that is already running cannot be interrupted.

Duplicate-work checks: searched open vllm-project/vllm PRs for Mooncake lookup, async/prefetch lookup, and on_new_request lookup. #48230 skips lookups for locally reused prefixes, which is a different change. #42321 (enqueue-time prefetch for LMCacheMPConnector via on_new_request) is the same idea for another connector; it was closed for inactivity, not rejected.

AI assistance: Codex and Claude Code assisted with implementation, source analysis, and validation.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI. CPU validation is complete; GPU serving/model evaluation and benchmarks remain pending.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

[mergify]

Documentation preview: https://vllm--61292.org.readthedocs.build/en/61292/

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Dao007forever.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[Sy0307]

Thanks for the prefetch work — it matches the head-of-line blocking we hit in production: the store lookup client is single-threaded with no RPC timeout, so one slow batch_is_exist (large key lists from long prompts × TP/PP/DCP rank namespaces) can defer every waiting prefill.

Upstream build_connector_meta already discards lookups for finished_req_ids/preempted_req_ids. Our #63932 only moves that discard into the request_finished callback (2 lines). The difference matters when a request is aborted or times out while its lookup is still queued: future.cancel() only works before the RPC starts, so discarding at finish drops the dead lookup before the single executor picks it up, instead of waiting for the next metadata batch — which never comes if the engine goes idle, leaving the dead lookup (and its future) occupying the only lookup slot until the next request arrives. That dead request's latency then lands on the next real request.

Would you be open to folding that two-line discard into this PR (with attribution), or should we close #63932 as low-value and keep the timeout work separate?

The above information was written by me and polished with AI assistance.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Dao007forever.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[Dao007forever]

Hey @Sy0307, I'll fold it in and let's do the RPC timeout as a separate PR.

[Alex-ai-future]

Thanks for the careful input-thread → scheduler handoff. The bounded prefetch and scheduler-owned adoption make the concurrency boundary easy to follow.

One small suggestion for the planned serving/TTFT validation: report the fraction of eligible requests whose prefetch has completed by their first scheduler probe, alongside the request wait time. That would directly show whether the change hides queue wait as intended, and how often the cap falls back to a scheduler-time lookup.

[njhill]

@Dao007forever I have opened #65466, please take a look. I think if we merge that first, this PR can be simplified quite a bit. In particular we should not longer need the new connector hook and request field, and can just initiate the lookup in the existing on_new_request().

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Dao007forever.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

Documentation preview: https://vllm--61292.org.readthedocs.build/en/61292/

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Dao007forever.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[njhill]

Thanks @Dao007forever

[Dao007forever]

/ci run

[github-actions]

✅ Triggered Buildkite CI #100095 for commit 2b1b48168a0a.