[KVConnector][Mooncake] Wire reset_cache cascade end-to-end

[aoshen02]

Summary

KVConnectorBase_V1.reset_cache (added in #27170) is currently a no-op for MooncakeStoreConnector. A caller hitting Scheduler.reset_prefix_cache(reset_connector=True) (or POST /reset_prefix_cache?reset_external=true) on a Mooncake engine gets the local prefix cache cleared but the external Mooncake master keeps all KV blocks computed against the previous weights. For RL post-training and other weight-update workflows this is a silent stale-cache correctness hole: the next request can read KV that was hashed against an old policy.

This PR implements the connector-side reset cascade.

Cascade

Scheduler.reset_prefix_cache(reset_connector=True)
  → Scheduler.reset_connector_cache()
    → MooncakeStoreConnector.reset_cache()        [new, SCHEDULER role]
      → MooncakeStoreScheduler.reset_store()      [new]
        → LookupKeyClient.reset() over ZMQ        [new]
          → LookupKeyServer typed dispatch        [new]
            → store_worker.store.remove_all(force=True)

Implementation notes

• The reset RPC reuses the existing scheduler↔worker-rank-0 ZMQ admin channel originally built for prefix lookups — no new sockets or handshake.
• The wire format is upgraded to a named-tag protocol instead of sentinel values aliasing the data field, mirroring how NixlConnector discriminates message types. Constants live in a new single-source-of-truth module vllm/distributed/kv_transfer/kv_connector/v1/mooncake/store/protocol.py. Future admin commands need only a new tag + new dispatch branch.
• MooncakeStoreConnector.reset_cache() is only meaningful on the SCHEDULER role; the WORKER role returns None (reset is driven from the scheduler-side ZMQ admin channel that lands on worker rank 0's LookupKeyServer).
• MooncakeStoreScheduler.reset_store() converts ZMQ RPC failure into a soft False return (rather than raising) to match the soft-dependency contract callers like verl use to keep training alive if Mooncake is unreachable.

Ordering caveat (documented in code)

The caller must ensure no in-flight Mooncake lookups/transfers when invoking reset. In RL workflows this holds naturally at the step boundary after weight updates + rollout drain. Outside that pattern the caller is responsible — the ZMQ socket is REQ/REP and serialises requests, but background KVCacheStoreSendingThread puts on each rank are independent of this admin channel. Documented inline at MooncakeStoreConnector.reset_cache, MooncakeStoreScheduler.reset_store, and LookupKeyClient.reset.

Why this is not a duplicate

• gh pr list --repo vllm-project/vllm --state open --search "MooncakeStoreConnector reset_cache" → empty
• gh pr list --repo vllm-project/vllm --state open --search "mooncake reset connector cache" → empty
• gh issue list --repo vllm-project/vllm --state open --search "MooncakeStoreConnector reset" → only the parent meta RFC [RFC]: Add Mooncake Store Connector for Shared KV Cache Reuse #38474 (Mooncake Store Connector overall), which this PR contributes to.

Related: #27170 introduced the connector-level reset_cache API and the reset_connector parameter on Scheduler.reset_prefix_cache. This PR is the Mooncake-specific implementation behind that API. #42693 (open) is a companion correctness fix for the no-connector branch of the same method — independent of this change.

Test plan

Added 8 new unit cases + 1 integration-shaped test in tests/v1/kv_connector/unit/test_mooncake_store_connector.py:

• test_reset_cache_scheduler_role_delegates_to_reset_store
• test_reset_cache_scheduler_role_propagates_failure
• test_reset_cache_worker_role_returns_none
• test_scheduler_reset_store_returns_client_reset_result
• test_scheduler_reset_store_handles_rpc_exception
• test_lookup_key_client_lookup_prepends_typed_tag
• test_lookup_key_client_reset_uses_typed_protocol
• test_protocol_tags_are_distinct_and_non_empty
• test_scheduler_reset_connector_cache_invokes_connector_reset (integration-shaped: mirrors Scheduler.reset_connector_cache call shape on a real SCHEDULER-role connector)

Commands run locally:

pre-commit run --files <all 5 changed paths>
# → ruff check, ruff format, mypy-3.10, SPDX, etc. all Passed

The full pytest invocation could not be executed in my local sandbox (the editable vllm install in the venv points at a different worktree whose precompiled CUDA flash-attention extensions don't match upstream main). The protocol module was sanity-checked standalone (tags non-empty + distinct, OK/ERR distinct). CI is the source of truth for the integration tests.

AI assistance

This patch was authored with assistance from Claude Opus 4.7 (1M context). Every changed line was reviewed by me end-to-end before opening this PR.

🤖 Generated with Claude Code

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[gemini-code-assist]

Code Review

This pull request introduces a reset_cache mechanism for the Mooncake store connector, which is essential for clearing the external store during RL weight updates. It establishes a formal ZMQ-based admin protocol between the scheduler and worker rank 0 to handle lookup and reset commands. Feedback identifies a critical need for robust error handling and input validation within the worker's request processing loop to prevent background thread termination and potential system hangs.

[gemini-code-assist]

The process_request loop lacks robust error handling and input validation. If all_frames is empty or if a LOOKUP_MSG is received without sufficient frames (e.g., missing the token_len or payload), an IndexError will be raised, causing the background thread to terminate. Since this thread is critical for prefix cache lookups and store resets, its failure would lead to a silent degradation of performance or hangs in the scheduler. Additionally, any exception during message decoding or processing (outside the RESET_MSG branch) will also kill the thread. The entire processing logic should be wrapped in a try-except block to ensure the thread remains alive and always sends a response to satisfy the ZMQ REQ/REP state machine.

        def process_request():
            while self.running:
                try:
                    all_frames = self.socket.recv_multipart(copy=False)
                    if not all_frames:
                        continue
                    msg_type = bytes(all_frames[0])

                    if msg_type == LOOKUP_MSG:
                        if len(all_frames) < 2:
                            raise ValueError("LOOKUP_MSG requires at least 2 frames")
                        token_len = int.from_bytes(all_frames[1], byteorder="big")
                        hash_frames = all_frames[2:]
                        hashes_str = self.decoder.decode(hash_frames)
                        result = self.store_worker.lookup(token_len, hashes_str)
                        self.socket.send(result.to_bytes(4, "big"))

                    elif msg_type == RESET_MSG:
                        try:
                            self.store_worker.store.remove_all(force=True)
                            logger.info(
                                "Mooncake store reset via remove_all(force=True) succeeded."
                            )
                            self.socket.send(RESP_OK)
                        except Exception as e:
                            logger.error("Mooncake remove_all failed: %s", e)
                            self.socket.send(RESP_ERR)

                    else:
                        logger.warning(
                            "LookupKeyServer received unknown msg_type: %r",
                            msg_type,
                        )
                        self.socket.send(RESP_ERR)
                except Exception as e:
                    logger.error("Error in LookupKeyServer: %s", e)
                    try:
                        # Ensure a response is sent to avoid hanging the client
                        self.socket.send(RESP_ERR)
                    except Exception:
                        pass

[ivanium]

maybe still logger.warning? when will we get into this path?

[aoshen02]

most of the time we will run into this path, as long as in rl, so I think debug is ok.

[ivanium]

nit: a bit verbose

[aoshen02]

fix

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @aoshen02.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[aoshen02]

CI failures on build 68061 are infrastructure / dependency issues, not PR regressions

All 8 failing jobs trace back to upstream HuggingFace transformers/Hub model-load failures, not to anything this PR touches. The diff is confined to:

• vllm/v1/core/sched/scheduler.py::reset_connector_cache — return True (no-op success) instead of False when no KV connector is configured.
• vllm/v1/engine/core.py::_reset_caches — add reset_connector: bool = True kwarg, threaded through to reset_prefix_cache.
• vllm/distributed/kv_transfer/kv_connector/v1/mooncake/store/* — Mooncake connector reset cascade (new code path; default-off for engines without a Mooncake connector).

None of the failing tests exercise those paths. Concrete evidence per job (from build 68061 logs):

Failing job	First / representative error
benchmarks-cli-test	pydantic ValueError: Unrecognized model in Qwen/Qwen3-0.6B. Should have a model_type key in its config.json
language-models-tests-standard	ValueError: Unrecognized model in meta-llama/Llama-3.2-1B-Instruct. Should have a model_type key in its config.json (2 failed)
model-runner-v2-examples	Same — meta-llama/Llama-3.2-1B-Instruct model_type missing in AutoConfig.from_pretrained
multi-modal-models-standard-4-other-plus-whisper	OSError: Can't load image processor for 'Qwen/Qwen2.5-VL-3B-Instruct' (HF Hub download failure)
rust-frontend-serve-slash-admin-coverage	pydantic ValidationError: Unrecognized model in Qwen/Qwen3-0.6B
v1-sample-plus-logits	20+ tests in v1/sample/test_logprobs.py all fail with Unrecognized model in meta-llama/Llama-3.2-1B-Instruct
entrypoints-integration-api-server-2	3 RuntimeError: Server exited unexpectedly errors — server can't boot because of the same HF model-load failure
distributed-dp-tests-4-gpus	test_run_eagle_dp[FLASH_ATTN] — Eagle DP, unrelated to KV connector
pytorch-compilation-unit-tests	Inductor autotuning compile failure inside triton_poi_fused_mul_0 (Triton/Inductor infra, unrelated)

All the model_type errors come from transformers/models/auto/configuration_auto.py:1528 from_pretrained — i.e., AutoConfig.from_pretrained(...) can't recognize the model. The same code paths pass on main (latest main-branch CI is green), so this is environment drift on the PR's CI image, not a regression.

Most likely causes (any one of these is sufficient):

1. transformers version pinned in the CI image is too old/new relative to the model configs cached on the runner.
2. HF Hub auth/rate-limit drop on the runner — partial config download produces an entry missing model_type.
3. CI image was rebuilt and no longer ships the local model snapshots, while the runner has no HF token to fetch them.

cc'ing the vLLM CI maintainers — could someone take a quick look at the CI image / HF auth on the H200 runners? Happy to retry once it's resolved; this PR is small and self-contained.

For reference: PR head is f365df1bd18f8072450bf5ba86660dada0a0b7d4; failing build is https://buildkite.com/vllm/ci/builds/68061.