[Mooncake] Add metrics for MooncakeStoreConnector operations

[Dao007forever]

Summary

Adds per-operation telemetry for the store-pool variant of the Mooncake KV connector — the sibling MooncakeConnector (P2P) already exposes MooncakeKVConnectorStats, but the store connector had nothing equivalent (TODO at vllm/distributed/kv_transfer/kv_connector/v1/mooncake/stats.py:15).

Every Mooncake RPC (save_exists, save_put, load_get, lookup_exists) now records duration + key count + byte count + status (ok / partial_failure / error) + failed-key count. These flow through the existing KVConnectorLogging pipeline to the engine logger, and through a new MooncakeStorePromMetrics (1 histogram + 4 counters, labelled by operation and status) to Prometheus.

What changed

• New vllm/distributed/kv_transfer/kv_connector/v1/mooncake/store/metrics.py — MooncakeStoreConnectorStats(KVConnectorStats) (per-op list[record] data model, aggregate / reduce for the logger) and MooncakeStorePromMetrics(KVConnectorPromMetrics).
• store/worker.py — KVTransferThread accepts an optional record_operation callback; send/recv threads wrap batch_is_exist / batch_put_from_multi_buffers / batch_get_into_multi_buffers with timing; MooncakeStoreWorker owns a locked stats accumulator, exposes get_kv_connector_stats() (swap-and-return), and times the batch_is_exist call in lookup().
• store/connector.py — adds get_kv_connector_stats() (instance) and the build_kv_connector_stats / build_prom_metrics classmethods that the metrics plumbing dispatches on.
• Tests: 5 new in test_mooncake_store_worker.py (send-thread + recv-thread metric capture, aggregate/reduce, worker swap-and-reset, lookup); 2 new in test_mooncake_store_connector.py (worker delegation, stats reconstruction from a dict).

Notes

• This repo's KVCacheStoreSendingThread calls batch_put_from_multi_buffers(keys, addrs, sizes, replicate_config) (4 positional args) — the metric wrap is around the existing call site unchanged.
• The recv thread has an existing VLLM_MOONCAKE_STORE_TIER_LOG branch that calls batch_get_replica_desc before the data fetch; the load_get timer starts after that diagnostic call so the metric measures only what the user actually waits for.
• Histogram buckets target Mooncake RPC latencies (1ms–4s) — sub-millisecond buckets dropped, longer-tail buckets added.

Why this is not a duplicate

Checked with:

gh pr list --repo vllm-project/vllm --state open --search "MooncakeStoreConnector in:body"
gh pr list --repo vllm-project/vllm --state open --search "MooncakeStorePromMetrics"
gh pr list --repo vllm-project/vllm --state open --search "mooncake_store_metrics"
gh issue list --repo vllm-project/vllm --state open --search "MooncakeStoreConnector metrics"

The 5 open MooncakeStoreConnector PRs (#42584 SupportsHMA, #42694 reset_cache, #42788 load failures, #43281 finish-after-preemption, #43371 save/load queue) are unrelated to telemetry. The mooncake/stats.py:15 TODO ("add MooncakePromMetrics ... in a follow-up PR") for the store connector is still open.

Test plan

• pre-commit run --files <5 changed files> — all hooks pass (ruff check, ruff format, mypy-local, SPDX, etc.).
• .venv/bin/python -m pytest tests/v1/kv_connector/unit/test_mooncake_store_worker.py tests/v1/kv_connector/unit/test_mooncake_store_connector.py — 52/52 in-scope tests pass.
  • 6 tests (test_requester_worker_init_*, test_topology_*) fail in my local env due to a pre-existing macOS sockaddr_un.sun_path 103-char limit in LookupKeyServer; unrelated to this change. These should run clean in CI / Linux.
• End-to-end on a real Mooncake deployment: scrape /metrics, confirm vllm:mooncake_store_operation_time_seconds, vllm:mooncake_store_operation_total, vllm:mooncake_store_operation_keys_total, vllm:mooncake_store_operation_bytes_total, vllm:mooncake_store_operation_failed_keys_total series appear with labels for save_exists, save_put, load_get, and lookup_exists.

AI assistance disclosure

Per AGENTS.md: this PR was produced with AI assistance (Claude). I (the submitter) have reviewed every changed line. The accompanying tests cover the new metric recording paths (success, partial failure, aggregate/reduce, and swap-and-reset semantics).

[gemini-code-assist]

Code Review

This pull request introduces per-operation telemetry for the MooncakeStoreConnector, enabling the tracking of duration, key/byte counts, and status for RPC operations such as save_exists, save_put, load_get, and lookup_exists. The implementation includes a new metrics module for data aggregation and Prometheus integration, instrumentation of worker threads, and comprehensive unit tests. Feedback identifies a potential AttributeError in the Prometheus observe method when stats are missing and a timing logic error in the receiving thread that could result in incorrect duration reporting during failures.

[njhill]

Thanks @Dao007forever