[Store] Add Mooncake KV Event Publisher for Dynamo-compatible KV events

[magicYang1573]

Description

This PR adds the Mooncake KV Event Publisher: Mooncake can now emit
Dynamo-compatible BlockStored / BlockRemoved KV cache events so Dynamo's
router can build its prefix index from KV blocks stored in Mooncake.

There are two interfaces:

• Input (engine → Mooncake): ReplicateConfig is extended with group-level
  kv_event_metadata (KvBlockEventMetadata / KvBlockComponentSpec) that
  carries the semantic information only the engine knows — sequence
  block_hash / parent_block_hash, token_ids, block_size, routing
  namespace (model_name / lora_name / dp_rank), and the physical layout of
  the logical block (group_ids + expected_components). Python bindings are
  included. When this metadata is absent and no publisher is installed, Mooncake
  behavior is unchanged.
• Output (Mooncake → Dynamo): since Mooncake is a centralized KV cache
  manager, it publishes a single ZMQ PUB stream for all workers
  (ZmqKvEventPublisher) using the SGLang/vLLM-compatible 3-frame msgpack wire
  format. Each event carries an extra worker_id field indicating the
  physical node that holds the block, so one stream can serve all workers. A
  Mooncake-specific Dynamo adapter (separate repo) parses worker_id.

Internals:

• A group manifest in MasterService coalesces the physical objects of a
  logical block and publishes exactly one BlockStored on completion and one
  BlockRemoved on removal, with per-worker de-duplication.
• A segment→worker_id registry resolves worker_id from the owning
  segment's transport endpoint (captured at mount time), with an optional
  explicit override for deployment/control-plane use.

Design and interface details, plus the remaining SGLang/Dynamo follow-up work,
are documented in:

• docs/source/design/kv-event-dynamo/mooncake_kv_event_publisher.md
• docs/source/design/kv-event-dynamo/mooncake_kv_event_publisher.zh.md

Module

• Transfer Engine (mooncake-transfer-engine)
• Mooncake Store (mooncake-store)
• Mooncake EP (mooncake-ep)
• Mooncake PG (mooncake-pg)
• Integration (mooncake-integration)
• P2P Store (mooncake-p2p-store)
• Python Wheel (mooncake-wheel)
• Common (mooncake-common)
• Mooncake RL (mooncake-rl)
• CI/CD
• Docs
• Other

Type of Change

• Bug fix
• New feature
• Refactor
• Breaking change
• Documentation update
• Performance improvement
• Other

How Has This Been Tested?

Built and tested in a conda environment with cppzmq + zeromq installed (the
ZMQ publisher is auto-detected; without ZeroMQ the manifest/encoder still build
and the live ZMQ publisher is omitted).

Test commands:

# Build the store unit tests (ZeroMQ auto-detected via CMake)
cmake --build build --target master_service_test -j

# KV event publisher, segment->worker_id registry, and ZMQ transport tests
./build/mooncake-store/tests/master_service_test \
  --gtest_filter='*KvEvent*:*Zmq*:*SegmentWorker*:*WorkerId*'

# Full master service suite (regression)
./build/mooncake-store/tests/master_service_test

Test results:

• Unit tests pass
• Integration tests pass (if applicable)
• Manual testing done (describe below)

All 149 master_service_test cases pass, including the new tests: manifest
lifecycle (create/complete/share/conflict/erase), publishing & de-duplication of
BlockStored / BlockRemoved, worker_id resolution
(SegmentWorkerRegistryCapturesEndpointAtMount,
KvEventWorkerIdReflectsSegmentEndpoint, KvEventWorkerIdExplicitOverrideWins),
and a real ZMQ pub/sub roundtrip (ZmqKvEventPublisherRoundTrip) where a SUB
socket receives the 3 frames over TCP and decodes them back into the original
event.

Checklist

• I have performed a self-review of my own code
• I have formatted my code using ./scripts/code_format.sh
• I have run pre-commit run --all-files and all hooks pass
• I have updated the documentation (if applicable)
• I have added tests to prove my changes are effective
• For changes >500 LOC: I have filed an RFC issue

Note: the diff is large mostly due to tests and design docs; the new
public surface is small (kv_event_metadata on ReplicateConfig, the
ZmqKvEventPublisher, and the worker registry / publisher hooks on
MasterService). Happy to file an RFC if preferred.

AI Assistance Disclosure

• No AI tools were used
• AI tools were used (specify below)

AI tooling (Cursor) assisted with implementation, unit tests, and the design
documentation. The submitter has reviewed and is responsible for all changes.

[gemini-code-assist]

Code Review

This pull request introduces the Mooncake KV Event Publisher, enabling Mooncake to publish Dynamo-compatible KV cache events over ZMQ. The changes include design documentation, C++ models and msgpack serialization for KV events, a ZMQ publisher implementation, integration with the master service to track logical block group manifests, Python bindings, and extensive unit tests. The review feedback suggests clarifying the Python examples in both the English and Chinese design documents to explicitly recommend using None to leave the parent_block_hash field unset.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

The comment on this line is a bit confusing. It shows setting a value for parent_block_hash (for a non-root block), but the comment 'leave unset for the root block' could be misinterpreted, for example as setting it to 0.

Since the underlying C++ type is std::optional, the idiomatic Python equivalent for 'unset' is None. To improve clarity, it would be better to explicitly mention this.

Suggested change

	meta.parent_block_hash = 0x8f3a_0000_0000_0000 # leave unset for the root block
	meta.parent_block_hash = 0x8f3a_0000_0000_0000 # For a root block, use `None` to leave it unset.

[gemini-code-assist]

The comment on this line is a bit confusing. It shows setting a value for parent_block_hash (for a non-root block), but the comment '根 block 则不设置' (don't set for root block) could be misinterpreted, for example as setting it to 0.

Since the underlying C++ type is std::optional, the idiomatic Python equivalent for 'unset' is None. To improve clarity, it would be better to explicitly mention this.

Suggested change

	meta.parent_block_hash = 0x8f3a_0000_0000_0000 # 根 block 则不设置
	meta.parent_block_hash = 0x8f3a_0000_0000_0000 # 对于根 block，使用 `None` 来表示不设置。

[ishandhanani]

This looks like the right semantic follow-up to #2214. Coalescing physical objects behind group_id and emitting BlockStored only on incomplete → complete and BlockRemoved only on complete → incomplete gives consumers an authoritative logical-block lifecycle; replica/media churn should not produce logical removal while the group remains complete.

For Dynamo, I think these should feed the global shared-cache side index keyed by group_id, not the worker-rooted prefix index. worker_id identifies physical placement inside Mooncake, but it is not routing affinity when every SGLang worker can fetch from the same Mooncake pool. It can remain useful as topology/cost metadata without making presence worker-local.

The remaining event-contract follow-up is recovery:

• add a publisher epoch/stream ID; both frame sequence and event_id are process-local and can repeat after restart;
• provide a snapshot of complete groups at a sequence watermark, or replay from a cursor, so late/gapped subscribers can rebuild;
• persist group manifests and dedup state in the HA snapshot.

With that contract, Dynamo can use group_id directly and avoid rebuilding every TP/PP K/V physical key on each request.

[magicYang1573]

Closing this PR as superseded by #2214 and ai-dynamo/dynamo#11239. Mooncake #2214 has landed the production KV event publisher, and Dynamo #11239 now consumes the object-level events and maintains the shared-cache index with group_id verification, making the publisher/transport and object-level indexing portions of this PR redundant. If authoritative logical-group lifecycle is still needed, it should be submitted as a focused follow-up on top of #2214, reusing the current wire protocol and covering publisher epoch, snapshot/replay, and HA recovery.