[kahalon]

Summary

A KV hint envelope can carry several actions; the format is shared by the framework RFCs (TensorRT-LLM #18153, vLLM #59708, SGLang #36224). submit_hint used only the first kv.fetch action, ignored the rest, and raised ValueError when that action was missing or invalid.

This PR adds a hint dispatcher that treats an envelope as a batch of independent hints. Every hint is parsed, routed to the handler for its type, and gets its own result. It is hint-agnostic infrastructure: kv.fetch is still the only hint type and keeps its behavior. Prefetch into KVCR DRAM, kv.invalidate and kv.evict are meant to plug into the same path.

Changes

• New kvcr.hints package, replacing kvcr/hint_parser.py:
  • dispatcher.py: HintDispatcher and the fixed _HINT_TYPES table. Each HintType pairs a parser with a handler factory, so a supported type always has both. Handlers run inline on the caller's thread, without the state lock.
  • parser.py: parse_hints returns every valid hint in envelope order, plus a REJECTED result for each hint it can't use. Only a malformed envelope raises.
  • fetch.py: the kv.fetch parser and handler. Each hint is one call to the existing per-request hint store.
  • results.py: HintStatus (ACCEPTED / REJECTED) and HintResult.
• core.py holds one dispatcher. submit_hint delegates to it, and handlers are closed with the other main-thread resources once the progress loop is quiescent.
• Each result is logged at debug level; rejections and handler errors also warn once per issue. No metric is added: integrations forward KVCR stats and may reject names they don't declare (vLLM's KVCR adapter asserts), so a hint counter waits until they declare it.
• The README and the design overview note that each hint of an envelope is handled independently, with a per-hint outcome.

No new configuration.

Behavior changes

• An invalid kv.fetch hint, or an envelope with none, no longer raises from submit_hint. Each unusable hint is reported REJECTED with a reason and logged once per (type, reason). A malformed envelope (not a mapping or struct, or actions not a list) still raises ValueError.
• Unknown action types are rejected per hint (unsupported_action_type) instead of being skipped silently.
• Every kv.fetch in an envelope is submitted, not only the first. The store's existing rules apply: a later hint from the same source replaces the hashes, and a different source marks the request failed so it isn't served remotely. One kv.fetch per envelope remains the supported shape.
• A handler exception rejects only its own hint (handler_error). A fatal progress-thread failure still propagates from submit_hint.

Validation

• ruff check src tests and ruff format --check src tests: clean.
• pytest tests/unit on Linux (the Dockerfile.ci image, real nixl), rebased on 34bb119: 649 passed, 0 failed (main: 627 passed). The 22 new tests are test_hints_parser.py, test_hints_dispatcher.py and test_hints_fetch.py.
• tests/e2e/run_vllm_kvcr.py on 2x B200 (vLLM nightly, Dynamo 1.6.0.dev20261023): passed. Rank 1 reused 2688 of 2713 prompt tokens via a peer transfer from rank 0.
• Performance on a B200 host, against main:
  • submit_hint followed by discard_hint, called as vLLM does: about 1.5 µs more per hinted request (2.8 → 4.3 µs with 1 block hash, 66.7 → 68.6 µs with 1024). Requests without a hint, and polling, lookup, fetch and deliver, are unchanged.
  • Serving A/B (2x B200, Qwen3-0.6B, data-parallel 2, Dynamo KV router; main and this PR alternated, 768 hinted peer-transfer and 768 plain requests each): no measurable difference. All hinted requests hit in both, and the latency gap was smaller than the run-to-run spread of main itself.

Commits are split for review: a pure move, the dispatcher, then the docs.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[oandreeva-nv]

/ok to test bd317ad

[oandreeva-nv]

/ok to test dda5a25

[mkhazraee]

LGTM, just needs the minor rebase after merging 67 which impacts the example and outcome logs discard their correlation fields.

[oandreeva-nv]

LGTM, thank you!