[Flasew]

Add hash resolution and independent KV prefetch/move

This PR adds kv.prefetch and kv.move request-independent hints, so a router can ask KVCR to copy or move cached blocks using a source endpoint and block hashes. KVCR uses framework's KeyAdaptor to resolve those hashes to full KVCR BlockKeys, and starts the transfer independently of the request carrying the hint. kv.fetch remains request-scoped advice: the framework supplies full keys and starts copying through fetch() or deliver().

Supersedes #154. The move and invalidation mechanism is included here.

Commits

Commit	Change
e6df929 — request advice	Make kv.fetch advice always mean copy. Supplied mode fields are ignored, and submit_hint() still starts no transfer.
76fb194 — hash resolution	Ask a source to translate advertised hashes into full keys and layouts through KeyAdapter.decode_from_router. Validate replies and handle lookup failures and timeouts.
93785b2 — move and invalidation internal mechanism	Allow internal fetches to name their source directly and request removal of its cached copies. Reuse existing transfers without adding a public framework move() API.
ca89731 — independent hints	Connect kv.prefetch and kv.move to lookup and transfer, check destination compatibility, and collect results without depending on an inference request.

Behavior

Independent hints. kv.prefetch copies blocks into KVCR-managed DRAM (G2); kv.move also requests removal of the source copies. Both carry source_control_endpoint and block_hashes. They can target blocks unrelated to the request carrying them, run without a request ID, and share an envelope with request advice. Discarding the carrier request does not cancel them.

Their source transfers read only complete G2 copies. If those copies disappear, the transfers fail instead of using framework memory. Ordinary request-scoped copying can still use framework memory.

Hash resolution. The destination first asks the source for full BlockKey values and layout/size information, adding one metadata round trip before transfer. This lookup does not reserve memory; the transfer checks availability again. The framework integration owns the hash-to-key map and decides how to handle collisions. Missing or ambiguous hashes and adapter failures are reported per hash, allowing the other hashes to proceed. KVCR adds no second index and does not reject inventory updates on collision. A worker cannot resolve against itself.

Move semantics. The destination includes move intent in the transfer request, and the source owns invalidation. Before transferring blocks eligible for removal, the source marks them for removal and stops accepting new reads and deposits. After a successful native transfer, it removes their G2 copies once existing readers and claims are released. This is eager removal: KVCR frees the source cache memory as soon as it is safe, rather than merely making the blocks evictable. A failed transfer cancels that move's pending removal and restores access after cleanup. If the destination already has complete copies, it sends an invalidate message and the source performs the same removal without another transfer.

G2 removal respects no_evict and does not touch framework-owned memory. Removing G3 copies as part of a move is not implemented. Inventory events reflect actual additions and removals. Destination completion means its data is ready; source cleanup may finish later.

For moves, the destination extends the completion allowance for the full-key fetch from operation_timeout_ms to abandon_timeout_ms, starting after hash resolution. The source still caps the transfer at its normal operation_timeout_ms. This gives the destination extra time to process completion after the source finishes. Source removal does not wait for a destination acknowledgement, so a destination that exceeds this longer deadline can still report a timeout after the source has removed its copy.

Independent hints use the existing KVCR.poll_completed() loop on both workers. Integrations must keep servicing that loop even when no inference request is active. At the source, polling admits moves and processes invalidation and cleanup. At the destination, it starts transfers after resolution, processes transfer completions, and releases temporary claims. Independent hint results are handled internally and do not appear in the returned framework completion list. See hash-addressed operations for detailed contracts.

Validation

• Source tests: lookup failures and collisions, self-source rejection, and when framework memory may be used.
• Destination tests: invalid or late replies, copied bytes, cached and partially cached moves, existing readers, no_evict, failures, and inventory updates.
• Hint tests: unrelated carrier requests, mixed actions, disappearing or incomplete source copies, delayed transfers, overlapping actions, different layouts, and cleanup.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.