[Flasew]

Summary

Add move semantics to request-scoped kv.fetch hints. The destination fetches or reuses its cached data, while the source removes eligible KVCR-owned DRAM (G2) copies. Destination completion does not wait for source cleanup.

Changes

• Select copy or move through the existing hint. Move applies only to matching hashes, even when opportunistic querying is enabled. The framework still supplies complete block keys; autonomous hash-only operations are outside this change.
• Carry move intent in the initial transfer request. The source retains its claims through native completion and owns removal, without a destination completion-confirmation message.
• Reuse destination cache hits and immediately request their source invalidation. Mixed batches transfer only missing data; unfinished destination pieces settle before the remaining pieces are moved.
• Keep admitted source advice stable across hint replacement or discard. Failed advice still permits local cache hits. deliver() remains a copy.

Behavior

• Pending removal rejects new overlapping reads and deposits; previously admitted work finishes before memory is reclaimed. Failed transfers reopen only the removal intent owned by that move.
• Honor deposit(no_evict=True) and preserve incompatible source layouts. Framework-owned memory and G3 copies are untouched.
• Give destination moves the existing abandon_timeout_ms allowance while sources keep their normal transfer timeout. A destination delayed beyond that allowance can still fail after source removal.
• Publish inventory events for actual source removals and newly complete destination copies. Removal remains best effort, with no separate public outcome; invalidation stays internal.

Validation

• Copy/move behavior across cached and partially cached keys, including transferred data, release handles, and reclaimed capacity.
• Retention and layout protection, overlapping operations, hint changes, and failure recovery—including preventing an old move from cancelling a newer invalidation.
• Delayed completion around timeout boundaries and cleanup during shutdown.
• Native transfers between independent KVCR workers, checking destination data and source/destination inventory events.

Native checks predate the final ownership fix, which is covered by focused unit regressions. Full vLLM/Dynamo serving was not rerun for this revision.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[Flasew]

When fetch starts, keys already pending invalidation are recorded as failures in op.results. Without this guard, _prepare_fetch could find their still-resident data and overwrite that failure with a successful cache hit.

[Flasew]

Existing G2 removal is tied to capacity eviction or retirement on release. Move needs targeted removal after previously admitted operations finish and no_evict protection is checked, without cancelling earlier work. This provides that final cleanup; its reclamation bookkeeping overlaps the existing paths and could be consolidated.

[PeaBrane]

Thanks for this. The move/invalidation design looks solid overall: reader admission against pending removal, no_evict and layout handling, lock ordering and shutdown all read well. A few notes inline: a possible lost-copy window at the deadline boundary, a fetch-failure regression under move hints, an API-shape question about fetch() inheriting move from the hint, and one softer, unreproduced generation/self-source concern.

These came from an AI-assisted read of the code at 778cdb3 and were checked against the source; none were reproduced at runtime.

[PeaBrane]

Possible lost copy at the deadline boundary. Removal is queued here when the source sees its write succeed. The destination, though, accepts a local-fill completion only while now < self.deadline (remote_ops.py around L227-231), and the source's deadline is received_at + remaining (remote_fw_dram.py around L769), so it lands roughly one control-message latency after the destination's. A write that completes in that gap succeeds at the source (copy removed) but is discarded at the destination, leaving neither side with a G2 copy. The destination's _expire_pending_ops running before the target op is applied in poll_completed seems to widen the window.

Suggestion: gate source removal on the destination's commit, for example by reusing the removal-only _StartWrite already sent for cached keys in moves.py complete() once the destination's fill commits; or, at minimum, make the source deadline strictly earlier than the destination's. A test that advances the target clock past its deadline between the source's DONE and the target's poll would cover it; the current test pair pins the target clock to 0.0 in test_kvcr_move.py.

[Flasew]

Good catch my earlier code review also caught this one. The update removes source blocks after destination confirmation.

[Flasew]

Updating my earlier reply: after a discussion internally, we switched back to source-owned removal after successful native writes. Source copies remain claimed and pending removal through completion, avoiding the unpin/redeposit gap before a later destination confirmation.

Move destinations now use abandon_timeout_ms for both transfer completion and pending local fills, while the source retains its normal operation_timeout_ms. Each worker requires the abandonment allowance to be at least twice its own operation timeout. write_done indicates that the native writes into the destination buffers completed successfully. The longer destination allowance gives it additional time to process completion and commit the local fill. This reduces the deadline-boundary failure window, although a sufficiently delayed destination can still time out after the source copy has been removed.

[PeaBrane]

Request poisoning under move advice: once the request's hint is marked failed, whether after any failed deliver() (_fail_request_hint) or a conflicting second hint (replace(..., failed=True, move=move)), move stays true. fetch() then enters begin(), gets False here, and the whole fetch completes FAILED, including keys that are local G2 hits. Before this change a failed hint only disabled remote fills.

Would it make sense for hint-inherited moves (as opposed to explicit move()) to fall back to a plain copy/local fill when the hint has failed, keeping the hard failure only for explicit move()?

[Flasew]

Fixed. If the move hint is failed or conflicting, it then behaves like any fetch with a failed copy hint: local cache hits are served, remote keys fail. We also stopped overwriting the stored hint's mode with the mode of the hint that got rejected.

[PeaBrane]

fetch() becomes a move whenever the request's hint says mode="move", while deliver() stays a copy. For integrations that use both in one request this splits semantics. For example, the SGLang HiCache backend (sgl-project/sglang#45647) delivers ordinary KV pages with deliver() but confirms hybrid trailing-state checkpoints with fetch(), so a move hint there would move the checkpoints while only copying the KV pages.

Consider making move opt-in per call (explicit move(), or a move= argument on fetch()) instead of inheriting it from the hint. That seems cheaper to settle now than after integrations depend on it.

[Flasew]

For this one, in the update we kept the hint mode as the trigger, but narrowed it. Only keys whose hashes are in a move hint get moved; any other key in the same fetch is a normal copy, and deliver always copies.

We didn't add a per-call move() because ultimately we want the router to decide move vs. copy, as expressed in the hint. For HiCache that means delivered pages are copied, and checkpoints fetched with fetch() are moved only if the router actually listed their hashes in a move hint.

[PeaBrane]

Possibly worth a look, but this needs human judgment and I haven't reproduced it. The removal intent here (and on the all-cached path in remote_fw_dram.py around L759-763) acts on whatever G2 copy is current when the message arrives, with no residency generation. If a cached-hit intent were delayed until after the key had been removed and freshly re-deposited, it could remove the new copy.

Separately, I didn't see a check that the hint's source isn't this node's own endpoint. With self-referencing advice, a node could drop its own copy once the fetch handle is released. Dynamo's source selection excludes a worker's own attachment, so this may be unreachable in practice. If you think either is real, a generation check and a self-source check look cheap.

[Flasew]

Hi Rudy, thanks for putting this together and for the thorough validation. I wanted to flag a direction change we’ve been discussing, since your work depends on PR #154.

We’re planning to make move hints always request-independent, so #154 may change fairly significantly. Our thinking is that a request-scoped hint labeled move doesn’t add much value when completion is asynchronous anyway: the caller would still need to poll to know the move has finished. Given that, we’d prefer to submit a separate, request-independent kv.move hint.

Before we settle on that, do you see value in retaining request-scoped move hints? I’d really appreciate your perspective on any use cases or benefits we might be overlooking. Thanks!

[PeaBrane]

Thanks for flagging this early. From our side, nearly every move we need is triggered by routing a request to a cold worker, so we'd lean toward keeping request-scoped move as the primary shape:

• The request needs those bytes now. A request-scoped move reuses that one transfer; a separate kv.move means the request either waits for it, copies the same pages in parallel, or recomputes.
• Completion falls out of the request: once the destination has delivered the page locally, it holds a complete copy. Only source cleanup is asynchronous, and best effort is fine there, so the caller never has to poll.
• Separating the two adds ordering and race cases (the move landing after the read, reads at the source rejected while a key is pending removal, router-side tracking of in-flight moves). We already hit the pending-removal interaction in a much simpler reverse-move test.

A request-independent kv.move does sound useful in addition, for drains, rebalancing or pre-warming, where no request exists.

Things we learned building the SGLang side (ziqifan617/sglang#9) that KVCR could absorb to make request-scoped move simpler for every framework:

1. Page-level atomicity across layouts. Today the adapter copy-fetches KV, indexer and trailing state, delivers locally, and only then re-fetches with move advice so the cached hits invalidate the source. An atomic "move this logical page" in KVCR would remove that.
2. Reads of a key pending removal could fall through to the remote source instead of being rejected while another claim holds it.
3. Rank fan-out: each TP rank moves independently, so a cross-rank completion step (or at least an outcome signal) would help.

Happy to share our test results or the native repro if useful. Totally open to other views; this is from one integration's perspective.

[Flasew]

Thanks so much, this is helpful. I see the benefit of reusing the request’s transfer and completion path. I wanted to clarify what we each mean by “polling.”

Looking at current main and #154, submit_hint records the advice, then fetch(..., request_id=...) returns an operation handle without waiting for the transfer. The destination still consumes poll_completed() before using the fetched data, and waits for deliver() completion when copying into framework buffers. #154’s API retains that contract, and I see the adapter’s _wait() doing that polling.

When you say the caller never has to poll, do you mean no additional framework-side move-completion polling on the source, and no waiting for source cleanup, because the framework already handles fetch/delivery completion?

On the race point, my understanding is that the delayed-invalidation/reverse-delivery issue can occur with the current request-scoped path too. Is your concern that a separate move introduces another operation that must be ordered or coordinated with the request’s read?

Does these descriptions match your understanding, or is there an important constraint I’m missing? Thanks again for working through this with us! :P @PeaBrane

Edit: updated PR at #181 which contains prefetch and move as request-independent hints. Adding request-scoped move is relatively straightforward on top of the infrastructure as well.