[PeaBrane]

Overview

Allow a fetch hint to extend a worker's existing GPU/host-pinned prefix from one source holding only the missing continuation. A target holding blocks 1–10 can fetch 11–15 from a source without blocks 1–10. This is standard behavior for existing hint-capable deployments, with no new configuration, CLI option, environment variable, or wire field.

Key Changes

Approximately 316 added non-test, non-comment Rust lines; no new production dependencies.

• lower_tier.rs / lower_tier_indexers.rs: discover exact fetch sources when an original local worker walk stops, using the edge already loaded by that walk. Active sources only shrink; a source can rejoin only at another destination's independent original frontier. There is no second root-to-tail scan. Single/Pair owner IDs and the existing Multi fingerprint skip repeated membership work; frontier handoffs move active maps and intervals.
• kv_hints/sources.rs: rank sources by start depth and choose at the destination's actual local frontier. Preserve topology and attachment exclusions, source/score consistency, and conservative prefill accounting. Filter intervals against both minimum and maximum local depths before sorting; move retained candidates into selection and refresh.
• Keep hint hashes root-aligned through the credited end. One hint names one source; fetch progress never becomes an assumed prefix for another source or for resuming detached local L2. Disk/External remain cache-scoring inputs, outside this HostPinned fetch plan. Queued requests refresh overlap and candidates together.

Validation

Destination/source inventory	Result
GPU 1–10; source B only 11–15	One B hint containing hashes 1–15
Local host only 1–10; source B only 11–15	Same continuation hint
No local hits; source holds a root prefix	Root-prefix fetch remains available
A source disappears and later reappears	Later coverage is usable only from another destination's independent original frontier

Behavioral tests cover gaps, eviction, dump/replay, cold/host-only/partial/full frontiers, topology, attachment exclusion, and independent owner-inventory oracles. The hash-consistent differential soak passes 20,000 cases with its oracle unchanged. The router suite passes 1,396 tests (3 ignored), with three builtin fetch-policy tests also passing. The soak was rerun after the map-move optimization; subsequent changes only remove redundant tests and a comment.

The separate CPU validation harness passed 32 real-KVCR suffix fetches with byte-for-byte delivery checks, rejected a wrong source, and verified partial success after a stale local prefix. It uses CPU buffers, a byte-copy NIXL adapter and loopback ZMQ. It does not run an engine scheduler or validate native NIXL, GPU/RDMA, inference, or serving performance. The receiver must use actual fetch outcomes and recompute any remaining gap; a hint is not a pin.

The completed Mooncake comparison and all trials use main 670019d948 versus the fused implementation on the same source: six fresh processes per arm, after six per A/A label. Lower-tier lookup p50 is 1.162→1.209 ms (+4.1%, 95% interval +0.3% to +6.2%); p99 is 3.835→3.692 ms (−3.7%, interval −12.3% to +2.6%). All 26 processes, including scouts and A/A, passed generator/keep-up and final membership checks.

This is directional x86 evidence on a public AgentX-shaped replay, not the reviewer's exact trace or Grace reproduction. A/A ratio intervals were 0.852–1.111 for p50 and 0.781–1.265 for p99: the observed p50 increase is smaller than that noise range, and the p99 decrease is inconclusive. Queueing is retained in the report: scheduled-to-finished p99 rose 410→443 ms and final drain 35→58 ms. Both arms handle the fixed offered load; this is not a capacity-knee or serving-throughput result. Earlier component results measured the removed global scan and do not describe this revision.

Issues and Ongoing Work

• No matching tracking issue found; no new issue is filed with this PR.
• Related: feat(mocker): opt-in KV state-agent (KVCR) mode #16423 adds mocker state-agent integration and hint delivery; its workers ignore the hints. The separate CPU harness checks fetch consumption.
• Applies to hint-capable workers with a local event-driven indexer, without predict-on-route. Preserves the shared block-hash consistency assumption and feat(router): credit KV blocks a worker can fetch, choosing the fetch source during selection #16701's source/score and conservative prefill-booking behavior.

Pull Request Checklist

• I used Dynamo's /pr-checklist skill.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a19ea2-719e-7143-b9f8-e070a19c0fa9.docs.buildwithfern.com/dynamo/dev

[coderabbitai]

Walkthrough

The router adds a default-off setting for continuation fetch hints. When enabled and supported, lower-tier queries collect source intervals, and selection can choose a source that extends beyond a target’s local prefix.

Changes

KV fetch continuation

Layer / File(s)	Summary
Configure continuation fetching components/src/dynamo/common/configuration/groups/kv_router_args.py, components/src/dynamo/common/tests/configuration/test_kv_router_args.py, lib/kv-router/src/scheduling/config.rs, lib/kv-router/src/scheduling/policy_config.rs, lib/bindings/python/..., docs/fern/pages/..., docs/fern/pages/reference/components/frontend-configuration.mdx, lib/kv-router/tests/data/router-settings.yaml	Adds router_fetch_continuation, defaulting to false, to router configuration, policy settings, and Python bindings. Tests cover CLI, environment, YAML precedence, and serialization. Documentation describes the setting and its requirements.
Discover continuation intervals lib/kv-router/src/kv_hints.rs, lib/kv-router/src/indexer/lower_tier.rs, lib/kv-router/src/indexer/lower_tier_indexers.rs, lib/kv-router/src/indexer/types.rs, lib/bench/kv_router/mooncake_lower_tier.rs, lib/llm/src/kv_router/indexer/recovery/state_agent.rs, lib/kv-router/src/scheduling/queue.rs, lib/kv-router/src/scheduling/selector/policy.rs, lib/router-plugins/builtin/...	Adds source intervals to transfer candidates and discovers contiguous source coverage while querying retained HostPinned chains. Updates query options and candidate initializers, with tests for interval discovery, replay, and owner removal.
Rank sources by local frontier lib/kv-router/src/kv_hints/sources.rs	Indexes continuation intervals by start depth and retrieves source rankings at or before a target’s local prefix. Tests compare results against generated inventories and cover cache-owner attachment.
Request continuation hints during routing lib/kv-router/src/services/indexer/backend.rs, lib/kv-router/src/services/selection/core/run.rs, lib/kv-router/src/services/selection/core/tests.rs, lib/kv-router/src/scheduling/overlap_refresh.rs	Gates continuation queries on chain-retention support and router configuration. Tests cover option forwarding, source selection, and island preference.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: Minimal · up to 6e43c

No actionable merge-blocking issue was established for the opt-in continuation feature. Merge after normal checks.

Pre-merge checks: 3 · 2

Failed checks: 2 warnings

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 27.59% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta…	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check		The description follows the template's Overview, Key Changes, Validation, and Issues and Ongoing Work sections, and it includes the checklist. It explains the feature (a single-source fetch hint that …	1) Correct the Overview. It says 'no new configuration, CLI option, environment variable, or wire field,' but the PR adds router_fetch_continuation, --router-fetch-continuation, DYN_ROUTER_FETCH_CONTINUATION, a YAML key, a Python KvRouterCo…

Passed checks: 3 passed

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Title check		The title clearly and concisely describes the main change: support for single-source fetch continuations in the router.

Full details: Docstring Coverage

Explanation

Docstring coverage is 27.59% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 29 functions across 22 files. (4 skipped: 4 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Full details: Description check

Explanation

The description follows the template's Overview, Key Changes, Validation, and Issues and Ongoing Work sections, and it includes the checklist. It explains the feature (a single-source fetch hint that extends a worker's existing GPU/host-pinned prefix), lists the affected files and their roles, and gives detailed validation results, including benchmark numbers, the A/A noise range, and a clear statement of what was not validated. The Overview says there is no new configuration, CLI option, environment variable, or wire field. This contradicts the Key Changes and the PR objectives, which add router_fetch_continuation, --router-fetch-continuation, DYN_ROUTER_FETCH_CONTINUATION, YAML settings, and a wire-visible setting. The Issues section says no tracking issue was found, and the template's Closes #XXXX line is not filled in. The PR objectives also say the author could not run Python binding tests. The description states the Python binding tests were not mentioned in the validation, so that gap is not disclosed.

Resolution

1. Correct the Overview. It says 'no new configuration, CLI option, environment variable, or wire field,' but the PR adds router_fetch_continuation, --router-fetch-continuation, DYN_ROUTER_FETCH_CONTINUATION, a YAML key, a Python KvRouterConfig argument, and a serialized v1.3-omitted field. Rewrite it to describe the opt-in default-off setting and its plumbing. 2) Fill in the 'Closes #XXXX' line in Issues and Ongoing Work with an issue number, or state explicitly that no issue applies. The template's PR Issue Link check is advisory today. 3) Add the Python binding validation status to Validation. The objectives say the Dynamo extension was not built and Python config/binding runtime tests did not run, so the reviewer needs that information. 4) Reconcile the Key Changes ('no second root-to-tail scan', '316 added lines') with the reported benchmark. Keep the benchmark claims tied to the revision under review.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[dynamo-review-agent]

Questions for the author:

• lib/kv-router/src/indexer/lower_tier.rs: With router_fetch_continuation enabled, each inference lookup adds a second full prompt-chain walk and builds per-request source vectors plus an FxHashMap whose entries scale with distinct owners. Please provide the expected owner/cardinality calculation and lookup-latency or throughput measurements for representative long-prompt, continuation-enabled workloads so the inference-path cost can be assessed.

[PeaBrane]

Following up on the owner/cardinality and lookup-cost question: I measured the current head (6e43c00d) against its base, with identical fixed inventories. Harness, complete results and report are on the separate validation branch.

Cardinality: destination count T does not bound source count S. S is the union of worker/rank residency owners and advertised exact cache owners observed along the chain; worker capability/topology filtering happens later. For B reachable blocks, discovery is O(B + sum of owner counts across changed owner-set runs). Stable identical Multi sets coalesce to roughly O(B+S); changing sets can approach O(BS). Staggered suffixes can cause O(S²) owner visits with only S final intervals. Fragmentation can produce I=O(BS) intervals. Summary construction sorts I references (O(I log I) upper bound), while each target lookup is O(log distinct starts), not a scan over S.

Measured cost: microseconds below are medians of six per-process percentiles. The combined component includes lower-tier lookup, the current candidate clone/drop handoff, summary construction and source choice for every target. Device matching, full worker scoring, queues, transport and inference are excluded. Blocks represent 16 tokens; these are synthetic inventories, not a production workload distribution.

Prompt / coverage	Sources / targets	Off p50	On p50	On p99
4k / shared suffix	8 / 8	2.28	7.37	7.86
32k / shared suffix	8 / 8	2.69	35.76	40.53
32k / shared suffix	256 / 8	2.67	61.81	67.71
32k / full root prefix	64 / 8	64.25	93.63	100.62
32k / staggered suffixes	64 / 8	2.71	90.09	96.50
32k / fragmented	64 / 8	2.68	253.60	261.99
32k / fragmented, all targets fully local	64 / 8	2.73	136.81	143.40
128k / shared suffix	256 / 64	14.75	176.57	185.49
128k / fragmented	64 / 64	14.62	1143.16	1164.96

Single/pair-owner 32k shared-suffix cases took 59.72/83.00 us p50: the fingerprint coalescing applies to Multi entries, so low source cardinality does not remove the per-block work. The 32k fragmented fixture emits 3,072 intervals and requests 550 KiB of allocation traffic per combined operation; the 128k fixture emits 12,288 and requests 2,184 KiB. Those are cumulative allocation/reallocation bytes, not peak heap.

The run used a Xeon 6975P-C CPU VM, one pinned physical core, identical release build settings and glibc, separate allocation instrumentation, interleaved fresh processes, and six-per-label A/A first. Default-off/base p50 ratios were 0.978–1.024 with every 95% interval including 1.0; A/A intervals spanned roughly 0.963–1.044. Enabled-cost differences clear that noise floor. All 34 processes × 15 inventories passed independent source/interval oracles, with base/off discrete outputs identical. Raw trials, hashes, allocation counts, and confidence intervals are included.

This measures router overhead, not sustained serving throughput or net fetch benefit. The fully-local control exposes useful follow-up work: prune intervals using actual destination frontiers, filter before sorting, and move candidates instead of cloning immediately before dropping their owner. None of those improvements is claimed in these measurements. The flag remains default-off.

[PeaBrane]

Follow-up to the cardinality/cost review: the updated implementation makes continuation standard fetch-hint behavior and removes the new flag and all its plumbing. The previous comment's measurements remain specific to 6e43c00d.

The follow-up report and raw trials compare that enabled implementation with the optimized one on the same 15 inventories. Candidate handoffs now move instead of clone; summary filtering precedes sorting; suffix discovery is skipped without positive partial device+local-host frontiers and emits only intervals usable at an original frontier. Root-prefix fetches remain available to cold workers, including workers absent from the hit map. No new destination-filter plumbing is needed.

Combined component p50 / p99 (µs, old → new):

• 32k fragmented: 256.46 / 266.46 → 122.19 / 129.21; 3,072 → 128 emitted intervals.
• 128k fragmented: 1,149.69 / 1,185.56 → 495.93 / 514.73; 12,288 → 128 intervals.
• Fully-local fragmented: 137.51 / 145.46 → 2.23 / 2.56; no additional suffix walk.

These reductions clear the A/A noise. Other cells moved roughly −1.3% to +3.4%; the single-owner case increased 60.26→62.29 µs, so this is not uniformly faster. All 420 process/fixture validations passed with identical source choices. Six fresh processes per timed arm, interleaved; allocation instrumentation ran separately. The unchanged source-run traversal can still approach O(B*S), and source count is not bounded by destination count.

The ordering is documented and regression-tested: L1 → contiguous local L2 → one remote L2 source. We do not resume detached local L2 after a remote fetch or plan multiple remote sources. Real-KVCR CPU validation also passed 32 byte-checked fetches plus a rejected wrong-source control. No serving-throughput or net fetch-benefit claim is made.

[jthomson04]

Perf-focused review. The selection logic looks right: cumulative top two per start depth, with expired intervals dropped by the prefix check. Moving the candidates instead of cloning them is a nice win on its own. The main problem is the cost of the second lower-tier walk now that it is on by default; measurements are on lower_tier_indexers.rs.

[dynamo-review-agent]

Questions for the author:

• lib/kv-router/src/indexer/lower_tier.rs: The published mooncake_bench regression predates the current fused-walk implementation in b330817, while the new default hint path still computes owner fingerprints per traversed block and retains active sources as owner sets change. Please rerun the same representative Lease/Book long-prompt workload against the reviewed head and provide lower-tier lookup p50/p99 (or equivalent throughput data) so the current inference-path cost can be assessed.

[jthomson04]

/ok to test 3677922

[devin-ai-integration]

✅ Dynamo PR CI passed — run 41546660668 (attempt 1) on 4b943ae122

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 41546660668. Updated on every full-CI run of this PR.}

[jthomson04]

/ok to test 4b943ae