[yifjiang]

Overview

Add experimental, default-off decode-to-prefill (D→P) prefix reuse. A later prefill can recover compatible sealed KV blocks retained by a decode worker, independently of execution selection and admission. An unavailable, incompatible or unprofitable donor leaves normal routing available.

This PR implements that D→P serving workflow. The companion KVCR #104 transfer primitive is role-neutral, but P→P/D→D serving orchestration and active-request migration are not implemented or qualified here.

Key Changes

Approximately 3,800 added non-test source lines, excluding comments and blank lines; this is a high-complexity change.

• Bounded donor selection requires a decode source, a prefill target, root-prefix coverage and compatible model/layout metadata. It does not book execution on the donor.
• Retained GPU banks keep sealed blocks and multimodal exclusion lineage independent of native cache clears, session frontiers and replay hits. Source pins and destination allocations remain reserved through native quiescence.
• Typed kv.fetch_gpu@1.0 hints deliver GPU block/part snapshots through the separately installed, source-locked NIXL/UCX stack: same-host CUDA IPC or opt-in MNNVL within a compatible NVLink fabric. Host traffic carries control metadata, not bulk KV payloads.

The GPU adapter supports any valid matching raw TP size (TPN→TPN), with one local multiprocessing engine per role and DP=PP=CP=1. Each role's TP group is local, including when MNNVL connects engines on different hosts. There is no TP resharding, vLLM canonical/replicated offload-layout conversion or new D→P RDMA path. Every rank has an explicit unique GPU ordinal, and all rank parts must complete before ready inventory is published. Budgets are divided across all ranks and the manifest is bounded to 64 KiB. The TP size must satisfy the model and framework's head-partition constraints. The certified domain is a caller-supplied layout contract, not automatic model compatibility proof. Mixed CPU hints and transformed multi-rank/MNNVL GPU domains are rejected before publication; compatible TP1 CPU hints remain supported.

Upgrade event publishers, routers, indexers and typed event consumers together before enabling gpu_direct: older storage-tier readers reject that value, and no downgrade is negotiated. Use per-incarnation engine IDs (the default generates a UUID). Automatic stale-bank reclamation and native peer-death recovery are unsupported; an unclean exit with a reused stable ID requires a new ID or independently quiesced operator cleanup.

Validation

This PR includes public main 2164107f, latest at the last integration refresh. Planner regressions exercise all eight reuse fields at nondefault values and use supported router modes and unambiguous flag prefixes. Four added cases cover CLI/captured-environment JSON round trips and fail-closed rejection of unmodeled reuse, removed modes and ambiguous prefixes. The resolution preserves both D→P inventory query methods, the optional source argument and retained-inventory projection while adopting upstream's canonical plugin APIs. All eight reuse settings now follow the planner's captured frontend environment; an added regression checks captured values against conflicting process settings and verifies environment-destination tracking. The five matching-TP production/helper files are byte-identical to e064d637, and the three TP unit files remain 50da69e2. No Clippy flags or CI gates are relaxed; current-head integration checks are reported separately in this PR's Checks tab. The added configuration and planner regressions are selected by the required current-image CI suites; their outcomes are tracked in Checks. Live retune continues to reject reuse settings it cannot model. The allocated results below retain their exact producer labels, and upstream integration changes are not relabeled as hardware-qualified.

Allocated SLURM validation of the e064d637 production source passes a fresh binding-wheel build, 1,451 router tests (three existing ignored), 3,774 LLM tests (13 existing ignored), 317 mocker tests, 21 default consolidator tests, four Clippy configurations, full formatting, docs/API generation, Fern validation and strict links. The committed 50da69e2 test projection passes 582 Python cases with zero failures/errors/skips, using the same verified read-only production site/core. These results retain their exact producer labels; current integration CI is separate.

The new unit cases cover TP2/3/4/8, TP-specific domains and budgets, multipart registration, all-rank pin expansion/readiness, permuted bank manifests, malformed topology and device maps, allocation failure, and manifest-size cleanup. They also retain TP1 wire/CPU-hint compatibility. The first fixed Python run passed 413 cases and failed eight parameters because the new fake multi-rank region lacked the aggregate strides field provided by the real class. 50da69e2 fixes only that fixture; a fresh full 582-case fixed suite passes with zero failures/errors/skips against the verified same core/site, and the original failed run remains preserved. Every selected node ID and source hash joins its JUnit result. The old-code TP4 negative failed for the intended whitelist error.

The TP4 harness passes all 36 allocated CPU contract tests. Fresh GB200 qualification on two four-GPU hosts with private IMEX channel 459 passes four-rank local worker D2D and native two-host MNNVL success/missing-key cases. Complete GPU byte checks and captured traces cover all four parts: 52 captured device-to-device payload copy events / 851,968 payload bytes, zero host payload copies. This uses production/core e064d637, current KVCR 47671498, fresh site 07147c0b and the declared historical 022e carrier/native KVBM dependency. The four-request Qwen3-4B TP4 serving smoke also passes: after prefill's native and retained caches are cleared, OFF computes 657 tokens and ON computes 81 plus 576 externally reused tokens. Full outputs match; nine aggregate rows transfer 84,934,656 bytes across all four rank banks, with one successful delivery and no partial transfers or failures. Decode's retained keysets remain unchanged during prefill's clear. This is functional qualification; no new latency or MiniMax performance claim is made. Arbitrary matching TP support is an adapter capability, not a claim that every model/TP combination has been run on hardware.

Historical allocated checks on 55a6774f passed a binding-wheel build, 3,764 LLM tests (13 existing ignored), both affected Clippy configurations, formatting, 404 Python cases in a composite run, and docs/API-generation/Fern/strict links. Earlier router evidence remains 1,450 passed/three ignored on 447fbf45. Those producer labels remain unchanged.

Regressions cover inactive-reuse card wire/checksum compatibility, rejection before mixed-domain publication, TP1 compatibility, framework-group versus router-key separation, retained multimodal/native-clear semantics, root-prefix coverage, and ownership/finalizer/timeout behavior. Optional KVBM cases have separate current-package CPU coverage. Hosted backend GPU/E2E success and CPU fake-driver tests do not qualify the new native D→P payload path.

Earlier Qwen/MiniMax native/model evidence remains pinned to Dynamo ef93ba10, KVCR 83e5e9e, image 022e and the declared native dependency locks. Controlled inherited-prefix performance is historical evidence for those producers, not a speedup, arbitrary-prefix or topology qualification of the current matching-TP head.

Issues and Ongoing Work

Part of #16747 and #16746. Refs #17939 for the cleanup-test contract and #17958 for the upstream CI feature-build split. Builds on #17657 fetch continuations while retaining cross-role root coverage. Related #17073 migrates active requests; this PR reuses sealed prefixes for a later prefill without moving execution.

Pull Request Checklist

• I used Dynamo's /pr-checklist skill.

[devin-ai-integration]

✅ Dynamo PR CI passed — run 41766415825 (attempt 1) on 333db2665f

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 41766415825. Updated on every full-CI run of this PR.}

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a1a496-9cc4-72f9-b105-6c982948c098.docs.buildwithfern.com/dynamo/dev