DEP: Model disaggregated KV handoff lifecycle faithfully in mocker

[PeaBrane]

Area

Mocker, disaggregated serving, router load accounting, vLLM, and SGLang.

Summary

Introduce one engine-neutral prefill-to-decode KV handoff lifecycle in dynamo-mocker, then use it from both the live mocker and replay harness.

The shared lifecycle must model the two real backend orderings without duplicating orchestration:

• vLLM / source-first: prefill may finish before decode has capacity, so source KV remains pinned until decode reserves destination blocks and completes the transfer.
• SGLang / destination-first: decode reserves destination capacity before prefill is dispatched, then source KV remains pinned while the transfer is in flight.

The implementation must separately represent:

1. prefill compute load,
2. router-side logical capacity booking,
3. engine-owned physical KV occupancy,
4. transfer completion and KV-index visibility.

Contributors: @nnshah1, @PeaBrane, and @dreamtalen.

Motivation

PR #10557 is trying to reproduce the overload cascade caused by stranded prefill KV. The failure mode is useful, but implementing it faithfully touches scheduler resource ownership, live rendezvous, replay timing, router accounting, metrics, cancellation, and both backend policies. Keeping those concerns inline in lib/llm/src/mocker.rs or making the bootstrap room the source of capacity truth would create separate live/replay paths and backend-specific duplication.

The current mocker also conflates several different milestones:

• A terminal prefill output reduces active_prefill_tokens.
• Request completion frees router ownership.
• Scheduler completion frees physical KV.
• Decode bootstrap happens before the request enters the scheduler, so it does not reserve physical decode capacity.
• OutputSignal.completed is used too broadly as compute completion, resource release, and lifecycle completion.

This makes it impossible to express the important interval where prefill compute is finished but its KV is still physically unavailable, or the inverse interval where decode capacity is reserved but incoming KV is not yet usable for overlap.

Current Behavior

As of current main, the mocker does not model handoff-owned source pinning or destination reservation. It has ordinary scheduler allocation for requests that are running, plus bootstrap sequencing and a modeled transfer delay, but those mechanisms do not preserve or reserve KV across the prefill-to-decode handoff.

Live mocker

The current bootstrap path is a rendezvous protocol rather than a capacity protocol:

1. Decode connects to the prefill bootstrap room before its DirectRequest is submitted to the decode scheduler.
2. Prefill waits for that connection before its own DirectRequest is submitted.
3. Prefill runs to scheduler completion. Its scheduler follows the normal terminal path and releases request-owned KV.
4. The live adapter optionally sleeps for the modeled handoff delay and marks the room complete.
5. Only after room completion does decode continue and enter its normal scheduler path.

Consequences:

• A bootstrap connection is treated as decode readiness even though the decode scheduler has not admitted the request or reserved physical KV pages.
• Decode routing already books the request's estimated block load in the decode router at worker selection. However, the decode worker has not admitted the request or reserved physical destination KV while it waits in bootstrap, so worker capacity does not yet match that logical booking.
• Prefill KV is released through normal scheduler completion before the bootstrap completion ACK; it is not pinned for a later reader.
• The current path cannot reproduce vLLM's interval where prefill computation is complete but source KV remains stranded while decode waits for capacity.
• It also does not faithfully reproduce SGLang's destination-first preallocation, because connection establishment is not an engine-owned request-slot/page reservation.

Replay harness

The current disaggregated replay path also has no cross-engine resource ownership:

1. A terminal prefill output calls on_prefill_completed, reducing prefill compute load.
2. It then immediately calls on_request_completed, freeing the prefill router's request/block ownership.
3. Decode is enqueued after the modeled handoff delay.
4. Decode allocates ordinary scheduler capacity only when it is later admitted and runs.

Consequences:

• Replay cannot retain source KV after prefill compute completion.
• Replay cannot reserve destination capacity before transfer or prefill dispatch.
• The modeled handoff delay represents elapsed transfer time, but not the resource states on either side of that transfer.
• Router accounting, engine occupancy, and transfer timing transition together instead of representing their distinct lifetimes.

Existing allocation is not a handoff reservation

Both scheduler implementations already allocate and free KV for normally admitted requests. Other subsystems, such as KV offload, may also reserve destinations for their own movement operations. Those mechanisms are not currently connected to disaggregated prefill-to-decode handoff ownership.

For this DEP, destination reservation specifically means decode owns the request slot and physical KV capacity for a particular HandoffId before incoming KV is written. Source pinning specifically means prefill's request-owned KV cannot be reused or evicted after compute completion until that handoff completes or aborts. Neither lifecycle is currently hooked up in live mocker or replay.

Ordinary scheduler allocation is a separate ownership class. #11016 defines ordinary MoveBlock::Use as an all-or-nothing transaction whose failed attempts leave no new request ownership or cache visibility. That rule must not roll back a valid handoff reservation: persistent ownership is permitted here only because it is keyed by HandoffId, has explicit activation/cancellation semantics, a concrete completion/error path, and exactly-once cleanup. KVBM transfer-pipeline settlement is separately tracked by #11018; replay-level liveness classification is tracked by #11015.

Correctness Model

Common invariant

Both real engines enforce the same fundamental rule:

Destination memory must exist before transfer bytes are written, and source KV cannot be reused until the transfer is complete or the handoff is explicitly aborted.

The mocker should share this invariant while allowing each backend to choose its natural ordering.

vLLM: source-first

At the checked vLLM revision:

1. Prefill completes and exposes source blocks.
2. Connector state delays freeing those source blocks.
3. Decode later allocates real destination blocks through the normal scheduler.
4. Decode waits for remote KV while owning those blocks.
5. Transfer completion activates decode.
6. Reader completion or lease expiry releases the source blocks.

This naturally permits stranded prefill KV: prefill compute is done, but source blocks remain pinned while decode waits for capacity.

SGLang: destination-first

At the checked SGLang revision:

1. Decode enters an explicit preallocation queue.
2. Decode checks KV capacity and reserves a request slot, destination pages, and decode headroom.
3. Decode sends destination metadata to prefill.
4. Only then is prefill dispatched.
5. Source ownership and radix-cache locks remain held while transfer is in flight.
6. Transfer success releases the source and activates the reserved decode request.

SGLang therefore shifts most capacity waiting ahead of prefill computation instead of naturally stranding completed prefill KV.

Required Invariants

1. Transfer cannot start without an engine-owned destination reservation.
2. Source KV cannot be freed, evicted, or reused before reader completion, cancellation, or lease expiry.
3. Destination reservations affect scheduler admission and worker capacity metrics immediately.
4. Reserved destination pages do not appear as stored/reusable KV overlap until transfer completes.
5. Prefill compute load may decrease independently of source physical occupancy.
6. Prefill completion and source pin acquisition are one scheduler-owned transition: the scheduler moves the completed request into held-prefill ownership before reporting completion.
7. Destination receive completion may activate decode before the source is released; source release requires a separate reader/sender completion acknowledgement or lease expiry.
8. Every source lease and destination reservation terminates exactly once on success, failure, cancellation, timeout, or owner/session loss; the associated existing router request booking must also reach its normal terminal cleanup.
9. Live and replay use the same role-local state machines and backend policies. Replay composes both roles in one process; live connects them with transport messages.
10. Bootstrap/rendezvous transports correlate a handoff but do not decide whether capacity exists.
11. Disabled mode preserves current behavior while the implementation is staged.

Proposed Architecture

Layering

flowchart LR
    subgraph Replay["Replay: one process"]
        RC["Replay orchestration<br/>virtual clock"]
        RS["Source role state"]
        RD["Destination role state"]
        RC --> RS
        RC --> RD
        RS <--> RD
    end

    subgraph Live["Live: separate worker processes"]
        LS["Prefill process<br/>source role state"]
        LD["Decode process<br/>destination role state"]
        LS <-->|"idempotent handoff facts<br/>and acknowledgements"| BT["Bootstrap / channel transport"]
        BT <--> LD
    end

    RS --> PS["Prefill scheduler resources"]
    RD --> DS["Decode scheduler resources"]
    LS --> PS2["Prefill scheduler resources"]
    LD --> DS2["Decode scheduler resources"]

Loading

Ownership boundaries:

• lib/mocker owns shared lifecycle types, role-local transition logic, backend policy, scheduler resource state, accounting snapshots, and replay integration.
• There is no process-global coordinator in live mode. Each worker owns the state for its local role and its local physical resources.
• Replay may compose source and destination role state in one process, but it must use the same role-local transitions and messages as live mode.
• Backend scheduler cores own physical source pins and destination reservations. The source process cannot release destination resources, and the destination process cannot release source resources.
• lib/llm contains thin live adapters that translate network/bootstrap messages into role-local lifecycle events and execute scheduler actions.
• Bootstrap/channel code carries idempotent facts and acknowledgements keyed by a non-reused handoff attempt ID. It must not own scheduler capacity, router lifecycle, or backend policy.
• Existing components retain their responsibilities: the decode router selects/books the request, its request guard owns router cleanup, and scheduler/KV-manager code owns KV publication.
• lib/mocker must not acquire Dynamo runtime or LLM dependencies.

Core Types

Exact names may change during implementation, but the ownership split should remain explicit:

struct HandoffId(Uuid); // Unique per attempt; never reused after restart/retry.
struct SourceLeaseId(u64);
struct DestinationReservationId(u64);

struct KvHandoffDemand {
    context_tokens: usize,
    max_output_tokens: usize,
}

enum HandoffPolicy {
    Disabled,
    SourceFirst,      // vLLM
    DestinationFirst // SGLang
}

The common lifecycle should be expressed as role-local state rather than one object that directly owns both live processes:

struct SourceHandoffState {
    id: HandoffId,
    policy: HandoffPolicy,
    state: SourceState,
}

struct DestinationHandoffState {
    id: HandoffId,
    policy: HandoffPolicy,
    state: DestinationState,
}

enum SourceState {
    NotStarted,
    Computing,
    Pinned(SourceLease),
    AwaitingReaderCompletion(SourceLease),
    Released,
}

enum DestinationState {
    NotRouted,
    Routed { worker: WorkerWithDpRank },
    Reserving,
    Reserved(DestinationReservation),
    Receiving(DestinationReservation),
    Active,
    Released,
}

The source scheduler must atomically convert terminal prefill ownership into a held-prefill lease. PrefillComputeCompleted must never be emitted while normal completion is still free to dereference the KV. The completion event therefore carries the already-acquired lease:

enum SourceEvent {
    PrefillPinned {
        id: HandoffId,
        lease: SourceLease,
        blocks: usize,
    },
    ReaderCompleted { id: HandoffId },
    LeaseRenewed { id: HandoffId },
    LeaseExpired { id: HandoffId },
    Cancelled { id: HandoffId, reason: CancelReason },
}

enum DestinationEvent {
    DestinationRouted { id: HandoffId, worker: WorkerWithDpRank },
    DestinationReserved {
        id: HandoffId,
        reservation: DestinationReservation,
    },
    ReceiveCompleted { id: HandoffId },
    Cancelled { id: HandoffId, reason: CancelReason },
    ReservationExpired { id: HandoffId },
}

ReceiveCompleted and ReaderCompleted are intentionally different:

• ReceiveCompleted means destination KV is valid and decode may activate.
• ReaderCompleted means the source has received the sender/all-readers completion acknowledgement and may release its pin.
• For vLLM, decode activation may precede source release.
• For SGLang, the transport may produce both facts close together, but they remain separate ownership transitions.

Role-local transition functions should validate events and return only handoff-specific local actions:

enum SourceAction {
    DispatchPrefill,
    ReleaseSource,
    SendSourceReady,
}

enum DestinationAction {
    ReserveDestination { request: DirectRequest },
    StartReceive,
    ActivateDestination { id: HandoffId },
    ReleaseDestination,
    SendReceiveCompleted,
}

Routing, router booking cleanup, and KV event publication are not handoff-state actions. They remain with the existing router guard and scheduler/KV-manager owners.

Scheduler-Owned Resources

Physical resources must live in the scheduler cores, not in bootstrap or lib/llm maps.

Conceptually, each backend needs scheduler-owned collections such as:

struct HeldPrefills<T> {
    by_handoff: HashMap<HandoffId, T>,
}

struct PendingDecodeReservations<T> {
    by_handoff: HashMap<HandoffId, T>,
}

Backend payloads are intentionally different:

• vLLM source state retains the completed sequence/block ownership that would otherwise be dereferenced.
• vLLM destination state uses the normal decode block allocator and sequence-slot admission.
• SGLang destination state retains the preallocated request slot, prompt pages, and decode headroom.
• SGLang source state retains request/cache ownership and the equivalent of its cache lock until transfer completion.

The destination scheduler receives the actual request when reserving it and retains that request in backend-native state keyed by HandoffId. Activation therefore uses only the ID; it does not reconstruct or resubmit the request:

enum SchedulerCommand {
    Submit(DirectRequest),
    ReserveDestination { id: HandoffId, request: DirectRequest },
    ActivateDestination { id: HandoffId },
    ReleaseSource { id: HandoffId },
    CancelHandoff { id: HandoffId },
}

A destination reservation is a backend-native aggregate, not merely a block count:

struct DestinationReservation {
    id: DestinationReservationId,
    physical_pages: usize,
    request_slot: Option<RequestSlot>,
    decode_headroom_tokens: usize,
    prefix_snapshot: Option<DestinationPrefixSnapshot>,
}

struct DestinationPrefixSnapshot {
    matched_tokens: usize,
    page_aligned_tokens: usize,
    // Backend-owned locked prefix/page identity needed through activation.
}

For SGLang specifically:

• request-slot availability is an independent admission constraint;
• only the page-aligned prompt delta is physically allocated and transferred;
• decode_headroom_tokens is admission headroom, not already allocated KV;
• the locked destination-prefix snapshot must remain valid through activation;
• partial-prefix and full-prefix-hit reservations must be supported.

Only physical_pages contributes to authoritative kv_used_blocks. Request slots and decode headroom must be exposed separately where needed for admission/diagnostics and must not be double-counted as physical occupancy.

Existing submission APIs can remain compatibility wrappers while the new lifecycle is introduced.

Router And Engine Accounting

The implementation must distinguish the decode router's existing selection-time booking from scheduler-owned physical occupancy. This DEP should not add a second router reservation mechanism.

struct KvCapacitySnapshot {
    active_blocks: usize,
    pinned_source_blocks: usize,
    reserved_destination_blocks: usize,
}

These physical-block categories must be mutually exclusive: when active blocks become pinned or a reservation becomes active, ownership moves between categories rather than being counted twice. Request slots and admission-only decode headroom are tracked separately and are not part of this block sum.

Required accounting behavior:

Lifecycle event	Prefill router	Decode router	Engine metrics	KV index
Prefill dispatched	Add prefill compute load	No change	Allocation appears normally	Publish computed source KV normally
Prefill compute completed	Remove active_prefill_tokens	No change	Source occupancy remains	No change
Source pinned	Keep request/block ownership	No change	Count as pinned/unavailable	No change
Decode routed	No change	Existing decode-router selection books estimated block load	No physical change yet	No change
Destination reserved	No change	Existing booking remains active	Count reserved blocks as unavailable	Do not publish destination overlap
Destination receive completed	No change	Existing booking remains active	Reservation becomes active; occupancy unchanged	Scheduler/KV manager publishes valid destination KV
Source reader completion	Free source request ownership	No change	Source usage decreases or becomes evictable cache	Normal source cache semantics
Decode completed	No change	Free decode ownership	Release decode capacity	Normal cache semantics

Worker-facing capacity must expose physical unavailability, including reservations and pins. The intended semantics are:

kv_used_blocks = active_blocks + pinned_source_blocks + reserved_destination_blocks
active_decode_blocks includes destination reservations on decode workers

The existing decode-router booking is intentionally allowed to precede physical reservation: it represents the request already assigned to that worker. The worker must then reserve physical capacity for the same request/HandoffId, or return a terminal failure that lets the existing router guard clean up. No new RouterCapacityReservation type or parallel booking path is required. A destination reservation consumes only its actually allocated physical pages and must not create a KV-cache hit until destination receive completion.

This also requires splitting the current overloaded terminal signal. Atomic prefill-pin completion, destination receive completion/decode activation, source reader completion/release, and full request completion must be separate observable events.

Live And Replay Adapters

The role-local lifecycle and scheduler resources are shared. The adapters differ only in how time and remote facts arrive:

• Replay: composes source and destination roles in one process with virtual time and deterministic messages. It must not use a separate transfer_id -> pinned request ownership map outside the shared role state.
• Live: each worker process owns one role and uses wall-clock timers plus bootstrap/channel messages. lib/llm/src/mocker.rs should translate request/network facts into local lifecycle events and execute local scheduler actions rather than containing backend-specific ownership logic inline.
• vLLM: selects SourceFirst policy.
• SGLang: selects DestinationFirst policy and uses the same role-local reservation and acknowledgement contracts instead of retaining an independent bootstrap lifecycle.

Failure And Cleanup

The common lifecycle must handle:

• decode no-show,
• source or destination cancellation,
• transfer failure,
• source lease expiry and optional renewal while decode waits,
• destination reservation expiry or owner/session teardown,
• duplicate or late completion messages,
• process/channel teardown or restart,
• replay teardown with unresolved handoffs.

Cleanup actions must be idempotent and local-resource owners must be able to reclaim resources without a final peer message. Source pins are lease-bound. Destination reservations must be owned by a request/session guard or an equivalent local deadline so decode-process restart, channel loss, or source disappearance cannot orphan capacity. Handoff attempt IDs are not reused, and restart invalidates stale handles/messages. A failed handoff produces a terminal decode outcome so the existing router request guard releases its selection-time booking. Physical cleanup must not be inferred from request stream EOF alone.

Implementation Plan

Land this as two dependent PRs. PR 1 makes replay the first consumer of the shared lifecycle, but it must not encode replay-specific ownership or timing assumptions. Its public shape must already be suitable for PR 2 to drive each role from bootstrap/channel facts without redesigning the role-state or scheduler APIs.

PR 1: Shared lifecycle, scheduler resources, and faithful replay

Build the engine-neutral foundation and use replay to prove the complete behavior deterministically.

• Add non-reused handoff attempt IDs, backend policy, source/destination role state, events, local actions, and transition validation in lib/mocker.
• Make terminal prefill completion atomically acquire a source lease before reporting the completion event.
• Split destination receive/decode activation from source reader completion/release and full request completion.
• Add scheduler command plumbing and scheduler-owned held-prefill/destination-reservation resources for both backend cores.
• Implement vLLM SourceFirst: prefill may complete and remain pinned while decode capacity is unavailable.
• Implement SGLang DestinationFirst: the existing decode routing step books estimated load, then the decode scheduler retains the request and reserves its request slot, locked prefix snapshot, physical delta pages, and admission headroom before prefill dispatch.
• Add KvCapacitySnapshot and exercise the existing decode-router booking so compute load, logical assignment, and physical occupancy transition independently without a duplicate booking path.
• Publish destination KV visibility only after destination receive completion.
• Wire replay through a virtual-clock adapter that composes the same source/destination role events and messages that live bootstrap will use.
• Add stranding, preallocation, accounting, lease expiry, cancellation, duplicate-event, and invalid-transition tests.

PR 1 is allowed and expected to improve disaggregated replay behavior. Non-disaggregated behavior should remain unchanged.

Bootstrap-ready shape required from PR 1

PR 1 must be implemented so the live adapter in PR 2 is an integration, not a second lifecycle implementation:

• Core records and scheduler resources are keyed by a stable, serializable HandoffId that can cross process boundaries.
• Source and destination role state are driven only by engine-neutral facts and return local handoff actions; they do not call replay scheduling code directly.
• Timeouts and lease deadlines are supplied through an adapter/clock boundary rather than calling a replay clock inside role-state logic.
• Source leases and destination reservations remain opaque scheduler-owned handles; adapters cannot mutate physical capacity directly.
• Replay correlation uses the shared handoff registry. There is no separate transfer_id -> pinned request ownership map that live code would need to replace.
• Source-ready, destination-reserved, receive-completed, reader-completed, abort, and lease-renewal facts can arrive in any valid order, including duplicates and late delivery.
• Router lifecycle and KV publication remain with their existing owners; replay observes their effects rather than reimplementing them as handoff-state actions.
• Core tests can drive each role independently and compose both roles with a message trace, independent of replay or network transport.

PR 2: Live bootstrap integration and end-to-end coverage

Connect the PR 1 lifecycle to distributed mocker processes and retire the duplicate live orchestration.

• Refactor lib/llm/src/mocker.rs into thin source-role and destination-role live adapters that translate request, scheduler, and network notifications into local handoff events and execute local actions.
• Reuse bootstrap/channel infrastructure only for handoff correlation and remote readiness/completion/abort/lease messages. A successful connection is not proof of decode capacity.
• Wire both vLLM and SGLang through the shared lifecycle and backend policies rather than retaining independent bootstrap and release paths.
• Add wall-clock source lease renewal/expiry, destination owner/session expiry, transfer failure, cancellation, stale/late-message rejection, and process/channel teardown handling.
• Keep the existing decode-router booking alive while the worker waits for or holds its destination reservation, and make worker metrics reflect physical destination reservations and pinned sources at the same lifecycle points as replay.
• Add live-versus-replay parity tests for representative request traces and capacity timelines.
• Add disaggregated E2E coverage across KV-aware, round-robin/random, least-loaded, P2C, and device-aware modes as applicable.

PR #10553, or an equivalent router fix, is needed for bootstrap-based live coverage in router modes that currently cannot select/track the prefill worker correctly.

Acceptance Criteria

• A vLLM replay can show atomic prefill completion/pinning, destination receive completion activating decode, and source KV remaining pinned until reader completion or lease expiry.
• An SGLang replay cannot dispatch prefill until the decode request slot, locked/page-aligned prefix snapshot, physical delta pages, and admission headroom are reserved.
• Existing decode routing books estimated router load at selection; the subsequent destination reservation affects scheduler admission and worker physical-capacity metrics before decode activation.
• Reserved physical destination pages are not reported as reusable KV overlap before receive completion; request slots and decode headroom are not counted as physical KV usage.
• Source router ownership is not freed merely because prefill emitted its terminal token.
• Cancellation, timeout, channel loss, or owner/session restart at every lifecycle phase returns all locally acquired resources exactly once without requiring a final peer message.
• The same source-role and destination-role transition tests run independently of live or replay adapters.
• Live and replay produce equivalent lifecycle/accounting transitions for the same policy and request trace.
• Default/disabled mode has no behavior regression while the PR train lands.
• lib/mocker gains no dependency on Dynamo runtime or dynamo-llm.

Test Scenarios

1. vLLM stranded source: fill decode capacity, atomically complete/pin a 4K-token prefill, verify prefill compute load falls while source physical usage remains; release decode capacity, reserve destination, complete receive and activate decode, then separately acknowledge readers and release source.
2. SGLang destination gate: fill decode capacity and verify prefill does not dispatch; release capacity, reserve a request slot, page-aligned prefix snapshot, physical delta pages, and admission headroom, then allow prefill and transfer. Cover zero, partial, and full destination-prefix hits.
3. Router/engine parity: verify existing selection-time decode booking, physical reservation, activation, and release remain aligned without double booking or double counting.
4. No false overlap: verify a destination reservation does not publish KV-index visibility until destination receive completion.
5. Failure matrix: cancel, lose the channel, or restart an owner before decode routing, after router booking but before physical reservation, after reservation, while source is pinned, after receive completion but before reader completion, and during transfer.
6. Duplicate/stale/late events: verify idempotent cleanup, attempt-ID isolation, and no resurrection of released handoffs.
7. Live/replay parity: run the same handoff trace against virtual and wall-clock adapters and compare state/accounting transitions.
8. Router modes: cover the supported disaggregated router modes once fix(routing): enable disagg-bootstrap for LeastLoaded / P2C / DeviceAwareWeighted #10553 or its replacement lands.

Future Transfer-Timing Extension

Keep transfer timing behind a replaceable adapter. StartReceive should describe source worker, destination worker, and actual transferred bytes; the adapter schedules ReceiveCompleted. The initial lifecycle may retain fixed configured delay. Later work may add processor-sharing bandwidth contention and optional NIC/topology constraints without changing source-pin or destination-reservation ownership. Transferred bytes should reflect destination prefix reuse and page alignment rather than always using full context size.

Non-Goals

• Modeling heterogeneous effective TP between prefill and decode, including rank fan-in/fan-out, KV head reshaping, backend/transport support matrices, and topology-dependent reader acknowledgement counts. This DEP initially assumes equal effective TP per DP group; heterogeneous TP can be added later as a separate transfer-topology layer without changing the source-pin/destination-reservation lifecycle.
• Simulating NIXL byte movement, exact remote memory addresses, or engine wire formats.
• Making vLLM and SGLang use the same ordering; they share a lifecycle and invariants but retain backend-faithful policies.
• Treating bootstrap connection establishment as proof of decode capacity.
• Publishing reserved-but-empty destination pages as KV-cache hits.
• Refactoring unrelated router or scheduler code beyond what is needed to establish these ownership boundaries.

Related Work

• feat(mocker): model prefill→decode KV stranding + abort for disagg cascade tests #10557: motivating mocker KV-stranding implementation. Its useful pieces should be decomposed onto this shared foundation rather than merged as one cross-cutting change.
• fix(routing): enable disagg-bootstrap for LeastLoaded / P2C / DeviceAwareWeighted #10553: enables disaggregated bootstrap routing for least-loaded, P2C, and device-aware modes; related to live E2E coverage but distinct from handoff resource modeling.
• #11016: transactional ordinary vLLM allocation; distinct from intentional handoff ownership.
• #11015: offline replay liveness ownership and diagnostics.
• #11018: causal settlement for asynchronous KVBM pipelines.

Upstream semantics reviewed at these revisions:

• vLLM d1586e1a1242754d2f6ac51f4f16680f7d4b129b
  • decode slot allocation and remote-KV wait
  • delayed source free
  • source release after finished sending
• SGLang bfba227f55f479dd7c53194b9aae143b1a9db907
  • decode preallocation queue
  • capacity check and destination reservation
  • destination metadata sent to prefill
  • source retained and released after transfer

[tedzhouhk]

The proposed lifecycle does not cover different TP sizes in P/D for GQA models. However, GQA models might not be important in the future so not sure whether we should spend time here.

Here's a summary from AI

Disaggregated KV Transfer With Heterogeneous TP

Assumption: P_TP and D_TP mean effective TP per DP group. If attention DP is enabled, use tp_size / dp_size.

High-Level Finding

Dynamo/mocker currently models lifecycle ordering and resource ownership, not TP reshaping. The real backend behavior differs by transfer path:

Backend / Path	P_TP > D_TP	P_TP < D_TP	Transform Location
vLLM NIXL	Supported with constraints	Supported with constraints	D-side pull/read descriptors
vLLM Mooncake	Unsupported	Unsupported	Reject hetero TP
SGLang NIXL	non-MLA unsupported; MLA special-case	non-MLA unsupported; MLA special-case	No general reshape
SGLang Mooncake	Supported	Supported	P-side slice/write
TRT-LLM C++ transceiver	Supported with constraints	Supported with constraints	P split/pack + D concat
TRT-LLM Python V2	Supported with constraints	Supported with constraints	P-side fragment mapping + WRITE

Case: P_TP > D_TP

Prefill has more TP shards than decode. A decode rank needs KV from multiple prefill ranks, or the prefill ranks must write into different offsets of the decode rank's larger destination shard.

vLLM NIXL

P-side extra work:

• Keep source KV pinned until all reader notifications complete.
• Expose per-rank source block metadata and memory descriptors.
• No active reshape or packing on P.

D-side extra work:

• Compute tp_ratio = -(P_TP / D_TP).
• Select multiple remote P ranks for one local D rank.
• Split local destination memory regions into abs(tp_ratio) logical chunks.
• Issue multiple NIXL READs.
• Notify all participating P ranks after read completion.
• Constraint: P_TP % D_TP == 0; for non-MLA, different block sizes are not supported in this direction.

SGLang Mooncake

P-side extra work:

• Each P rank sends its local KV heads into an offset inside the D buffer.
• Uses dst_head_start_offset = local_tp_rank * src_heads_per_rank.
• Sends num_heads_to_send = src_heads_per_rank.
• Builds fine-grained page/token/head slice transfer params and writes to D.

D-side extra work:

• Fetch prefill parallel info.
• Target multiple P ranks.
• Send destination metadata: dst pointers, dst indices, dst_tp_rank, dst_tp_size, dst_kv_item_len.
• Wait for P_TP / D_TP prefill responses.

TRT-LLM C++ Transceiver

P-side extra work:

• CacheFormatter::format() calls splitKVCacheDispatch().
• Each P rank packs/splits its local KV into per-target transfer buffers.
• Sends those buffers to the corresponding D rank.

D-side extra work:

• Request/receive from multiple P ranks.
• Allocate receive split buffers sized per source domain.
• Call concatKvCacheV2Dispatch() to merge received fragments into D local KV layout.

Case: P_TP < D_TP

Decode has more TP shards than prefill. Multiple decode ranks consume different slices from the same prefill rank.

vLLM NIXL

P-side extra work:

• Keep a larger source shard pinned until all D readers finish.
• Expose source descriptors.
• No active reshape or packing.

D-side extra work:

• Compute tp_ratio = D_TP / P_TP.
• Multiple D ranks map to the same P rank.
• Each D rank reads only its head slice using a remote rank offset.
• Issue one NIXL READ per D rank.
• Optional local post-processing for layout/block-size conversion when supported.
• Constraint: D_TP % P_TP == 0.

SGLang Mooncake

P-side extra work:

• One P rank serves multiple D ranks.
• For each D rank, compute src_head_start_offset = dst_tp_rank * dst_heads_per_rank.
• Sends num_heads_to_send = dst_heads_per_rank.
• Destination offset is zero because each D rank owns its own local shard.

D-side extra work:

• Each D rank targets one P rank.
• Sends destination metadata and waits for its slice write to complete.

TRT-LLM C++ Transceiver

P-side extra work:

• Fan out from one P rank to multiple D ranks.
• splitKVCacheDispatch() slices local KV by destination TP domain.
• Pack per-D target buffers and send, optionally in parallel.

D-side extra work:

• Receive the slice intended for this D rank.
• concatKvCacheV2Dispatch() writes it into D local runtime layout.

Unsupported / Special Cases

vLLM Mooncake

• Heterogeneous TP is explicitly unsupported.
• Both P_TP > D_TP and P_TP < D_TP should be modeled as rejected configs.

SGLang NIXL

• non-MLA heterogeneous TP is unsupported.
• MLA has a special replicated/limited path, but it is not a general KV head reshape path.
• P still writes same-size KV items; D manages target rank/dummy connection behavior.

Mocker DEP Gap

DEP #10736 currently models:

• source-first vs destination-first lifecycle,
• source pinning,
• destination reservation,
• transfer completion,
• router/engine accounting.

It does not model:

• P_TP vs D_TP,
• rank fanout/fanin,
• backend-specific hetero-TP support or rejection,
• P-side split/pack vs D-side pull/concat,
• reader-completion counts changing with TP topology.

To cover this, the mocker would need a separate transfer-topology layer, for example:

struct HandoffTopology {
    source_tp_per_dp: usize,
    destination_tp_per_dp: usize,
    transfer_path: KvTransferPath,
    tp_relation: TpRelation,
}

enum TpRelation {
    Equal,
    SourceWider,      // P_TP > D_TP
    DestinationWider, // P_TP < D_TP
    Unsupported,
}

[PeaBrane]

Thanks, added heterogeneous prefill/decode TP handling as an explicit non-goal for this DEP. The initial scope assumes equal effective TP per DP group; rank fan-in/fan-out and backend-specific reshaping/support can be layered on separately later.

[nnshah1]

Thanks, added heterogeneous prefill/decode TP handling as an explicit non-goal for this DEP. The initial scope assumes equal effective TP per DP group; rank fan-in/fan-out and backend-specific reshaping/support can be layered on separately later.

I think this makes sense to have as future scope - from a failure / ft mode perspectivs we mainly need the capacity to be correctly pinned umtil the peer picks it up - the exact timing isn't critical - maybe the tp transform can be modeled as a fixed cost?

[PeaBrane]

Implementation update: source hold and destination reservation

Changelog: This supersedes the earlier source-hold-only scope update. PR #10831 now implements both atomic source retention and persistent physical destination reservation/activation. Replay and live orchestration are still unchanged.

What PR #10831 implements

• A stable HandoffId and internal scheduler commands for source registration/release/cancel and destination reserve/activate/cancel.
• Atomic source completion ownership:
  • vLLM retains the completed request and defers its terminal Deref sequence.
  • SGLang retains the completed request and defers cache/free/radix-unlock cleanup.
  • Early cancellation restores ordinary terminal cleanup; held cleanup occurs exactly once.
• Persistent WaitingForRemoteKv destination ownership:
  • Reservation acquires a reusable cached prefix and fresh unpublished physical KV for the missing prompt footprint.
  • Reservation does not consume runnable sequence capacity.
  • Activation publishes newly valid KV and queues the same request as normal waiting/prebuilt-ready work.
  • Normal scheduling later promotes that request without recomputing or reallocating its prompt.
  • Cancellation and allocation failure unwind retained ownership exactly once.
• Physical occupancy remains authoritative. Retained source handles and destination allocations are visible through existing KV-manager occupancy and load reporting. There is no parallel logical reservation ledger, pinned-block counter, capacity snapshot, or router accounting path.
• KV-event timing follows physical validity:
  • Cached-prefix acquisition and empty unpublished handles emit no Stored.
  • Real inactive-cache eviction may emit normal Removed events.
  • Activation emits Stored once for newly valid transferred blocks/pages; reused prefix KV is not republished.
• Separate quiescence predicates:
  • is_empty() continues to mean no runnable scheduler work.
  • is_drained() additionally requires no retained source, destination, or offload ownership.
  • num_requests() semantics are unchanged.
• Direct-core lifecycle regressions cover source hold, destination reservation, activation while runnable capacity is full, later promotion with the same UUID and handles, final cleanup, and reservation-failure unwind.

Two deliberate backend details are documented:

• vLLM mocker uses BlockDuplicationPolicy::Reject and one canonical physical block per hash/PLH. If a suffix becomes cached after reservation, activation discards the redundant reserved block and adopts the canonical handle. This can change the reserved block ID and reduce occupancy, but avoids unsafe duplicate offload/event bookkeeping in the mocker's current hash-owned model.
• SGLang currently reserves the page-rounded incoming prompt footprint. Its additional upstream logical decode allowance and completion guard remain deferred.

TRT-LLM rejects destination reservation as unsupported until its completion-headroom semantics are modeled. Retrying cannot make that backend capability available.

Local validation for the PR: cargo test -p dynamo-mocker --lib passed (381 passed, 1 ignored); clippy with warnings denied and formatting passed.

Clean lifecycle replacing the current handoff hack

The shared orchestration should become:

SubmitHandoffPrefill
  -> source terminal HeldPrefill
ReserveDestination
  -> destination WaitingForRemoteKv
transfer completion
  -> ActivateDestination
  -> destination Waiting/prebuilt-ready
ReleaseSource
  -> deferred source cleanup
normal scheduling
  -> destination Running

Activation is the readiness barrier: release the source only after the destination has valid KV and the same request is safely queued. There is no byte-copy API in this foundation; the orchestrator may initially treat the existing handoff delay as simulated transfer completion.

Backend ordering remains policy, not a second ownership model:

• vLLM can use source-first flow: finish and hold the source, then reserve destination capacity.
• SGLang can reserve the destination before starting source compute, matching its destination-first flow.

Both use the same scheduler commands and physical ownership rules.

Offline replay follow-up

One focused follow-up PR should replace the current UUID-plus-delay decode enqueue path:

1. Add HandoffId and an explicit handoff phase to offline disaggregated request state.
2. Expose scheduler-command application through ReplayWorkerCore / OfflineWorkerState; ordinary requests continue through the existing receive path.
3. Dispatch disaggregated prefills with SubmitHandoffPrefill.
4. Select/book the decode worker using the existing router/admission lifecycle, then call ReserveDestination in the backend-appropriate order. DestinationUnavailable keeps the source held and retries when physical capacity changes; it must not enqueue an ordinary decode request.
5. Replace DecodeHandoff { uuid, delay } -> enqueue_decode with an explicit transfer-completion event keyed by HandoffId. On that event, call ActivateDestination; after Applied, call ReleaseSource.
6. Split router lifecycle timing cleanly: prefill compute completion may still update prefill statistics, but the prefill worker's request ownership is completed only when the source is released. Do not introduce a second router-side KV reservation or pinned-load metric.
7. Keep worker readiness based on core.is_empty(), but make replay termination/drain checks use core.is_drained() so retained ownership neither looks finished nor causes a busy loop.
8. On cancellation or failure, unwind destination and source ownership plus existing router/admission guards exactly once.

Initial transfer timing can retain the current configured delay. If later timing needs the exact uncached suffix size, return a compact reservation summary from ReserveDestination; do not infer it from the full prompt or add a capacity ledger.

Offline validation should include:

• Full disaggregated source-to-destination lifecycle for both backends.
• Reservation-unavailable retry/cancel behavior.
• No Stored before activation and no duplicate prompt allocation/compute after activation.
• Existing non-disaggregated replay report/stats remain byte-for-byte stable; disaggregated report JSON should reflect the explicit lifecycle without changing unrelated accounting.

Live mocker follow-up

A subsequent focused PR should replace the current bootstrap rendezvous plus fixed sleep:

1. Add a scheduler control sender beside the ordinary DirectRequest sender and process lifecycle commands in the same core task.
2. Make destination reservation—not connection establishment—the decode-readiness signal. The destination reports reservation-ready metadata keyed by HandoffId only after physical ownership is established.
3. Submit the prefill with source-hold registration according to backend ordering. Transfer completion calls ActivateDestination; an activation acknowledgement then permits ReleaseSource.
4. Keep bootstrap/transport thin: it carries IDs, readiness, completion, and cancellation acknowledgements. Capacity and KV ownership remain exclusively inside the scheduler core.
5. When a core has no runnable work, wait on both request and command channels. Retained ownership keeps is_drained() false but must not cause polling or spin.
6. Add cross-process lifecycle protection at this layer: disconnect/timeout cancellation, session generations or bounded stale-message protection, and status reconciliation. The core primitive intentionally keeps no permanent tombstones.
7. Preserve existing router admission guards and physical worker load reporting. No pinned-block category or special local-balancing path is needed.

Live/offline parity coverage should then verify the same lifecycle states, KV-event timing, occupancy behavior, source-release barrier, cancellation unwind, and final drained state.

Still deferred

• Actual KV byte movement, bandwidth contention, transfer leases, and expiry.
• Heterogeneous TP reshaping.
• TRT-LLM destination reservation and completion headroom.
• SGLang's extra logical decode-token allowance/completion guard.
• Router metrics or pinned/reserved occupancy categories.
• New Python/CLI configuration beyond what the orchestration needs.