Skip to content

feat(kv-router): DisaggCoordinator with linked-booking compensation - #20

Closed
ishandhanani wants to merge 1 commit into
idhanani/p2-direct-dispatchfrom
idhanani/p3-disagg-coordinator
Closed

ishandhanani wants to merge 1 commit into
idhanani/p2-direct-dispatchfrom
idhanani/p3-disagg-coordinator

Conversation

@ishandhanani

Copy link
Copy Markdown
Owner

Phase 3, step 1 (contract 5) of the runtime-optional frontend roadmap. Stacked on #19.

Summary

lib/kv-router/src/services/selection/coordinator.rs: the linked-booking state machine for disaggregated serving over two selection pools, implemented once (runtime-free) so the frontend, the EPP, and the standalone service share it. This is the roadmap's highest-severity risk item, so it lands first as a self-contained, exhaustively tested component before the production PrefillRouter is switched onto it.

  • Order (decode-anchored): advisory decode select (uses feat(kv-router): advisory select with worker load and busy evaluation #12's potential_decode_blocks / decode_busy / cached tokens) -> optional advisory prefill busy read (worker_load.prefill_busy) only when the policy consumes it -> ConditionalDisaggPolicy (conditional_disagg.rs) + the decode busy gate (same rule as conditional_bypass.rs::decode_gate_allows_bypass) -> decode select_and_reserve pinned to the previewed worker -> prefill select_and_reserve with routing_constraints extended by the decode worker's KV-transfer domain (kv_transfer_domain + topology_domains -> dynamo.topology/<domain>=<value>, required or preferred per kv_transfer_enforcement, same derivation as ModelManager::get_kv_transfer_routing_constraints).
  • Compensation, explicit and recorded (LinkedBookingState transitions on the plan): decode failure books nothing; prefill failure after decode is booked follows PrefillFailurePolicy (FreeDecode default, or BypassOnDecode); prefill_complete frees prefill early; release frees whatever remains. Cross-pool atomicity is waived by design; decode-first greedy pairing accepted.
  • SelectionPool trait is the seam; SelectionService implements it. Selection ids derive as <id>/decode and <id>/prefill.
  • Decode override semantics match the frontend's build_decode_router_override (no prompt-side load, no KV-reuse assumption, zero overlap credit unless bypass).
  • Documented in standalone-selection.md ("Disaggregated coordination", with the compensation table).

Not in this PR: switching PrefillRouter/EPP onto the coordinator. The production path still books prefill first and merges topology constraints into decode; flipping that order changes booking behavior under NIXL handoff and is gated on the replay-parity disaggregated lifecycle campaign (Phase 3 exit gate).

Validation

  • cargo test -p dynamo-kv-router --features standalone-selection --lib: 1077 passed. New (8): linked_plan_books_both_pools_and_releases_in_order (exact transition sequence Idle -> DecodePreviewed -> DecodeCommitted -> Linked -> PrefillReleased -> Released, per-pool active-request counts at each step, idempotent prefill_complete), policy_bypass_books_decode_only, decode_busy_gate_vetoes_a_policy_bypass, prefill_failure_frees_decode_by_default, prefill_failure_can_bypass_on_the_decode_worker, decode_failure_books_nothing, prefill_is_constrained_to_the_decode_kv_transfer_domain (6 plans all land in the decode worker's zone), incomplete_kv_transfer_metadata_is_an_error_and_frees_decode.
  • cargo clippy --features standalone-selection --all-targets: only the pre-existing registry.rs dead-code warning. cargo fmt and docs_lint.py clean.

Phase 3 step 1 of the runtime-optional frontend roadmap: the contract-5
disaggregated coordination as a runtime-free component over two selection
pools, shared by every host (frontend, EPP, standalone service).

DisaggCoordinator books a request decode-first: advisory decode preview
(potential_decode_blocks, decode_busy, cached tokens), optional advisory
prefill busy read, conditional-disaggregation policy plus the decode busy
gate, decode commit pinned to the previewed worker, then prefill commit
constrained to the decode worker's KV-transfer domain (its kv_transfer_domain
and topology_domains become a required or preferred dynamo.topology taint per
kv_transfer_enforcement). Cross-pool atomicity is explicitly waived: the two
ledgers are independent and every partial state is compensated and recorded in
the plan's LinkedBookingState transitions. Prefill failure after decode is
booked follows PrefillFailurePolicy (free decode and fail, or bypass on the
decode worker); prefill_complete frees prefill early; release frees the rest.

SelectionPool is the seam: SelectionService implements it; the EPP can
implement it over its own client.

Not yet wired into PrefillRouter (the production path still books prefill
first); that switch is gated on the replay-parity disaggregated lifecycle
campaign.

Tests: linked path with exact transition sequence and per-pool load checks,
policy bypass, decode-gate veto, prefill failure under both compensation
policies, decode failure booking nothing, KV-transfer-domain constraint
steering prefill, and incomplete KV-transfer metadata failing closed.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani

Copy link
Copy Markdown
Owner Author

Closed while Stack 1 (#45 to #53) and Stack 2 (#40) go upstream; the branch stays for later stacks (direct dispatch, catalog leases, runtime features). See #33.

This branch had an error being deployed

1 failed deployment
external_collaborator — dc948574 Deployed Sep 3, 2026 by ishandhanani via ok-to-test #41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation feat

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant