Skip to content

feat(router): decode-first disaggregated coordination and catalog replay endpoint - #24

Closed
ishandhanani wants to merge 2 commits into
idhanani/p4-zero-infra-docsfrom
idhanani/p3-decode-first
Closed

ishandhanani wants to merge 2 commits into
idhanani/p4-zero-infra-docsfrom
idhanani/p3-decode-first

Conversation

@ishandhanani

Copy link
Copy Markdown
Owner

Stacked on #23 (idhanani/p4-zero-infra-docs). Roadmap: Phase 3 frontend wiring + Phase 4 catalog replay endpoint.

Summary

  • Decode-first coordination in PrefillRouter (contract 5) behind --router-disagg-decode-first / DYN_ROUTER_DISAGG_DECODE_FIRST. The decode RoutingHost previews and books the decode worker first, the prefill request is constrained to that worker's KV-transfer domain via ModelManager::get_kv_transfer_routing_constraints, and decode is dispatched on the held RoutePlan after prefill completes. The conditional-disagg decision is evaluated on the same preview; a bypass runs prefill on the booked decode worker.
  • Compensation policy --router-disagg-bypass-on-prefill-failure / DYN_ROUTER_DISAGG_BYPASS_ON_PREFILL_FAILURE: default frees the decode booking (Compensated) and fails the request; opt-in runs prefill on the decode worker (BypassAfterPrefillFailure). Every transition is logged with its LinkedBookingState.
  • Fallback to the unchanged prefill-first path when the decode pool is not KV routed, the request pins a prefill worker, or the decode host is unavailable. RoutePlan::release is now a non-test method.
  • Flags plumbed through KvRouterConfig (serde + env + compat struct), PyO3 KvRouterConfig constructor, .pyi, and the frontend argument group.
  • Sidecar catalog registration now publishes the rehosted ZMQ replay_endpoint from GetKvEventSources (dp rank 0), so a standalone selection service can replay after a late subscribe.
  • Docs: frontend-configuration.mdx param fields; standalone-selection.md describes the frontend wiring and the encoder pool as a third embedded instance (each per-role KvRouter starts its own partition under --router-embedded-selection; no extra coordinator needed).

Validation

  • cargo check --workspace --all-targets and cargo check in lib/bindings/python: clean.
  • cargo clippy -p dynamo-llm -p dynamo-kv-router -p dynamo-vllm-sidecar --all-targets: no warnings.
  • cargo test -p dynamo-llm prefill_router: 36 passed, including new decode_first_falls_back_to_prefill_first_without_a_decode_host and decode_first_plan_is_none_when_a_prefill_worker_is_pinned.
  • cargo test -p dynamo-kv-router --features standalone-selection disagg_coordination_flags: passed.
  • cargo test -p dynamo-vllm-mocker sidecar_registers: passed (catalog registration with replay endpoint field).
  • Python arg group parses --router-disagg-decode-first --router-disagg-bypass-on-prefill-failure (both true; defaults false).
  • docs_lint.py: 0 errors.

Not validated end to end against a real prefill+decode deployment with KV-transfer topology; the flag defaults to off.

…rations

The vLLM sidecar's selection-catalog registration now carries the
rehosted ZMQ replay endpoint reported by GetKvEventSources for
data-parallel rank 0, so a standalone selection service can request a
replay after a late subscribe without runtime discovery.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Wire contract 5 into the frontend behind two new flags:

- `--router-disagg-decode-first` (`DYN_ROUTER_DISAGG_DECODE_FIRST`): the
  decode RoutingHost previews and books the decode worker before prefill
  selection, the prefill request is constrained to that worker's
  KV-transfer domain through the model manager's topology constraints,
  and decode is dispatched on the held plan after prefill completes. The
  conditional-disagg decision is carried along; a bypass runs prefill on
  the booked decode worker.
- `--router-disagg-bypass-on-prefill-failure`
  (`DYN_ROUTER_DISAGG_BYPASS_ON_PREFILL_FAILURE`): compensation policy for
  "decode booked, prefill failed". Default releases the decode booking and
  fails the request.

Every transition is logged with its `LinkedBookingState`. Requests that pin
a prefill worker, a decode pool that is not KV routed, or a missing decode
host fall back to the unchanged prefill-first path. `RoutePlan::release`
is now a non-test method used for compensation.

Flags are exposed through KvRouterConfig (serde + env), the PyO3
KvRouterConfig constructor, the `.pyi` stub, and the frontend argument
group. Docs cover the flags and the encoder pool as a third embedded
instance.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani

Copy link
Copy Markdown
Owner Author

Closed while Stack 1 (#45 to #53) and Stack 2 (#40) go upstream; the branch stays for later stacks (direct dispatch, catalog leases, runtime features). See #33.

This branch had an error being deployed

1 failed deployment
external_collaborator — f93f7644 Deployed Sep 3, 2026 by ishandhanani via ok-to-test #45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation feat router

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant