Repository navigation
feat(router): decode-first disaggregated coordination and catalog replay endpoint - #24
Closed
ishandhanani wants to merge 2 commits into
Closed
ishandhanani wants to merge 2 commits into
ishandhanani wants to merge 2 commits into
Conversation
…rations The vLLM sidecar's selection-catalog registration now carries the rehosted ZMQ replay endpoint reported by GetKvEventSources for data-parallel rank 0, so a standalone selection service can request a replay after a late subscribe without runtime discovery. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Wire contract 5 into the frontend behind two new flags: - `--router-disagg-decode-first` (`DYN_ROUTER_DISAGG_DECODE_FIRST`): the decode RoutingHost previews and books the decode worker before prefill selection, the prefill request is constrained to that worker's KV-transfer domain through the model manager's topology constraints, and decode is dispatched on the held plan after prefill completes. The conditional-disagg decision is carried along; a bypass runs prefill on the booked decode worker. - `--router-disagg-bypass-on-prefill-failure` (`DYN_ROUTER_DISAGG_BYPASS_ON_PREFILL_FAILURE`): compensation policy for "decode booked, prefill failed". Default releases the decode booking and fails the request. Every transition is logged with its `LinkedBookingState`. Requests that pin a prefill worker, a decode pool that is not KV routed, or a missing decode host fall back to the unchanged prefill-first path. `RoutePlan::release` is now a non-test method used for compensation. Flags are exposed through KvRouterConfig (serde + env), the PyO3 KvRouterConfig constructor, the `.pyi` stub, and the frontend argument group. Docs cover the flags and the encoder pool as a third embedded instance. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
had a problem deploying
to
external_collaborator
September 3, 2026 09:49 — with
GitHub Actions
Failure
Owner
Author
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #23 (
idhanani/p4-zero-infra-docs). Roadmap: Phase 3 frontend wiring + Phase 4 catalog replay endpoint.Summary
PrefillRouter(contract 5) behind--router-disagg-decode-first/DYN_ROUTER_DISAGG_DECODE_FIRST. The decodeRoutingHostpreviews and books the decode worker first, the prefill request is constrained to that worker's KV-transfer domain viaModelManager::get_kv_transfer_routing_constraints, and decode is dispatched on the heldRoutePlanafter prefill completes. The conditional-disagg decision is evaluated on the same preview; a bypass runs prefill on the booked decode worker.--router-disagg-bypass-on-prefill-failure/DYN_ROUTER_DISAGG_BYPASS_ON_PREFILL_FAILURE: default frees the decode booking (Compensated) and fails the request; opt-in runs prefill on the decode worker (BypassAfterPrefillFailure). Every transition is logged with itsLinkedBookingState.RoutePlan::releaseis now a non-test method.KvRouterConfig(serde + env + compat struct), PyO3KvRouterConfigconstructor,.pyi, and the frontend argument group.replay_endpointfromGetKvEventSources(dp rank 0), so a standalone selection service can replay after a late subscribe.frontend-configuration.mdxparam fields;standalone-selection.mddescribes the frontend wiring and the encoder pool as a third embedded instance (each per-roleKvRouterstarts its own partition under--router-embedded-selection; no extra coordinator needed).Validation
cargo check --workspace --all-targetsandcargo checkinlib/bindings/python: clean.cargo clippy -p dynamo-llm -p dynamo-kv-router -p dynamo-vllm-sidecar --all-targets: no warnings.cargo test -p dynamo-llm prefill_router: 36 passed, including newdecode_first_falls_back_to_prefill_first_without_a_decode_hostanddecode_first_plan_is_none_when_a_prefill_worker_is_pinned.cargo test -p dynamo-kv-router --features standalone-selection disagg_coordination_flags: passed.cargo test -p dynamo-vllm-mocker sidecar_registers: passed (catalog registration with replay endpoint field).--router-disagg-decode-first --router-disagg-bypass-on-prefill-failure(both true; defaults false).docs_lint.py: 0 errors.Not validated end to end against a real prefill+decode deployment with KV-transfer topology; the flag defaults to off.