feat(router): unify frontend and standalone selection core

[ishandhanani]

Summary

Unifies frontend, standalone SelectionService and EPP routing around one SelectionCore with shared scheduling, worker membership, KV indexing and session affinity. This removes the parallel runtime scheduler and makes reservation, catalog-update and affinity lifecycle behavior consistent across selection hosts.

This draft exposes the complete aggregate-serving stack from #45–#53 against this fork's main, implementing DEP #14310. It uses the exact published #53 tip, retains the existing stack for review history, and excludes the separate DisaggCoordinator PR #40.

Implementation

• Embeds the shared selection partition in the frontend, deletes the old KvScheduler, and replaces selector generics and temporary adapters with SelectionPolicySource and SelectionHost.
• Shares worker registration through WorkerCatalogSource / CatalogReconciler, including EPP data-parallel ranks; metadata updates preserve surviving bookings and unchanged listeners.
• Uses one Indexer and KvIndexSource contract for event-driven, approximate and remote indexing, with RuntimeIngress and ZmqDirectIngress providing host-specific event transport.
• Moves enabled session-affinity tables into core partitions, scopes replica events by partition, and owns affinity leases in reservation records; exposes affinity through the standalone service, Python API and EPP.
• Fixes reservation expiry, cancellation, worker scale-to-zero/rejoin and invalid TTL handling while preserving main's deferred replica observer and frontend lease-expiry ownership; removes redundant wrappers, arguments and repeated filtering.

Walkthrough

flowchart LR
  Frontend[Frontend KvRouter] --> Core[SelectionCore partition]
  Standalone[SelectionService and EPP] --> Core
  Discovery[Runtime discovery or pod reflector] --> Catalog[CatalogReconciler]
  Catalog --> Core
  Events[RuntimeIngress or ZmqDirectIngress] --> Index[Shared Indexer]
  Index --> Core
  Core --> Scheduler[Scheduler and policy]
  Core --> Affinity[Partition affinity table]
  Core --> Reservation[Reservation record]
  Reservation --> Lease[Owned affinity lease]

Loading

Selection and state ownership

SelectionPolicySource resolves the configured registry policy or an explicit factory when the partition is built. SelectionHost supplies the runtime-specific load, eligibility, shared-cache, telemetry and replication inputs while the core owns scheduling and bookings. Indexed overlap and optional shared-cache hits feed selection alongside caller constraints. LoRA narrowing preserves the caller's allow-set and allowed pin; structured session metadata takes precedence over the legacy flat session ID.

The same partition index serves initial selection, dequeue-time overlap refresh and routing-decision recording. Event transport remains behind the ingress interface: runtime recovery stays in the frontend ingress, and standalone ingress owns direct ZMQ worker subscriptions. Remote indexing reads the primary without opening local worker subscriptions.

Membership and request lifecycle

CatalogReconciler applies desired worker snapshots, retries incomplete registration and removes withdrawn workers. Upserts prepare records before taking the mutation lock, so independent policy construction can proceed concurrently; committed updates reconcile only removed or changed ranks and endpoints. An empty partition keeps its index so a rejoined worker feeds the index the scheduler still holds.

A booked request with a session ID and no explicit pin or affinity target acquires the partition's binding before selection. The selected worker commits a new binding; a stale binding can be invalidated and rebound after worker withdrawal. Waiting for another initializer honors cancellation. Advisory selection reads existing bindings and projected load without queue admission, booking or affinity initialization.

Reservation insertion follows fallible endpoint resolution. Explicit free and stale-record cleanup drop the reservation's affinity lease through the same ownership path. Standalone bookings retain periodic expiry; frontend bookings retain RequestLeaseManager expiry ownership and deferred replica Add/Free replay. Enabled affinity is partition-scoped, preventing equal session IDs in different models or routing groups from colliding; frontend wrappers borrow that table and share replica initialization and coordinator lifetime.

DEP feedback and remaining design differences

Tom's affinity ownership proposal is implemented at the partition boundary, but tables still initialize lazily when enabled rather than existing unconditionally. Builtin routing modes without a selection partition retain their host-owned table. EPP uses hard affinity because it cannot observe the actual dispatch outcome.

The other proposed host-boundary removals are not implemented in this draft. Availability and overload callbacks retain transport inhibition, immediate overload feedback, backend KV occupancy and the two-signal latch, which catalog and slot state alone do not contain. The prefill estimator still resides in HostLoad; moving it to partition construction remains outstanding.

Controller-aware LoRA filtering still carries replica_set and is_active semantics beyond loaded-adapter metadata. ModelManager still owns shared-cache subscriptions to preserve configuration gates, overrides and reconnect behavior; the proposed config-source/core-owned-client split remains outstanding. The LoRA scorer/plugin redesign was explicitly a follow-up. These are reviewed design differences, not resolved DEP comment threads.

The reservation lease leak, cross-partition affinity collision, sole-worker rejoin failure, metadata-update state loss and Python TTL panic were reproduced before their fixes.

Validation

Head: b91015f183bc159e1fbf6968f2cc967cbbc0b23f. Base: fork main at ab58558c2dff28f4ff1ea4e219030eac5da3f48f. Creating this combined PR changes neither source nor tested tree.

• All-target check passed: cargo check -p dynamo-kv-router -p dynamo-llm -p dynamo-ext-proc --no-default-features --features dynamo-kv-router/standalone-selection,dynamo-kv-router/standalone-slot-tracker --all-targets.
• Python bindings check passed: cargo check --manifest-path lib/bindings/python/Cargo.toml --features select-service --all-targets.
• Rust suites: 1,115 KV-router tests, 2,420 LLM tests with no default features, and 154 EPP tests passed (3,689 total; three ignored). Strict core Clippy passed.
• Python: 23 targeted routing, affinity, custom-policy and stream-error checks passed. All 16 focused SelectionService bindings tests passed, including 13 TTL cases; the isolated Rust shutdown test passed with explicit libpython linking.
• Regular/sidecar and custom-policy binding checks passed. Bindings all-target Clippy passed with kv-indexer,slot-tracker,select-service. All nine stack boundaries passed the Rust compile check above; bindings with select-service also passed at feat(kv-router): approximate, remote, and hint-aware indexing in SelectionService #46–feat(kv-router): session affinity for the standalone service and EPP #53.

Aggregated end-to-end validation

Two B200 GPUs ran two TP1 aggregated SGLang 0.5.19 workers serving Qwen3-0.6B, with a KV-aware Dynamo frontend, hard session affinity and worker ZMQ KV events enabled. Baseline was ab58558c2dff28f4ff1ea4e219030eac5da3f48f. The GPU candidate was built at 63e998744f53194c7a1a6a99adbc6e5c5c6bb317, whose Git tree exactly equals this published head b91015f183bc159e1fbf6968f2cc967cbbc0b23f; the later change only moved a bindings compatibility fix to its earliest stack boundary. Both arms used identical installed package versions, model revision and deployment settings.

• 49 frontend completions: 16 sessions exercised unary/streaming/unary requests across two workers, followed by stream cancellation and a successful same-session follow-up.
• 34 standalone-selector completions: SelectionService booked workers, requests were forwarded through the frontend with explicit worker pins, and every response's actual worker matched the selection. The probe exercised repeated sessions, metadata refresh, catalog withdrawal/failover, catalog scale-to-zero/rejoin and reservation release, finishing with zero active bookings.
• 11,264 AIPerf profiling requests plus 704 warmups: all completed with zero benchmark errors, cancellations or context skips. Every profiling record was joined to a successful frontend completion; all server outputs were exactly 128 tokens and requests were balanced across both workers.

The standalone probe used approximate indexing and pinned frontend forwarding. EPP was covered by its Rust suite, not a live Envoy/Kubernetes deployment. This validation was aggregated only; no disaggregated serving test or separate DisaggCoordinator work is claimed. All benchmark jobs completed and compute was released.

Prompt artifacts were byte-identical between matched arms. Short-run prompts were 512 client tokens / 520 server tokens after chat framing; two prompts per longer arm retokenized to 513/521, with matching histograms on both arms. Client output retokenization varies slightly around the verified 128 server tokens, so request throughput is the primary comparison.

The small measured throughput reduction consistently favors main. These runs do not establish zero overhead, statistical equivalence or production-scale capacity.

GitHub CI is not green across this series. Recurring failures are the missing fork Fastchecker runner group, helm-tester image and approval-bot token, plus published-docs composition and an unchanged external documentation link. The local compilation and final-stack validation above are recorded separately from those CI failures.