feat(sglang): add ephemeral KV session routing

[ishandhanani]

Summary

• Add sticky session routing and worker session lifecycle control for ephemeral KV cache reuse
• Wire session control through streaming sessions and the SGLang frontend path
• Session control is request-driven: no --enable-agent-controller flag needed. AgentController and StickySessionRouter activate lazily when requests carry nvext.session_control
• Graceful degradation: if no worker has --enable-streaming-session, the router warns once and ignores session_control. Requests proceed without isolation.

Design Changes (from review feedback)

Addressed review feedback:

• Removed --enable-agent-controller flag -- session control is per-request opt-in, not startup-gated
• AgentController uses OnceCell<Option<EventPlaneClient>> -- lazy init on first session request, caches unavailability if no endpoint exists
• StickySessionRouter always created but only activates when requests carry session_control
• Handler guard -- _session_kwargs() checks enable_streaming_session before injecting session_params into SGLang calls, preventing errors when sessions are off

KV Pressure Benchmark

Controlled benchmark with interleaved main agent + subagent traffic. GLM-4.7-Flash TP=2 on 2x L40S. 60 requests: 3 main agent turns + 5 subagent sessions (9-15 turns each) replayed from OpenCode traces.

New metric sglang:kv_physical_usage = 1 - (available_size / total_pool) captures physical GPU memory occupied including evictable radix nodes.

Streaming Sessions vs No Sessions

Subagent       STREAMING (peak -> post-close)     NO-STREAMING (peak, no drop)
--------       --------------------------------   ----------------------------
Sub 1          0.437 -> 0.048  (89% reclaimed)    monotonic climb
Sub 2          0.345 -> 0.049  (86% reclaimed)    monotonic climb
Sub 3          0.277 -> 0.050  (82% reclaimed)    monotonic climb
Sub 4          0.437 -> 0.071  (84% reclaimed)    monotonic climb

Final          0.134 (13%)                        0.958 (96%)
Avg HIT_RATE   0.86                               0.004

• Streaming = sawtooth: each session peaks, drops to ~0.05 on close (main agent's retained KV). 82-89% reclamation per session.
• No-streaming = staircase: KV accumulates monotonically to 96%, never freed.

Priority Eviction Ablation

Same workload without streaming sessions but with --radix-eviction-policy priority. Main agent requests at priority=50, subagent requests at priority=1.

Result: priority eviction does not solve the problem.

• PHYS_USE climbs monotonically from 0.075 to 0.999 with zero drops between sessions
• Low-priority subagent KV lingers until the pool is physically full (~97%)
• Under pressure, eviction churns between 0.957-0.999 but never meaningfully drops
• Main agent KV is evicted alongside subagent KV, destroying prefix reuse (HIT_RATE: 0.004)
• Priority eviction answers "what to evict when full" but not "when to free resources"

Streaming sessions are fundamentally different -- they free KV proactively on session close, keeping headroom so the main agent's prefix cache survives across subagent cycles.

Validation

• cargo check -p dynamo-llm + cargo check -p dynamo-py3 (bindings)
• cargo test -p dynamo-kv-router (269 tests pass)
• Streaming session smoke test: components/src/dynamo/sglang/tests/test_streaming_session_smoke.py
• python -m pytest -q components/src/dynamo/frontend/tests/test_sglang_processor_unit.py
• End-to-end with OpenCode against GLM-4.7-Flash (session open/close/KV release verified in logs)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Walkthrough

This pull request transitions cache control infrastructure from TTL-based prefix pinning to session-based lifecycle management. It replaces the pin_prefix endpoint with open_session/close_session handlers, introduces sticky session routing for worker affinity, adds retention_seconds injection for priority-based KV eviction with time decay, and updates related configurations and documentation.

Changes

Cohort / File(s)	Summary
Configuration & Documentation Help Text components/src/dynamo/common/configuration/groups/kv_router_args.py, lib/bindings/python/src/dynamo/_core.pyi, lib/kv-router/src/scheduling/config.rs	Updated --enable-cache-control / router_enable_cache_control help descriptions to reflect agent-aware cache control covering session lifecycle RPCs, sticky routing, and retention_seconds injection instead of prior PIN-with-TTL semantics.
Frontend Token Normalization & Parser Configuration components/src/dynamo/frontend/sglang_prepost.py, components/src/dynamo/frontend/sglang_processor.py, components/src/dynamo/frontend/tests/test_sglang_processor_unit.py	Added _normalize_prompt_token_ids helper to standardize tokenizer output into list[int]; added _runtime_config_parser_name helper to resolve parser names from runtime config; extended unit tests for both helpers.
Request Handler Session & Retention Support components/src/dynamo/sglang/request_handlers/handler_base.py, components/src/dynamo/sglang/request_handlers/llm/decode_handler.py, components/src/dynamo/sglang/request_handlers/llm/prefill_handler.py	Replaced pin_prefix endpoint with open_session, close_session, and session_control handlers; added _retention_kwargs and _session_kwargs helpers; updated generation calls to inject session and retention parameters.
SGLang Endpoint & Service Configuration components/src/dynamo/sglang/init_llm.py, components/src/dynamo/sglang/publisher.py, components/src/dynamo/sglang/register.py	Added new session_control endpoint registration in init_decode; minor import and formatting adjustments.
Protocol Definitions for Session Control lib/llm/src/protocols/openai/nvext.rs, lib/llm/src/protocols/common/preprocessor.rs, lib/llm/src/preprocessor.rs	Introduced SessionControl struct with session_id, action (open/close), and timeout fields; added session_control field to RoutingHints and NvExt; propagated session control data through preprocessor.
Core Session & Affinity Management (New Modules) lib/llm/src/kv_router/agent_controller.rs, lib/llm/src/kv_router/sticky_sessions.rs	Added AgentController for session lifecycle management with event-plane client initialization and deferred close actions; added StickySessionRouter with InMemoryAffinityStore for session-to-worker affinity tracking with TTL expiration and background reaper.
KV Router Refactoring lib/llm/src/kv_router.rs, lib/llm/src/kv_router/push_router.rs	Removed cache_control module; added re-exports for approx, protocols, scheduling, selector from external crate; exposed AgentController and StickySessionRouter; refactored KvPushRouter to replace PIN infrastructure with sticky session and agent controller integration, injecting retention_seconds into request extra args.
Removed Cache Control Implementation lib/llm/src/kv_router/cache_control.rs	Deleted entire cache-control module including PinState, CacheControlClient, create_cache_control_client, and spawn_pin_prefix function.
Example Configuration & Documentation examples/backends/sglang/launch/agg_router.sh, docs/backends/sglang/agents.md	Updated launch script with larger model, configurable context length, session control flags, parser configurations, and radix eviction policy; replaced cache-pinning documentation with session-control semantics, priority-based retention, lifecycle actions, and updated limitations.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main change: adding ephemeral KV session routing, which is the primary objective of this large PR.
Docstring Coverage	✅ Passed	Docstring coverage is 93.15% which is sufficient. The required threshold is 80.00%.
Description check	✅ Passed	The PR description is comprehensive and well-structured, including summary, design changes, benchmarks, and validation details that clearly explain the feature additions and their impact.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[ishandhanani]

Follow-up on this branch:

• renamed the router/frontend gate from --enable-cache-control to --enable-agentic-controller
• renamed the config field to router_enable_agentic_controller
• kept nvext.cache_control as an ignored Anthropic-compatibility input only
• removed the active frontend/feature/session docs references to cache-control so the branch now presents a session-only model

The current PR body / CodeRabbit summary is stale on the cache-control point after this update.

[ishandhanani]

Follow-up correction:

• the gate is now --enable-agent-controller (not --enable-agentic-controller)
• config / bindings field is now router_enable_agent_controller
• nvext.cache_control has been removed from the OpenAI NvExt struct
• Anthropic cache_control parsing remains accepted for request compatibility, but conversion to the internal chat request drops it, so it is a no-op

Validation rerun after this change:

• python3 -m compileall ... on the touched Python files
• cargo fmt --all
• cargo test -p dynamo-llm sticky_sessions -- --nocapture
• cargo test -p dynamo-llm cache_control_passthrough -- --nocapture

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-7c34eb1c-a11c-45c0-9b86-dbe29f333e04.docs.buildwithfern.com/dynamo/dev

[PeaBrane]

One design question on feature gating: session_control is already a per-request opt-in, and requests that do not carry it naturally want the existing KV-routing behavior. Given that, it seems like a cleaner model would be to keep normal KV routing as the unconditional default and make the session-control machinery request-driven/lazy instead of startup-gated: only initialize and use sticky-session state plus the AgentController when a request actually carries nvext.session_control, and fail only those requests if the worker side does not expose session_control or streaming sessions. That would preserve today's behavior for ordinary traffic, remove one frontend config knob, and make the feature boundary line up more closely with the API surface.