feat(backends): handle conditional disagg bypass annotation

[karen-sy]

Important

This PR is [2/5] in the conditional-disaggregation stack.
Merge order is bottom-up: #11718 merges to main first, and #11723 merges last.
Do not merge this PR until every earlier PR in the checklist has merged and this branch has been rebased/retargeted as needed.

Link to the DEP: #11514

Stack merge order:

• feat(router): add conditional disagg policy kernel #11718 — feat(router): add conditional disagg policy kernel
• feat(backends): handle conditional disagg bypass annotation #11719 — feat(backends): handle conditional disagg bypass annotation
• feat(router): add decode affinity scoring #11720 — feat(router): add decode affinity scoring
• feat(router): orchestrate conditional disagg bypass #11725 — feat(router): orchestrate conditional disagg bypass
• feat(router): expose conditional disagg configuration #11723 — feat(router): expose conditional disagg configuration

Reference to original (closed) combined PR, with addressed review comments: ai-dynamo/dynamo#11357

Overview:

Details:

Where should the reviewer start?

Related Issues

⚠️ This section is required. Choose one path below and delete the other.

🔗 This PR is linked to an issue:

• Closes #XXXX

🚫 This PR is NOT linked to an issue:

• Confirmed — no related issue

Summary by CodeRabbit

• New Features
  • Added conditional remote-prefill bypass for eligible decode requests, allowing local prefill and generation when configured.
  • Added configurable bypass policies based on prompt size, prefill worker load, decode worker load, or a combination.
  • Added configuration options and validation for bypass thresholds and worker-busy conditions.
  • Improved decode-worker support for prefix caching and key-value cache event publishing.
• Bug Fixes
  • Unsupported bypass requests now fail early with a clear error message.
• Tests
  • Added coverage for bypass behavior, multimodal requests, configuration boundaries, and load-based decisions.

[devin-ai-integration]

Devin Review found 1 potential issue.

[datadog-official]

⚠️ Warnings

🚦 3 Pipeline jobs failed

Docs link check | lychee     

PR | backend-status-check     

PR | trtllm-runtime / Test cuda13.1, amd64     

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: faee8da | Docs | Datadog PR Page | Give us feedback!}

[coderabbitai]

Walkthrough

The change adds configurable conditional-disaggregation policies to the KV router and enables annotated remote-prefill bypass handling in vLLM and TensorRT-LLM. SGLang rejects the annotation early, while router configuration, KV events, and backend tests are updated.

Changes

Conditional disaggregation

Layer / File(s)	Summary
Router configuration and public wiring lib/kv-router/src/scheduling/config.rs, lib/kv-router/src/lib.rs, lib/bindings/python/rust/llm/entrypoint.rs	Adds conditional-disaggregation policy settings, environment parsing, serde conversion, defaults, validation, logging, public exports, and default initialization for the underlying router config.
Conditional bypass policy engine lib/kv-router/src/conditional_disagg.rs	Adds decision inputs, policy traits, ISL and worker-load policies, policy composition, factories, and async unit tests.
vLLM local bypass execution components/src/dynamo/vllm/handlers.py, components/src/dynamo/vllm/args.py, components/src/dynamo/vllm/main.py, components/src/dynamo/vllm/tests/test_vllm_worker_handler.py	Annotated decode requests use aggregated token/text generation paths; decode workers can also configure KV event publishing, with tests covering multimodal, text-only, and abort behavior.
TensorRT-LLM bypass execution components/src/dynamo/trtllm/request_handlers/handler_base.py, components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py	Annotated decode requests reuse or create context-and-generation parameters, bypass decode-only validation and deferred abort handling, and add focused tests.
SGLang unsupported-path guard components/src/dynamo/sglang/request_handlers/llm/decode_handler.py	Annotated requests fail immediately with a runtime error before SGLang generation proceeds.

Estimated code review effort: 4 (Complex) | ~60 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description keeps the required headings, but the sections are empty and the required Related Issues section is not completed.	Fill in Overview, Details, and reviewer-start guidance, and complete the required Related Issues section with a real issue link or confirmed no-issue choice.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly matches the main backend change: handling the conditional disagg bypass annotation.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/src/dynamo/trtllm/request_handlers/handler_base.py`:
- Around line 719-732: Update the bypass handling around
use_request_disagg_params and the request-parameter replacement near
request_type so multimodal EPD requests reuse decoded ep_disaggregated_params
with context_and_generation when available, preserving embedding handles. Only
construct the minimal request_type-based LlmDisaggregatedParams for text-only
bypasses, and add an integration test verifying generate_async receives the
expected engine argument.

In `@lib/kv-router/src/scheduling/config.rs`:
- Around line 186-214: Update the DYN_ROUTER_CONDITIONAL_DISAGG_POLICY handling
to emit a tracing::warn! diagnostic when value.parse() fails, while preserving
the existing assignment for valid policies and default behavior for invalid
values. Follow the warning behavior used by the deprecated-key path immediately
above.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 775b9d41-c428-436c-8ee7-f3d217419c7b

📥 Commits

Reviewing files that changed from the base of the PR and between 4f15eb6 and ea30ca8.

📒 Files selected for processing (11)

• components/src/dynamo/sglang/request_handlers/llm/decode_handler.py
• components/src/dynamo/trtllm/request_handlers/handler_base.py
• components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py
• components/src/dynamo/vllm/args.py
• components/src/dynamo/vllm/handlers.py
• components/src/dynamo/vllm/main.py
• components/src/dynamo/vllm/tests/test_vllm_worker_handler.py
• lib/bindings/python/rust/llm/entrypoint.rs
• lib/kv-router/src/conditional_disagg.rs
• lib/kv-router/src/lib.rs
• lib/kv-router/src/scheduling/config.rs

💤 Files with no reviewable changes (2)

• components/src/dynamo/vllm/main.py
• components/src/dynamo/vllm/args.py