feat: Support deployment-level thinking defaults for chat templates

[indrajit96]

Summary

Adds a deployment-level default thinking mode for chat-template rendering. This lets operators choose whether requests should default to thinking enabled or disabled while preserving request-level override priority.

The setting is applied at the prompt-preprocessing layer.
It does not directly toggle reasoning inside vLLM, SGLang, or TensorRT-LLM. Instead, Dynamo publishes the deployment setting through model runtime metadata and converts it into compatible chat-template arguments before rendering the model prompt.

Overview

Precedence

Thinking controls are resolved in the following order:

1. Explicit request-level control (highest priority)
2. Deployment-level default from this PR
3. Model or chat-template native default (lowest priority)

The following request controls suppress the deployment default when present:

• Root-level thinking
• chat_template_args.thinking
• chat_template_args.enable_thinking
• chat_template_args.thinking_mode
• chat_template_args.reasoning_effort
• The equivalent fields supplied through the chat_template_kwargs alias
• Top-level reasoning_effort

Request control	Deployment default	Effective behavior
thinking=false	enabled	Request disables thinking
enable_thinking=true	disabled	Request enables thinking
reasoning_effort=high	disabled	Request controls reasoning behavior
None	disabled	Thinking defaults to disabled
None	enabled	Thinking defaults to enabled
None	Unset	Model/chat-template native default is preserved

Details

• Adds --dyn-default-thinking-mode enabled|disabled and the equivalent
  DYN_DEFAULT_THINKING_MODE environment variable.
• Publishes the configured value through model runtime metadata as
  default_thinking_mode.
• Wires metadata publication through the vLLM, SGLang, and TensorRT-LLM worker
  registration paths.
• Applies the deployment default in the Python vLLM/SGLang frontend processors and the
  Rust OpenAI preprocessor.
• Injects compatible chat-template arguments only when the request does not already
  contain a thinking control:
  • thinking
  • enable_thinking
  • thinking_mode
• Preserves request precedence for root-level thinking, reasoning_effort,
  chat_template_args, and chat_template_kwargs.
• Leaves behavior unchanged when the deployment setting is unset.

The setting remains best-effort at the model level. A model may still generate reasoning
if it ignores the corresponding chat-template control.

Validation

Unit coverage includes Rust preprocessor default injection and Python processor coverage
for the vLLM and SGLang paths. Runtime validation used the ARM64 vLLM image with
Qwen/Qwen3-0.6B; earlier E2E coverage also exercised Qwen3 and SmolLM3 deployment
defaults and request overrides.

Layer	Test	Result
Rust unit	Deployment disabled is injected as all compatible template arguments	Passed
Rust unit	Deployment enabled is injected as all compatible template arguments	Passed
Rust unit	Explicit request controls are not overwritten	Passed
Rust unit	Deployment disabled survives Kimi's implicit enabled normalization	Passed
Rust unit	Kimi renderer, SGLang gate, and postprocessor use synchronized normalized values	Passed
Rust unit	Root OpenAI thinking normalization suite	Passed: 4/4
Python unit	vLLM deployment default and request-precedence cases	Coverage added
Python unit	SGLang deployment default and request-precedence cases	Coverage added
E2E	CLI default disabled, no request override	HTTP 200; no reasoning; 13 completion tokens
E2E	CLI default enabled, no request override	HTTP 200; reasoning present; 256-token test limit reached
E2E	Deployment default unset	HTTP 200; Qwen3 native thinking default preserved
E2E	DYN_DEFAULT_THINKING_MODE=disabled	HTTP 200; no reasoning; 13 completion tokens
E2E	Invalid CLI value	Rejected during argument parsing with exit code 2
E2E	Enabled deployment plus explicit normalized disable override	HTTP 200; request override won; no reasoning; 13 completion tokens
E2E	Qwen3 and SmolLM3 deployment defaults enabled/disabled	Passed
E2E	Override through chat_template_kwargs.enable_thinking	Passed

Tracking

• Linear: DIS-2508

[devin-ai-integration]

Devin Review found 1 potential issue.

[coderabbitai]

Walkthrough

Adds a deployment-level default_thinking_mode option (enabled/disabled) surfaced via --dyn-default-thinking-mode CLI flag. The value flows from CLI config through Rust WorkerConfig, engine runtime metadata (vLLM, SGLang, TRT-LLM), Python/Rust preprocessors, and frontend processors, injecting thinking/enable_thinking/thinking_mode template kwargs only when the request does not already specify them.

Changes

Default Thinking Mode Feature

Layer / File(s)	Summary
thinking.py helpers and Rust preprocessor logic components/src/dynamo/frontend/thinking.py, lib/llm/src/preprocessor.rs	New thinking.py defines DEFAULT_THINKING_MODE_RUNTIME_KEY, THINKING_CONTROL_KEYS, runtime_default_thinking_mode, and apply_default_thinking_mode_to_template_kwargs. Rust preprocessor.rs adds parallel helpers and calls apply_default_thinking_mode on inbound chat-completion requests before preprocessing.
CLI config and Rust WorkerConfig field components/src/dynamo/common/configuration/groups/runtime_args.py, lib/backend-common/src/worker.rs	DynamoRuntimeConfig adds dyn_default_thinking_mode and the corresponding --dyn-default-thinking-mode CLI arg. Rust WorkerConfig adds the default_thinking_mode field and build_local_model injects it into runtime_data.
Engine runtime metadata publication components/src/dynamo/vllm/main.py, components/src/dynamo/sglang/register.py, components/src/dynamo/trtllm/workers/llm_worker.py	Each engine worker conditionally JSON-serializes dyn_default_thinking_mode and publishes it as an engine-specific runtime_config key.
Python WorkerConfig binding and PyO3 stub components/src/dynamo/common/backend/worker.py, lib/bindings/python/rust/backend.rs, lib/bindings/python/src/dynamo/_core.pyi	Python WorkerConfig dataclass adds default_thinking_mode, from_runtime_config reads it, Worker.run() forwards it. The PyO3 constructor and .pyi stub expose the new parameter.
Frontend processors wiring components/src/dynamo/frontend/prepost.py, components/src/dynamo/frontend/sglang_prepost.py, components/src/dynamo/frontend/vllm_processor.py, components/src/dynamo/frontend/sglang_processor.py	All four frontend modules accept default_thinking_mode from runtime config and thread it into preprocess_chat_request / apply_default_thinking_mode_to_template_kwargs. SGLang processor also propagates it into worker pool initargs.
Tests components/src/dynamo/common/backend/tests/test_backend_bindings.py, lib/backend-common/src/worker.rs, lib/llm/src/preprocessor.rs, components/src/dynamo/frontend/tests/test_sglang_processor_unit.py, components/src/dynamo/frontend/tests/test_vllm_processor_unit.py	Tests cover WorkerConfig binding constructor/defaults, Rust build_local_model runtime_data injection, Rust preprocessor injection/non-override cases, and Python frontend disabled/non-override behavior.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	⚠️ Warning	The description is detailed, but it misses the required 'Where should the reviewer start?' section and the required Related Issues template.	Add a 'Where should the reviewer start?' section and replace Tracking with the required Related Issues block, including either Closes #XXXX or the no-issue checkbox.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the main change: deployment-level thinking defaults for chat-template rendering.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

lib/bindings/python/rust/backend.rs (1)

297-345: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Append the new backend.WorkerConfig parameter instead of inserting it mid-signature.

This changes the positional constructor contract for backend.WorkerConfig(...): every argument after reasoning_parser now shifts one slot. Please move default_thinking_mode to the end of the PyO3 signature/parameter list and update lib/bindings/python/src/dynamo/_core.pyi to match.

Suggested fix

     #[pyo3(signature = (
         namespace,
         component = "backend".to_string(),
         endpoint = "generate".to_string(),
         model_name = String::new(),
         served_model_name = None,
         model_input = ModelInput::Tokens,
         endpoint_types = "chat,completions".to_string(),
         custom_jinja_template = None,
         tool_call_parser = None,
         reasoning_parser = None,
-        default_thinking_mode = None,
         exclude_tools_when_tool_choice_none = true,
         enable_local_indexer = true,
         enable_kv_routing = true,
         metrics_labels = Vec::new(),
         runtime = None,
         disaggregation_mode = DisaggregationMode::Aggregated,
         health_check_payload = None,
         structural_tag_mode = "off".to_string(),
         structural_tag_scope = "auto".to_string(),
         structural_tag_schema = "auto".to_string(),
         route_to_encoder = false,
+        default_thinking_mode = None,
     ))]
@@
         custom_jinja_template: Option<String>,
         tool_call_parser: Option<String>,
         reasoning_parser: Option<String>,
-        default_thinking_mode: Option<String>,
         exclude_tools_when_tool_choice_none: bool,
         enable_local_indexer: bool,
         enable_kv_routing: bool,
         metrics_labels: Vec<(String, String)>,
         runtime: Option<RuntimeConfig>,
         disaggregation_mode: DisaggregationMode,
         health_check_payload: Option<PyObject>,
         structural_tag_mode: String,
         structural_tag_scope: String,
         structural_tag_schema: String,
         route_to_encoder: bool,
+        default_thinking_mode: Option<String>,
     ) -> PyResult<Self> {

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/bindings/python/rust/backend.rs` around lines 297 - 345, The
backend.WorkerConfig constructor signature changed the positional argument order
by inserting a new parameter in the middle, which breaks existing callers.
Update the PyO3 constructor in backend.rs so default_thinking_mode is appended
at the end of the signature and parameter list after route_to_encoder, and
mirror the same ordering in the backend.WorkerConfig type stub in
lib/bindings/python/src/dynamo/_core.pyi to keep the Python API contract stable.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/src/dynamo/common/backend/worker.py`:
- Line 120: Move the new WorkerConfig field default_thinking_mode to the end of
the class so the dataclass stays append-only and existing positional callers
keep binding correctly. Update the WorkerConfig definition by appending
default_thinking_mode after the current optional fields, rather than inserting
it before exclude_tools_when_tool_choice_none or any existing parameters.

---

Outside diff comments:
In `@lib/bindings/python/rust/backend.rs`:
- Around line 297-345: The backend.WorkerConfig constructor signature changed
the positional argument order by inserting a new parameter in the middle, which
breaks existing callers. Update the PyO3 constructor in backend.rs so
default_thinking_mode is appended at the end of the signature and parameter list
after route_to_encoder, and mirror the same ordering in the backend.WorkerConfig
type stub in lib/bindings/python/src/dynamo/_core.pyi to keep the Python API
contract stable.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b0055b16-3d34-4118-ad8d-6ab9157ae694

📥 Commits

Reviewing files that changed from the base of the PR and between 1e05725 and 3443e08.

📒 Files selected for processing (17)

• components/src/dynamo/common/backend/tests/test_backend_bindings.py
• components/src/dynamo/common/backend/worker.py
• components/src/dynamo/common/configuration/groups/runtime_args.py
• components/src/dynamo/frontend/prepost.py
• components/src/dynamo/frontend/sglang_prepost.py
• components/src/dynamo/frontend/sglang_processor.py
• components/src/dynamo/frontend/tests/test_sglang_processor_unit.py
• components/src/dynamo/frontend/tests/test_vllm_processor_unit.py
• components/src/dynamo/frontend/thinking.py
• components/src/dynamo/frontend/vllm_processor.py
• components/src/dynamo/sglang/register.py
• components/src/dynamo/trtllm/workers/llm_worker.py
• components/src/dynamo/vllm/main.py
• lib/backend-common/src/worker.rs
• lib/bindings/python/rust/backend.rs
• lib/bindings/python/src/dynamo/_core.pyi
• lib/llm/src/preprocessor.rs

[datadog-official]

🎯 Code Coverage (details)
• Patch Coverage: 52.73%
• Overall Coverage: 45.11% (+1.13%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a839dd0 | Docs | Datadog PR Page | Give us feedback!}

[hhzhang16]

Approving as a gms-codeowner

[rmccorm4]

Pulling in 0e1d1d8 to see if it helps with build timeouts

[rmccorm4]

Pulling in the grove deploy fix be9dd5a

[rmccorm4]

#12203 (just merged) should help with TRTLLM CI test failures

  [w5]         error_data = {'code': 400, 'message': 'ValueError: node_id must be in range [0, 256)', 'type': 'Bad Request'}