fix(trtllm-sidecar): enable guided decoding for required tool_choice

[KrishnanPrash]

Overview:

The TensorRT-LLM sidecar launch scripts and deploy examples start the OpenEngine server without a guided-decoding backend. Dynamo sends a required or named tool_choice to the engine as a JSON schema, so those requests now fail with a 500. The launch scripts before #14349 set no backend either, so the schema was never enforced: with no backend the engine drops the guide, and the OpenEngine servicer from #14349 rejects the request instead. This PR sets guided_decoding_backend: xgrammar in the engine configuration of both launch scripts, both deploy examples and the README's manual engine command, so TensorRT-LLM enforces the schema and the frontend parses the tool call.

Repro (TensorRT-LLM 1.3.0rc27 with the openengine extra, Qwen3-0.6B, one TensorRT-LLM sidecar worker):

DYN_TOOL_CALL_PARSER=hermes DYN_REASONING_PARSER=qwen3 \
  lib/sidecar/trtllm/launch/agg.sh --model Qwen/Qwen3-0.6B

curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "Qwen/Qwen3-0.6B", "max_tokens": 512, "temperature": 0,
  "messages": [{"role": "user", "content": "What is the weather in Paris right now?"}],
  "tools": [{"type": "function", "function": {"name": "get_weather",
    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}],
  "tool_choice": "required",
  "chat_template_kwargs": {"enable_thinking": false}}'

Before:

{"message":"Internal server error","type":"Internal Server Error","code":500}

The worker log shows the cause:

Generate Error: Internal: Generate: guided decoding mode 'json_schema' is not supported by the 'configured' grammar backend (Unimplemented)

After:

{"content": "", "tool_calls": [{"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"}], "finish_reason": "tool_calls"}

Disaggregated (launch/disagg.sh, TensorRT-LLM 1.3.0rc29, prefill and decode engines on one GPU, NIXL transceiver), with a stricter schema that unconstrained output is unlikely to match: unit must be kelvin or rankine, and days must be 3. Each request went through the prefill worker and then the decode worker. Prefill samples only the first token, so the arguments show that the decode engine enforced the schema. The auto row has no guide and is the control.

tool_choice	arguments	matches the schema
required	{"city": "Paris", "unit": "kelvin", "days": 3}	yes
required, streamed	{"city": "Paris", "unit": "kelvin", "days": 3}	yes
named get_weather	{"city": "Paris", "unit": "kelvin", "days": 3}	yes
auto (no guide)	{"city": "Paris", "unit": "kelvin", "days": 1}	no, days is 1

Details:

TensorRT-LLM applies a JSON schema, regex, grammar or structural tag only when the LLM API option guided_decoding_backend is set. The OpenEngine launch path never set it, so every guided request reached an engine that could not enforce it.

launch/agg.sh now always writes the engine configuration file and adds the backend to it. Before, it wrote the file only when a memory override was set. launch/disagg.sh adds the backend next to the cache transceiver setting, so the prefill and the decode engine both have it. The deploy ConfigMaps get the same key.

agg.sh now refuses a forwarded --extra_llm_api_options or --config, as disagg.sh already does, because either one would replace the generated file and drop the backend. The refusal message of disagg.sh now names both keys that its file carries.

The README's manual engine command now passes an options file with the backend, and its disaggregation section says that both engines need it.

Out of scope: a guided request that reaches an engine without a backend still returns a 500 instead of a 400 with the engine's reason.

Related Issues

None

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Required and named tool choices are now enforced through guided decoding in TensorRT-LLM deployments.
• Configuration Changes
  • TensorRT-LLM deployments now use the XGrammar guided-decoding backend across aggregated and disaggregated configurations.
  • GPU-memory options are no longer passed through the aggregated deployment launcher.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 6b6db8b7-90e3-45b5-b125-96683fb4e3dc

📥 Commits

Reviewing files that changed from the base of the PR and between a4f4f71 and 20cb046.

📒 Files selected for processing (4)

• lib/sidecar/trtllm/deploy/agg.yaml
• lib/sidecar/trtllm/deploy/disagg.yaml
• lib/sidecar/trtllm/launch/agg.sh
• lib/sidecar/trtllm/launch/disagg.sh

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

The TensorRT-LLM deployment configurations and launch scripts now set guided_decoding_backend to xgrammar. The aggregate launcher always passes a generated options file and no longer adds GPU-memory options through that path.

Changes

TensorRT-LLM guided decoding

Layer / File(s)	Summary
Deployment engine configurations lib/sidecar/trtllm/deploy/agg.yaml, lib/sidecar/trtllm/deploy/disagg.yaml	The aggregate and disaggregated deployment configurations set guided_decoding_backend to xgrammar. Comments describe enforcement of required or named tool_choice JSON schemas.
Launch-time engine overrides lib/sidecar/trtllm/launch/agg.sh, lib/sidecar/trtllm/launch/disagg.sh	The aggregate launcher always creates and passes a temporary options file with the xgrammar backend, replacing conditional GPU-memory option generation. The disaggregated launcher adds xgrammar to its engine override alongside the cache transceiver backend.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 20cb0

The configurations enable xgrammar for the guided-decoding path, and aggregate profile-memory overrides remain forwarded. No actionable merge risk is evident in the supplied changes.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 2…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the main change: enabling guided decoding for required tool_choice requests in the TensorRT-LLM sidecar.
Description check	✅ Passed	The description gives a detailed overview, implementation details, and reproduction results. It omits the reviewer-start section and does not select the required related-issue option; these gaps do no…

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}

[devin-ai-integration]

✅ Dynamo PR CI passed — run 37416041191 (attempt 1) on f203a3c55d

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 37416041191. Updated on every full-CI run of this PR.}

[tanmayv25]

Disagg verification request: does the guide reach the decode leg?

In TensorRT-LLM's OpenEngine disagg path, the grammar matcher state is not part of the KV handoff. KvSessionRef carries the KV state and the first/draft tokens, but no matcher state or schema. On its first generation-only iteration, the decode worker builds a new matcher from that request's own guided_decoding_params. If the decode-leg request has no guided field, decode output is unconstrained, even with guided_decoding_backend: xgrammar set on both engines.

Sidecar side is fine. Prefill and decode use the same request builder, which always sets guided: guided_decoding(request)? (lib/sidecar/trtllm/src/convert.rs:128). Only kv differs between the phases.

Suggested check: run the PR's repro against lib/sidecar/trtllm/launch/disagg.sh with a schema an unconstrained model is unlikely to satisfy by chance. Prefill samples only the first token, so the result shows whether decode enforced the schema. For example:

curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "Qwen/Qwen3-0.6B", "max_tokens": 512, "temperature": 0,
  "messages": [{"role": "user", "content": "What is the weather in Paris right now?"}],
  "tools": [{"type": "function", "function": {"name": "get_weather",
    "parameters": {"type": "object",
      "properties": {
        "city": {"type": "string"},
        "unit": {"type": "string", "enum": ["kelvin", "rankine"]},
        "days": {"type": "integer", "minimum": 3, "maximum": 3}},
      "required": ["city", "unit", "days"], "additionalProperties": false}}}],
  "tool_choice": "required",
  "chat_template_kwargs": {"enable_thinking": false}}'

What to expect:

• Guide reaches decode: tool_calls with arguments that match the schema exactly, e.g. {"city": "Paris", "unit": "kelvin", "days": 3}.
• Guide dropped: free text, invalid JSON, or arguments that break the enum or range.

It would be worth adding the disagg result to the PR description next to the agg before/after.

[KrishnanPrash]

Disagg verification request: does the guide reach the decode leg?

Yes. I ran launch/disagg.sh at ebf3568 with your schema: TensorRT-LLM 1.3.0rc29, Qwen3-0.6B, NIXL transceiver, prefill and decode engines on one GPU (TRTLLM_PREFILL_GPU=0 TRTLLM_DECODE_GPU=0). Each request went through the prefill worker and then the decode worker.

tool_choice	arguments	matches the schema
required	{"city": "Paris", "unit": "kelvin", "days": 3}	yes
required, streamed	{"city": "Paris", "unit": "kelvin", "days": 3}	yes
named get_weather	{"city": "Paris", "unit": "kelvin", "days": 3}	yes
auto (no guide)	{"city": "Paris", "unit": "kelvin", "days": 1}	no, days is 1

The auto row is the control: without a guide the model breaks the range, so the guided rows come from the decode engine's grammar. This matches the engine code: on the first generation-only iteration, GuidedDecoder._init_disagg_gen_requests builds the matcher from the decode request's own guided_decoding_params, and the sidecar sets them on both legs (convert.rs:128).

I added this result to the PR description. f203a3c also adds the backend to the README's manual engine command and names it in the disagg.sh refusal message. It changes no launch behavior, so the result above still applies.