feat(session): pass request chat_template_kwargs to apply_chat_template by guapisolo · Pull Request #1820 · radixark/miles

guapisolo · 2026-07-27T07:33:21Z

Summary

Pass request chat_template_kwargs to apply_chat_template when rendering session input_ids.

Motivation

Session requests accepted chat_template_kwargs, but local apply_chat_template calls did not receive them.

Usage

POST /sessions/{session_id}/v1/chat/completions
Content-Type: application/json

{"messages": [{"role": "user", "content": "Hi"}], "chat_template_kwargs": {"enable_thinking": false}}

For this request, Miles passes enable_thinking=false to apply_chat_template while rendering input_ids.

Request values replace matching startup chat_template_kwargs for this render.

A request value that conflicts with a model family's fixed value returns HTTP 400.

Design Notes

Key choices: SessionCore.chat_completions merges request values over startup chat_template_kwargs.
TITOTokenizer.clone_with_chat_template_kwargs creates a request-scoped tokenizer over the same HF tokenizer.
TITOTokenizer.apply_chat_template passes the merged values to template.apply_chat_template.
The request-scoped clone has negligible overhead.

Verification

Tests added: test_chat_template_kwargs_override_reaches_render_and_backend verifies a request override changes locally rendered input_ids.

Review Focus

Scrutinize the merge order in SessionCore.chat_completions.
Scrutinize whether TITOTokenizer.clone_with_chat_template_kwargs preserves required model-family state.

gemini-code-assist · 2026-07-27T07:33:24Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

… outbound The session server forwarded request chat_template_kwargs to SGLang but rendered input_ids with the startup tokenizer's kwargs, so a client setting its response mode split the local token stream from SGLang's parser view. Merge the request kwargs over the launch-time values into a request-scoped TITO tokenizer (a clone over the shared HF tokenizer) so the same resolved kwargs drive both the locally rendered input_ids and the outbound SGLang request; family constructors re-derive dependent kwargs (DeepSeek's effective thinking), so a stale derived value never survives the override. Family-fixed history kwargs stay authoritative: a conflicting value fails as a clear 400 via the constructor's fixed-template validation. Known accepted gaps, marked FIXME at the merge site: top-level reasoning/reasoning_effort request fields are not mirrored into the kwargs; mid-session mode-change safety is a per-family property with no guard yet (DeepSeek V3.2/V4 render history under one mode, so a flip strands old-mode prefix tokens, while Nemotron 3 and Qwen3/3.5 carry the setting per turn or only in the generation prompt and are safe — the guard belongs on FixedTemplate); and compute_session_mismatch still renders with the startup tokenizer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Keep only the end-to-end request override test, relying on existing constructor coverage for conflicts and derived modes. Shorten the clone overhead note and clarify the SGLang field-mapping FIXME.

…server v2 Add an opt-in tree-serving implementation while keeping the bare flag and v1 on the linear server. SessionState positions each request at the deepest matching node, appends divergent work as branches, supports a strict single-chain guard, and rejects extension of length-truncated paths. The v2 samples operation folds each leaf, applies the picker and merger hooks, masks completions exactly once, joins rewards by response id, and layers fold < agent < server metadata. The core reuses the v1 HTTP plumbing and honors request-scoped TITO template kwargs from the included #1820 changes. Shared seams are inert for v1: route setup dispatches by version and forwards agent metadata only to v2; the codec accepts an explicit field allowlist, with v2 adding reward to the base wire that already carries per-sample metadata; v2-only flags require --use-session-server v2; fixed-template role capabilities remain authoritative; and session verification preserves the explicit version. Tests cover tree search, branching and strict-mode HTTP behavior, truncation and compaction, golden multi-leaf assembly, reward and metadata layering, hook failures, and the unchanged v1 surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Shi-Dong

LGTM with nit.

Shi-Dong · 2026-07-29T02:24:41Z

+            #    - DeepSeek V3.2/V4: unsafe — history think blocks and the
+            #      prompt-head reasoning_effort marker render under one mode,
+            #      so a flip strands old-mode prefix tokens and the history no
+            #      longer matches the requested mode;


Nit: it took me a long time to parse this sentence lol. Perhaps ask the agent to rephrase the sentence using clearer terms.

Updated in e8cc944. I removed the model-by-model discussion and kept the FIXME focused on the current request-shape behavior: only nested chat_template_kwargs reach the local renderer; top-level reasoning and reasoning_effort are not mapped to template kwargs.

sry I make it too long

Shi-Dong · 2026-07-29T02:40:04Z

Actually, is it possible that the server is launched with enable_thinking=False, but a subsequent request sends thinking=True? Then the merged kwargs dict has both enable_thinking and thinking keys, with conflicting values.

guapisolo · 2026-07-29T05:36:22Z

Fixed in 9b74231. Request chat_template_kwargs now merge through template.merge_chat_template_kwargs: when a DeepSeek request supplies any of thinking_mode, enable_thinking, or thinking, startup values from that alias group are replaced before the request layer is applied. Other families keep the normal same-key merge. Added one focused V3.2/V4 regression test for startup enable_thinking=False plus request thinking=True.

Treat DeepSeek mode aliases as one override group when merging request kwargs over startup defaults. This lets request thinking=True replace startup enable_thinking=False while keeping alias validation in the renderer. Co-authored-by: Shi Dong <shi.dong@radixark.ai>

Replace the model-by-model FIXME with the single request-shape constraint enforced by this block. Co-authored-by: Shi Dong <shi.dong@radixark.ai>

…server v2 Add an opt-in tree-serving implementation while keeping the bare flag and v1 on the linear server. SessionState positions each request at the deepest matching node, appends divergent work as branches, supports a strict single-chain guard, and rejects extension of length-truncated paths. The v2 samples operation folds each leaf, applies the picker and merger hooks, masks completions exactly once, joins rewards by response id, and layers fold < agent < server metadata. The core reuses the v1 HTTP plumbing and honors request-scoped TITO template kwargs from the included #1820 changes. Shared seams are inert for v1: route setup dispatches by version and forwards agent metadata only to v2; the codec accepts an explicit field allowlist, with v2 adding reward to the base wire that already carries per-sample metadata; v2-only flags require --use-session-server v2; fixed-template role capabilities remain authoritative; and session verification preserves the explicit version. Tests cover tree search, branching and strict-mode HTTP behavior, truncation and compaction, golden multi-leaf assembly, reward and metadata layering, hook failures, and the unchanged v1 surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…server v2 Add an opt-in tree-serving implementation while keeping the bare flag and v1 on the linear server. SessionState positions each request at the deepest matching node, appends divergent work as branches, and rejects extension of length-truncated paths. The v2 samples operation folds each leaf, applies the picker and merger hooks, masks completions exactly once, joins rewards by response id, and layers fold < agent < server metadata. The core reuses the v1 HTTP plumbing and honors request-scoped TITO template kwargs from the included #1820 changes. Shared seams are inert for v1: route setup dispatches by version and forwards agent metadata only to v2; the codec accepts an explicit field allowlist, with v2 adding reward to the base wire that already carries per-sample metadata; v2-only flags require --use-session-server v2; fixed-template role capabilities remain authoritative; and session verification preserves the explicit version. Tests cover tree search, branching HTTP behavior, truncation and compaction, golden multi-leaf assembly, reward and metadata layering, hook failures, and the unchanged v1 surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

guapisolo requested review from Shi-Dong, Zhichenzzz, fzyzcjy, jybsuper, maocheng23 and yueming-yuan as code owners July 27, 2026 07:33

guapisolo force-pushed the refactor/session-generated-checkpoints branch from bf1445c to 9b335c1 Compare July 27, 2026 20:53

Base automatically changed from refactor/session-generated-checkpoints to main July 28, 2026 07:04

guapisolo force-pushed the feat/session-request-chat-template-kwargs branch from f2ae6e8 to f1be8e9 Compare July 28, 2026 07:26

session: trim template override comments and tests

d083455

Keep only the end-to-end request override test, relying on existing constructor coverage for conflicts and derived modes. Shorten the clone overhead note and clarify the SGLang field-mapping FIXME.

guapisolo added the run-ci-sglang label Jul 28, 2026

guapisolo changed the title ~~feat(session): honor request chat_template_kwargs in local render and outbound~~ feat(session): pass request chat_template_kwargs to apply_chat_template Jul 28, 2026

Shi-Dong approved these changes Jul 29, 2026

View reviewed changes

guapisolo force-pushed the feat/session-request-chat-template-kwargs branch from f4f46f6 to 9530a52 Compare July 29, 2026 05:35

guapisolo and others added 2 commits July 29, 2026 05:49

docs(session): simplify template override comment

e8cc944

Replace the model-by-model FIXME with the single request-shape constraint enforced by this block. Co-authored-by: Shi Dong <shi.dong@radixark.ai>

guapisolo force-pushed the feat/session-request-chat-template-kwargs branch from 9530a52 to e8cc944 Compare July 29, 2026 05:51

guapisolo merged commit 5cc5ff9 into main Jul 29, 2026
41 checks passed

guapisolo deleted the feat/session-request-chat-template-kwargs branch July 29, 2026 06:57

fzyzcjy mentioned this pull request Jul 29, 2026

Quote the model args miles inlines into the launch command #1911

Merged

Navigation Menu

Uh oh!

feat(session): pass request chat_template_kwargs to apply_chat_template - #1820

feat(session): pass request chat_template_kwargs to apply_chat_template#1820
guapisolo merged 4 commits into
mainfrom
feat/session-request-chat-template-kwargs

guapisolo commented Jul 27, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Jul 27, 2026

Uh oh!

Shi-Dong left a comment

Uh oh!

Shi-Dong Jul 29, 2026

Uh oh!

guapisolo Jul 29, 2026 •

edited

Loading

Uh oh!

guapisolo Jul 29, 2026

Uh oh!

Shi-Dong commented Jul 29, 2026

Uh oh!

guapisolo commented Jul 29, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

guapisolo commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Usage

Design Notes

Verification

Review Focus

Uh oh!

gemini-code-assist Bot commented Jul 27, 2026

Uh oh!

Shi-Dong left a comment

Choose a reason for hiding this comment

Uh oh!

Shi-Dong Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

guapisolo Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guapisolo Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Shi-Dong commented Jul 29, 2026

Uh oh!

guapisolo commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

guapisolo commented Jul 27, 2026 •

edited

Loading

guapisolo Jul 29, 2026 •

edited

Loading

guapisolo commented Jul 29, 2026 •

edited

Loading