Skip to content

fix(frontend): gate SGLang tool-call parsing on delimiter token ids #17467

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ydaiming wants to merge 12 commits into
base: main
Choose a base branch
from
Open

fix(frontend): gate SGLang tool-call parsing on delimiter token ids #17467

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
c89bdee
fix(frontend): gate SGLang tool-call parsing on delimiter token ids
ydaiming Oct 28, 2026
98af701
fix(frontend): address review of the delimiter-id tool-call gate
ydaiming Oct 28, 2026
cb5e59b
fix(frontend): route stop-buffered text per delimiter boundary
ydaiming Oct 28, 2026
09b543d
fix(frontend): reject marker isolation on the DeepSeek-V4 encoder path
ydaiming Oct 28, 2026
3c40f83
fix(frontend): merge split-segment tool-call deltas by index
ydaiming Oct 28, 2026
44a185e
fix(frontend): locate a segment's delimiter from the end of its decod…
ydaiming Oct 28, 2026
3ce2101
fix(frontend): hold a tool call whole between its delimiter ids; deri…
ydaiming Oct 28, 2026
f969544
test(frontend): drop a restating test docstring
ydaiming Oct 28, 2026
6fb4315
fix(frontend): honor tokenizer controls on the isolated prompt path
ydaiming Oct 29, 2026
18883e4
Merge branch 'main' into ydaiming/frontend-tool-call-marker-gate
ydaiming Nov 10, 2026
fee2a19
Merge branch 'main' into ydaiming/frontend-tool-call-marker-gate
ydaiming Nov 10, 2026
07e5316
docs(frontend): restructure the echoed-delimiters docs after review
ydaiming Nov 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions components/src/dynamo/frontend/frontend_args.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,7 @@ class FrontendConfig(RouterConfigBase, KvRouterConfigBase, AisPerfConfigBase):
enable_streaming_reasoning_dispatch: bool
reasoning_field_name: str
exclude_tools_when_tool_choice_none: bool
isolate_tool_call_markers: bool
preprocess_workers: Optional[int]
tokenizer_backend: str
tokenizer_fallback: bool
Expand Down Expand Up @@ -288,6 +289,10 @@ def validate(self) -> None:
f"--tokenizer: invalid value '{self.tokenizer_backend}' "
f"(choose from {sorted(self._VALID_TOKENIZER_BACKENDS)})"
)
if self.isolate_tool_call_markers and self.chat_processor != "sglang":
raise ValueError(
"--isolate-tool-call-markers requires --dyn-chat-processor=sglang"
)
if self.ais_perf_config is not None and self.router_prefill_load_model != "ais":
raise ValueError(
"--ais-perf-config requires --router-prefill-load-model=ais"
Expand Down Expand Up @@ -761,6 +766,21 @@ def add_arguments(self, parser) -> None:
"tool calls in the content field."
),
)
add_negatable_bool_argument(
g,
flag_name="--isolate-tool-call-markers",
env_var="DYN_ISOLATE_TOOL_CALL_MARKERS",
default=False,

@akshatha-k akshatha-k Nov 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] (8) Docs impact if the default flips to on.

@biswapanda as maintainer: the PR description asks whether this should default to on for the SGLang processor. Could you weigh in? If it flips, three places need to change together: the KB table default (chat-processor.md), the ParamField default (frontend-configuration.mdx), and this --help text.

@ydaiming ydaiming Nov 10, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on the three places. Nothing changes in this commit: the default stays off until a maintainer decides. If it flips, I will update the KB table, the ParamField default and the --help text in the same commit.

help=(
"Tokenize message content that contains the tool-call parser's "
"delimiters (for example a literal <tool_call> in a tool result) as "
"plain sub-words instead of the delimiter tokens, so the model cannot "
"copy a delimiter token out of untrusted input and have it parsed as "
"a real tool call. Requests whose content the chat template rewrites "
"are rejected rather than encoded unsafely. Requires "
"--dyn-chat-processor=sglang."

@akshatha-k akshatha-k Nov 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] (7) --help text lacks the default and the per-model caveat.

The docs say it's off by default and should be evaluated per model; the help text, which operators see first, says neither. Suggestion: append "Off by default; evaluate against your model before enabling."

@ydaiming ydaiming Nov 10, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 07e53163ee: the help text ends with 'Evaluate against your model before enabling.' The default is already appended by the argument helper (... env var: DYN_ISOLATE_TOOL_CALL_MARKERS | default: False), so I did not repeat 'off by default' in the sentence.

),
)
add_argument(
g,
flag_name="--dyn-chat-processor",
Expand Down
Loading
Loading