Skip to content

[Tool Parser] Add an Inkling structural tag so forced tool choice is enforced - #54120

Open
sungsooha wants to merge 5 commits into
vllm-project:mainfrom
sungsooha:inkling/structural-tag
Open

[Tool Parser] Add an Inkling structural tag so forced tool choice is enforced#54120
sungsooha wants to merge 5 commits into
vllm-project:mainfrom
sungsooha:inkling/structural-tag

Conversation

@sungsooha

@sungsooha sungsooha commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Problem

Inkling's tool parser sets structural_tag_model = None and supports_required_and_named = False, so tool_choice="required" and a named tool choice install no constraint and silently degrade to auto.

Measured on Inkling-NVFP4 (GB300, TP4): forcing get_weather on a prompt that matches a second tool returned calculate 5/5.

This is the Inkling instance of the family tracked in #50477 (which names inkling_tool_parser.py explicitly), alongside #49712 and #53363.

Why not a JSON-schema constraint

Inkling wraps its tool JSON in content markers:

<|message_model|><|content_invoke_tool_json|>{"name":"get_weather","args":{...}}<|end_message|>

Constraining raw JSON conflicts with that wire format. A structural tag constrains markers and payload together, which is exactly what hermes and kimi_k3 already do.

Change

Register an inkling builder modelled on the existing vLLM-owned ones:

tool_choice grammar
auto tool calls OR a text block (the model may answer instead)
required at least one call from the offered set, no text alternative
forced exactly the named tool, in the same container format as required

forced deliberately does not use a bare TagFormat at the root: that is a triggered constraint, so it never anchors and the model stays free to call a different tool. kimi_k3 wraps forced the same way for the same reason.

Validation

With the tag registered, on Inkling-NVFP4 / GB300 TP4, speculative decoding (MTP8) enabled:

  • forced get_weather honoured 5/5 (was 0/5)
  • tool_choice=required returns a single call (was 1–7 mixed calls)
  • 0 Failed to advance FSM errors

⚠ Deployer note — reasoning models need enable_in_reasoning

A reasoning model additionally needs --structured-outputs-config '{"enable_in_reasoning": true}'. Otherwise StructuredOutputManager.should_fill_bitmask never fills the grammar bitmask (it is gated on reasoning_ended, and enable_in_reasoning defaults to False), so the tag compiles and is completely inert — HTTP 200, no error, unconstrained output.

Inkling's is_reasoning_end scans token ids backwards and returns False on <|message_model|>, so a tool block never flips it. The same mechanism was described in #37388 (self-closed with "I realized that I can use enable_in_reasoning: True") and fixed model-side for Muse Glimmer in #52390.

A follow-up making Inkling's is_reasoning_end recognise a tool-block opening — as #50528 and #49876 did for adjacent Inkling cases — would remove the need for the flag. Happy to take that on in a separate PR if maintainers prefer it here.

Notes

Draft — opened for early feedback. ⚠ #53099 refactors structural_tag_registry.py; happy to rebase onto it.

@mergify mergify Bot added ci/build multi-modality Related to multi-modality (#4194) new-model Requests to new models qwen Related to Qwen models inkling nvidia rocm Related to AMD ROCm labels Aug 27, 2026
@mergify mergify Bot added the intel-gpu Related to Intel GPU label Aug 27, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Aug 27, 2026
@mergify mergify Bot added cpu Related to CPU backends speculative-decoding dflash mrv2 Model Runner V2 specific labels Aug 27, 2026
@mergify mergify Bot added the tool-calling label Aug 27, 2026
@sungsooha
sungsooha force-pushed the inkling/structural-tag branch from 08b6e9d to 533e800 Compare August 27, 2026 23:35
@sungsooha sungsooha mentioned this pull request Aug 28, 2026
Merged
sungsooha added a commit to sungsooha/dynamo that referenced this pull request Aug 28, 2026
@sungsooha
fix(container): anchor the Inkling structural tag at the content-kind…
ac199a6 
… marker

Refreshes the vendored patch from vllm-project/vllm#54120, which now begins the
tool/text tags at `<|content_invoke_tool_json|>` / `<|content_text|>` instead of
at `<|message_model|>`.

Inkling's chat template prefills the role marker into the generation prompt (it
ends `...<|end_message|><|message_model|>`), so generation resumes inside the
model message and never re-emits it. The tags are `tags_with_separator` with
`at_least_one=true`, so they anchor rather than trigger -- the old grammar
therefore required a SECOND role marker. Compiling the tag with xgrammar shows
the inversion, for forced/required/auto alike:

                        suffix-only   with duplicate marker
    before                 rejected              accepted
    after                  accepted              rejected

It appeared to work only because the parser absorbed the duplicate, while
manufacturing exactly the unconsumed-control-marker residue ai-dynamo#12510 strips.

Validated on Inkling-NVFP4 / GB300, vLLM 0.28.0, MTP 8, 2-replica AGG, applied
over the shipped image via ConfigMap: forced tool choice honoured 5/5 against a
prompt matching the other tool; required = 1 call; parallel_tool_calls true/false
= 2/1; no control-marker residue in content -- all in both streaming and
non-streaming. Full p0 blocking count unchanged at 2 (the unrelated
reasoning_budget_zero vLLM limitation).

Note the probe suite could NOT have caught this: control_marker_residue passed
before the fix too, because the parser was absorbing the duplicate. The
grammar-level acceptance test is what exposed it.

Reported by dynamo-review-agent on ai-dynamo#13954.

Signed-off-by: Sungsoo Ha <sungsooh@nvidia.com>
@sungsooha
sungsooha marked this pull request as ready for review August 28, 2026 05:27
claude[bot]
claude Bot reviewed Aug 28, 2026
View reviewed changes

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

sungsooha added a commit to sungsooha/dynamo that referenced this pull request Aug 28, 2026
@sungsooha
fix(container): parser-counter reasoning accounting; refresh the Inkl…
6908a25 
…ing patch

Syncs both of our unmerged fixes to their reviewed state and re-validates the
container on real hardware.

frontend (ai-dynamo#13947, P1 review): reasoning tokens are now counted by
the pinned vLLM parser's own `count_reasoning_tokens` over a per-choice ledger of
the ORIGINAL generated ids, appended before any branch in process_output. The
per-chunk classifier and the terminal `tokenizer.encode(saved_reasoning)`
retokenisation are both removed -- the latter drifted because detokenisation is
not injective (generated id 77150 decodes to '１０' and re-encodes to two tokens).

vLLM patch (vllm-project/vllm#54120): regenerated against the v0.28.0 tree, now
also accepting `<|content_model_end_sampling|>`. The Inkling parser treats it as
a co-equal terminator -- `is_reasoning_end` returns True on it and
`count_reasoning_tokens` closes a block on either -- so permitting only
`<|end_message|>` masked a valid natural stop. Applies clean with the
Dockerfile's own `patch --batch --forward -p1`.

Validated on Inkling-NVFP4 / GB300, vLLM 0.28.0, MTP 8, on BOTH topologies with
the full patch set overlaid on the shipped image. p0: 13/15 with 2 blocking on
each, and the two blockers are the same unpatchable vLLM limitation
(reasoning_budget_zero, NVBug 6678449a):

  2-replica AGG  structural_tag_grammar pass · reasoning ratio 1.00 · kv_routing pass
  1P1D disagg    structural_tag_grammar pass · reasoning ratio 1.00

p0 was extended to cover these changes, because it structurally could not before:
the tag can be wrong while every wire-level probe passes (the parser absorbs the
malformed output -- control_marker_residue passed before BOTH tag defects), and
reasoning_tokens can be non-zero but the wrong order (the pre-fix code reported a
constant 7 against 324 real tokens).

Signed-off-by: Sungsoo Ha <sungsooh@nvidia.com>
sungsooha added a commit to sungsooha/dynamo that referenced this pull request Aug 28, 2026
@sungsooha
build(container): add Inkling vLLM 0.28.0 runtime support
ba7fa4c 
Bumps the vLLM runtime base to v0.28.0-ubuntu2404 and layers the Inkling
structural-tag patch onto it, following the Nemotron-3 Ultra pattern (ai-dynamo#10234).

Why 0.28.0: Inkling's tool parser and the structural-tag registry the patch
extends only exist from that release. The tag is a multi-arch manifest list
(linux/arm64 + linux/amd64), so GB300 aarch64 is covered. xpu and cpu stay at
0.27.1 -- untested at 0.28.0, and Inkling does not use them.

Why the patch: vLLM ships an Inkling tool parser but registers no structural tag
for it, so Dynamo finds no grammar to install for a tool request and silently
degrades to tool_choice="auto" -- HTTP 200, no warning, and a forced tool choice
is simply not honoured. Upstream as vllm-project/vllm#54120; the .patch here is
cut against the exact v0.28.0 tree and applies with `patch --forward -p1` clean.

validate_inkling_runtime.py asserts the POSTCONDITIONS rather than trusting
patch's exit code: 'inkling' registered in both the builtin set and the builder
registry, structural_tag_model == "inkling", and supports_required_and_named
equal to the value AbstractToolParser.__init_subclass__ derives from
VLLM_ENFORCE_STRICT_TOOL_CALLING (setting it by hand is redundant and silently
overwritten). Verified against the real patched runtime, and verified to FAIL
when the tag is unregistered or the vLLM version differs.

The build-time version test is load-bearing: `patch --forward` would otherwise
skip or fuzz a hunk on a different base and leave a half-patched image that
still builds successfully.

NOTE: the image alone is not sufficient. The worker must also run with
--dyn-enable-structural-tag AND
--structured-outputs-config '{"enable_in_reasoning": true}'; without the second,
vLLM never fills the grammar bitmask for a reasoning model and the grammar is
inert. That belongs in the recipe, not here.

Signed-off-by: Sungsoo Ha <sungsooh@nvidia.com>
sungsooha added a commit to sungsooha/dynamo that referenced this pull request Aug 28, 2026
@sungsooha
fix(container): anchor the Inkling structural tag at the content-kind…
087d000 
… marker

Refreshes the vendored patch from vllm-project/vllm#54120, which now begins the
tool/text tags at `<|content_invoke_tool_json|>` / `<|content_text|>` instead of
at `<|message_model|>`.

Inkling's chat template prefills the role marker into the generation prompt (it
ends `...<|end_message|><|message_model|>`), so generation resumes inside the
model message and never re-emits it. The tags are `tags_with_separator` with
`at_least_one=true`, so they anchor rather than trigger -- the old grammar
therefore required a SECOND role marker. Compiling the tag with xgrammar shows
the inversion, for forced/required/auto alike:

                        suffix-only   with duplicate marker
    before                 rejected              accepted
    after                  accepted              rejected

It appeared to work only because the parser absorbed the duplicate, while
manufacturing exactly the unconsumed-control-marker residue ai-dynamo#12510 strips.

Validated on Inkling-NVFP4 / GB300, vLLM 0.28.0, MTP 8, 2-replica AGG, applied
over the shipped image via ConfigMap: forced tool choice honoured 5/5 against a
prompt matching the other tool; required = 1 call; parallel_tool_calls true/false
= 2/1; no control-marker residue in content -- all in both streaming and
non-streaming. Full p0 blocking count unchanged at 2 (the unrelated
reasoning_budget_zero vLLM limitation).

Note the probe suite could NOT have caught this: control_marker_residue passed
before the fix too, because the parser was absorbing the duplicate. The
grammar-level acceptance test is what exposed it.

Reported by dynamo-review-agent on ai-dynamo#13954.

Signed-off-by: Sungsoo Ha <sungsooh@nvidia.com>
sungsooha added a commit to sungsooha/dynamo that referenced this pull request Aug 28, 2026
@sungsooha
fix(container): parser-counter reasoning accounting; refresh the Inkl…
af53999 
…ing patch

Syncs both of our unmerged fixes to their reviewed state and re-validates the
container on real hardware.

frontend (ai-dynamo#13947, P1 review): reasoning tokens are now counted by
the pinned vLLM parser's own `count_reasoning_tokens` over a per-choice ledger of
the ORIGINAL generated ids, appended before any branch in process_output. The
per-chunk classifier and the terminal `tokenizer.encode(saved_reasoning)`
retokenisation are both removed -- the latter drifted because detokenisation is
not injective (generated id 77150 decodes to '１０' and re-encodes to two tokens).

vLLM patch (vllm-project/vllm#54120): regenerated against the v0.28.0 tree, now
also accepting `<|content_model_end_sampling|>`. The Inkling parser treats it as
a co-equal terminator -- `is_reasoning_end` returns True on it and
`count_reasoning_tokens` closes a block on either -- so permitting only
`<|end_message|>` masked a valid natural stop. Applies clean with the
Dockerfile's own `patch --batch --forward -p1`.

Validated on Inkling-NVFP4 / GB300, vLLM 0.28.0, MTP 8, on BOTH topologies with
the full patch set overlaid on the shipped image. p0: 13/15 with 2 blocking on
each, and the two blockers are the same unpatchable vLLM limitation
(reasoning_budget_zero, NVBug 6678449a):

  2-replica AGG  structural_tag_grammar pass · reasoning ratio 1.00 · kv_routing pass
  1P1D disagg    structural_tag_grammar pass · reasoning ratio 1.00

p0 was extended to cover these changes, because it structurally could not before:
the tag can be wrong while every wire-level probe passes (the parser absorbs the
malformed output -- control_marker_residue passed before BOTH tag defects), and
reasoning_tokens can be non-zero but the wrong order (the pre-fix code reported a
constant 7 against 324 real tokens).

Signed-off-by: Sungsoo Ha <sungsooh@nvidia.com>
@sungsooha
[Tool Parser] Add an Inkling structural tag so forced tool choice is …
a094188 
…enforced

Inkling's tool parser sets `structural_tag_model = None` and
`supports_required_and_named = False`, so `tool_choice="required"` and a named
tool choice install no constraint and silently degrade to `auto`. Measured on
Inkling-NVFP4: forcing `get_weather` on a prompt matching a second tool returned
`calculate` 5/5.

A bare JSON-schema constraint is the wrong fix -- Inkling wraps its tool JSON in
content markers (`<|message_model|><|content_invoke_tool_json|>{...}<|end_message|>`),
so constraining raw JSON conflicts with the wire format. Register an `inkling`
structural tag constraining markers and payload together, modelled on the
existing vLLM-owned `hermes` / `kimi_k3` builders:

  auto     -- tool calls OR a text block (the model may answer instead)
  required -- at least one call from the offered set, no text alternative
  forced   -- exactly the named tool, wrapped in the same container format as
              required; a bare TagFormat at the root is a *triggered* constraint
              and never anchors

With the tag registered, forced `get_weather` is honoured 5/5 and
`tool_choice=required` returns a single call instead of 1-7 mixed calls, with
speculative decoding (MTP8) enabled and no FSM failures.

NOTE for deployers: a reasoning model additionally needs
`--structured-outputs-config '{"enable_in_reasoning": true}'`. Otherwise
`StructuredOutputManager.should_fill_bitmask` never fills the grammar bitmask
(gated on `reasoning_ended`; `enable_in_reasoning` defaults to False), so the tag
compiles and is inert -- HTTP 200, no error, unconstrained output. Inkling's
`is_reasoning_end` scans backwards and returns False on `<|message_model|>`, so a
tool block never flips it.

Signed-off-by: Sungsoo Ha <sungsooh@nvidia.com>
@sungsooha
[Tool Parser] Do not set supports_required_and_named alongside struct…
5531e37 
…ural_tag_model

`AbstractToolParser.__init_subclass__` forces `supports_required_and_named = False`
whenever `structural_tag_model` is set and `VLLM_ENFORCE_STRICT_TOOL_CALLING` is
on (abstract_tool_parser.py:65-71). That is the intended design: the structural
tag supersedes the generic required/named JSON path.

Setting the flag to True in the subclass was therefore both redundant and
silently overwritten. Drop it and document why, so the next reader does not
re-add it.

No behaviour change -- the measured results (forced get_weather honoured 5/5
with MTP8) were obtained with the effective value already False.

Signed-off-by: Sungsoo Ha <sungsooh@nvidia.com>
@sungsooha
[Tool Parser] Anchor the Inkling structural tag at the content-kind m…
527afc3 
…arker

The tag constrains the GENERATED SUFFIX, but it began at `<|message_model|>`.
Inkling's chat template prefills that role marker into the generation prompt --
the prompt ends `...<|end_message|><|message_model|>` -- so generation resumes
inside the model message and never re-emits it.

Because the tool/text tags are `tags_with_separator` with `at_least_one=true`,
they anchor rather than merely trigger, so the grammar required a SECOND role
marker. Compiling the tag with xgrammar and feeding it token sequences shows the
inversion directly, for forced/required/auto alike:

                        suffix-only   with duplicate marker
    before                 rejected              accepted
    after                  accepted              rejected

It appeared to work only because the parser absorbed the duplicate -- while
manufacturing exactly the unconsumed-control-marker residue that dynamo#12510
exists to strip.

Begin the tags at `<|content_invoke_tool_json|>` / `<|content_text|>` instead;
`_INKLING_MESSAGE_MODEL` is then unused and removed.

Validated on Inkling-NVFP4 / GB300 (vLLM 0.28.0, MTP 8), 2-replica AGG: forced
tool choice honoured 5/5 against a prompt matching the other tool, required = 1
call, parallel_tool_calls true/false = 2/1, and no control-marker residue in
content -- in both streaming and non-streaming. p0 blocking count unchanged at 2
(the unrelated reasoning_budget_zero limitation).

Reported by dynamo-review-agent on ai-dynamo/dynamo#13954.

Signed-off-by: Sungsoo Ha <sungsooh@nvidia.com>
@sungsooha
[Tool Parser] Accept both Inkling block terminators in the structural…
b8ae1fc 
… tag

The tag permitted only `<|end_message|>`, but the Inkling parser treats
`<|content_model_end_sampling|>` as a co-equal terminator:

  * the parser docstring states "sampling may also end a block with the
    standalone <|content_model_end_sampling|> token";
  * `is_reasoning_end()` returns True on it (`token_id in (text_id,
    end_sampling_id)`);
  * `count_reasoning_tokens()` closes a reasoning block on either
    (`end_ids = {END_MESSAGE, CONTENT_MODEL_END_SAMPLING}`).

Constraining to one terminator therefore masks a valid natural stop and forces
the model to keep generating to reach the permitted token -- and it removes one
of the two paths that flip `is_reasoning_end`, which the reasoning plumbing
keys on.

`TagFormat.end` accepts `list[str]`, so both are permitted directly rather than
via an OrFormat; `AnyTextFormat.excludes` now lists both so the text block
cannot swallow either terminator.

Verified by compiling the tag with xgrammar, forced/required/auto:

                          <|end_message|>   <|content_model_end_sampling|>
    before                       accepted                        REJECTED
    after                        accepted                        accepted

The duplicate-role-marker fix from the previous commit is unaffected (a leading
`<|message_model|>` stays rejected).

Reported by dynamo-review-agent on ai-dynamo/dynamo#13954.

Signed-off-by: Sungsoo Ha <sungsooh@nvidia.com>
@sungsooha
sungsooha force-pushed the inkling/structural-tag branch from cebd319 to b8ae1fc Compare August 28, 2026 16:03
@mergify

mergify Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @sungsooha.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Aug 29, 2026
@sungsooha
Merge remote-tracking branch 'upstream/main' into inkling/structural-…
60804dc 
…tag-v2

# Conflicts:
#	vllm/tool_parsers/structural_tag_registry.py

Signed-off-by: Sungsoo Ha <sungsooh@nvidia.com>
@mergify mergify Bot removed the needs-rebase label Sep 2, 2026
@mergify

mergify Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @sungsooha.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Sep 10, 2026
@mergify mergify Bot removed the needs-rebase label Sep 18, 2026
@mergify

mergify Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @sungsooha.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Sep 29, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@claude claude[bot] claude[bot] left review comments
@aarnphm aarnphm Awaiting requested review from aarnphm aarnphm is a code owner
@chaunceyjiang chaunceyjiang Awaiting requested review from chaunceyjiang chaunceyjiang is a code owner
@sfeng33 sfeng33 Awaiting requested review from sfeng33 sfeng33 is a code owner
@bbrowning bbrowning Awaiting requested review from bbrowning bbrowning is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

ci/build cpu Related to CPU backends dflash inkling intel-gpu Related to Intel GPU mrv2 Model Runner V2 specific multi-modality Related to multi-modality (#4194) needs-rebase new-model Requests to new models nvidia qwen Related to Qwen models rocm Related to AMD ROCm speculative-decoding tool-calling

Projects

AMD
Status: Todo
NVIDIA
Status: No status
Sprint - DFlash
Status: Backlog
Tool Calling
Status: No status

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@sungsooha