Repository navigation
Conversation
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
A `repeats` field appends into the default list object owned by the compiled template (and, through the shallow copy, into the caller's spec dict), so one parse's tool calls leak into every later parse sharing the template. Deep-copy defaults at template load and at parser init, and add a regression test.
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Delimiter text comes from the input offsets, and open events keep named captures. Unmatched text around XML tags no longer fails the parse.
Keep the event contract to what adapters need: open, close and malformed events carry the span of input_text they consumed, explicit opens carry their captures, and malformed events carry the original error. - Drop per-chunk offsets, prefix_end, close_start and closed; the prefix boundary is len(input_text) before the first feed(). - parse_response re-raises the original exception, like Transformers. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
`parse_response` only checked events from the generated text, so a region the prefix left malformed was silently ignored. Include `initial_events` in the check. Co-authored-by: Cursor <cursoragent@cursor.com>
Connect checkpoint `response_template` metadata to SGLang reasoning and tool parsing through the `chat_parsing` streaming events. Co-authored-by: Cursor <cursoragent@cursor.com>
- Configure response-template detokenization in `_process_messages`, so the Responses API gets the same special-token settings as Chat and no longer inspects `ResponsesRequest` as a chat request. - Honor `force_nonempty_content` in the reasoning detector. - Define `think_excluded_tokens` and `get_think_end_token_ids`, which strict thinking and scheduler startup read from every reasoning detector. - Register `response_template` in the CLI parser name lists. Co-authored-by: Cursor <cursoragent@cursor.com>
With tool_choice "required" or a named tool, a response-template tool parser has no native constraint, so serving uses the generic JSON schema and parses the JSON array the model writes after its reasoning. Two things broke that path: - When every field of the template has an opener, the reasoning parser has no field for text outside them, so it dropped the JSON. - Detokenization keeps a tool-call closer that stops generation, so a JSON array ended by one failed to parse, or streamed the closer as content. Serving now records on the request whether a grammar governs the output (tool JSON schema, response_format, regex, ebnf or structural tag). Only then does the reasoning parser pass text outside the template's fields through, and a kept closer is trimmed like any other matched stop. This covers Chat and Responses, streaming and non-streaming.
Serving now records where an output grammar takes over from the response template: after the reasoning when the grammar backend gates on it, or from the first token. The response-template reasoning parser stops at the reasoning closer, as the grammar backend does, and passes the rest through unparsed, so a JSON string containing template delimiters is no longer rewritten. Unconstrained requests keep the template's own parsing. JsonArrayParser also gains finish(). An increment emits a call's name before its arguments, so a call that arrived whole in the last chunk lost its arguments; Chat and Responses streams now flush it at stream end like the other tool parsers.
The grammar backend defers an output grammar until the reasoning ends only when the reasoning parser's think_end_token encodes to token ids; otherwise the grammar applies from the first token. A response-template thinking field that closes with a pattern has no such token, yet a request asking for reasoning was still marked to parse reasoning first, so the JSON the grammar produced from the start was dropped. Check the same condition the scheduler checks before recording where the grammar takes over.
Only JSON-schema output replaces the response template's framing. A structural tag, regex or EBNF constraint can spell the template's own delimiters, so its output keeps the template's parsing: content wrappers are removed and a native tool call that ends on its kept closer is still returned. The JSON repairs for required and named tool choice and response_format are unchanged. Also shorten the related comments and give the regression tests black-box docstrings in place of assertions on private state.
The two non-streaming Gemma 4 tool-call parity tests differed only in their input, so they are now subtests of one method, as the reasoning parity test already is. _collect_tool_stream repeated its call accumulation for the finish() result; one loop now covers every result. The Gemma 4 call's expected arguments were written out three times and are now named once next to the call. Every input, detector and assertion is kept.
The result of feeding "}" was assigned to closed and immediately overwritten by the "</call>" result; only the call itself matters.
`has_tool_call` streamed the text through the adapter, which holds back an opener whose pattern could still grow. A response cut off right after a call header was then treated as plain content and returned the raw opener. Parse the text as complete instead, so the cut-off call is dropped like any other malformed call. Signed-off-by: Yoni Gozlan <yonigozlan@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Jiminator
changed the title
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
github-actions
Bot
added
documentation
Jiminator
changed the title
This was referenced Oct 27, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
#45361 makes the cuTile BF16x3 router GEMM Beam's default router on Hopper and SM10x Blackwell.
cuda-tilecomes in throughflashinfer-python(cuda-tile>=1.4.0), but it compiles its kernels with thetileirascompiler. The CUDA 13.0 toolkit in the release image doesn't shiptileiras, and no current dependency installs it. Without it, a Beam server fails during prefill CUDA-graph capture on every TP rank withFileNotFoundError: 'tileiras' compiler not found.Split out of #44747 so the dependency can be reviewed on its own. Merge it right before #45361, and only if #45361 lands.
Modifications
Adds
nvidia-cuda-tileiras==13.4.92topython/pyproject.toml. Validation usedtileiras,nvidia-cuda-nvccandnvidia-nvvm13.4.92, which must share their major.minor.Validation
b57b5e2286without conflicts ([Feat] parser: Add Beam reasoning, tool calls and response templates (5/7) #45257 merges that main, which includes [Feat] parser: Enrich chat_parsing streaming events #40478; the others are on6efdd33d07).reflection/Beam-1.0-501B-A23B-NVFP4checkpoint and the cookbook GB300 NVFP4 low-latency recipe, serving main6efdd33d07+ [Fix] nccl: Disable graph buffer registration whenever a captured graph can hold a collective #43216, [Feat] parser: Enrich chat_parsing streaming events #40478, [Fix] parser: Accept whole-number floats for integer tool arguments #43941 and [Feat] parser: Add checkpoint-driven response-template adapters #40479 + this stack:/tokenizeall pass. A profiler trace shows the cuTile router kernel running.Merge order
Each part is based on the one before it, so its diff shows only its own changes. Merge bottom-up; after each part merges, the next part is retargeted to
mainandmainis merged into it. #44747 stays open, unchanged, as the full reference. Merge in this order:nvidia-cuda-tileiras, needed by [Perf] beam: Add optional cuTile BF16x3 router GEMM #45361#45292 (Dynamo and
grpcio-toolspins for the Beam image) is not in the merge order for now.CI States
Latest PR Test (Base): ❌ Run #39665206721
Latest PR Test (Extra): ❌ Run #39665206514
Latest PR Test (AMD ROCm 10): ❌ Run #39665206582