Enrich chat_parsing streaming events by yonigozlan · Pull Request #40478 · sgl-project/sglang

yonigozlan · 2026-09-20T16:21:38Z

Extend chat_parsing streaming events with just enough context for serving adapters, while keeping parsing results unchanged.

Context

Transformers response templates describe how generated text is split into structured message fields. The core parser can already parse those fields, but its original events are intentionally minimal.

A serving adapter needs a little more: where each delimiter sits in the parser input, the tool name captured by an opener before the region body exists, and a way to recover when one region fails to parse. Without that, every serving library has to reconstruct parser state on its own.

Event contract

Event	Keys added by this PR	Meaning
`region_open`	`start`, `end`, `captures`	Span of the opening delimiter in `input_text` (empty for implicit opens). Explicit opens also carry the opener's named groups.
`region_chunk`	none	`text` is raw region text. `dirty=False` only for untransformed `text` fields; whitespace trimming still happens at close.
`region_close`	`start`, `end`	Span of the closing delimiter (empty at implicit boundaries and end of stream), alongside the parsed `value`.
`region_malformed`	new event	Replaces `region_close` when a region's value fails to parse or transform. Carries `start`, `end`, and the original `error`.

ResponseParser.input_text exposes the raw input after start-anchor truncation. Before the first feed() it is exactly the truncated prefix, so its length marks where generated text begins.
Prompt bytes held back at the end of the prefix as a possible delimiter are not re-emitted as generated region_chunk text.
Named opener captures let an adapter resolve a tool name as soon as the opener matches, then wait for the close event before emitting parsed arguments.

Malformed output

Streaming: a region that fails to parse ends with region_malformed and parsing continues, so later content and valid calls still parse.
Non-streaming: parse_response() re-raises the original exception from the first malformed region, even when required fields are also missing.

Scope

The grammar, transforms, and tool-argument coercion are unchanged. This PR only enriches the event and recovery contract consumed by the adapters in the next layer.

Stacked on Port chat_parsing core.

CI States

Latest PR Test (Base): ❌ Run #37720990537
Latest PR Test (Extra): ❌ Run #37720990107
Latest PR Test (AMD ROCm 10): ❌ Run #37720990395

Rocketknight1

A lot of these changes seem useful! A couple of them change the spec, though, like strict for xml-inline, and I'm not sure if that's necessary or not. Also, in some cases there are probably more changes than necessary (I flagged a couple of examples, there may be more!)

Overall I like the integration, though, and if there are necessary changes then I'm willing to upstream them to transformers! But it'd be cool if we could make sure that the changes are 100% minimal and focused, rather than a big sprawling update with unnecessary changes!

yonigozlan · 2026-09-22T20:29:20Z

Thanks for the reviews @Rocketknight1 ! I simplified the additions a bit following your comments

Rocketknight1 · 2026-09-23T16:20:07Z

Got it! If you're using agents for this, Opus 5.5 seems like it's a big improvement for simplifying/reviewing code. Might be worth handing the whole thing to it and asking if it can find conceptual simplifications or ways to reduce the diff without breaking funcitonality

yonigozlan · 2026-09-24T19:17:28Z

Thanks @Rocketknight1 ! Made some more simplifications, the diff is now +52/-18 to response_parser.py, and is mostly adding support for returning malformed region instead of raising, +exact positions of closing delimiters. I don't think we can do less without compromising on correctness now. Indeed Opus 5.5 helped a lot there!

Rocketknight1

Yes, that sounds great now! Thanks for the cleanup!

resubmitting with commenti nstead of changes requested

Jiminator

Just left one comment, thank you for the contribution!

Jiminator · 2026-09-26T08:45:34Z

+    stream = ResponseParser(response_template, prefix=prefix, tools=tools)
+    events = stream.feed(text)
+    message, final_events = stream.finalize()
+    for event in events + final_events:


Could you include stream.initial_events in this malformed-event check. A malformed JSON region closed in the assistant prefill currently returns {} instead of raising, while the implementation in #40477 raises for the same input.

Co-authored-by: Cursor <cursoragent@cursor.com>

A `repeats` field appends into the default list object owned by the compiled template (and, through the shallow copy, into the caller's spec dict), so one parse's tool calls leak into every later parse sharing the template. Deep-copy defaults at template load and at parser init, and add a regression test.

Co-authored-by: Cursor <cursoragent@cursor.com>

Delimiter text comes from the input offsets, and open events keep named captures. Unmatched text around XML tags no longer fails the parse.

Keep the event contract to what adapters need: open, close and malformed events carry the span of input_text they consumed, explicit opens carry their captures, and malformed events carry the original error. - Drop per-chunk offsets, prefix_end, close_start and closed; the prefix boundary is len(input_text) before the first feed(). - parse_response re-raises the original exception, like Transformers. Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

`parse_response` only checked events from the generated text, so a region the prefix left malformed was silently ignored. Include `initial_events` in the check. Co-authored-by: Cursor <cursoragent@cursor.com>

Jiminator

LGTM

JustinTong0323

The prefix fix and the simplifications look good. A few small things before this lands — the first one should be settled before the serving adapter builds on it.

JustinTong0323 · 2026-09-29T00:20:03Z

+        # Prefix bytes held back as a possible delimiter belong to the prompt, not the chunk.
+        text = text[max(0, self._prefix_len - self._pos) :]
+        if text:
+            dirty = field.content not in STREAMABLE_PARSERS


Clean chunks still aren't final-value fragments: <n>1/2a/</n> emits two dirty=False chunks and then goes malformed at close, and a text field with a transform emits clean chunks whose close value is a dict. Settle this before the adapter relies on it — either restrict clean to text without transform, or redefine dirty and make code, docs and tests agree.

JustinTong0323 · 2026-09-29T00:20:03Z

-    stream.feed(text)
-    message, _ = stream.finalize()
+    events = stream.feed(text)
+    message, final_events = stream.finalize()


finalize() runs before any malformed event is inspected, so a missing required field masks the region's real error: template with required x and y, input <x>{</x> raises "fields missing: ['y']" instead of the JSON parse error. Check the recorded malformed events before finalizing.

JustinTong0323 · 2026-09-29T00:20:03Z

                    f"Field '{field.name}': 'join' requires each match to parse to a string, "
                    f"got {type(value).__name__}."
                )
+        except (KeyError, TypeError, ValueError) as error:


RecursionError escapes this handler — a region body with deep nesting ('[' * 20000) aborts the whole stream instead of recovering (same through _coerce via the JSON content parser). Convert it to ValueError at the decode boundary so recover-and-continue holds.

JustinTong0323 · 2026-09-29T00:20:03Z

        self._opened = True
-        events.append({"type": "region_open", "field": field.name})
+        events.append(
+            {"type": "region_open", "field": field.name, "start": m.start(), "end": m.end(), "captures": self._captures}


This now aliases self._captures — a consumer mutating the event's captures dict changes the parsed value at close. Use a dict(...) copy to keep the snapshot semantics.

JustinTong0323 · 2026-09-29T00:20:03Z

        self.assertEqual(result, {"role": "assistant"})
        self.assertEqual(final_events, [])

+    def test_region_events_expose_delimiter_offsets(self):


Non-blocking: offsets are only exercised on single feeds. A mutation that mis-reports implicit-open offsets passes the whole suite — worth adding a chunking-invariance check on non-chunk events (compare full event dicts across chunkings, offsets included).

Jiminator · 2026-10-05T23:44:00Z

/tag-and-rerun-ci

JustinTong0323

LGTM

yonigozlan requested review from Fridge003, JustinTong0323, ispobock and merrymercy as code owners September 20, 2026 16:21

github-actions Bot added dependencies Pull requests that update a dependency file npu labels Sep 20, 2026

yonigozlan mentioned this pull request Sep 20, 2026

Add checkpoint-driven response-template adapters #40479

Open

yonigozlan force-pushed the feat/chat-parsing-events branch 2 times, most recently from 43c3c83 to a4904b5 Compare September 20, 2026 16:30

Rocketknight1 reviewed Sep 22, 2026

View reviewed changes

Comment thread python/sglang/srt/parser/chat_parsing/response_parser.py Outdated

Comment thread python/sglang/srt/parser/chat_parsing/response_parser.py Outdated

Comment thread python/sglang/srt/parser/chat_parsing/response_parser.py Outdated

yonigozlan force-pushed the feat/chat-parsing-events branch from a4904b5 to ada8f35 Compare September 22, 2026 19:09

yonigozlan requested a review from Rocketknight1 September 22, 2026 20:29

Rocketknight1 approved these changes Sep 25, 2026

View reviewed changes

Jiminator previously requested changes Sep 26, 2026

View reviewed changes

Comment thread python/sglang/srt/parser/chat_parsing/response_parser.py Outdated

Jiminator reviewed Sep 26, 2026

View reviewed changes

yonigozlan and others added 10 commits September 27, 2026 15:51

Port chat_parsing core

9b20eb9

Co-authored-by: Cursor <cursoragent@cursor.com>

Keep already-decoded array arguments unchanged.

2295531

Co-authored-by: Cursor <cursoragent@cursor.com>

Enrich response parser streaming events

75fc072

Co-authored-by: Cursor <cursoragent@cursor.com>

Simplify response region close offsets

2af08ac

Co-authored-by: Cursor <cursoragent@cursor.com>

Clarify response region close handling

6a463a0

Co-authored-by: Cursor <cursoragent@cursor.com>

Trim response event contract tests

5bb1485

Co-authored-by: Cursor <cursoragent@cursor.com>

Drop copied response event fields and xml-inline strict mode.

b2d9a1e

Delimiter text comes from the input offsets, and open events keep named captures. Unmatched text around XML tags no longer fails the parse.

Keep held-back prompt bytes out of streamed region chunks

261ec8a

Co-authored-by: Cursor <cursoragent@cursor.com>

Raise for malformed regions produced by the prompt prefix

106b751

`parse_response` only checked events from the generated text, so a region the prefix left malformed was silently ignored. Include `initial_events` in the check. Co-authored-by: Cursor <cursoragent@cursor.com>

yonigozlan force-pushed the feat/chat-parsing-events branch from 4fce556 to 106b751 Compare September 27, 2026 20:23

Merge main into feat/chat-parsing-events

30546df

Jiminator added run-ci CI: run the baseline test suite on this PR and removed run-ci CI: run the baseline test suite on this PR labels Sep 28, 2026

Jiminator reviewed Sep 28, 2026

View reviewed changes

JustinTong0323 reviewed Sep 29, 2026

View reviewed changes

Jiminator added 3 commits October 3, 2026 02:35

Fix chat parsing event contracts and error recovery

5ef3746

Merge branch 'main' into feat/chat-parsing-events

0bb6c88

Merge branch 'main' into feat/chat-parsing-events

499c3bd

github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Oct 5, 2026

JustinTong0323 approved these changes Oct 6, 2026

View reviewed changes

Jiminator added the bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) label Oct 8, 2026

Merge branch 'main' into feat/chat-parsing-events

5e781f7

Conversation

yonigozlan commented Sep 20, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Event contract

Malformed output

Scope

CI States

Uh oh!

Rocketknight1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yonigozlan commented Sep 22, 2026

Uh oh!

Rocketknight1 commented Sep 23, 2026

Uh oh!

yonigozlan commented Sep 24, 2026

Uh oh!

Rocketknight1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Jiminator left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jiminator left a comment

Choose a reason for hiding this comment

Uh oh!

JustinTong0323 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jiminator commented Oct 5, 2026

Uh oh!

JustinTong0323 left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yonigozlan commented Sep 20, 2026 •

edited by github-actions Bot

Loading