[Bugfix][Structured Output] Stop XGrammar token batches at termination

[sfeng33]

Purpose

Prevent XGrammar from receiving additional tokens after accepting a terminating stop token within the same batch.
Fixes #52767.

This covers two related problems:

1. accept_tokens() could return early after rejection without synchronizing LLM's cached _is_terminated state.
2. Batched speculative acceptance and validation could continue past a terminating token, producing XGrammar warnings and possible FSM state inconsistencies.

Test Plan

Unit test:

pytest tests/v1/spec_decode/test_mtp_structured_output.py

Live model test:

vllm serve Qwen/Qwen3.8-27B     
--tensor-parallel-size 2     
--enable-auto-tool-choice     
--tool-call-parser qwen3_coder     
--reasoning-parser qwen3     
--mm-encoder-tp-mode data     
--speculative-config '{"method":"mtp","num_speculative_tokens":3}

Scenario	How the bug was triggered	Before fix	After fix
Tokens after termination	Aligned EOS with MTP draft slot 1 or 2, leaving additional draft tokens after EOS	Matcher processed tokens after termination and emitted warnings	20/20 requests returned HTTP 200 without matcher errors
Second advance after termination	Used strict JSON with ignore_eos=true to force another grammar advance after EOS	HTTP 500	5/5 requests returned HTTP 200
Normal tool calling	Sent strict-tool requests using tool_choice=auto and required	Baseline behavior	10/10 requests returned correct tool calls

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[russellb]

Code review

Reviewed with the patch applied locally. Verification: both new tests fail on main (accept_tokens('mtp_req', [50256, 198]) returns False; xgrammar logs trying to accept new token with id 198) and pass with the fix. With the patch, tests/v1/spec_decode/test_mtp_structured_output.py (18 passed) and tests/v1/structured_output/ (59 passed) are green. I also checked the assumption the new validate_tokens depends on: in xgrammar 0.2.1, rollback(1) correctly undoes a stop-token acceptance (is_terminated() goes True -> rollback -> False -> re-accept works), so the break-then-rollback is safe.

The approach looks right. It brings xgrammar in line with backend_guidance.py, which already returns True from accept_tokens and [] from validate_tokens once stopped. Moving the _is_terminated sync inside the loop makes the desync structurally impossible rather than patching it after the fact, and the reset() change is a real latent bug fix (today a reset matcher stays flagged as terminated forever).

One issue outside the diff:

The overlapping open PR is not disclosed. #37506 has been open since March, touches the same two files, and addresses the same desync (syncing _is_terminated on the failed-accept path rather than preventing the overshoot). On that thread @chubby-cutie proposed essentially this approach. AGENTS.md asks that you explain the difference when an open PR already covers the fix. This looks like the better fix and should supersede #37506, but saying so lets a maintainer close the other one.

Minor scope note: backend_outlines.py and backend_lm_format_enforcer.py have no equivalent guard, and #52767's reporters hit this across many models. Keeping the fix xgrammar-only is reasonable, just worth stating.

Three inline comments below.

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[russellb]

agent comments were very minor. I'm happy with this when you're ready to merge it

[sfeng33]

/ci run

[github-actions]

✅ Triggered Buildkite CI #84456 for commit dd25c941229b.

[gaby]

@sfeng33 Thank you!