[Bugfix][Structured Output] Mask request stop tokens in xgrammar until grammar terminates

[sfeng33]

Purpose

Fixes #42403.

With structured outputs (e.g. JSON schema), a model can sample a stop token while the grammar FSM is still mid-object, truncating generation into invalid JSON. On Gemma this is <end_of_turn> (id 106).

Root cause: xgrammar only knows the tokenizer's single eos, but a request's real stop set is generation_config's eos list plus any user stop_token_ids. On Gemma-4 that gap is concrete — generation_config eos is [1, 106, 50] while the tokenizer eos is just <eos> (1). Tokens 106/50 are invisible to the grammar, so they escape the bitmask in non-terminal states.

Fix

Forward the request's sampling_params.all_stop_token_ids through compile_grammar(...) into xgrammar's native
GrammarMatcher(override_stop_tokens=...). xgrammar then gates every one of the request's stop tokens to the grammar-terminal state — masked mid-object, allowed only once the schema is satisfied.

Test Plan

 vllm serve google/gemma-4-31B-it \
  --served-model-name gemma4 \
  --structured-outputs-config '{"backend":"xgrammar"}' 

Force <end_of_turn> (id 106) via logit_bias + greedy:

from openai import OpenAI
c = OpenAI(base_url="http://localhost:8000/v1", api_key="x")
schema = {"type":"object","properties":{"name":{"type":"string"},
          "bio":{"type":"string"}},"required":["name","bio"]}
r = c.chat.completions.create(
    model="gemma4", temperature=0, max_tokens=128, logit_bias={"106": 100},
    messages=[{"role":"user","content":"Return a person as JSON: name and a one-line bio."}],
    extra_body={"structured_outputs": {"json": schema}})
print(r.choices[0].message.content, "| stop_reason:", r.choices[0].stop_reason)

Test Result

	Output	valid JSON
before	{"name": " (truncated, stop_reason=106)	❌
after	{"name": "Elara Vance", "bio": "..."} (stop_reason=106)	✅

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @sfeng33.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[yzong-rh]

LGTM. Reproduces cleanly, test passes.

The issue occur in guidance as well, but it doesn't seem mask out stop tokens even if we override eos_token in llguidance_hf.from_tokenizer. Separate fix required.

[sfeng33]

/ci run

[github-actions]

✅ Triggered Buildkite CI #83175 for commit c2b5c3268ca4.

[bbrowning]

Pulled locally, reviewed, and confirmed the new test properly catches this regression. Looks good!