[Fix] Simple fix: pass tokenizer stop tokens to XGrammar so constrained requests can terminate

[qifanchriswu001]

Motivation

Fixes #31533.

With the xgrammar backend, a structured-outputs request (regex / JSON schema / EBNF) never terminates when the model's real chat EOS is declared only in tokenizer_config.json and not in the model config's eos_token_id — as with checkpoints fine-tuned from a base model and shipped without generation_config.json, where config.json keeps the base EOS. XGrammar only unmasks stop tokens from the list SGLang passes it, and create_grammar_backend forwarded only model_config.hf_eos_token_id, while server-side stop detection (Req._check_token_based_finish) additionally honors tokenizer.eos_token_id and tokenizer.additional_stop_token_ids. Any EOS covered by stop detection but missing from the grammar's list stays masked at accepting states, so the request runs until max_new_tokens.

See #31533 for a minimal repro that drives xgrammar exactly as the scheduler does, plus the production symptom (17\n2\n3\n... at temperature 0 until the length cap; identical request without the constraint stops at 3 tokens). The fix has been verified end to end on the production setup — see Accuracy Tests below.

Environment where this was hit in production:

• SGLang lmsysorg/sglang:v0.5.15.post1-cu130 (the affected code path is unchanged on current main, 37f94cb)
• NVIDIA B200, CUDA 13.0, speculative decoding disabled, temperature 0
• Model: fine-tuned Qwen3.5-MoE (35B-A3B) checkpoint — config.json eos 248044 <|endoftext|>, tokenizer eos <|im_end|> 248046, no generation_config.json
• Grammar backend: xgrammar (default). The same weights and regex served with vLLM v0.22.0 terminate normally.

Modifications

• create_grammar_backend (xgrammar branch): pass the union of model-config EOS ids, tokenizer.eos_token_id, and tokenizer.additional_stop_token_ids as model_eos_token_ids — the same set Req._check_token_based_finish accepts. When no EOS information exists at all, keep passing None so XGrammar auto-detects from the tokenizer.
• Two unit tests in test/registered/unit/constrained/test_base_grammar_backend.py covering the union and the None passthrough.

Not addressed (possible follow-up): per-request sampling_params.stop_token_ids still does not reach the grammar matcher; supporting that would need per-request override_stop_tokens on the GrammarMatcher.

Accuracy Tests

No change to model forward or kernels. Verified end to end on the production setup (B200, SGLang nightly dev build 2026-06-19, xgrammar backend, speculative decoding disabled, temperature 0) by hand-applying the exact stop-token union this PR computes: adding a generation_config.json with "eos_token_id": [248046, 248044] to the affected checkpoint, which makes model_config.hf_eos_token_id contain the tokenizer EOS — the same set create_grammar_backend passes to xgrammar after this change. Everything else (image, weights, request) identical:

Request	Tokenizer EOS missing from stop list (before)	Tokenizer EOS in stop list (after)
regex-constrained	17\n2\n3\n..., finish_reason=length, 64/64 tokens	"17", finish_reason=stop, 3 tokens
unconstrained control	"17", finish_reason=stop, 3 tokens	unchanged

Speed Tests and Profiling

Not applicable — a few set operations at grammar-backend construction time (once per server start).

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

---

CI States

Latest PR Test (Base): ❌ Run #34447546015
Latest PR Test (Extra): ❌ Run #34447545847
Latest PR Test (AMD ROCm 10): ❌ Run #34447546000

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[qifanchriswu001]

@hnyls2002 @DarkSharpness @JustinTong0323 would appreciate a review ^