Add EBNF and Lark sampling temperature support

[Ubospica]

Summary

• add [temperature=VALUE] attributes for Lark terminals and subgrammars, including nested override and ambiguous-path handling
• expose effective temperatures through GrammarMatcher, BatchGrammarMatcher, and draft-tree traversal APIs
• preserve temperature metadata through grammar transformations and serialization, with documentation and 28 focused Python tests

Test plan

• pre-commit run --all-files
• ruff check python/xgrammar/matcher.py tests/python/test_temperature.py tests/python/test_serialization.py
• HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 pytest -q tests/python/test_temperature.py tests/python/test_lark.py tests/python/test_grammar_matcher_basic.py tests/python/test_speculative_decoding.py tests/python/test_serialization.py tests/python/test_grammar_parser.py (349 passed)

[Copilot]

Pull request overview

This PR adds end-to-end support for rule-scoped sampling temperatures in Lark grammars, surfaces the effective temperature through matcher APIs (single, batch, and speculative draft-tree traversal), and preserves the metadata through grammar transformations, printing/parsing, and JSON serialization.

Changes:

• Add [temperature=VALUE] attributes for eligible Lark rules/subgrammars and propagate “active temperature” through the Earley parsing state.
• Expose effective temperatures via GrammarMatcher.temperature, return temperatures from BatchGrammarMatcher.batch_fill_next_token_bitmask, and optionally fill per-node temperatures in GrammarMatcher.traverse_draft_tree.
• Persist temperature through grammar transforms + EBNF printing/parsing + JSON serialization, with new docs and focused Python tests.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/python/test_temperature.py	Adds focused tests for default/override/ambiguity behavior, batch API return values, traversal temperature filling, and validation.
tests/python/test_serialization.py	Updates expected serialized rules shape to include the new temperature field.
python/xgrammar/matcher.py	Adds default_temperature, temperature property, traversal temperatures output tensor, and batch API return value.
include/xgrammar/matcher.h	Extends C++ matcher APIs with default_temperature, GetTemperature, traversal temperatures, and batch-return temperatures.
docs/defining_structures/lark_grammar.md	Documents %json/%lark and adds new “Sampling Temperature” rule option documentation.
cpp/tvm_ffi/tvm_ffi.cc	Adds optional-float marshaling, threads temperature data through FFI, and returns temperatures from batch bitmask fill.
cpp/lark_converter.cc	Parses temperature rule attributes, restricts applicability, and persists per-rule temperature into the grammar builder.
cpp/grammar_printer.cc	Prints [temperature=...] on rules with temperature, using a precise float representation.
cpp/grammar_parser.h	Adds FloatLiteral token type to support parsing printed temperatures.
cpp/grammar_parser.cc	Extends lexer/parser to read rule attributes and float literals; persists parsed temperature into the builder.
cpp/grammar_matcher.cc	Stores default_temperature, tracks active_temperature_rule_id through parsing, computes max temperature across ambiguous scan states, returns temperatures from batch fill, and fills traversal temperatures.
cpp/grammar_impl.h	Adds temperature to rule metadata and includes it in reflection-based serialization.
cpp/grammar_functor.h	Preserves rule temperature during grammar functor transformations.
cpp/grammar_functor.cc	Preserves rule temperature across multiple grammar mutators/transformers and avoids inlining/optimizations that would lose it.
cpp/grammar_builder.h	Adds UpdateRuleTemperature API.
cpp/grammar_builder.cc	Implements UpdateRuleTemperature.
cpp/earley_parser.h	Adds active_temperature_rule_id to parser state and introduces resolver for inherited temperature.
cpp/earley_parser.cc	Propagates active_temperature_rule_id through rule expansion/completion/prediction to support correct runtime temperature reporting.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cjackal]

Hi, I think the std::optional<float> type does not convert correctly to Optional[float] on xgrammar-python side in tvm_ffi.cc - on latest main one get a SyntaxError at import time like:

...
  def __c_ffi_init__(_0: Object, _1: Any, _2: bool, _3: int, _4: std::optional[float], /) -> Object: ...