Initial structural_tag support for tool calling

[mgoin]

Purpose

My initial start to attempt progress on #32142

Adds structural tag support for tool calling so that tool_choice="required" constrains only the JSON arguments region, not the entire output. This allows models to generate their native tool tokens (e.g., <tool_call>) and any reasoning before the constrained region. Implemented for Hermes parser as reference; other parsers can override get_structure_info(). Currently disabled for tool_choice="auto" due to test regressions - needs further investigation into xgrammar trigger behavior.

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request introduces support for structural tags in tool calling, which is a great enhancement. It allows for more flexible and native tool call generation by constraining only the JSON arguments instead of the entire model output. The implementation adds a new abstraction in ToolParser and provides a concrete implementation for the Hermes tool format. A new utility module structural_tag_utils.py is added to build the configuration for the backend.

The overall approach is solid, but I've found a critical issue in the new structural_tag_utils.py file that could lead to a runtime crash due to a missing None check. I've provided comments with suggestions to fix this. Please address this to make the implementation more robust.

[chaunceyjiang]

Yesterday I tried testing structural_tag on minimax_m2 and found that structural_tag doesn’t support its minimax:tool_call very well.
The minimax:tool_call format is quite special, and it seems that xgrammar doesn’t support it yet.
#32232

[mgoin]

Do you know if this might work with the guidance backend?