[Feature] Support Latest Xgrammar Structural Tag

[Hanchenli]

Purpose

This PR is created to support the upcoming XGrammar Structural Tag feature. The latest tags may be in the format of:

{
"type": "structural_tag",
"format": {
"type":
"triggered_tags",
"tags": [{
"begin": "hello_flag",
"content": {
"type": "any_text"
},
"end": "hello"
}],
"triggers": ["hello"],
"stop_after_first":
False
}
}

This requires us to update the format checking for the API format and the construction of structural tag at XGrammar Backend.

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request adds support for a new structural tag format in XGrammar, including updates to the API protocol and the backend implementation. The changes introduce backward compatibility for the legacy format while enabling the new one. My review focuses on improving the type safety of the new API definition and reducing code duplication in the backend logic. I've identified two high-severity issues related to these points.

[aarnphm]

quick notes.

[chaunceyjiang]

Xgrammar has not officially released this feature yet. Should we wait for the official release before integrating this new feature?

[aarnphm]

Xgrammar has not officially released this feature yet. Should we wait for the official release before integrating this new feature?

yes, this should be held, at least after 0.10.2

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Hanchenli.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[Hanchenli]

I did some merging and XGrammar version has been bumped. Maybe we can merge this now after review? @aarnphm @chaunceyjiang

[aarnphm]

Can you check the failure? has to do with the args rename recently.

[Hanchenli]

Can you check the failure? has to do with the args rename recently.

The tests seem to be passing for me in local.

[chaunceyjiang]

I suggest adding some usage examples in the examples directory.

[Hanchenli]

Closing as the functionality has been added through #25515