[BugenZhao]

Overview

Python rejects a structured-output JSON schema nested more than 64 levels deep, and a structural tag nested more than 64 + 16 levels, with a 400 (check_json_nesting in vllm/sampling_params.py, #60036). The Rust frontend has no cap and forwards any schema to EngineCore, where grammar conversion cost grows roughly with size × depth²: #60036 measured 98 s for a depth-512 structural tag. This PR applies the same caps and message in the Rust frontend.

Claims

• A JSON schema (object or JSON string) nested more than 64 levels, or a structural tag nested more than 80, returns 400 "The JSON schema is nested too deeply: objects and arrays may nest at most 64 levels" before the request reaches EngineCore.
• This covers chat response_format, structured_outputs on chat, completions, and generate, and the tool-call structural tags that output parsers build.
• A structural tag that is not valid JSON now returns 400 "Invalid structural tag specification." instead of being forwarded to EngineCore, as Python's backend validation does.

Validation

Commands from .buildkite/scripts/run-rust-frontend-cargo-ci.sh (style-clippy and test), run from the repo root:

cargo fmt --manifest-path rust/Cargo.toml --all -- --check   # ok
cargo sort --workspace --check rust                          # ok
cargo deny --manifest-path rust/Cargo.toml --config rust/deny.toml check bans   # ok
cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --all-features --locked -- -D warnings   # ok
cargo nextest run --manifest-path rust/Cargo.toml --workspace --all-features --locked --no-fail-fast
# 2187 tests run: 2187 passed, 13 skipped

• validate_caps_json_schema_nesting (vllm-engine-core-client): 64 levels pass and 65 fail with Python's message, both as a schema object and as a JSON string.
• validate_caps_structural_tag_nesting_with_wrapper_allowance: a structural tag passes at 80 levels and fails at 81.
• validate_rejects_malformed_structural_tag: a non-JSON structural tag returns "Invalid structural tag specification."
• prepare_chat_request_rejects_deeply_nested_json_schema (vllm-server): a chat response_format with a 65-level schema returns 400 with the message.

The behavioral tests fail with the check removed from validate.

Details

The check lives in StructuredOutputsParams::validate (vllm-engine-core-client), the existing owner of Python's _validate_structured_outputs rules. It already runs at every entry point, including lower_sampling_params on the final constraint, so parser-built tool-call structural tags are covered as in Python. A new nesting submodule sits next to the existing xgrammar one. Every constraint is checked on its parsed value: the schema check runs at the start of validate_json_schema, which both JSON branches already call, and a structural tag is parsed and then checked. serde_json stops parsing at 128 levels, so a document deeper than that is rejected as invalid JSON (still a 400, with a different message than Python's). For the same reason a structural tag that fails to parse is rejected, so a tag past the parse limit cannot skip the check.

One difference: Rust's parser-built structural tags nest tool argument schemas up to 20 wrapper levels deep (Python's stay within 15), so for that parser a strict tool's parameter schema is effectively capped near 62 levels instead of 64. Real tool schemas are far below either limit, so this PR keeps Python's single 16-level allowance.

Not a duplicate: no open upstream PR adds a nesting cap to the Rust frontend; #62506 validates JSON Schema keyword values in the Python frontend.

AI assistance (Claude Code) was used to prepare this change; the human submitter reviewed every changed line.

🤖 Generated with Claude Code

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[BugenZhao]

/ci run

[github-actions]

✅ Triggered Buildkite CI #96155 for commit 1773cd431224.

ℹ️ This PR is 9 commits behind upstream main. Merged upstream main into this branch as 1773cd431224 before starting CI. Pull the branch before pushing more changes.