[BugenZhao]

Overview

A structured-output constraint that EngineCore fails to compile finishes the request as FINISHED_ERROR with no error attached. The Python frontend rarely sees this, because it validates constraints with the backend library before submitting. The Rust frontend cannot, so its clients get HTTP 500 for an invalid regex or grammar, and an empty 200 when llguidance builds a matcher in an error state (for example a schema with uniqueItems). This PR reports compile failures with the validation_error code that #49625 introduced for backend mismatches, so both frontends return 400, and makes the guidance backend fail compilation instead of continuing with a broken matcher.

Claims

• When a request's grammar fails to compile, its error output carries RequestError(code="validation_error", message="Invalid structured outputs: <compile error>"). Python already maps that code to VLLMValidationError (400) since [Bugfix] Reject mixed structured output backends during validation #49625.
• The guidance backend raises on a matcher that reports an error (Grammar error: ...) instead of logging a warning and generating nothing.
• The Rust frontend maps validation_error to HTTP 400 (an SSE BadRequestError event when streaming) and gRPC INVALID_ARGUMENT. This also covers [Bugfix] Reject mixed structured output backends during validation #49625's backend-mismatch rejection, which the Rust frontend reported as 500.
• Rust non-streaming /inference/v1/generate fails the request on every engine-typed error, as Python raises every RequestError; before, only retryable ones did.

Validation

GPU smoke on GB200, Qwen3-8B with --reasoning-parser qwen3 --tool-call-parser hermes via VLLM_USE_RUST_FRONTEND=1 vllm serve, main (e9cc6244a0) against this PR. On main the Rust frontend sends native HTTP constraints as guidance and parser-built output grammars as xgrammar, and EngineCore keeps the first request's backend, so two request orders cover both backends and #49625's mismatch rejection. Each order ends with a valid request; the engine stays healthy on both builds.

Order	Request	main	This PR
guidance first	completions regex: "("	200, empty text	400 Grammar error: ... unclosed group
	completions grammar: "root ::= ("	500	400 Failed to convert the grammar from GBNF to Lark ...
	completions json {"type": "array", "uniqueItems": true}	200, empty text	400 Grammar error: Unimplemented keys: ["uniqueItems"]
	chat regex (composed into an xgrammar tag, mismatch)	500	400 ... only supports one backend per engine ...
xgrammar first	chat regex: "(" (composed)	500	400 Regex parsing error at position 2 ...
	chat streaming grammar: "root ::= ("	SSE InternalServerError	SSE BadRequestError
	completions regex (guidance, mismatch)	500	400 ... only supports one backend per engine ...

Tests:

• Python (tests/v1/core/test_scheduler.py, test_async_scheduler.py, tests/v1/structured_output/test_backend_guidance.py, test_validation.py, tests/test_outputs.py): 104 passed. test_grammar_compile_error_finishes_only_request now asserts the error output's RequestError; the new test_backend_guidance_rejects_grammar_matcher_cannot_build compiles a uniqueItems schema. Both fail with the engine changes reverted (error is None; compilation does not raise).
• Rust: cargo nextest run --workspace (2229 passed), clippy, fmt. engine_request_errors_map_to_http_status (was retryable_engine_error_returns_service_unavailable) runs a retryable and a validation error through completions, chat, and raw generate against the mock engine; engine_errors_map_to_status_codes covers gRPC.

Details

Root cause. Grammar compilation runs asynchronously after StructuredOutputManager.grammar_init. Since #47312, a failed compilation finishes only that request, through grammar_compile_error_reqs, but without a RequestError, so every frontend reports an internal error. #49625 added validation_error for the separate case where a request names a backend other than the one the engine already uses; that check runs in grammar_init, before compilation, and does not cover a constraint that the right backend fails to compile. The guidance backend creates its LLMatcher without failing on a build error: check_error() only logs it, and the matcher allows nothing, so the request ends with empty output.

Why validation_error. It is the existing "the request itself is invalid" signal on RequestError, and Python's to_exception already maps it to 400; a compile failure of the request's own constraint is the same kind of error. Python's frontend validation already returns 400 for these constraints when it can check them, so this only changes requests that reach the engine unvalidated. The Rust side adds RequestError::VALIDATION_ERROR and is_validation_error() next to the existing retryable handling rather than an enum: code is an open string on the Python side, and no other code needs distinct Rust behavior.

Not changed.

• A streaming request still gets HTTP 200 with an SSE error event, as with Python, since compilation finishes after the response starts.
• xgrammar error messages include its log prefix ([time] /project/cpp/...:line:). Python's HTTP layer rewrites such paths with sanitize_message; the Rust frontend passes the message through.

Not a duplicate.

• [Bugfix] Reject mixed structured output backends during validation #49625 (merged) rejects backend mismatches before compilation; this PR reports compilation failures with the same code.
• Handle structured output grammar compilation failures #45012 (open, inactive since 2026-08-19) handles failed grammar futures leaving requests blocked and reports them through the internal-error path (500); [Bugfix] handle grammar compilation failures to avoid engine crash #47312 has since fixed the hang. This PR changes the status of those failures to 400.
• [Rust Frontend] Select one structured outputs backend for every route #62456 (open) gives every Rust route one backend, which removes the mismatch case for the Rust frontend; the two PRs are independent.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

AI assistance (Claude) was used to investigate, write, and test this change. I reviewed every changed line and ran the tests and GPU smoke above.

🤖 Generated with Claude Code

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[BugenZhao]

/ci run

[github-actions]

✅ Triggered Buildkite CI #96538 for commit 91d70c34d21f.

ℹ️ This PR is 43 commits behind upstream main. Merged upstream main into this branch as 91d70c34d21f before starting CI. Pull the branch before pushing more changes.