[QwertyJack]

Purpose

Fixes #43920.

The V1 structured-output manager initializes one engine-level backend from the first structured-output request. However, backend=auto still resolves every request independently. A valid xgrammar-compatible schema followed by a valid schema using multipleOf resolves to xgrammar and guidance respectively, while EngineCore continues compiling both with the first backend. After #47312, this no longer kills EngineCore, but the second request still reaches grammar compilation through the wrong backend and returns HTTP 500.

This change adds two consistency checks:

• The frontend records the first resolved backend on its engine config and raises VLLMValidationError for a later mismatch, preserving an admission-time HTTP 400 for the common single-frontend path.
• StructuredOutputManager records the backend name after successful backend creation and rejects any later mismatch before backend construction or grammar compilation. This is the authoritative guard for multi-API-server and data-parallel topologies where frontend config state is process-local.

Requests without structured outputs do not execute either check.

This is not a duplicate of existing work:

• [Bugfix] handle grammar compilation failures to avoid engine crash #47312 handles engine-side grammar compilation exceptions and keeps EngineCore alive; it intentionally reports FinishReason.ERROR, which becomes HTTP 500.
• [Bugfix] Don't crash EngineCore when structured output grammar compilation fails #44401 is a broader scheduler and HTTP error-propagation change. This PR instead prevents a valid auto-fallback request from being routed to an incompatible cached backend.

Test Plan

PYTHONPATH=$PWD .venv/bin/python -m pytest tests/v1/structured_output/ -q
.venv/bin/pre-commit run --files vllm/sampling_params.py vllm/v1/structured_output/__init__.py tests/v1/structured_output/test_validation.py

The frontend regression test uses real xgrammar and guidance validation. The first JSON schema resolves to xgrammar; a second valid schema containing multipleOf resolves to guidance and must be rejected because the engine already selected xgrammar.

The manager regression test models two independent frontends sending requests resolved to xgrammar and guidance into one EngineCore. It verifies that the second backend is not constructed and the guidance grammar is not compiled through the cached xgrammar backend.

Test Result

Pre-fix, order-dependent repro on unmodified main (one engine config with backend=auto, real xgrammar 0.2.2):

1) frontend per-request auto resolution on one engine config:
  valid JSON schema resolves to: xgrammar
  duplicate-root EBNF resolves to: guidance
2) engine-side compilation with the latched backend:
  JSON: compiled by latched xgrammar
  GRAMMAR: FAILED under latched xgrammar -> RuntimeError

The same request succeeds on a fresh engine (guidance accepts the EBNF) and fails only after an xgrammar request latched the backend, which is why this shows up as intermittent 500s rather than as a deterministic failure of a single request.

Regression tests:

• tests/v1/structured_output/test_validation.py: 20 passed (2 new tests; all pre-existing validation tests still pass).
• tests/v1/structured_output/: 74 passed.
• pre-commit run --files ... for the three touched files: all hooks passed, including ruff, mypy, typos, config validation, and SPDX checks.
• Model evaluation is not applicable because generation logic and model outputs are unchanged; this only rejects unsupported mixed-backend routing before it reaches an incompatible compiler.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• Documentation update is not required for this request-validation bug fix.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[sfeng33]

Thanks for the PR! The fix is correct for a single frontend but silently incomplete for the default multi-API-server / data-parallel topology.

I think the single-backend check belongs in StructuredOutputManager.grammar_init() inside the shared EngineCore, immediately before backend creation/grammar compilation, conceptually:

if self.backend is None:
    self.backend_name = request_backend
    self.backend = create_backend(request_backend)
elif request_backend != self.backend_name:
    reject_request(...)

Wdyt?

[QwertyJack]

Good catch, agreed. The frontend config state is process-local, so it cannot be the only guard.

I updated the PR to record the selected backend in StructuredOutputManager after successful backend creation and reject a different request backend before backend construction or grammar compilation. I kept the frontend check as an early HTTP 400 path when requests share one frontend; the manager check is now authoritative for multi-API-server / DP topologies.

I also added a regression test that models two independent frontends resolving requests to xgrammar and guidance before sending them to one manager. It verifies that the guidance backend is not constructed and the second grammar is not compiled through the cached xgrammar backend. The focused suite now passes 6 tests. Thanks for pointing this out.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @QwertyJack.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[arpera]

Hi @QwertyJack, thank you for your PR! I see you implemented approach with rejection of requests that needs to use different grammar backends.
What do you think of an approach that would process these queries and not reject them? For example, to store multiple grammar backends per StructuredOutputManager. I believe this would be a better approach not to reject such requests. What do you think about it, could this be implemented? Or there are some obstacles that stops us from this idea in general? Would be glad to hear your thoughts on this first.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @QwertyJack.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[QwertyJack]

@arpera Thanks for the suggestion! I agree that supporting multiple backends would be preferable for backend=auto.

This PR preserves the existing single-backend constraint and prevents requests from reaching an incompatible compiler. Removing that constraint would also require validating mixed-backend batches, bitmask compatibility, and async/speculative decoding paths. I expect that work to extend beyond this PR’s scope and be better addressed separately.

[arpera]

/ci run

[github-actions]

✅ Triggered Buildkite CI #96098 for commit 1e5bd327060b.

[arpera]

/ci retry

[github-actions]

✅ Triggered Buildkite CI #96121 for commit a5fba8b7cccf, running 5 failed step(s) from Buildkite CI #96098.

[arpera]

@QwertyJack, please, look at this failed ci job first
https://buildkite.com/vllm/ci/builds/96121/list?jid=01a158b5-7a2c-49a3-aa2b-50d40be0f607&tab=output
It looks like this PR broke some of the tests

[arpera]

/ci run

[github-actions]

✅ Triggered Buildkite CI #96225 for commit ec442299940d.