Skip to content

[Feat] Structured decisions endpoint (/v1/systemone) - #59299

Merged
mgoin merged 32 commits into
vllm-project:mainfrom
mmastrac:structured-decisions
Oct 8, 2026
Merged

mgoin merged 32 commits into
vllm-project:mainfrom
mmastrac:structured-decisions

Conversation

@mmastrac

@mmastrac mmastrac commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

This PR was prepared with the help of AI, but reviewed by myself. This PR description is fully human-written with the exception of the curl command below and the test result one-liner.

The current structured decisions server lives as an example. This example is untuned, requires additional setup and could run more effectively inside of vLLM itself.

This implements a new endpoint, with a single question type (choice), for AR models only: POST /v1/systemone

Example:

curl -s localhost:8000/v1/systemone -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-0.6B",
  "state": {"team": "billing", "language": "French"},
  "questions": {
    "team": {
      "type": "choice",
      "instructions": "Which team does the message name?",
      "criteria": {"billing": null, "shipping": null, "security": null}
    },
    "lang": {
      "type": "choice",
      "instructions": "Which language does the message name?",
      "criteria": {"English": null, "French": null}
    }
  },
  "seed": 7,
  "chat_template_kwargs": {"enable_thinking": false}
}'
{
  "id": "decision-80bd0a693f255e1e",
  "object": "structured_decision",
  "created": 1790714266,
  "model": "Qwen/Qwen3-0.6B",
  "answers": {
    "team": {
      "type": "choice",
      "choice": "billing",
      "probabilities": {"billing": 0.955, "shipping": 0.008, "security": 0.037},
      "confidence": 0.955
    },
    "lang": {
      "type": "choice",
      "choice": "French",
      "probabilities": {"English": 0.0006, "French": 0.9994},
      "confidence": 0.9994
    }
  },
  "usage": {"input_tokens": 254, "output_tokens": 2},
  "diagnostics": {
    "team": {"label_mass": 0.998, "argmax_is_label": true},
    "lang": {"label_mass": 0.9999, "argmax_is_label": true}
  }
}

Limits: 64 questions (only choice), and options capped at the smaller of the tokenizer's supported label set (see below) and 128.

For a generic endpoint, we need to consider a few things:

Tokenization

Each model tokenizes differently, so we must ensure that all choice-type questions provide the model a choice between N single-token options in a way that supports the answer format. The server tries labels A to ZZ in the template's answer format and keeps the largest group

Templates

Decision templates are fixed - we may decide to offer a template in the future if someone needs it.

Strategies

Decision strategies vary by model: next_token covers AR, an upcoming diffusion strategy will cover diffusion and other decision-specific models may wish to ship their own. The model uses decision_read_strategy to map the strategy.

Models without a registered read strategy return 501.


What's upcoming in future patches in the series:

  • Diffusion will be handled in an upcoming patch
  • Other question types, including those shipped by other providers were not included to keep this patch smaller
  • Probabilities are currently uncalibrated, and it is currently unplanned how we will make those pluggable (punted to the future)

Test Plan

Tests included, against Qwen, proving the request type works.

pytest tests/entrypoints/generate/structured_decisions

Test Result

28 passed on a GB10 (DGX Spark), with this branch's files mounted over a vllm-openai nightly image.


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

mmastrac and others added 10 commits September 29, 2026 11:12
POST /v1/systemone answers typed questions about a state with a probability
for every allowed answer, in the request and answer shapes of Jev's decision
API. Until now this lived only in the DiffusionGemma example server, which
calls vLLM over HTTP.

Each question is one read on the engine client: the chat prompt with the
assistant reply prefilled up to "id:", one generated token, and
logprob_token_ids set to the question's label tokens. Softmax over the labels
gives the answer. The reads share the system prompt and the state, so prefix
caching prefills the state once per request.

Question types are plugins: a QuestionType subclass registered with
register_question_type parses its criteria, names its labels and shapes its
answer, and the prompt, slot resolution and reads are shared. This PR
registers "choice" only. depends_on, ask_if and alone are refused with a 400
until they are implemented.

The route is registered with the other generate routes and served only when
the model supports "generate".

Tests: unit tests for the registry, the choice type and label resolution on
the Qwen3 tokenizer, and an end-to-end test on Qwen/Qwen3-0.6B (13 and 7
passing on a GB10 against the nightly-ddd6fbca image with this package
overlaid).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
A question with a field other than type, instructions and criteria now gets
a 400 naming the fields, replacing the fixed list of refused fields
(depends_on, ask_if, alone). An ignored field could be one that changes the
answer, so unknown fields are errors whatever they are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
…rategies

Decision templates. The system prompt of a decision now comes from a Jinja
template rendered in the same sandboxed environment as chat templates. The
built-in template keeps the previous wording. --decision-template takes a file
path or inline text, loaded like --chat-template, and a request may carry its
own decision_template when the server runs with --trust-request-chat-template.
A template may define an answer_prefix(question) macro, which the server calls
to prefill each read, so a template that asks for a different reply format
reads at the matching place. Without the macro the prefix is "<id>:". Per-type
option wording moved from QuestionType.describe into the template.

Read strategies. A registry picks a ReadStrategy per model at startup: the
first whose supports() accepts the model serves every decision. The existing
next-token read moved into NextTokenStrategy, which declines diffusion models,
so DiffusionGemma gets no decision route until a canvas strategy exists.
Each strategy reports DecisionLimits (questions per request, choices per
question) for its model, and the server enforces them. For next-token reads
that is 64 questions and the engine's 128 logprob_token_ids.

The 26-alternative bound is now a property of letter labels in
QuestionType.labels, not a global.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
The request-template test asserted which option a 0.6B model picks under a
terse custom prompt, which is the model's call, not the server's. It now
checks that the macro's prefix lands the read on a label (label mass above
0.5) and that the probabilities differ from the default template's on the
same input. Adds a 400 case for a template that does not compile.

Docs: decision templates, the answer_prefix macro, the limits, and that
diffusion models are not served yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Answer macro. A decision template now says how an answer is written with
answer(question, label), "<id>: <label>" by default, in place of
answer_prefix(question). The server renders the answer once per label and
finds the one token where the labels differ: the tokens before it are the
read's prefill, and the differing token is each label's id. A template can
therefore use any answer format ("(A)", "id -> A") without the server
assuming a space before the label, and the system prompt can call the same
macro to show the reply format. The default template does.

Templates render once per request with the real variables, so a template
body that uses questions at the top level works, and its macros see them.
Loading only compiles the template.

Review cleanups: "alternatives" is now "options" everywhere, to match the
template variables, the docs and the Jev criteria. The strategy docstring
says an unsupported model gets a 501 from the route, which is what happens.
The serving class catches only StructuredDecisionError, so other errors
reach the server's handler with their own status. The router documents the
501. The strategy resolves every prompt before it submits any request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Drops comments that describe a piece by comparing it with another (decision
templates against chat templates, the next-token strategy against diffusion)
and docstrings that restate their declaration.

Drops tests that another test already covers or that check trivial math:
label softmax, the template without instructions, the default answer, a
label changing two tokens, answer repeatability, the unknown-model 404 and
the HTTP form of a template compile error. The HTTP validation test keeps
one case each for unknown fields and the question limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
A model class now names the structured-decision read strategy it needs with
a decision_read_strategy class variable, "next_token" by default, recorded
in _ModelInfo beside the other capability flags. The server registers
strategies by name and picks the one the model names. A name with no
registered strategy gets the 501. DiffusionGemma names "canvas", so it keeps
the 501 until a canvas strategy is registered.

This replaces supports(), which tested ModelConfig.is_diffusion and relied
on registration order. limits() is now an instance method, so a strategy
can size its limits from the served config (a canvas from the diffusion
config) through its context.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
…zer, shuffle per question

Letter labels were assumed to be one token after "id: ". That holds for
Qwen3 and Gemma with the default template, but not for every template:
without the space, Qwen3 fuses the colon into 15 of the 26 letters (":A"),
so a template like "id:label" would read a mix of fused and unfused slots.

The server now tries A..Z then AA..ZZ in the template's answer the first
time it uses a template with a tokenizer. It groups the labels that land in
one token by the tokens around them and by what else that token holds
(a fused space, a fused colon, nothing), and keeps the largest group. On
Qwen3-0.6B the default format keeps 552 labels, and "q:L" keeps the 528
unfused ones over the fused colon group. Digits give only 10, and
letter-digit or two-digit labels are never one token on Qwen3 or
DiffusionGemma, so letters stay.

Each question takes single letters first, shuffled with a seed from a
SHA-256 of its id, type, instructions and options, and falls back to
two-letter labels only past 26 options. Shuffling the whole group made
things worse: two-letter labels carry meaning (the language question gave
English "TM" and French "LT", and the model answered French at 0.999 on
an English ticket). Seeding from the question keeps the prompt stable for
prefix caching. The default prompt now says labels are chosen randomly.

A question may have at most min(labels kept, strategy limit) options, and
the error names that number. The next-token strategy's limit is still the
128 logprob_token_ids cap.

The e2e tests used to pass because both right answers had label A. With
shuffled labels Qwen3-0.6B gets the second of two semantic questions right
0 to 2 times in 10 random label draws (Qwen3-4B: 9 to 10), so the tests
now ask copy questions whose answer is in the state (0.94 and 0.997 on the
real labels), and the request-template test no longer asserts label mass,
which depends on how the model reads a custom format.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
A request may set `seed`, which is mixed into each question's label
shuffle. Reading a question under several seeds and averaging the answers
cancels the model's preference for particular labels. Without a seed the
labels still come from the question's content alone, so repeated requests
keep one prompt for prefix caching.

Measured on the original semantic e2e questions (a double-charge ticket:
team billing/shipping/security, language English/French), 320 seeds:

- Qwen3-4B is right on 310/320 single reads and on every average of 2 or
  more seeds.
- Qwen3-0.6B gets worse with averaging (team 65/320 single, 0/20 at 16
  seeds; language 28/320, 0/20). Its bias is option position, not labels:
  it picks the last-listed option. With the options reversed it is right
  on 240/320 and 309/320; with the order shuffled per seed its answers sit
  near chance (billing 0.38, English 0.53). The e2e tests keep the copy
  questions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
The e2e test asserted one read's answers, which only proves the one label
draw its question hash picks. A new parametrized test averages each
question's probabilities over seeds 0 to 15 and asserts the argmax, on
three question sets Qwen3-0.6B gets right in every 16-seed group
(10 of 10 groups over 160 label draws each):

- copy team + copy language from a JSON state (150 and 148 of 160 single
  reads right)
- "mentions a card?" yes + sentiment angry (147 and 134 of 160)
- sky color blue (159 of 160)

The same battery found what 0.6B cannot do under shuffled labels:
questions whose answer is "no" (5 of 160, a yes bias) and naming a
message's language (near chance), so neither is in the test.
test_choice_decision keeps the response-shape checks without asserting
answers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
@mergify

mergify Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Documentation preview: https://vllm--59299.org.readthedocs.build/en/59299/

@mergify mergify Bot added documentation Improvements or additions to documentation frontend labels Sep 29, 2026
mmastrac and others added 4 commits September 29, 2026 14:55
Drops the references to Jev's decision API, which vLLM readers have no
context for. Rewrites comments and docs that used nouns as verbs ("names
the strategy", "shapes the answer", "the group wins", "every label reads
the same way") into plain statements, and cuts docstrings that restated
a field or return type (AnswerSlot fields, RenderedDecision.slot). Adds
one comment on why the label sweep skips text that does not round-trip
through the tokenizer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Request-supplied Jinja is the part of the first PR a security review
would slow down on, and nothing needs it yet. The server now compiles
its one decision template (--decision-template or the built-in) at
startup, which also makes a broken template stop startup. The
decision_template request field, the --trust-request-chat-template
plumbing, select_template/compile_template and their tests go.

The startup check for fewer than two labels goes too: every question
needs at least two options, so a template that leaves fewer labels
already fails each request with "at most N options for this model".

Also cuts two comments that argued instead of stating a condition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
…docs

Replaces figures of speech and wording that speaks of the model as if it
wanted things ("wanted to say something else", "deserves less trust",
"sit inside one token", "left open", "line up") with the concrete
condition. Claims about seeds now say averaging reduces label bias
rather than cancels it. The choice-criteria error now says a value may
be null, which the parser already accepted. The docs no longer mention
request templates in the error sentence, and the cost line says what
each further question prefills.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
pre-commit's mypy hook failed on four files:

- The read loop indexed output.logprobs after a check mypy could not
  narrow through result.outputs[0]. The check now binds output first
  and tests result, output and output.logprobs directly.
- BaseServing._check_model and _maybe_get_adapters take a closed union
  of request types. The decision request is passed with
  type: ignore[arg-type], as generative_scoring already does.
- The strategy test's SimpleNamespace stand-in for ModelConfig is typed
  Any.
- max(t, key=t.get) has an overloaded key type; it is a lambda now.

Checked with tools/pre_commit/mypy.py and mypy 1.20.2 for Python 3.10
and 3.13.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

A vLLM developer reported accuracy dropping with every question after the
first. Every read rendered all of the request's questions into the
system prompt, which asks for "one line per question, in this order",
then prefilled the reply at the read's own line. For any question after
the first, the reply started at a line the prompt said comes later.

Measured with four questions (copy a team name, "mentions a card?" yes,
sentiment, sky color), each rotated through all four positions, 48
label seeds each, 192 reads per position:

  layout                          Qwen3-0.6B pos 1-4      Qwen3-1.7B pos 1-4
  all questions (before)          156 129 117 115         159 114 163 171
  all questions, "in any order"   157 134 128 121         170 114 164 169
  prefill earlier answers         156 158 151 171         159 137 181 188
  own question, after the state   165 165 165 165         185 185 185 185

Each read now puts the state first, then the decision template rendered
for its question alone, in the user turn. Accuracy no longer depends on
position, and the total is the highest of the four (0.6B 86% against
67%, 1.7B 96% against 79%). Reads still run in parallel. The shared
prefix is now the state rather than the system prompt plus the state,
so each read prefills its own question text, a few dozen tokens.

The default template text is unchanged, since it is what was measured.
RenderedDecision.system_text becomes text, and the docs describe the
template as the text of one question's read.

Tested on a GB10 (gx10) with this branch's files over a vllm-openai
nightly image: 26 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Comment thread vllm/entrypoints/generate/structured_decisions/templates.py Outdated
Comment thread vllm/entrypoints/generate/structured_decisions/question_types.py Outdated
Comment thread vllm/model_executor/models/registry.py Outdated
Comment thread vllm/entrypoints/launchers/cli_args.py Outdated
Comment thread vllm/entrypoints/generate/structured_decisions/strategies.py Outdated
Comment thread vllm/entrypoints/generate/structured_decisions/question_types.py Outdated
…usion, confidence)

From Lucas Wilkinson's review:

- The one-token label read moves to vllm/entrypoints/generate/label_reads.py
  as next_token_label_reads(). /generative_scoring and the decision
  endpoint both call it. Each caller still builds its own SamplingParams,
  so /generative_scoring keeps its sampling settings, request ids, input
  logging and error messages.
- The read strategy comes from model_config.is_diffusion. The model
  registry hook (decision_read_strategy on the model class, _ModelInfo,
  interfaces.py, DiffusionGemma's "canvas") and the strategy name
  registry go until the diffusion PR has a second strategy to register.
- --decision-template goes, so the template text stays internal. The
  docs drop the decision template section.
- confidence was the top label's share among the labels, which can read
  about 1.0 when label_mass is 2e-14. It is now that share times
  label_mass: the chosen label's probability over the whole vocabulary.

Tested on a GB10 (gx10) with this branch's files over a vllm-openai
nightly image: structured_decisions and generative_scoring, 42 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>

@IRONICBo IRONICBo left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One compatibility edge case in the request validator:

Comment thread vllm/entrypoints/generate/structured_decisions/question_types.py Outdated
Comment thread vllm/entrypoints/generate/structured_decisions/api_router.py Outdated
@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #93493 for commit fd64fb067160, running 1 failed step(s) from Buildkite CI #93475.

@chaunceyjiang chaunceyjiang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!

I tested it locally, and it works well.

I have two minor, non-blocking suggestions that we can address in a follow-up PR:

  1. I suggest adding a cache_salt parameter and passing it to the core engine.
  2. I suggest adding get_trace_headers handling, consistent with the other endpoints.

Comment thread vllm/entrypoints/generate/structured_decisions/strategies.py Outdated
@chaunceyjiang

Copy link
Copy Markdown
Contributor

/ci run

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown

❌ @chaunceyjiang, Only reviewers with write access can use CI commands before CI is delegated to the PR author.

mmastrac and others added 2 commits October 8, 2026 07:29
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
@mmastrac

mmastrac commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #93598 for commit 35c56839c9ed.

@mgoin mgoin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just one small issue we can deal with in a followup



def state_text(state: Any) -> str:
return state if isinstance(state, str) else json.dumps(state)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should use ensure_ascii=False in dumps to not scramble non-ASCII text. we can do this as a quick followup

@mgoin
mgoin enabled auto-merge (squash) October 8, 2026 14:30
@mgoin
mgoin merged commit d33872d into vllm-project:main Oct 8, 2026
92 checks passed
vllm-bot pushed a commit that referenced this pull request Oct 8, 2026
…state prompts (#60651)

Signed-off-by: Francisco Javier Arceo <4163062+franciscojavierarceo@users.noreply.github.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
kerrylu-berri pushed a commit to BerriAI/litellm that referenced this pull request Oct 9, 2026
* feat(decisions): add hosted_vllm provider

Adds HostedVLLMDecisionsConfig so hosted_vllm/<served model> works on
POST /v1/systemone and POST /v1/decisions against a vLLM server that
serves the System One body (vllm-project/vllm#59299, after v0.31.0).
vLLM only supports choice questions and rejects others, so the base
decisions config gains a per-provider health_check_questions attribute
and the evaluation-mode health probe sends a choice question for
hosted_vllm while every other provider keeps the noul probe.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(decisions): move the hosted_vllm placeholder key to constants

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
cursor Bot pushed a commit to opengfx1030/vllm-rdna that referenced this pull request Oct 11, 2026
Align answers, confidence, and error bodies with llama.cpp and SGLang,
and step aside when vLLM vllm-project#59299 already owns the route. Refuse a request
whose label token ids do not match the loaded decision model.

Co-authored-by: Grok 4.7 <cursoragent@cursor.com>
DeoLeung pushed a commit to aiyou178/litellm that referenced this pull request Oct 11, 2026
* feat(decisions): add hosted_vllm provider

Adds HostedVLLMDecisionsConfig so hosted_vllm/<served model> works on
POST /v1/systemone and POST /v1/decisions against a vLLM server that
serves the System One body (vllm-project/vllm#59299, after v0.31.0).
vLLM only supports choice questions and rejects others, so the base
decisions config gains a per-provider health_check_questions attribute
and the evaluation-mode health probe sends a choice question for
hosted_vllm while every other provider keeps the noul probe.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(decisions): move the hosted_vllm placeholder key to constants

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 61e5f2d)
@noooop noooop mentioned this pull request Oct 12, 2026
6 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation frontend ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants