Skip to content

[Feature] Router: let decision questions read prior user turns - #4778

Open
subin9 wants to merge 5 commits into
vllm-project:mainfrom
subin9:feat/3771-prior-user-turns
Open

subin9 wants to merge 5 commits into
vllm-project:mainfrom
subin9:feat/3771-prior-user-turns

Conversation

@subin9

@subin9 subin9 commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Purpose

Decision signals read only the current user turn, so a follow-up such as "Rewrite your previous answer in fewer words" routes on almost nothing. This implements Option A of #3771 for decision-model questions. prior_user_turns: N (0 to 8) on a routing.signals.decision rule makes its question read up to N earlier user turns, oldest first, then the current turn, one blank line apart. The earlier turns share 1,760 characters, filled from the latest turn back, so however many turns a question reads they add at most that much to the state. Questions to one deployment share a call only when they read the same number of turns, so single-turn requests still make one call. Assistant and tool messages are left out: on the measurement below, the whole conversation scored no better than the user turns on all but the smallest model, at about twice the tokens. The decision selector and the built-in signals are unchanged, and #3771's Option B (a continuity signal family) is not covered.

On MT-Bench's 80 two-turn conversations (prompts Apache-2.0), an eight-way Choice over its categories, asked on the second turn:

Model second turn alone both user turns
Decision 2.0 Kai-0.6B 41.3% 77.5%
Decision 2.0 Eos-0.8B 52.5% 71.3%
Decision 2.0 Nox-4B 55.0% 81.3%
Decision 2.0 Lux-9B 55.0% 83.8%
Vela 2.0 0.3B 35.0% 67.5%
Vela 2.0 0.8B 50.0% 77.5%
Vela 2.0 4B 50.0% 81.2%
Vela 2.0 9B 51.2% 87.5%

Related #3771

Test Plan

Unit tests for the state, the oldest-first order, the shared limit, the call grouping and the first-turn case. The field reaches the validator, the DSL decompiler, the CLI model, the generated schema, the dashboard type and the decision signal tutorial. The new E2E case decision-prior-user-turns runs in the model-runtime profile: it finds a first turn for which the fixture runtime answers the follow-up differently alone and after that turn, then checks that the Router routes on the second answer.

Test Result

The model-runtime profile passed locally with all 13 cases on main with #4734. In the new case the fixture answered code for the follow-up alone and chat after "Prove that the square root of two is irrational.", and the Router matched follow_up_kind:chat. The E2E case asks a Decision 2.0 fixture; the Vela 2.0 numbers above come from the runtime directly, not through the Router.

@github-actions github-actions Bot added the pr/blocked Blocked on a named decision, dependency, or required check. label Oct 9, 2026
Decision signals read only the current user turn. prior_user_turns: N puts
up to N earlier user turns, oldest first, before it in the state. On
MT-Bench follow-ups that raised right answers from 41-55% to 71-84% on
Decision 2.0 and from 35-51% to 67.5-87.5% on Vela 2.0.

Related vllm-project#3771

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
@subin9
subin9 force-pushed the feat/3771-prior-user-turns branch from 65b5f8b to ad94ec5 Compare October 9, 2026 15:54

@Xunzhuo Xunzhuo left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for clarifying the long-input behavior. My earlier statement that the current turn was truncated, and the reference to a blocking review, were incorrect: Decision rejects an over-limit state. The revised shared history budget preserves the current turn; both classification regressions, config/DSL checks, and all three Decision long-input rejection controls pass. I found no remaining source blocker in this change.

Eight earlier turns of 1,760 characters each could push a state past the
model's input limit, and Decision models refuse such an input. The
earlier turns now share 1,760 characters, filled from the latest turn.

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
…turns

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
The overview now says the standalone frontend or an ExtProc gateway
presents the request, so the contract looked for a sentence that main
no longer has and Router Contracts failed on every pull request.

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
The Chinese guide now has to show the English page's snippets, so it gets
the prior_user_turns section too.

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
@subin9

subin9 commented Oct 10, 2026

Copy link
Copy Markdown
Collaborator Author

You're right. Eight earlier turns of up to 1,760 characters each could take the state past the model's input limit, and the Decision models refuse such an input instead of cutting it, so the question failed even when the current turn alone fit. The earlier turns now share 1,760 characters, filled from the latest turn back. I couldn't find the blocking review you mentioned, so if your regression shows a different case, could you paste it here?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr/blocked Blocked on a named decision, dependency, or required check.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants