Skip to content

Enforce required and named tool_choice by pinning the call's opening - #9

Open
mmastrac wants to merge 2 commits into
mainfrom
tool-choice-pin
Open

mmastrac wants to merge 2 commits into
mainfrom
tool-choice-pin

Conversation

@mmastrac

@mmastrac mmastrac commented Sep 23, 2026 •

Copy link
Copy Markdown
Owner

vLLM enforces tool_choice: "required" and a named function with structured outputs, and it rejects structured outputs for diffusion models ("not yet supported for diffusion language models"). On the chat path both choices were therefore ignored: with tool_choice naming get_time, the model called get_weather. required only appeared to work because the model happened to call something.

A diffusion canvas can be seeded and pinned instead, which is what this server already does for its reads. For those two choices, the relay now pins the empty thought block and the opening of a tool call on the canvas, <|tool_call>call:NAME{, and sends tool_choice: "auto" so vLLM's tool parser reads the call. The model can only fill in the arguments.

For required, the name is the only function offered, or the one a one-step choice read picks over the offered functions' names and descriptions (pick_tool). The first version pinned only <|tool_call>call: and let the model write the name. When no offered tool fit the request, it wrote nonsense: get_, the punchline of a requested joke, or no call at all. Requests that seed their own canvas, and names not among the offered tools, are passed to vLLM unchanged.

The README notes the vLLM flags this assumes: --enable-auto-tool-choice --tool-call-parser gemma4, and --exclude-tools-when-tool-choice-none so that "none" gets a plain answer rather than a call the parser strips.

Tested live on a GB10 (nightly-e9757321, gemma4 tool parser), with a second server process on the same vLLM:

tool_choice before after
auto get_weather({"city": "Paris"}) same
auto, no tool needed "Bonjour !" same
required get_weather, unenforced get_weather, pinned
required, "Tell me a joke." the joke as a function name get_time, an offered tool
required, "Say hello in French." no call get_time
none (with --exclude-tools-when-tool-choice-none) empty content a text answer
named get_time get_weather get_time({"city": "Paris"})

Also checked:

  • A streamed named call arrives as get_time with its arguments.
  • A full round trip works: a required call, then a tool result, then the final answer ("The current weather in Paris is 14°C with light rain.").

🤖 Generated with Claude Code

mmastrac and others added 2 commits September 23, 2026 09:55
vLLM enforces tool_choice "required" and a named function with structured
outputs, which it does not support for diffusion models, so both were ignored:
a named get_time came back as a get_weather call. For those two choices the
relay now pins the empty thought block and the opening of a tool call
("<|tool_call>call:", plus the name and brace when one is named) on the seeded
canvas, and sends tool_choice "auto", so the model can only fill in a call and
vLLM's tool parser reads it. Requests that seed their own canvas and unknown
function names are left for vLLM.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pinning only "<|tool_call>call:" left the name to the model, and when no
offered tool fit the request it wrote one anyway: "get_", the answer to the
question, or no call at all. "required" now pins a name too: the only function
offered, or the one a one-step choice read over the functions' names and
descriptions picks, then the model fills in the arguments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mmastrac

Copy link
Copy Markdown
Owner Author

(this won't land as-is - just a stopgap for missing vLLM features)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant