Repository navigation
Conversation
vLLM enforces tool_choice "required" and a named function with structured
outputs, which it does not support for diffusion models, so both were ignored:
a named get_time came back as a get_weather call. For those two choices the
relay now pins the empty thought block and the opening of a tool call
("<|tool_call>call:", plus the name and brace when one is named) on the seeded
canvas, and sends tool_choice "auto", so the model can only fill in a call and
vLLM's tool parser reads it. Requests that seed their own canvas and unknown
function names are left for vLLM.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pinning only "<|tool_call>call:" left the name to the model, and when no offered tool fit the request it wrote one anyway: "get_", the answer to the question, or no call at all. "required" now pins a name too: the only function offered, or the one a one-step choice read over the functions' names and descriptions picks, then the model fills in the arguments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner
Author
|
(this won't land as-is - just a stopgap for missing vLLM features) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
vLLM enforces
tool_choice: "required"and a named function with structured outputs, and it rejects structured outputs for diffusion models ("not yet supported for diffusion language models"). On the chat path both choices were therefore ignored: withtool_choicenamingget_time, the model calledget_weather.requiredonly appeared to work because the model happened to call something.A diffusion canvas can be seeded and pinned instead, which is what this server already does for its reads. For those two choices, the relay now pins the empty thought block and the opening of a tool call on the canvas,
<|tool_call>call:NAME{, and sendstool_choice: "auto"so vLLM's tool parser reads the call. The model can only fill in the arguments.For
required, the name is the only function offered, or the one a one-stepchoiceread picks over the offered functions' names and descriptions (pick_tool). The first version pinned only<|tool_call>call:and let the model write the name. When no offered tool fit the request, it wrote nonsense:get_, the punchline of a requested joke, or no call at all. Requests that seed their own canvas, and names not among the offered tools, are passed to vLLM unchanged.The README notes the vLLM flags this assumes:
--enable-auto-tool-choice --tool-call-parser gemma4, and--exclude-tools-when-tool-choice-noneso that"none"gets a plain answer rather than a call the parser strips.Tested live on a GB10 (nightly-e9757321, gemma4 tool parser), with a second server process on the same vLLM:
autoget_weather({"city": "Paris"})auto, no tool neededrequiredget_weather, unenforcedget_weather, pinnedrequired, "Tell me a joke."get_time, an offered toolrequired, "Say hello in French."get_timenone(with--exclude-tools-when-tool-choice-none)get_timeget_weatherget_time({"city": "Paris"})Also checked:
get_timewith its arguments.requiredcall, then a tool result, then the final answer ("The current weather in Paris is 14°C with light rain.").🤖 Generated with Claude Code