Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -334,7 +334,15 @@ The **Upstream name** column shows where the vLLM or SGLang parser name differs

**Notes:** Channel grammar: reasoning in `<\|channel\|>analysis<\|message\|>...<\|end\|>`, the answer in `<\|channel\|>final<\|message\|>...<\|return\|>`, and one `<\|start\|>assistant to=NAME<\|channel\|>analysis <\|constrain\|>xml<\|message\|>...<\|call\|>` message per tool call with `<rfl:parameter name="KEY">VALUE</rfl:parameter>` arguments typed from the tool schema. Pair with `--dyn-reasoning-parser beam`; both flags are required and the pair routes to the unified parser without `DYN_PARSER_VERSION=2`. The route accepts the request field `preserve_thinking` (supplied prior assistant reasoning always stays in the rendered history; `true` and omitted are the same, `false` is accepted and has no effect — omit `reasoning_content` to drop it) and defaults `allow_missing_thinking=true` so history turns without reasoning render an empty analysis block instead of failing and `chat_template_kwargs.builtin_tools` (names resolved against `tools` for rendering only).

**Per-turn grammar:** on the Beam route the frontend holds every chat turn to this channel grammar with a structural tag (a non-empty analysis channel, then either tool calls whose `rfl:parameter` values are held to their JSON-schema types or the final answer), mirroring the model builder's SGLang serving layer; `required` and named tool choices reuse the same tag, so they are enforced on the Harmony wire. A `response_format` (`json_object` or `json_schema`) is held inside the final channel behind the analysis, so the response keeps its reasoning and the channel structure the parser expects; a forced tool choice ignores `response_format`, and the `nvext` guided-decoding fields (`guided_json`, `guided_regex`, ...) stay bare constraints from the first token. Set `DYN_BEAM_TURN_GRAMMAR=0` on the frontend to disable it for automatic tool choice (forced choices keep theirs; `response_format` then falls back to the bare JSON constraint). A tool declared `strict: true` whose schema is outside OpenAI's strict subset is served as non-strict on this route (logged as a warning) instead of the HTTP 400 other routes return; the grammar holds every typeable tool either way.
**Per-turn grammar:** on the Beam route the frontend holds every chat turn to this channel grammar with a structural tag (a non-empty analysis channel, then either tool calls whose `rfl:parameter` values are held to their JSON-schema types or the final answer), mirroring the model builder's SGLang serving layer; `required` and named tool choices reuse the same tag, so they are enforced on the Harmony wire. A `response_format` (`json_object` or `json_schema`) is held inside the final channel behind the analysis, so the response keeps its reasoning and the channel structure the parser expects; a forced tool choice ignores `response_format`, and the `nvext` guided-decoding fields (`guided_json`, `guided_regex`, ...) stay bare constraints from the first token. Set `DYN_BEAM_TURN_GRAMMAR=0` on the frontend to disable it for automatic tool choice (forced choices keep theirs; `response_format` then falls back to the bare JSON constraint).

**Rejected requests** (`Beam cannot enforce the requested tool constraints with this schema and prompt configuration`): the frontend returns HTTP 400 with this message instead of serving the request best-effort when the request asks for tool enforcement that the turn grammar cannot provide. That happens when the request has:

- a forced tool choice (`required` or a named function), but the last assistant message you send as a prefill has already started the final answer, so a tool call can no longer be forced;
- an applicable `strict: true` tool (any strict tool under `auto` or `required`, the named one under a named choice; `tool_choice: "none"` makes none applicable) whose schema the grammar cannot hold, for example one using keywords outside its JSON-schema subset such as `pattern`;
- `parallel_tool_calls: false` with usable tools together with an `nvext` guided-decoding field (`guided_json`, `guided_regex`, ...), which takes the output away from the grammar.

To resolve it, drop `strict` or fix the schema, remove the assistant prefill before forcing a call, or drop the guided-decoding field. Independently of the grammar, a `strict: true` tool whose schema is outside OpenAI's strict subset is rejected with HTTP 400 on this route as on every other.
</Accordion>
</AccordionGroup>
</Accordion>
Expand Down
13 changes: 3 additions & 10 deletions lib/llm/src/http/service/openai.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3727,21 +3727,14 @@ async fn chat_completions(
err_response
})?;

// Beam route, before generic validation (which rejects what these normalize): a
// `strict` tool whose schema is outside OpenAI's strict subset is served as
// non-strict instead of rejected, and a legacy `builtin_tools` declaration is
// promoted into `tools` (see `tool_choice::relax_noncompliant_strict_tools_for_beam`
// and `tool_choice::declare_legacy_builtin_tools_for_beam`).
// Beam route, before generic validation (which rejects what this normalizes): a
// legacy `builtin_tools` declaration is promoted into `tools` (see
// `tool_choice::declare_legacy_builtin_tools_for_beam`).
crate::preprocessor::tool_choice::declare_legacy_builtin_tools_for_beam(
&mut request,
parsing_options.tool_call_parser.as_deref(),
parsing_options.reasoning_parser.as_deref(),
);
crate::preprocessor::tool_choice::relax_noncompliant_strict_tools_for_beam(
&mut request,
parsing_options.tool_call_parser.as_deref(),
parsing_options.reasoning_parser.as_deref(),
);

// Handle Rest of Validation Errors
if let Err(err_response) = validate_chat_completion_fields_generic(&request) {
Expand Down
5 changes: 0 additions & 5 deletions lib/llm/src/preprocessor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8588,11 +8588,6 @@ impl
self.tool_call_parser.as_deref(),
self.runtime_config.reasoning_parser.as_deref(),
);
crate::preprocessor::tool_choice::relax_noncompliant_strict_tools_for_beam(
&mut request,
self.tool_call_parser.as_deref(),
self.runtime_config.reasoning_parser.as_deref(),
);

// create a response generator
let response_generator = request.response_generator(context.id().to_string());
Expand Down
Loading
Loading