Server-side tool-call emulation for models without native tool support - #946
Conversation
Small / non-tool-trained models served through the mesh /v1 endpoint
handle the OpenAI tools field poorly: the chat template ignores the
schemas or the model loops re-issuing the same call. goose solved this
in its local-inference provider, but that path is bypassed when a client
talks to a mesh, and every other OpenAI client hits the same wall.
The serving node is the only party that knows the loaded model's actual
chat-template capability, so it emulates tool calling when the template
cannot do it natively:
- Detect capability from the runtime chat-template metadata
(parse_tool_calls + non-empty chat_parser), not model size. Tool-capable
templates are unchanged.
- Adapt the request: strip tools/tool_choice, inject a compact instruction
(name + description + compact parameter schema) teaching the
TOOL_CALL {json} convention, and rewrite history so the template never
sees tool roles.
- Parse the response: scan for TOOL_CALL lines into OpenAI tool_calls with
finish_reason tool_calls, tolerant of prose and <think> blocks; hold back
partial markers while streaming.
Ported from goose local-inference (tool_emulation.rs, tool_parsing.rs,
tiny_model_system.md). Adds 19 unit + 5 integration tests.
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughAdds server-side tool-call emulation for chat templates without native tool support in skippy-server. Introduces a ChangesTool-call emulation feature
Estimated code review effort: 4 (Complex) | ~60 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/skippy-server/src/frontend/prompting.rs`:
- Around line 26-41: The chat prompt rendering in prompting.rs is probing
template capability by rendering the full native prompt first, which can expose
unsupported tool roles before emulation is chosen. Update the flow around
render_chat_prompt, tool_emulation::template_supports_native_tool_calls, and
tool_emulation::rewrite_history_for_emulation so capability is checked using
sanitized/minimal data or metadata before any full render, then render only once
with either the native messages or the rewritten emulation history.
- Around line 35-41: The emulation path in render_chat_prompt currently rewrites
history with build_emulation_instruction(tools) but drops request.tool_choice
semantics, so the forced-tool requirement is lost. Update
tool_emulation::build_emulation_instruction (and any caller like
rewrite_history_for_emulation in prompting.rs) to encode tool_choice, especially
“required” and any explicit function selection, into the emulation instruction
so the model is constrained to the requested tool. Ensure the rewritten prompt
preserves the original tool_choice behavior instead of allowing free-form
answers or different tools.
- Around line 290-293: The partial parsing logic in prompting::scan_text
currently strips everything after the last newline whenever is_partial is true,
which suppresses normal one-line prose during streaming. Update the scan_text
handling so it only withholds the trailing segment when it might be an
incomplete TOOL_CALL marker, and otherwise lets prose flow through normally; use
the existing is_partial path in prompting.rs to distinguish tool-call fragments
from regular text.
In `@docs/specs/server-side-tool-call-emulation.md`:
- Around line 54-56: The fenced example in the docs spec is unlabeled and will
fail MD040; update the code block around the TOOL_CALL example to use a language
tag like text. Make the change in the markdown snippet itself so the example
stays CI-clean, and keep the TOOL_CALL sample content unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 32931fab-7309-4bbb-a255-c455032a0a48
📒 Files selected for processing (5)
crates/skippy-server/src/frontend.rscrates/skippy-server/src/frontend/prompting.rscrates/skippy-server/src/frontend/tests.rscrates/skippy-server/src/frontend/tool_emulation.rsdocs/specs/server-side-tool-call-emulation.md
| let native = self.render_chat_prompt(request, &options, &marker, None, true)?; | ||
|
|
||
| // If the request carries tools but the template does not support native | ||
| // tool calling, re-render with server-side tool-call emulation: strip | ||
| // tools, inject a text-convention instruction, and rewrite history so | ||
| // the template never sees tool roles. Tool-capable templates keep the | ||
| // native prompt unchanged. | ||
| if tool_calls_requested(request) | ||
| && !tool_emulation::template_supports_native_tool_calls(&native.metadata_json) | ||
| && let Some(tools) = request.tools.as_ref() | ||
| && let Some(instruction) = tool_emulation::build_emulation_instruction(tools) | ||
| { | ||
| let rewritten = | ||
| tool_emulation::rewrite_history_for_emulation(&request.messages, &instruction); | ||
| let emulated = | ||
| self.render_chat_prompt(request, &options, &marker, Some(&rewritten), false)?; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Avoid probing unsupported templates with the original tool-role history.
Line 26 renders the native prompt before emulation is selected, so non-tool-capable templates can still see role: "tool" / assistant tool_calls and fail before the rewrite at Lines 38-41 runs. Probe capability with sanitized/minimal messages or metadata first, then render either the native or rewritten prompt once.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/skippy-server/src/frontend/prompting.rs` around lines 26 - 41, The
chat prompt rendering in prompting.rs is probing template capability by
rendering the full native prompt first, which can expose unsupported tool roles
before emulation is chosen. Update the flow around render_chat_prompt,
tool_emulation::template_supports_native_tool_calls, and
tool_emulation::rewrite_history_for_emulation so capability is checked using
sanitized/minimal data or metadata before any full render, then render only once
with either the native messages or the rewritten emulation history.
| && let Some(tools) = request.tools.as_ref() | ||
| && let Some(instruction) = tool_emulation::build_emulation_instruction(tools) | ||
| { | ||
| let rewritten = | ||
| tool_emulation::rewrite_history_for_emulation(&request.messages, &instruction); | ||
| let emulated = | ||
| self.render_chat_prompt(request, &options, &marker, Some(&rewritten), false)?; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Preserve tool_choice semantics in the emulation instruction.
The emulation path strips tool_choice at Line 41, but build_emulation_instruction(tools) does not encode tool_choice: "required" or a forced function choice. This can let the model answer normally or call another tool even when the request requires a specific tool.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/skippy-server/src/frontend/prompting.rs` around lines 35 - 41, The
emulation path in render_chat_prompt currently rewrites history with
build_emulation_instruction(tools) but drops request.tool_choice semantics, so
the forced-tool requirement is lost. Update
tool_emulation::build_emulation_instruction (and any caller like
rewrite_history_for_emulation in prompting.rs) to encode tool_choice, especially
“required” and any explicit function selection, into the emulation instruction
so the model is constrained to the requested tool. Ensure the rewritten prompt
preserves the original tool_choice behavior instead of allowing free-form
answers or different tools.
| let scan_text = if is_partial { | ||
| text.rsplit_once('\n') | ||
| .map(|(head, _)| head) | ||
| .unwrap_or_default() |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Do not withhold all partial prose while streaming.
This drops every trailing line during partial parsing, so a normal one-line answer streams no content until finalization. Only hold the trailing segment when it could be a partial TOOL_CALL marker; otherwise parse/emit prose normally.
Suggested shape
- let scan_text = if is_partial {
- text.rsplit_once('\n')
- .map(|(head, _)| head)
- .unwrap_or_default()
+ let scan_text = if is_partial && trailing_segment_may_be_tool_call(text) {
+ text.rsplit_once('\n').map(|(head, _)| head).unwrap_or_default()
} else {
text
};🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/skippy-server/src/frontend/prompting.rs` around lines 290 - 293, The
partial parsing logic in prompting::scan_text currently strips everything after
the last newline whenever is_partial is true, which suppresses normal one-line
prose during streaming. Update the scan_text handling so it only withholds the
trailing segment when it might be an incomplete TOOL_CALL marker, and otherwise
lets prose flow through normally; use the existing is_partial path in
prompting.rs to distinguish tool-call fragments from regular text.
| ``` | ||
| TOOL_CALL {"name": "the_tool_name", "arguments": {"arg": "value"}} | ||
| ``` |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Label the fenced example.
This block will trip MD040 in docs lint. Add a language tag (for example text) so the spec stays CI-clean.
♻️ Proposed fix
-```
+```text
TOOL_CALL {"name": "the_tool_name", "arguments": {"arg": "value"}}
-```
+```📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| ``` | |
| TOOL_CALL {"name": "the_tool_name", "arguments": {"arg": "value"}} | |
| ``` |
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 54-54: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/specs/server-side-tool-call-emulation.md` around lines 54 - 56, The
fenced example in the docs spec is unlabeled and will fail MD040; update the
code block around the TOOL_CALL example to use a language tag like text. Make
the change in the markdown snippet itself so the example stays CI-clean, and
keep the TOOL_CALL sample content unchanged.
Source: Linters/SAST tools
The skippy CI smoke already sends a tools request to SmolLM2-135M, whose chat template does not support native tool calling, so it exercises the new server-side tool-call emulation path. It previously only asserted HTTP 200 and role==assistant. Strengthen it: use a tool-forcing prompt and a real token budget, and assert that any tool_calls returned by the emulation path are well-formed (non-empty function name with arguments that parse as a JSON object). Requiring a 135M model to reliably emit a tool call would be flaky, so a call is not forced, but malformed emulated calls now fail the smoke.
Live spot check (local, real model)Verified end-to-end against a real non-tool-trained model served through the mesh —
CI coverageThe existing |
The ported goose heuristic (parse_tool_calls && non-empty chat_parser) does not work against mesh-llm's patched llama.cpp: parse_tool_calls is true for every tools request and chat_parser is always a non-empty PEG structure, so the check was always true and emulation never fired. Detect native support from grammar_triggers instead: a tool-capable jinja template yields a tool-call grammar trigger (e.g. <tool_call>) when applied with tools, while a template with no native tool support (e.g. SmolLM2-135M) yields an empty grammar_triggers list. Verified live: SmolLM2-135M routes to emulation (native_supported=false), Qwen2.5-0.5B and Qwen3.5-0.8B keep native tool calling (grammar trigger present).
Correction: detection signal fixed (grammar_triggers)While verifying your question ("do real-tool-calling models skip this path?") I found a real bug in the first version: the ported goose heuristic — Fixed to detect native support from Verified live with a temporary branch trace:
So the answer to your question is confirmed with the corrected detector: tool-capable templates do not trigger emulation. (Note: my earlier "spot check" comment mis-attributed Qwen2.5-0.5B as emulated — it was actually native. This commit corrects the detection so the routing is now right.) Trace removed; unit test updated; clippy/fmt/tests clean. |
- Add MESH_FORCE_TOOL_EMULATION override (goose's ToolCallingMode::
ForceEmulated analogue) so emulation can be exercised against strong
models and used as an escape hatch when a native template misbehaves.
Routed through should_emulate_tool_calls().
- Revert scripts/skippy-ci-smoke.sh to main: the two-node/binary smoke
runs on tiny models (SmolLM2-135M) that cannot reliably emit a tool
call, so probing for emulated tool_calls there would be flaky. The
emulation path is covered by unit/integration tests and verified live.
Verified end-to-end: with MESH_FORCE_TOOL_EMULATION=1, Qwen2.5-3B emitted
the raw text 'TOOL_CALL {"name": "get_weather", "arguments":
{"city": "Paris"}}', parsed into a real OpenAI tool_call with
finish_reason tool_calls.
…tool call
Three changes that make server-side emulation actually work for the weak /
non-tool-trained models it targets, proven live on gemma-4-E4B:
- Prompt dominance (goose-style, server-safe): the emulation instruction now
leads the system message, with the client's original system content preserved
below it under '# Task context'. This gives the tool-calling frame the
dominant position goose gets by replacing the system prompt, without
discarding the client's authoritative prompt (a serving node must not do
that). Verified: gemma kept a pirate persona AND emitted the tool call.
- Robust parsing: scan for the TOOL_CALL marker anywhere (with balanced-JSON
extraction) instead of only at line start, so a call emitted right after a
reasoning marker (e.g. gemma's <|channel>thought...channel|>TOOL_CALL {...})
is still parsed. Handles multiple calls and trailing prose.
- Early stop (Jasper's tool_call_emitted -> Stop): stop generation once a
complete emulated tool call is produced, so the model does not ramble past
the call. Only active on emulated tools requests; native paths unaffected.
Verified end-to-end with MESH_FORCE_TOOL_EMULATION=1: gemma-4-E4B emitted a
parseable get_weather call (78 tokens, early-stopped) under a competing system
prompt; native path (no force) unchanged.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/skippy-server/src/frontend/generation_flow.rs`:
- Around line 387-389: The split multimodal generation path is missing the
emulation early-stop wiring that the non-split collector already uses. Update
generate_split_multimodal_text and the SplitMultimodalGeneration flow so
hook_request/emulation_active are carried through, then apply
TextGenerationCollector::with_emulation_stop(...) for the split collector as
well. This keeps split-path tool requests from continuing after an emulated
TOOL_CALL has completed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: cad5db15-3641-40c7-ba4d-c6a45bf2a441
📒 Files selected for processing (4)
crates/skippy-server/src/frontend.rscrates/skippy-server/src/frontend/generation_flow.rscrates/skippy-server/src/frontend/prompting.rscrates/skippy-server/src/frontend/tool_emulation.rs
🚧 Files skipped from review as they are similar to previous changes (1)
- crates/skippy-server/src/frontend/prompting.rs
Thread emulation_active through SplitMultimodalGeneration so the split multimodal generation path (embedded stage-0 with downstream lanes + media + tools for a non-tool-capable model) also stops generating once a complete emulated TOOL_CALL is produced, matching the local and non-split multimodal paths. Computed from hook_request/prompt at the construction site, which already has both in scope.
1 participant
Closes #944.
What you can now do
Point any OpenAI client — goose, curl, an SDK — at a mesh model that is not
tool-trained (e.g. a small model whose chat template ignores
tools), pass thestandard
toolsfield, and get back realtool_callswithfinish_reason: "tool_calls". Previously those models ignored the schemas orlooped re-issuing the same call.
The serving node is the only party that knows the loaded model's actual
chat-template capability, so it emulates tool calling when — and only when — the
template can't do it natively. Tool-capable models (including lean ones like
Qwen3-0.6B with a small toolset) keep native tool calling and see zero
behavior change.
How it works
template is applied, the staged runtime returns
metadata_json; nativesupport means
parse_tool_calls == trueand a non-emptychat_parser(the same signal goose checks).
tools, theprompt is re-rendered:
tools/tool_choicestripped, a compact instructioninjected (tool name + description + compact parameter schema with
?foroptional args), and history rewritten so the template never sees tool roles
(assistant
tool_calls→TOOL_CALL {json}text;role:"tool"→ user"Tool result:" text). The compact schema matters: the live prototype showed
name+description alone made a 0.6B model hallucinate argument names.
TOOL_CALL {json}lines →OpenAI
tool_calls, tolerant of surrounding prose and<think>blocks.Streaming holds back the trailing incomplete line and withholds calls until
finalization, matching native tool-call streaming semantics. Emulated calls
ride the same downstream assembly as native ones, so
call_mesh_*ids andstreaming behave identically.
Ported from goose's local-inference provider
(
crates/goose-local-inference/src/tool_emulation.rs,tool_parsing.rs,prompts/tiny_model_system.md).Architecture
crates/skippy-server/src/frontend/tool_emulation.rs(detection, request adaptation, response parsing).
frontend/prompting.rswires it intoprepare_chat_prompt(adaptation) andparse_chat_output(parsing), with the streaming-awareparse_emulated_chat_outputbridge.tools, so the flowing
metadata_jsonnaturally reports non-native and theparse path branches on the same signal.
docs/specs/server-side-tool-call-emulation.md.Validation
cargo test -p skippy-server --lib— 183 passed (19 new unit tests intool_emulation, 5 new integration tests infrontend::tests).cargo clippy -p skippy-server --all-targets -- -D warnings— clean.cargo fmt --all --check— clean.cargo check -p mesh-llm— clean (reachable from the shipped binary).Tests cover: capability detection (both signals required), compact schema with
required/optional flags, single/multiple/multi-turn calls,
<think>-blocktolerance, disallowed-name filtering, malformed JSON treated as prose, history
rewrite (system merge/insert, tool→user, assistant tool_calls→text), streaming
partial withholding, and
parallel_tool_calls:falsetruncation.Summary by CodeRabbit