feat: ollama tool shim #1448

alicehau · 2025-02-28T22:46:15Z

Tool shim that uses ollama models to "interpret" LLM responses generated by other models particularly without native tool calling e.g., deepseek. The shim is currently a feature in the openrouter and ollama providers.

The shim instructs the primary model to output json for intended tool usage, the interpretive model uses ollama structured outputs to translate the primary model's message into valid json, and then that json is translated into valid tool calls to be invoked.

To use, set the env var GOOSE_TOOLSHIM=1

The default interpreter model is mistral, but you can override this with the env var GOOSE_TOOLSHIM_MODEL=[name of ollama model].

GOOSE_TOOLSHIM=1 GOOSE_TOOLSHIM_MODEL=llama3.2 cargo run --bin goose session

Recommend running ollama server with env var OLLAMA_CONTEXT_LENGTH set to something higher than the 2048 default limit. e.g., OLLAMA_CONTEXT_LENGTH=50000 ollama serve. This feature is only available from the ollama source build as it hasn't been released yet.

The most promising combination right now seems to be full deepseek-r1 via openrouter + toolshim.

michaelneale

very cool - surprised to also see it with openrouter - I assume it works ok there for similar models?

michaelneale · 2025-03-03T22:06:05Z

@alicehau I think we should get this in when ollama release is ready, as looks promising.

ahau-square · 2025-03-03T22:10:45Z

very cool - surprised to also see it with openrouter - I assume it works ok there for similar models?

yes, I've tested this out with at least deepseek r1 models on openrouter and it works the same!

salman1993 · 2025-03-05T13:42:42Z

crates/goose/src/providers/toolshim.rs

+use super::errors::ProviderError;
+use crate::message::{Message, MessageContent};
+use crate::model::ModelConfig;
+use crate::providers::formats::openai::create_request;


we can also consider copying the formats/openai.rs file into a new formats/ollama.rs and having some of these changes there. either is fine

my thought here was to leave open the possibility of a toolshim backed by a non-ollama model

crates/goose/src/providers/openai.rs

crates/goose/src/providers/mod.rs

crates/goose/src/providers/openrouter.rs

crates/goose/src/agents/truncate.rs

salman1993 · 2025-03-10T16:08:24Z

crates/goose/src/agents/truncate.rs

do we also need to add to reference and summarize agents?

thought those were just legacy at this point? I don't recall any way for folks to switch their agents?

crates/goose/src/providers/ollama.rs

salman1993

left a minor comment on moving the toolshim check up in the agent

crates/goose/src/providers/openrouter.rs

ahau-square · 2025-03-10T22:16:04Z

Made some big refactoring changes to pull all the shim logic out of the individual providers and put it into the agent instead

baxen

LGTM! Excited to try this

I think we should eventually move this logic into the creation of a "Model" where the model always has tool calling supported, either by a shim like this or natively. that i think we can revisit with a config upgrade in the near future. This seems like a good place to start trying it out now

baxen · 2025-03-12T04:19:47Z

crates/goose/src/agents/truncate.rs

+        let config = capabilities.provider().get_model_config();
+        let mut system_prompt = capabilities.get_system_prompt().await;
+        let mut toolshim_tools = vec![];
+        if config.interpret_chat_tool_calls {


nit: it sounds like i missed a version where this was handled in the provider instead? I like that separation of concerns actually. But also this definitely works

Yeah the original version was implemented in each provider. Moved in this direction of putting logic into the agent as a lot of the code was just duplicated across providers to check if the shim was on, then modify the tools passed, system prompt, and interpret the response received back. It can also be nicer to have in the agent as then you don't have to modify each new provider to add the shim.

baxen · 2025-03-12T04:35:22Z

crates/goose/src/model.rs

    /// Optional maximum tokens to generate
    pub max_tokens: Option<i32>,
+    /// Whether to interpret tool calls
+    pub interpret_chat_tool_calls: bool,


nit: i'd maybe just have tool_call_interpreter_model be an Option and it being Some/None replaces this field?

I currently have a default tool shim model, mistral-nemo set so you don't have to pass in the model. But I'm open to requiring you to pass a toolshim model also

crates/goose/src/model.rs

baxen · 2025-03-12T04:54:25Z

crates/goose/src/providers/toolshim.rs

+            let content = response["message"]["content"].as_str().unwrap_or_default();
+
+            // Try to parse the content as JSON
+            if let Ok(content_json) = serde_json::from_str::<Value>(content) {


i wonder if there are any ways to incrementally improve error handling here.

if we can't parse the json should we retry the structured generation? it should very rarely happen/never?

same as above for empty tool calls

same as above for a tool call with no name field

if we have a tool call with a name but arguments field should we surface that as a tool call error

not sure if any of this comes up in practice or does the structured generation just always consistently solve that?

the structured outputs from ollama ensure you conform to the schema you pass in every time as they are masking tokens that don't conform in the sampling stage (https://blog.danielclayton.co.uk/posts/ollama-structured-outputs/).

empty tool calls are actually ok, as not every response should have a tool call (i.e., agent asks for clarification or just completed the task)

if there are tool calls, they will always have "name" and "arguments" fields since those are passed as required to ollama's structured outputs

* upstream/main: feat(google_drive): move credentials into keychain, add optional fallback (block#1603) feat: add session list command in cli (block#1586) feat: google sheets support (in google drive builtin MCP server) (block#1601) fix: deep link opening when window is closed (block#1633) docs: edits to docker guide (block#1639) feat: ollama tool shim (block#1448) feat: add write approve mode (block#1628) ui: auto update card upon config (block#1610) fix: fix tool output expansion checks (block#1634) fix: remove conditional that breaks output display for tool calls (block#1631) docs: Persistent Command History (block#1627) change to make build work on windows, macos, linux (block#1618) chore(release): release version 1.0.13 (block#1623) fix: handle mac screenshots with the image tool (block#1622) feat: write eval results to eval dir (block#1620) [fix] fix model config logging to remove api key (block#1619) fix: ensure repeating benches return to initial run-dir (block#1617)

Co-authored-by: Alice Hau <ahau@squareup.com>

alicehau requested review from baxen, michaelneale, salman1993 and zakiali February 28, 2025 22:47

michaelneale approved these changes Mar 3, 2025

View reviewed changes

ahau-square requested a review from laanak08 March 4, 2025 14:54

salman1993 reviewed Mar 5, 2025

View reviewed changes

crates/goose/src/providers/openai.rs Outdated Show resolved Hide resolved

zakiali reviewed Mar 5, 2025

View reviewed changes

crates/goose/src/providers/mod.rs Outdated Show resolved Hide resolved

zakiali reviewed Mar 5, 2025

View reviewed changes

crates/goose/src/providers/openrouter.rs Outdated Show resolved Hide resolved

ahau-square force-pushed the ahau/toolshim-refactor branch from 123f057 to 65d041c Compare March 7, 2025 23:26

ahau-square marked this pull request as ready for review March 10, 2025 15:43

salman1993 reviewed Mar 10, 2025

View reviewed changes

crates/goose/src/agents/truncate.rs Outdated Show resolved Hide resolved

salman1993 reviewed Mar 10, 2025

View reviewed changes

crates/goose/src/providers/ollama.rs Outdated Show resolved Hide resolved

salman1993 requested changes Mar 10, 2025

View reviewed changes

ahau-square force-pushed the ahau/toolshim-refactor branch from 1a31146 to fdd46c3 Compare March 10, 2025 16:56

ahau-square mentioned this pull request Mar 10, 2025

feat: add additional goosebench evals #1571

Merged

zakiali reviewed Mar 10, 2025

View reviewed changes

crates/goose/src/providers/openrouter.rs Outdated Show resolved Hide resolved

alicehau added 9 commits March 10, 2025 17:56

ollama model + structured output shim

e1b5ae9

tool shim generic

220f926

refactoring toolshim

b9b6fe8

refactor

d666bcb

ollama toolshim ready

73407cd

clean up

05530b3

lint

bba2741

format

acfbf83

add toolshim to openai provider

48544e3

Alice Hau added 13 commits March 10, 2025 17:56

refactor

50a1a5f

undo change to reference agent

dc67538

undo ref agent changes

4b8805d

refactor config

004d032

refactor

6877687

check env vars

5b035cb

fix tests

23a00e3

clean up

bbdbe66

start moving interpret chat tool check into agent

9ad5d3a

fully move shim logic into agent

8f76da0

minor semicolon fix

374ede5

refactor toolshim logic out of providers

255203d

revert providers and rebase

190a26c

ahau-square force-pushed the ahau/toolshim-refactor branch from 56b986f to 190a26c Compare March 10, 2025 22:00

refactor

ed61236

Alice Hau and others added 2 commits March 10, 2025 19:35

lint

e6866c5

Merge branch 'main' into ahau/toolshim-refactor

b22340d

baxen approved these changes Mar 12, 2025

View reviewed changes

Alice Hau added 3 commits March 12, 2025 10:08

rename var

f7f71ee

rename vars to toolshim

6de0be6

renaming

f0bc9f6

salman1993 approved these changes Mar 12, 2025

View reviewed changes

alicehau merged commit 259ccd5 into main Mar 12, 2025
6 checks passed

alicehau deleted the ahau/toolshim-refactor branch March 12, 2025 15:01

acekyd mentioned this pull request Mar 19, 2025

docs: add experimental features guide with Ollama tool shim documentation #1774

Closed

zakiali mentioned this pull request Mar 24, 2025

Unhelpful responses for Tool Calling functionality with Ollama #1817

Closed

ahau-square pushed a commit that referenced this pull request May 2, 2025

feat: ollama tool shim (#1448)

b156ac1

Co-authored-by: Alice Hau <ahau@squareup.com>

cbruyndoncx pushed a commit to cbruyndoncx/goose that referenced this pull request Jul 20, 2025

feat: ollama tool shim (block#1448)

a758e8d

Co-authored-by: Alice Hau <ahau@squareup.com>

feat: ollama tool shim #1448

feat: ollama tool shim #1448

Uh oh!

Conversation

alicehau commented Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michaelneale left a comment

Choose a reason for hiding this comment

Uh oh!

michaelneale commented Mar 3, 2025

Uh oh!

ahau-square commented Mar 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

salman1993 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ahau-square commented Mar 10, 2025

Uh oh!

baxen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

alicehau commented Feb 28, 2025 •

edited

Loading