Skip to content

docs: local LLMs context size tip #3454

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
angiejones merged 5 commits into from
Jul 24, 2025
Merged

docs: local LLMs context size tip #3454

angiejones merged 5 commits into from
Jul 24, 2025

Conversation

@jjjuk
Copy link
Contributor

@jjjuk jjjuk commented Jul 16, 2025 •
edited
Loading

Hi everyone! Personally had issues with context size and Ollama, and saw several other cases. So I've extended tool calling warning with information about context size. Updated this one for Ollama and Ramalama.

Before:
image

After, for Ollama:
image

Note: Ollama docs poorly explain how to set this variable in the desktop app. I'm using systemd and cli but I guess most of the users will go for UI. There will be a reference by now, but it's not really useful. Need to contribute to their docs too 🥲

After, for Ramalama:
image

and small change in Ramalama example:

image

If you wan't to change something I'm happy to help!

@jjjuk jjjuk force-pushed the main branch 3 times, most recently from 1bd2b35 to 428dca4 Compare July 16, 2025 17:01
@jjjuk
Copy link
Contributor Author

jjjuk commented Jul 16, 2025 •
edited
Loading

Also, the context size increases VRAM usage, so there is a LLM-Model-VRAM-Calculator that estimates how much space is required for a model and its context mentioned in #1817

@DOsinga DOsinga requested a review from angiejones July 17, 2025 08:02
@jjjuk
Copy link
Contributor Author

jjjuk commented Jul 17, 2025 •
edited
Loading

Also I think we need to edit "(e.g. DeepSeek-R1)" 'cause DeepSeek-R1 already supports tool calling although it didn't before

michaelneale
michaelneale approved these changes Jul 18, 2025
View reviewed changes
Copy link
Collaborator

@michaelneale michaelneale left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for this, yeah there are so many little permutations and variables, this makes sense to me (I never could remember how to set the size).

@michaelneale
Copy link
Collaborator

michaelneale commented Jul 18, 2025

deepseek-r1 works on openrouter at least with tool calling (the ones I have tried) and I think it may on ollama now but haven't tried it, so yeah.

@angiejones
Copy link
Collaborator

angiejones commented Jul 20, 2025

i want to fix this up a bit before merging

@jjjuk
Copy link
Contributor Author

jjjuk commented Jul 20, 2025

i want to fix this up a bit before merging

Ok, which model is the example of a popular model that doesn't support tool calling? I suggest gemma3 for this one.

@angiejones
Copy link
Collaborator

angiejones commented Jul 24, 2025

thanks so much for this!

@jjjuk
Copy link
Contributor Author

jjjuk commented Jul 24, 2025

@angiejones Wait I didn't change the example! Haha

@jjjuk
Copy link
Contributor Author

jjjuk commented Jul 24, 2025 •
edited
Loading

@angiejones we need to change this part:

As an alternative, you can use a custom DeepSeek-r1 model we've made specifically for Goose.

Need your review to decide what to do with this part. Just delete this?

@angiejones angiejones changed the title docs: Tool calling support and Context size warning docs: local LLMs context size warning Jul 24, 2025
@angiejones angiejones changed the title docs: local LLMs context size warning docs: local LLMs context size tip Jul 24, 2025
@angiejones
Copy link
Collaborator

angiejones commented Jul 24, 2025

@angiejones we need to change this part:

As an alternative, you can use a custom DeepSeek-r1 model we've made specifically for Goose.

Need your review to decide what to do with this part. Just delete this?

how come delete?

angiejones
angiejones requested changes Jul 24, 2025
View reviewed changes
Copy link
Collaborator

@angiejones angiejones left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

want to split the warning into two

@angiejones angiejones merged commit a52e11a into block:main Jul 24, 2025
8 checks passed
katzdave added a commit that referenced this pull request Jul 24, 2025
@katzdave
Merge branch 'main' of github.com:block/goose into dkatz/goose-compact2
fcf0f27 
* 'main' of github.com:block/goose:
  Adds the `WaitingForUserInput` state (#3620)
  docs: update extensions library (#3612)
  Fixing grants path (#3632)
  docs: June 2024 Community All-Stars Spotlight (#3631)
  grant program (#3630)
  Lifei/sub recipe desktop temp (#3576)
  docs: local LLMs context size tip (#3454)
  fix: Handle non-default base path for OpenAI compatible model fetching (#3566)
  Goose security updates (#3579)
  fix: multi-tool calls in streamed openai-compatible responses (#3609)
  feat: subagent turn count, simple agent loop (#3597)
  feat: subagent independent extension manager (#3596)
  Improve session history loading resiliency (#3588)
  Added logging and changed default route case to not redirect to home when there is an invalid route (#3610)
  fix: chat sidebar layout overlapping content occasionally (#3590)
  fix: loading shared sessions (#3607)
atarantino pushed a commit to atarantino/goose that referenced this pull request Aug 5, 2025
@jjjuk @angiejones
docs: local LLMs context size tip (block#3454)
044a2b3 
Signed-off-by: jjjuk <[email protected]>
Co-authored-by: angiejones <[email protected]>
Signed-off-by: Adam Tarantino <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@michaelneale michaelneale michaelneale approved these changes

@angiejones angiejones angiejones approved these changes

Assignees

@angiejones angiejones

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jjjuk @michaelneale @angiejones