Make token counter safer #4924
Merged
Make token counter safer #4924
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jamadeo
approved these changes
Oct 1, 2025
crates/goose/src/token_counter.rs
Outdated
| if let Some(item_str) = item.as_str() { | ||
| func_token_count += enum_item; | ||
| func_token_count = | ||
| func_token_count.saturating_add_signed(ENUM_INIT); |
Collaborator
There was a problem hiding this comment.
is this not supposed to be ENUM_ITEM?
crates/goose/src/token_counter.rs
Outdated
| if let Some(item_str) = item.as_str() { | ||
| func_token_count += enum_item; | ||
| func_token_count = | ||
| func_token_count.saturating_add_signed(ENUM_INIT); |
Collaborator
There was a problem hiding this comment.
also here.
How is this different from the one in AsyncTokenCounter?
Collaborator
Author
There was a problem hiding this comment.
this code is clearly duplicated
jamadeo
reviewed
Oct 1, 2025
crates/goose/src/token_counter.rs
Outdated
| Err(e) => panic!("Failed to initialize o200k_base tokenizer: {}", e), | ||
| } | ||
| tiktoken_rs::o200k_base().map(Arc::new).unwrap_or_else(|e| { | ||
| eprintln!("Failed to initialize o200k_base tokenizer: {}", e); |
Collaborator
There was a problem hiding this comment.
this looks like a log message the user wouldn't know what to do with
Collaborator
Author
There was a problem hiding this comment.
yeah, I know, but I don't know what else we should do. maybe still crash?
Collaborator
|
Where are you seeing this actually get used, local models? The counting should be done generally in provider metadata with this as a fallback. |
Collaborator
Author
we see crashes here and there. most recently: #4513 |
katzdave
added a commit
that referenced
this pull request
Oct 1, 2025
* 'main' of github.com:block/goose: fix: session timestamps (#4913) feat: lazy infinite scroller for session history view (#4922) chore: properly identify when to try oauth (#4918) Make token counter safer (#4924) Rename Hacktoberfest Blog to Hacktoberfest Content (#4926) Include Session ID appropriately in UI (#4901)
zanesq
added a commit
that referenced
this pull request
Oct 1, 2025
…-deeplink * 'main' of github.com:block/goose: (57 commits) Don't set agent props twice (#4872) fix: conversation fixer merges assistant text blocks and drops empty text messages (#4898) Batch fetch remaining issues for documentation updates fix: session timestamps (#4913) feat: lazy infinite scroller for session history view (#4922) chore: properly identify when to try oauth (#4918) Make token counter safer (#4924) Rename Hacktoberfest Blog to Hacktoberfest Content (#4926) Include Session ID appropriately in UI (#4901) fix mcp integration test flakiness (#4871) Add type field to empty schemas for anthropic (#4911) docs: remove subagents from experimental (#4907) CLI: dont show logs to user (#4902) patching the security scanner to redact environment variables (#4908) rmcp upgrade (#4792) feat: Use the screen, goose (#4905) added claude-sonnet-4-5 (#4906) Tiny: fix github casing (#4903) remove anyOf from create_task tool (#4897) chore(deps): bump tracing-subscriber from 0.3.19 to 0.3.20 (#4442) ...
zanesq
added a commit
that referenced
this pull request
Oct 1, 2025
* 'main' of github.com:block/goose: Don't set agent props twice (#4872) fix: conversation fixer merges assistant text blocks and drops empty text messages (#4898) Batch fetch remaining issues for documentation updates fix: session timestamps (#4913) feat: lazy infinite scroller for session history view (#4922) chore: properly identify when to try oauth (#4918) Make token counter safer (#4924) Rename Hacktoberfest Blog to Hacktoberfest Content (#4926) Include Session ID appropriately in UI (#4901) fix mcp integration test flakiness (#4871) Add type field to empty schemas for anthropic (#4911) docs: remove subagents from experimental (#4907) CLI: dont show logs to user (#4902) patching the security scanner to redact environment variables (#4908) rmcp upgrade (#4792) feat: Use the screen, goose (#4905) # Conflicts: # ui/desktop/src/components/ui/scroll-area.tsx
michaelneale
added a commit
that referenced
this pull request
Oct 2, 2025
* main: Fix auto scroll to bottom during chat (#4923) Fix Typo, Add Description to Hacktoberfest Content Issue Template (#4931) Don't set agent props twice (#4872) fix: conversation fixer merges assistant text blocks and drops empty text messages (#4898) Batch fetch remaining issues for documentation updates fix: session timestamps (#4913) feat: lazy infinite scroller for session history view (#4922) chore: properly identify when to try oauth (#4918) Make token counter safer (#4924) Rename Hacktoberfest Blog to Hacktoberfest Content (#4926) Include Session ID appropriately in UI (#4901) fix mcp integration test flakiness (#4871)
michaelneale
added a commit
that referenced
this pull request
Oct 2, 2025
* main: (44 commits) Add PR template (#4934) Using --resume with --name should still accept session IDs (#4937) Fix auto scroll to bottom during chat (#4923) Fix Typo, Add Description to Hacktoberfest Content Issue Template (#4931) Don't set agent props twice (#4872) fix: conversation fixer merges assistant text blocks and drops empty text messages (#4898) Batch fetch remaining issues for documentation updates fix: session timestamps (#4913) feat: lazy infinite scroller for session history view (#4922) chore: properly identify when to try oauth (#4918) Make token counter safer (#4924) Rename Hacktoberfest Blog to Hacktoberfest Content (#4926) Include Session ID appropriately in UI (#4901) fix mcp integration test flakiness (#4871) Add type field to empty schemas for anthropic (#4911) docs: remove subagents from experimental (#4907) CLI: dont show logs to user (#4902) patching the security scanner to redact environment variables (#4908) rmcp upgrade (#4792) feat: Use the screen, goose (#4905) ...
zanesq
added a commit
that referenced
this pull request
Oct 2, 2025
…-unification * 'main' of github.com:block/goose: (24 commits) Fix auto scroll to bottom during chat (#4923) Fix Typo, Add Description to Hacktoberfest Content Issue Template (#4931) Don't set agent props twice (#4872) fix: conversation fixer merges assistant text blocks and drops empty text messages (#4898) Batch fetch remaining issues for documentation updates fix: session timestamps (#4913) feat: lazy infinite scroller for session history view (#4922) chore: properly identify when to try oauth (#4918) Make token counter safer (#4924) Rename Hacktoberfest Blog to Hacktoberfest Content (#4926) Include Session ID appropriately in UI (#4901) fix mcp integration test flakiness (#4871) Add type field to empty schemas for anthropic (#4911) docs: remove subagents from experimental (#4907) CLI: dont show logs to user (#4902) patching the security scanner to redact environment variables (#4908) rmcp upgrade (#4792) feat: Use the screen, goose (#4905) added claude-sonnet-4-5 (#4906) Tiny: fix github casing (#4903) ... # Conflicts: # ui/desktop/src/components/BaseChat.tsx
wpfleger96
added a commit
to wpfleger96/goose
that referenced
this pull request
Oct 2, 2025
* main: (30 commits) feat(nightly): build nightlies from main shas (block#4888) Add missing library for fedora/rhel/centos docs (block#4819) feat(process): Add GOVERNANCE and MAINTAINERS documents (block#4962) Pause test finder, have it run cargo fmt (block#4958) Disable the issue comment trigger on pr-comment-bundle (block#4961) fix(providers): update Claude Sonnet 4 model identifier (block#4884) fix redirect to extensions page after deeplink install and show toast with success message (block#4863) Remove wait-for-ready log (block#4956) docs: add a new goose tip (block#4940) Add PR template (block#4934) Using --resume with --name should still accept session IDs (block#4937) Fix auto scroll to bottom during chat (block#4923) Fix Typo, Add Description to Hacktoberfest Content Issue Template (block#4931) Don't set agent props twice (block#4872) fix: conversation fixer merges assistant text blocks and drops empty text messages (block#4898) Batch fetch remaining issues for documentation updates fix: session timestamps (block#4913) feat: lazy infinite scroller for session history view (block#4922) chore: properly identify when to try oauth (block#4918) Make token counter safer (block#4924) ...
HikaruEgashira
pushed a commit
to HikaruEgashira/goose
that referenced
this pull request
Oct 3, 2025
Co-authored-by: Douwe Osinga <[email protected]> Signed-off-by: HikaruEgashira <[email protected]>
This was referenced Oct 8, 2025
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fixes things up a bit. most crucially dies less