chore: remove tiktoken dependency and use API-reported token counts - #1526
Merged
Conversation
…ad of local tokenizer
tanzhenxin
requested review from
DennisYu07,
LaZzyMan,
Mingholy,
gwinthis and
pomelo-nwu
as code owners
January 18, 2026 04:55
Contributor
📋 Review SummaryThis PR successfully removes the 🔍 General Feedback
🎯 Specific Feedback🟡 High
🟢 Medium
🔵 Low
✅ Highlights
|
Contributor
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
Mingholy
approved these changes
Jan 21, 2026
xaelistic
pushed a commit
to xaelistic/qwen-code
that referenced
this pull request
Jun 7, 2026
chore: remove tiktoken dependency and use API-reported token counts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Remove
tiktokendependency and switch to using API-reported token counts forsessionTokenLimitand chat compression features. This simplifies the codebase and improves accuracy by leveraging the model's actual token counts fromusageMetadata.Dive Deeper
Problem
The
tiktokenlibrary has been causing issues:.wasmfiles at runtime, requiring special-case handling in the bundling processSolution
sessionTokenLimit: Now uses
lastPromptTokenCountfrom the API'susageMetadatainstead of locally estimating tokens. This is more accurate since it reflects the model's actual tokenization.Chat compression: Now uses API
usageMetadatafor token calculations, leveraging the actual token usage statistics returned by LLM responses.Fallback estimation: For providers without native token counting APIs (OpenAI, Anthropic), a lightweight character-based estimator is used:
Benefits
Reviewer Test Plan
Please verify that
sessionTokenLimitand chat compression still work as expected.Testing Matrix
Linked issues / bugs
Fixes #1289