Skip to content

docs(count_tokens): document TOKEN_COUNTER_MAX_EXACT_CHARS and off-loop local counting #1259

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mateo-berri merged 4 commits into from
Sep 10, 2026
Merged

docs(count_tokens): document TOKEN_COUNTER_MAX_EXACT_CHARS and off-loop local counting #1259

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
98065f3
docs(count_tokens): document TOKEN_COUNTER_MAX_EXACT_CHARS and off-lo…
mateo-berri Sep 8, 2026
29a4acc
docs(count_tokens): describe the sampled estimate above TOKEN_COUNTER…
mateo-berri Sep 8, 2026
ba892f9
Merge remote-tracking branch 'origin/main' into litellm_lit_5546_toke…
mateo-berri Sep 10, 2026
6e8e368
docs(count_tokens): document TOKEN_COUNTER_MAX_CONCURRENT_COUNTS
mateo-berri Sep 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/count_tokens.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,8 @@ result = await litellm.acount_tokens(
print(result.tokenizer_type) # "local_tokenizer"
```

On the proxy, local counting runs in a worker thread, so a large payload does not hold up other requests. Each worker process counts at most `TOKEN_COUNTER_MAX_CONCURRENT_COUNTS` payloads at a time (default 4) and queues the rest, which bounds the memory a burst of large counts can take. Strings longer than `TOKEN_COUNTER_MAX_EXACT_CHARS` characters (default 4,000,000, roughly a million tokens) are estimated by tokenizing 16 evenly spaced samples that together total that many characters and scaling the result by the string's length, which keeps the cost of the largest payloads bounded.

## Proxy Usage

### OpenAI Format: `/v1/responses/input_tokens`
Expand Down
2 changes: 2 additions & 0 deletions docs/proxy/config_settings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1457,6 +1457,8 @@ router_settings:
| TOGETHER_AI_110_B | Size parameter for Together AI 110B model. Default is 110
| TOGETHER_AI_EMBEDDING_150_M | Size parameter for Together AI 150M embedding model. Default is 150
| TOGETHER_AI_EMBEDDING_350_M | Size parameter for Together AI 350M embedding model. Default is 350
| TOKEN_COUNTER_MAX_CONCURRENT_COUNTS | Local token counts each worker process runs at the same time before the rest queue. Default is 4
| TOKEN_COUNTER_MAX_EXACT_CHARS | Characters per string above which the local token counter tokenizes 16 evenly spaced samples that together total that many characters and scales the count by the string's length. Default is 4000000
| TOOL_CHOICE_OBJECT_TOKEN_COUNT | Token count for tool choice objects. Default is 4
| TOOL_POLICY_CACHE_TTL_SECONDS | TTL in seconds for caching tool policy guardrail results. Default is 60
| UI_LOGO_PATH | Path to the logo image used in the UI
Expand Down
Loading