diff --git a/docs/count_tokens.md b/docs/count_tokens.md index da5c1f887..d061ea86c 100644 --- a/docs/count_tokens.md +++ b/docs/count_tokens.md @@ -110,6 +110,8 @@ result = await litellm.acount_tokens( print(result.tokenizer_type) # "local_tokenizer" ``` +On the proxy, local counting runs in a worker thread, so a large payload does not hold up other requests. Each worker process counts at most `TOKEN_COUNTER_MAX_CONCURRENT_COUNTS` payloads at a time (default 4) and queues the rest, which bounds the memory a burst of large counts can take. Strings longer than `TOKEN_COUNTER_MAX_EXACT_CHARS` characters (default 4,000,000, roughly a million tokens) are estimated by tokenizing 16 evenly spaced samples that together total that many characters and scaling the result by the string's length, which keeps the cost of the largest payloads bounded. + ## Proxy Usage ### OpenAI Format: `/v1/responses/input_tokens` diff --git a/docs/proxy/config_settings.md b/docs/proxy/config_settings.md index aff4c0b40..c29cbe2dc 100644 --- a/docs/proxy/config_settings.md +++ b/docs/proxy/config_settings.md @@ -1457,6 +1457,8 @@ router_settings: | TOGETHER_AI_110_B | Size parameter for Together AI 110B model. Default is 110 | TOGETHER_AI_EMBEDDING_150_M | Size parameter for Together AI 150M embedding model. Default is 150 | TOGETHER_AI_EMBEDDING_350_M | Size parameter for Together AI 350M embedding model. Default is 350 +| TOKEN_COUNTER_MAX_CONCURRENT_COUNTS | Local token counts each worker process runs at the same time before the rest queue. Default is 4 +| TOKEN_COUNTER_MAX_EXACT_CHARS | Characters per string above which the local token counter tokenizes 16 evenly spaced samples that together total that many characters and scales the count by the string's length. Default is 4000000 | TOOL_CHOICE_OBJECT_TOKEN_COUNT | Token count for tool choice objects. Default is 4 | TOOL_POLICY_CACHE_TTL_SECONDS | TTL in seconds for caching tool policy guardrail results. Default is 60 | UI_LOGO_PATH | Path to the logo image used in the UI