Repository navigation
feat(openai): add litellm.acount_tokens() public API + OpenAI token counting support #22809
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Chesars
merged 15 commits into
BerriAI:litellm_oss_staging_03_04_2026
from
Chesars:worktree-count-tokens-api
Mar 5, 2026
Merged
Changes from 11 commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
43d2a19
feat(openai): add OpenAI token counting API support and public litellβ¦
Chesars d33dec8
fix: address Greptile review feedback
Chesars 018750e
Update litellm/llms/openai/responses/count_tokens/transformation.py
Chesars 3976298
Update litellm/proxy/response_api_endpoints/endpoints.py
Chesars cb54215
Update litellm/proxy/response_api_endpoints/endpoints.py
Chesars c185652
fix: avoid chat-format fallback for empty input_items, remove duplicaβ¦
Chesars 28a20c1
fix: add ProxyException handling to count_tokens endpoint
Chesars 91928d9
Update litellm/llms/openai/responses/count_tokens/transformation.py
Chesars 59c64cb
Update litellm/proxy/response_api_endpoints/endpoints.py
Chesars 1fb38df
Update litellm/llms/openai/responses/count_tokens/transformation.py
Chesars 13dbcb1
Update transformation.py
Chesars bbec0f7
Update litellm/proxy/response_api_endpoints/endpoints.py
Chesars 92b0585
Update litellm/llms/openai/responses/count_tokens/handler.py
Chesars 8786e67
fix: address PR review feedback β F821, double auth, strict field, doβ¦
Chesars abc381c
fix: use chat format in tools test, include tools/system in local falβ¦
Chesars File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,187 @@ | ||
| import Tabs from '@theme/Tabs'; | ||
| import TabItem from '@theme/TabItem'; | ||
|
|
||
| # Token Counting | ||
|
|
||
| ## Overview | ||
|
|
||
| LiteLLM provides exact token counting by calling provider-specific token counting APIs. This gives you accurate token counts before sending requests, helping with cost estimation and context window management. | ||
|
|
||
| | Feature | Details | | ||
| |---------|---------| | ||
| | SDK Method | `litellm.acount_tokens()` | | ||
| | Proxy Endpoints | `/v1/messages/count_tokens` (Anthropic format), `/v1/responses/input_tokens` (OpenAI format) | | ||
| | Fallback | Local tiktoken-based counting for unsupported providers | | ||
|
|
||
| ## Supported Providers | ||
|
|
||
| | Provider | Token Counting API | Format | | ||
| |----------|-------------------|--------| | ||
| | OpenAI | [Responses API `/input_tokens`](https://platform.openai.com/docs/api-reference/responses/input-tokens) | OpenAI Responses | | ||
| | Anthropic | [Messages `/count_tokens`](https://docs.anthropic.com/en/docs/build-with-claude/token-counting) | Anthropic Messages | | ||
| | Vertex AI (Claude) | Vertex AI Partner Models Token Counter | Anthropic Messages | | ||
| | Bedrock (Claude) | AWS Bedrock CountTokens API | Anthropic Messages | | ||
| | Gemini | Google AI Studio countTokens API | Anthropic Messages | | ||
| | Vertex AI (Gemini) | Vertex AI countTokens API | Anthropic Messages | | ||
| | Other providers | Local tiktoken fallback | N/A | | ||
|
|
||
| ## SDK Usage | ||
|
|
||
| ### Basic Usage | ||
|
|
||
| ```python | ||
| import asyncio | ||
| import litellm | ||
|
|
||
| async def main(): | ||
| # OpenAI | ||
| result = await litellm.acount_tokens( | ||
| model="openai/gpt-4o", | ||
| messages=[{"role": "user", "content": "Hello, how are you?"}], | ||
| ) | ||
| print(f"Token count: {result.total_tokens}") | ||
| print(f"Tokenizer: {result.tokenizer_type}") # "openai_api" | ||
|
|
||
| # Anthropic | ||
| result = await litellm.acount_tokens( | ||
| model="anthropic/claude-3-5-sonnet-20241022", | ||
| messages=[{"role": "user", "content": "Hello, how are you?"}], | ||
| ) | ||
| print(f"Token count: {result.total_tokens}") | ||
| print(f"Tokenizer: {result.tokenizer_type}") # "anthropic_api" | ||
|
|
||
| asyncio.run(main()) | ||
| ``` | ||
|
|
||
| ### With Tools and System Message | ||
|
|
||
| ```python | ||
| import asyncio | ||
| import litellm | ||
|
|
||
| async def main(): | ||
| result = await litellm.acount_tokens( | ||
| model="openai/gpt-4o", | ||
| messages=[{"role": "user", "content": "What's the weather in Paris?"}], | ||
| tools=[{ | ||
| "type": "function", | ||
| "name": "get_weather", | ||
| "description": "Get weather for a city", | ||
| "parameters": { | ||
| "type": "object", | ||
| "properties": {"city": {"type": "string"}}, | ||
| }, | ||
| }], | ||
| system="You are a helpful weather assistant.", | ||
| ) | ||
| print(f"Token count (with tools): {result.total_tokens}") | ||
|
|
||
| asyncio.run(main()) | ||
| ``` | ||
|
|
||
| ### Response Format | ||
|
|
||
| `litellm.acount_tokens()` returns a `TokenCountResponse`: | ||
|
|
||
| ```python | ||
| TokenCountResponse( | ||
| total_tokens=15, # Token count | ||
| request_model="openai/gpt-4o", # Model requested | ||
| model_used="gpt-4o", # Model used for counting | ||
| tokenizer_type="openai_api", # "openai_api", "anthropic_api", "local_tokenizer" | ||
| original_response={"input_tokens": 15}, # Raw API response | ||
| error=False, # True if counting failed | ||
| error_message=None, # Error details if failed | ||
| ) | ||
| ``` | ||
|
|
||
| ### Fallback Behavior | ||
|
|
||
| If a provider doesn't support a token counting API, or if the API key is missing, `acount_tokens()` automatically falls back to local tiktoken-based counting: | ||
|
|
||
| ```python | ||
| # Unsupported provider β automatic fallback | ||
| result = await litellm.acount_tokens( | ||
| model="together_ai/meta-llama/Llama-3-8b-chat-hf", | ||
| messages=[{"role": "user", "content": "Hello"}], | ||
| ) | ||
| print(result.tokenizer_type) # "local_tokenizer" | ||
| ``` | ||
|
|
||
| ## Proxy Usage | ||
|
|
||
| ### OpenAI Format β `/v1/responses/input_tokens` | ||
|
|
||
| <Tabs> | ||
| <TabItem value="curl" label="curl"> | ||
|
|
||
| ```bash | ||
| curl -X POST "http://localhost:4000/v1/responses/input_tokens" \ | ||
| -H "Content-Type: application/json" \ | ||
| -H "Authorization: Bearer sk-1234" \ | ||
| -d '{ | ||
| "model": "gpt-4o", | ||
| "input": "Hello, how are you?" | ||
| }' | ||
| ``` | ||
|
|
||
| </TabItem> | ||
| <TabItem value="python" label="Python (httpx)"> | ||
|
|
||
| ```python | ||
| import httpx | ||
|
|
||
| response = httpx.post( | ||
| "http://localhost:4000/v1/responses/input_tokens", | ||
| headers={ | ||
| "Content-Type": "application/json", | ||
| "Authorization": "Bearer sk-1234" | ||
| }, | ||
| json={ | ||
| "model": "gpt-4o", | ||
| "input": "Hello, how are you?" | ||
| } | ||
| ) | ||
|
|
||
| print(response.json()) | ||
| # {"input_tokens": 7} | ||
| ``` | ||
|
|
||
| </TabItem> | ||
| </Tabs> | ||
|
|
||
| **Response:** | ||
| ```json | ||
| {"input_tokens": 7} | ||
| ``` | ||
|
|
||
| ### Anthropic Format β `/v1/messages/count_tokens` | ||
|
|
||
| See [Anthropic Token Counting](./anthropic_count_tokens.md) for full documentation. | ||
|
|
||
| ```bash | ||
| curl -X POST "http://localhost:4000/v1/messages/count_tokens" \ | ||
| -H "Content-Type: application/json" \ | ||
| -H "Authorization: Bearer sk-1234" \ | ||
| -d '{ | ||
| "model": "claude-3-5-sonnet-20241022", | ||
| "messages": [ | ||
| {"role": "user", "content": "Hello, how are you?"} | ||
| ] | ||
| }' | ||
| ``` | ||
|
|
||
| ## Proxy Configuration | ||
|
|
||
| ```yaml | ||
| model_list: | ||
| - model_name: gpt-4o | ||
| litellm_params: | ||
| model: openai/gpt-4o | ||
| api_key: os.environ/OPENAI_API_KEY | ||
|
|
||
| - model_name: claude-3-5-sonnet | ||
| litellm_params: | ||
| model: anthropic/claude-3-5-sonnet-20241022 | ||
| api_key: os.environ/ANTHROPIC_API_KEY | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| """ | ||
| OpenAI Responses API token counting implementation. | ||
| """ | ||
|
|
||
| from litellm.llms.openai.responses.count_tokens.handler import ( | ||
| OpenAICountTokensHandler, | ||
| ) | ||
| from litellm.llms.openai.responses.count_tokens.token_counter import ( | ||
| OpenAITokenCounter, | ||
| ) | ||
| from litellm.llms.openai.responses.count_tokens.transformation import ( | ||
| OpenAICountTokensConfig, | ||
| ) | ||
|
|
||
| __all__ = [ | ||
| "OpenAICountTokensHandler", | ||
| "OpenAICountTokensConfig", | ||
| "OpenAITokenCounter", | ||
| ] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,105 @@ | ||
| """ | ||
| OpenAI Responses API token counting handler. | ||
|
|
||
| Uses httpx for HTTP requests to OpenAI's /v1/responses/input_tokens endpoint. | ||
| """ | ||
|
|
||
| import json | ||
| from typing import Any, Dict, List, Optional, Union | ||
|
|
||
| import httpx | ||
|
|
||
| import litellm | ||
| from litellm._logging import verbose_logger | ||
| from litellm.llms.custom_httpx.http_handler import get_async_httpx_client | ||
| from litellm.llms.openai.common_utils import OpenAIError | ||
| from litellm.llms.openai.responses.count_tokens.transformation import ( | ||
| OpenAICountTokensConfig, | ||
| ) | ||
|
|
||
|
|
||
| class OpenAICountTokensHandler(OpenAICountTokensConfig): | ||
| """ | ||
| Handler for OpenAI Responses API token counting requests. | ||
| """ | ||
|
|
||
| async def handle_count_tokens_request( | ||
| self, | ||
| model: str, | ||
| input: Union[str, List[Any]], | ||
| api_key: str, | ||
| api_base: Optional[str] = None, | ||
| timeout: Optional[Union[float, httpx.Timeout]] = None, | ||
| tools: Optional[List[Dict[str, Any]]] = None, | ||
| instructions: Optional[str] = None, | ||
| ) -> Dict[str, Any]: | ||
| """ | ||
| Handle a token counting request to OpenAI's Responses API. | ||
|
|
||
| Returns: | ||
| Dictionary containing {"input_tokens": <number>} | ||
|
|
||
| Raises: | ||
| OpenAIError: If the API request fails | ||
| """ | ||
| try: | ||
| self.validate_request(model, input) | ||
|
|
||
| verbose_logger.debug( | ||
| f"Processing OpenAI CountTokens request for model: {model}" | ||
| ) | ||
|
|
||
| request_body = self.transform_request_to_count_tokens( | ||
| model=model, | ||
| input=input, | ||
| tools=tools, | ||
| instructions=instructions, | ||
| ) | ||
|
|
||
| endpoint_url = self.get_openai_count_tokens_endpoint(api_base) | ||
|
|
||
| verbose_logger.debug(f"Making request to: {endpoint_url}") | ||
|
|
||
| headers = self.get_required_headers(api_key) | ||
|
|
||
| async_client = get_async_httpx_client( | ||
| llm_provider=litellm.LlmProviders.OPENAI | ||
| ) | ||
|
|
||
| request_timeout = timeout if timeout is not None else litellm.request_timeout | ||
|
|
||
| response = await async_client.post( | ||
| endpoint_url, | ||
| headers=headers, | ||
| json=request_body, | ||
| timeout=request_timeout, | ||
| ) | ||
|
|
||
| verbose_logger.debug(f"Response status: {response.status_code}") | ||
|
|
||
| if response.status_code != 200: | ||
| error_text = response.text | ||
| verbose_logger.error(f"OpenAI API error: {error_text}") | ||
| raise OpenAIError( | ||
| status_code=response.status_code, | ||
| message=error_text, | ||
| ) | ||
|
|
||
| openai_response = response.json() | ||
| verbose_logger.debug(f"OpenAI response: {openai_response}") | ||
| return openai_response | ||
|
|
||
| except OpenAIError: | ||
| raise | ||
| except httpx.HTTPStatusError as e: | ||
| verbose_logger.error(f"HTTP error in CountTokens handler: {str(e)}") | ||
| raise OpenAIError( | ||
| status_code=e.response.status_code, | ||
| message=e.response.text, | ||
| ) | ||
| except (httpx.RequestError, json.JSONDecodeError) as e: | ||
| verbose_logger.error(f"Error in CountTokens handler: {str(e)}") | ||
| raise OpenAIError( | ||
| status_code=500, | ||
| message=f"CountTokens processing error: {str(e)}", | ||
| ) | ||
|
Chesars marked this conversation as resolved.
Resolved
|
||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.