Litellm oss staging 03 04 2026

[ghost]

Relevant issues

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Type

🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test

Changes

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Error		Mar 11, 2026 1:03pm

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bdb02c092e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Count system-only instructions in local token fallback

When provider-side counting is unavailable, the fallback only prepends the system prompt if fallback_messages is non-empty, so calls that pass system without any messages return a token count that ignores the instructions entirely. This undercounts valid inputs (for example Responses-style calls that only use instructions) and can break budgeting/rate-limit logic that depends on accurate token accounting.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Enforce tool allowlist on unmapped route aliases

The allowlist check exits early whenever get_call_types_for_route(route) returns None, which leaves tool-capable aliases outside API_ROUTE_TO_CALL_TYPES (for example /openai/v1/responses) completely unguarded. In those routes, requests with disallowed tools bypass allowed_tools enforcement, so keys/teams can invoke tools they are supposed to be blocked from using.

Useful? React with 👍 / 👎.

[greptile-apps]

Greptile Summary

This is a large staging PR that bundles several independent features: full Black Forest Labs (BFL) image generation and image edit support (with polling), OpenAI Responses API token counting, Mistral Voxtral audio transcription, Gemini 2.x global mediaResolution support, Vertex AI context-caching minimum-token guard, a Delta.reasoning → reasoning_content alias, automatic budget_reset_at computation in customer endpoints, and a batch of UI static-file reorganisations.

Key issues found:

• Hardcoded "gemini-2" in model check (litellm/llms/vertex_ai/gemini/transformation.py:684) — the mediaResolution feature gate uses a string substring match instead of a capability flag in model_prices_and_context_window.json, violating the project rule. New Gemini generations will not work without a code change.
• Hardcoded BFL model-to-endpoint mappings (litellm/llms/black_forest_labs/common_utils.py:26–42) — IMAGE_EDIT_MODELS and IMAGE_GENERATION_MODELS are static dicts in code rather than being driven by the JSON config, so adding a new BFL model requires a code change on top of the JSON update.
• image_url content blocks silently dropped in OpenAI token-count transformation (litellm/llms/openai/responses/count_tokens/transformation.py:124–132) — for multimodal user messages, only "text" parts are forwarded to the /v1/responses/input_tokens endpoint; image blocks are discarded, causing the returned token count to be substantially under-counted.

Confidence Score: 3/5

• Mergeable with caution — the BFL and Mistral additions are largely additive, but the OpenAI token-count undercounting and the hardcoded Gemini-2 flag are correctness issues that should be addressed before shipping.
• Score reflects two rule violations (hardcoded model flags) and one correctness bug (image tokens dropped from token-count requests) that affect the reliability of new features. The existing codebase is not broken by this PR, but the new OpenAI token-counting API will silently return wrong values for any multimodal conversation.
• litellm/llms/openai/responses/count_tokens/transformation.py (image content dropped), litellm/llms/vertex_ai/gemini/transformation.py (hardcoded gemini-2 check), litellm/llms/black_forest_labs/common_utils.py (hardcoded model-endpoint map)

Important Files Changed

Filename	Overview
litellm/llms/black_forest_labs/image_edit/handler.py	New BFL image-edit handler with sync/async polling. Logic is well-structured; the previously flagged timeout-not-forwarded issue is the main concern but is tracked elsewhere.
litellm/llms/black_forest_labs/image_generation/handler.py	New BFL image-generation handler mirroring the image-edit handler; same polling pattern and same previously flagged timeout concern.
litellm/llms/black_forest_labs/common_utils.py	Hardcodes IMAGE_EDIT_MODELS and IMAGE_GENERATION_MODELS dicts mapping model names to BFL API paths, violating the rule to store model-specific metadata in model_prices_and_context_window.json.
litellm/llms/vertex_ai/gemini/transformation.py	Adds global mediaResolution support for Gemini 2.x using a hardcoded "gemini-2" in model string check, violating the project rule against hardcoding model-specific flags.
litellm/llms/openai/responses/count_tokens/transformation.py	New OpenAI Responses API token-counting transformation; silently drops image_url content blocks from user messages, leading to incorrect (under-counted) token counts for multimodal conversations.
litellm/llms/openai/responses/count_tokens/handler.py	Clean async handler for the OpenAI /v1/responses/input_tokens endpoint; error handling and logging are appropriate.
litellm/llms/openai/responses/count_tokens/token_counter.py	Implements OpenAITokenCounter; correctly falls back when no input items are produced and uses a global handler instance. The system-only message fallback silently returning None is acceptable.
litellm/llms/mistral/audio_transcription/transformation.py	New Mistral Voxtral audio transcription transformation; clean implementation following the base class pattern with correct form-data handling.
litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py	Fixes user-provided HTTPHandler being incorrectly forwarded to the async make_call context by using a separate gemini_client parameter; logic looks correct.
litellm/llms/vertex_ai/context_caching/vertex_ai_context_caching.py	Adds minimum token count guard before attempting Vertex AI context caching to avoid API errors; uses existing is_prompt_caching_valid_prompt utility correctly.
litellm/main.py	Adds acount_tokens public API with provider-specific routing and local tiktoken fallback; the system-prompt-not-prepended-when-messages-empty issue is tracked elsewhere.
model_prices_and_context_window.json	Adds 8 Black Forest Labs models with correct pricing, modes, and supported_endpoints metadata.
litellm/proxy/management_endpoints/customer_endpoints.py	Adds automatic budget_reset_at calculation from budget_duration when not explicitly set; still uses deprecated datetime.utcnow() (tracked separately in previous threads).

Sequence Diagram

sequenceDiagram
    participant Caller
    participant images/main.py
    participant BFLHandler as BFL Handler (image_edit / image_generation)
    participant BFLConfig as BFL Transformation Config
    participant BFLAPI as BFL API (api.bfl.ai)

    Caller->>images/main.py: image_edit() / image_generation()
    images/main.py->>BFLHandler: bfl_image_edit.image_edit() / bfl_image_generation.image_generation()
    BFLHandler->>BFLConfig: validate_environment() → headers
    BFLHandler->>BFLConfig: get_complete_url() → url
    BFLHandler->>BFLConfig: transform_image_edit_request() → data
    BFLHandler->>BFLAPI: POST /v1/{model-path} (JSON body)
    BFLAPI-->>BFLHandler: {polling_url, id}
    loop Poll until Ready / timeout
        BFLHandler->>BFLAPI: GET polling_url
        BFLAPI-->>BFLHandler: {status: "Pending"|"Ready"|"Error"}
    end
    BFLHandler->>BFLConfig: transform_image_edit_response() → ImageResponse
    BFLHandler-->>images/main.py: ImageResponse
    images/main.py-->>Caller: ImageResponse

Loading

_{Last reviewed commit: 3dab620}

[greptile-apps]

The timeout parameter is declared in the _poll_for_result_sync and _poll_for_result_async method signatures but is never forwarded to the actual polling GET requests. This means each request uses the HTTP client's default timeout instead of the user-specified one, risking stuck polling loops.

Both the sync and async polling paths in image_edit/handler.py and image_generation/handler.py have this issue.

Suggested change

	url=complete_url,
	headers=headers,
	json=data,
	while time.time() - start_time < max_wait:
	response = sync_client.get(
	url=polling_url,
	headers=polling_headers,
	timeout=timeout,
	)

[greptile-apps]

The timeout parameter is declared in the _poll_for_result_sync and _poll_for_result_async method signatures but is never forwarded to the actual polling GET requests. This means each request uses the HTTP client's default timeout instead of the user-specified one, risking stuck polling loops.

Suggested change

	)
	except Exception as e:
	raise BlackForestLabsError(
	while time.time() - start_time < max_wait:
	response = sync_client.get(
	url=polling_url,
	headers=polling_headers,
	timeout=timeout,
	)

[greptile-apps]

The condition if system and fallback_messages: only prepends the system prompt when fallback_messages is non-empty. When the caller passes messages=None (empty list) together with a system prompt, the system prompt is silently dropped and token_counter receives an empty input, returning an incorrect token count.

Suggested change

	fallback_messages = messages or []
	if system and fallback_messages:
	fallback_messages = [{"role": "system", "content": system}] + fallback_messages
	local_count = litellm.token_counter(
	# Fallback to local tiktoken-based token counting
	fallback_messages = messages or []
	if system:
	fallback_messages = [{"role": "system", "content": system}] + fallback_messages
	local_count = litellm.token_counter(
	model=model,
	messages=fallback_messages,
	tools=tools,
	)

[greptile-apps]

datetime.utcnow() is deprecated as of Python 3.12 and should be replaced with datetime.now(timezone.utc).

Suggested change

	budget_request.budget_reset_at = datetime.utcnow() + timedelta(
	seconds=duration_in_seconds(duration=budget_request.budget_duration)
	)
	return budget_request
	if budget_request.budget_reset_at is None and budget_request.budget_duration is not None:
	from datetime import timezone
	budget_request.budget_reset_at = datetime.now(timezone.utc) + timedelta(
	seconds=duration_in_seconds(duration=budget_request.budget_duration)
	)

[greptile-apps]

Hardcoded model-version check violates provider-agnostic pattern

The check if "gemini-2" in model: hardcodes a model family name to gate the mediaResolution feature. Per the project's custom rule, model-specific capability flags should be stored in model_prices_and_context_window.json and read via get_model_info, so that support for new models works without requiring a code change.

For example, you could add a "supports_global_media_resolution": true field to the relevant Gemini 2.x entries in the JSON and then check it here:

model_info = litellm.get_model_info(model=model, custom_llm_provider="vertex_ai")
if model_info and model_info.get("supports_global_media_resolution"):
    max_media_resolution = _extract_max_media_resolution_from_messages(messages)
    ...

This way, future Gemini versions that adopt (or drop) global mediaResolution can be handled by updating the JSON only.

Rule Used: What: Do not hardcode model-specific flags in the ... (source)

[greptile-apps]

Hardcoded model-to-endpoint mapping requires code changes to support new models

IMAGE_EDIT_MODELS and IMAGE_GENERATION_MODELS are static dicts that map model names to BFL-specific API paths. Adding a new Black Forest Labs model therefore requires a code change here in addition to updating model_prices_and_context_window.json.

Per the project rule, model-specific metadata should be stored in the JSON config and loaded via get_model_info. Consider adding a BFL endpoint path field (e.g. "api_path") to each entry in model_prices_and_context_window.json and reading it through get_model_info at runtime:

"black_forest_labs/flux-kontext-pro": {
    "litellm_provider": "black_forest_labs",
    "mode": "image_edit",
    "api_path": "/v1/flux-kontext-pro",
    ...
}

Then in _get_model_endpoint:

model_info = litellm.get_model_info(model=model, custom_llm_provider="black_forest_labs")
if model_info and model_info.get("api_path"):
    return model_info["api_path"]
raise ValueError(f"Unknown BFL model: {model}")

This way, adding a new BFL model only requires updating the JSON.

Rule Used: What: Do not hardcode model-specific flags in the ... (source)

[greptile-apps]

image_url content blocks are silently dropped from the token count request

When a user message contains a list of content blocks (as in a multimodal request), only "text" blocks are forwarded to the OpenAI Responses API token-counting endpoint. Any "image_url" or "image_file" blocks are silently discarded.

Since image tokens can be substantial, this leads to a significant undercount for multimodal conversations. The OpenAI Responses API does support image inputs, so image blocks should be forwarded too. For example:

if isinstance(content, list):
    transformed_parts = []
    for block in content:
        if not isinstance(block, dict):
            if isinstance(block, str):
                transformed_parts.append(block)
            continue
        if block.get("type") == "text":
            transformed_parts.append(block.get("text", ""))
        elif block.get("type") == "image_url":
            # Forward image_url to the Responses API for accurate token counting
            image_url = block.get("image_url", {})
            transformed_parts.append({
                "type": "input_image",
                "image_url": image_url.get("url", ""),
            })
    content = transformed_parts if any(isinstance(p, dict) for p in transformed_parts) else "\n".join(p for p in transformed_parts if isinstance(p, str))