refactor(types): replace Any with proven types in 7 files by devin-ai-integration[bot] · Pull Request #43844 · BerriAI/litellm

devin-ai-integration · 2026-09-30T09:38:56Z

TLDR

Problem this solves:

Any silently disables type checking wherever it lands
9,013 Any errors across the backend tree
Reviewers cannot tell which payloads are actually untyped

How it solves it:

Replaces Any with proven types in 7 files
Removes 8 Any errors, 14 basedpyright errors total
Behavior is unchanged: annotations plus one same-object import move

User Flow

This change only replaces Any annotations with proven types in seven files, so a developer calling the proxy sees the same flow at the merge base and at the tip, observed live against a 2-worker proxy on each commit

Before (merge base): every call a developer makes behaves the same as it does on main today

POST https://litellm-domain/v1/chat/completions with qa-gemini and "Reply with exactly PONG" returns 200 with PONG, finish_reason=stop and non-zero prompt and completion tokens
POST https://litellm-domain/v1/chat/completions with qa-openai and "stream": true returns 200 as SSE chunks assembling the model's short reply with a final usage chunk
POST https://litellm-domain/v1/embeddings with qa-embed returns 200 with one 3072-dimension vector and 5 prompt tokens
POST https://litellm-domain/key/generate with the master key returns a fresh sk-... virtual key limited to qa-gemini
POST https://litellm-domain/v1/chat/completions with that key and qa-gemini returns 200 with OK
The same key asking for qa-openai returns 403
GET https://litellm-domain/key/info?key=sk-... shows spend above 0 for the key
POST https://litellm-domain/key/delete for that key returns 200, and the next chat call with it returns 401
GET https://litellm-domain/guardrails/list returns 200 with an empty guardrails list
GET https://litellm-domain/v1/models returns 200 listing qa-embed, qa-fireworks, qa-gemini and qa-openai
POST https://litellm-domain/v1/responses with qa-gemini, streaming and not, returns 200 with PONG (9 events from response.created to response.completed when streaming), and with qa-openai returns 200 with PONG and non-zero usage
POST https://litellm-domain/v1/chat/completions with qa-fireworks returns 412 because the provider account is suspended
POST https://litellm-domain/v1/chat/completions with qa-gemini and a get_weather JSON-schema tool returns 200 with one tool call, arguments {"city": "Paris"}, finish_reason=tool_calls

After (this PR): the same developer walking the same 13 steps sees the same status codes and response shapes

POST https://litellm-domain/v1/chat/completions with qa-gemini and "Reply with exactly PONG" returns 200 with PONG, finish_reason=stop and non-zero prompt and completion tokens
POST https://litellm-domain/v1/chat/completions with qa-openai and "stream": true returns 200 as SSE chunks assembling the model's short reply with a final usage chunk
POST https://litellm-domain/v1/embeddings with qa-embed returns 200 with one 3072-dimension vector and 5 prompt tokens
POST https://litellm-domain/key/generate with the master key returns a fresh sk-... virtual key limited to qa-gemini
POST https://litellm-domain/v1/chat/completions with that key and qa-gemini returns 200 with OK
The same key asking for qa-openai returns 403
GET https://litellm-domain/key/info?key=sk-... shows spend above 0 for the key
POST https://litellm-domain/key/delete for that key returns 200, and the next chat call with it returns 401
GET https://litellm-domain/guardrails/list returns 200 with an empty guardrails list
GET https://litellm-domain/v1/models returns 200 listing qa-embed, qa-fireworks, qa-gemini and qa-openai
POST https://litellm-domain/v1/responses with qa-gemini, streaming and not, returns 200 with PONG (9 events from response.created to response.completed when streaming), and with qa-openai returns 200 with PONG and non-zero usage
POST https://litellm-domain/v1/chat/completions with qa-fireworks returns 412 because the provider account is suspended
POST https://litellm-domain/v1/chat/completions with qa-gemini and a get_weather JSON-schema tool returns 200 with one tool call, arguments {"city": "Paris"}, finish_reason=tool_calls

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup for both runs. Same config, same payloads, same live providers, same Postgres; only the checked-out commit differs

model_list:
  - model_name: qa-gemini
    litellm_params:
      model: gemini/gemini-2.5-flash-lite
      api_key: os.environ/GEMINI_API_KEY
  - model_name: qa-openai
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
  - model_name: qa-fireworks
    litellm_params:
      model: fireworks_ai/accounts/fireworks/models/gpt-oss-120b
      api_key: os.environ/FIREWORKS_AI_API_KEY
  - model_name: qa-embed
    litellm_params:
      model: gemini/gemini-embedding-001
      api_key: os.environ/GEMINI_API_KEY

general_settings:
  master_key: os.environ/QA_MASTER_KEY
  database_url: os.environ/QA_DATABASE_URL

litellm_settings:
  drop_params: true

git checkout <hash>
litellm --config config.yaml --port $PORT --num_workers 2

Before (`62a1b7f`)

Type-check count

Run basedpyright over the tree the way scripts/type_check_gate.py does:

.venv-typecheck/bin/basedpyright --outputjson --pythonpath .venv-typecheck/bin/python

Observed:

reportAny             7174
reportExplicitAny     1839
Any total             9013
all errors          141601

Chat completions

Send a non-streaming chat completion to qa-gemini:

curl -X POST http://127.0.0.1:$PORT/v1/chat/completions \
  -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-gemini","messages":[{"role":"user","content":"Reply with exactly: PONG"}],"max_tokens":16,"temperature":0}'

Observed:

HTTP 200
content='PONG'
finish_reason=stop model=qa-gemini
usage keys=['completion_tokens', 'completion_tokens_details', 'prompt_tokens'] prompt>0=True completion>0=True
id_prefix=G3G_as6oE

Send a streaming chat completion with usage to qa-openai:

curl -N -X POST http://127.0.0.1:$PORT/v1/chat/completions \
  -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-openai","messages":[{"role":"user","content":"Count: 1 2 3"}],"max_tokens":24,"temperature":0,"stream":true,"stream_options":{"include_usage":true}}'

Observed:

HTTP 200
n_chunks=11
assembled='1, 2, 3.'
final_usage_present=True prompt>0=True

Send a non-streaming chat completion to qa-fireworks:

curl -X POST http://127.0.0.1:$PORT/v1/chat/completions \
  -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-fireworks","messages":[{"role":"user","content":"Reply with exactly: PONG"}],"max_tokens":512,"temperature":0}'

Observed:

HTTP 412
Traceback (most recent call last):
  File "<string>", line 4, in <module>
KeyError: 'choices'

Tool calling

Send a chat completion to qa-gemini carrying a get_weather function tool:

curl -X POST http://127.0.0.1:$PORT/v1/chat/completions \
  -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-gemini","messages":[{"role":"user","content":"What is the weather in Paris? Use the tool."}],"tools":[{"type":"function","function":{"name":"get_weather","description":"Get weather for a city","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],"tool_choice":"auto","max_tokens":64,"temperature":0}'

Observed:

HTTP 200
n_tool_calls=1
name=get_weather arguments={"city": "Paris"}
finish_reason=tool_calls

Responses API

Send a non-streaming responses request to qa-gemini:

curl -X POST http://127.0.0.1:$PORT/v1/responses \
  -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-gemini","input":"Reply with exactly: PONG","max_output_tokens":16,"temperature":0}'

Observed:

HTTP 200
object=response status=completed model=qa-gemini
text='PONG'
usage input>0=True output>0=True

Send a non-streaming responses request to qa-openai:

curl -X POST http://127.0.0.1:$PORT/v1/responses \
  -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-openai","input":"Reply with exactly: PONG","max_output_tokens":16,"temperature":0}'

Observed:

HTTP 200
object=response status=completed model=qa-openai
text='PONG'
usage input>0=True output>0=True

Send a streaming responses request to qa-gemini:

curl -N -X POST http://127.0.0.1:$PORT/v1/responses \
  -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-gemini","input":"Reply with exactly: PONG","max_output_tokens":24,"temperature":0,"stream":true}'

Observed:

HTTP 200
n_events=9
first_type=response.created last_type=response.completed
assembled='PONG'
completed_event_present=True

Embeddings

Send an embedding request to qa-embed:

curl -X POST http://127.0.0.1:$PORT/v1/embeddings \
  -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-embed","input":"litellm any reduction"}'

Observed:

HTTP 200
object=list n_vectors=1 dim=3072
first3=[-0.024564, 0.00778, 0.001819]
prompt_tokens=5

Virtual key lifecycle, model access control and spend

Generate a key scoped to qa-gemini, use it on both models, read its spend, delete it and reuse it:

curl -X POST http://127.0.0.1:$PORT/key/generate -H "Authorization: Bearer $QA_MASTER_KEY" \
  -H 'Content-Type: application/json' -d '{"models":["qa-gemini"],"max_budget":1.0,"key_alias":"qa-<label>-<n>"}'
# then /v1/chat/completions with that key on qa-gemini and on qa-openai,
# then GET /key/info?key=<key>, POST /key/delete, and one more chat call

Observed:

generated key prefix: sk-n_ie...
chat-with-virtual-key HTTP 200
content='OK'
denied-model HTTP 403
error_mentions_key_not_allowed=False
key spend recorded > 0: True
key-delete HTTP 200
reuse-deleted-key HTTP 401

Guardrails and model list

List the guardrails and the models:

curl http://127.0.0.1:$PORT/guardrails/list -H "Authorization: Bearer $QA_MASTER_KEY"
curl http://127.0.0.1:$PORT/v1/models -H "Authorization: Bearer $QA_MASTER_KEY"

Observed:

HTTP 200
guardrails_key=True count=0
HTTP 200
models=['qa-embed', 'qa-fireworks', 'qa-gemini', 'qa-openai']

After (`85baad6`)

Type-check count

Run basedpyright over the tree the way scripts/type_check_gate.py does:

.venv-typecheck/bin/basedpyright --outputjson --pythonpath .venv-typecheck/bin/python

Observed:

reportAny             7170
reportExplicitAny     1835
Any total             9005
all errors          141587

Chat completions

Send a non-streaming chat completion to qa-gemini:

curl -X POST http://127.0.0.1:$PORT/v1/chat/completions \
  -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-gemini","messages":[{"role":"user","content":"Reply with exactly: PONG"}],"max_tokens":16,"temperature":0}'

Observed:

HTTP 200
content='PONG'
finish_reason=stop model=qa-gemini
usage keys=['completion_tokens', 'completion_tokens_details', 'prompt_tokens'] prompt>0=True completion>0=True
id_prefix=VnG_aoLwC

Send a streaming chat completion with usage to qa-openai:

curl -N -X POST http://127.0.0.1:$PORT/v1/chat/completions \
  -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-openai","messages":[{"role":"user","content":"Count: 1 2 3"}],"max_tokens":24,"temperature":0,"stream":true,"stream_options":{"include_usage":true}}'

Observed:

HTTP 200
n_chunks=11
assembled='1, 2, 3.'
final_usage_present=True prompt>0=True

Send a non-streaming chat completion to qa-fireworks:

curl -X POST http://127.0.0.1:$PORT/v1/chat/completions \
  -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-fireworks","messages":[{"role":"user","content":"Reply with exactly: PONG"}],"max_tokens":512,"temperature":0}'

Observed:

HTTP 412
Traceback (most recent call last):
  File "<string>", line 4, in <module>
KeyError: 'choices'

Tool calling

Send a chat completion to qa-gemini carrying a get_weather function tool:

curl -X POST http://127.0.0.1:$PORT/v1/chat/completions \
  -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-gemini","messages":[{"role":"user","content":"What is the weather in Paris? Use the tool."}],"tools":[{"type":"function","function":{"name":"get_weather","description":"Get weather for a city","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],"tool_choice":"auto","max_tokens":64,"temperature":0}'

Observed:

HTTP 200
n_tool_calls=1
name=get_weather arguments={"city": "Paris"}
finish_reason=tool_calls

Responses API

Send a non-streaming responses request to qa-gemini:

curl -X POST http://127.0.0.1:$PORT/v1/responses \
  -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-gemini","input":"Reply with exactly: PONG","max_output_tokens":16,"temperature":0}'

Observed:

HTTP 200
object=response status=completed model=qa-gemini
text='PONG'
usage input>0=True output>0=True

Send a non-streaming responses request to qa-openai:

curl -X POST http://127.0.0.1:$PORT/v1/responses \
  -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-openai","input":"Reply with exactly: PONG","max_output_tokens":16,"temperature":0}'

Observed:

HTTP 200
object=response status=completed model=qa-openai
text='PONG'
usage input>0=True output>0=True

Send a streaming responses request to qa-gemini:

curl -N -X POST http://127.0.0.1:$PORT/v1/responses \
  -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-gemini","input":"Reply with exactly: PONG","max_output_tokens":24,"temperature":0,"stream":true}'

Observed:

HTTP 200
n_events=9
first_type=response.created last_type=response.completed
assembled='PONG'
completed_event_present=True

Embeddings

Send an embedding request to qa-embed:

curl -X POST http://127.0.0.1:$PORT/v1/embeddings \
  -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-embed","input":"litellm any reduction"}'

Observed:

HTTP 200
object=list n_vectors=1 dim=3072
first3=[-0.024564, 0.00778, 0.001819]
prompt_tokens=5

Virtual key lifecycle, model access control and spend

Generate a key scoped to qa-gemini, use it on both models, read its spend, delete it and reuse it:

curl -X POST http://127.0.0.1:$PORT/key/generate -H "Authorization: Bearer $QA_MASTER_KEY" \
  -H 'Content-Type: application/json' -d '{"models":["qa-gemini"],"max_budget":1.0,"key_alias":"qa-<label>-<n>"}'
# then /v1/chat/completions with that key on qa-gemini and on qa-openai,
# then GET /key/info?key=<key>, POST /key/delete, and one more chat call

Observed:

generated key prefix: sk-Bo3e...
chat-with-virtual-key HTTP 200
content='OK'
denied-model HTTP 403
error_mentions_key_not_allowed=False
key spend recorded > 0: True
key-delete HTTP 200
reuse-deleted-key HTTP 401

Guardrails and model list

List the guardrails and the models:

curl http://127.0.0.1:$PORT/guardrails/list -H "Authorization: Bearer $QA_MASTER_KEY"
curl http://127.0.0.1:$PORT/v1/models -H "Authorization: Bearer $QA_MASTER_KEY"

Observed:

HTTP 200
guardrails_key=True count=0
HTTP 200
models=['qa-embed', 'qa-fireworks', 'qa-gemini', 'qa-openai']

Live A/B on changed paths (real Gemini and OpenAI, 2 proxy workers, fresh Postgres per leg)

Same config and probes on both legs: a complexity router with a live Gemini judge, a Responses API call mixing a gateway MCP tool (a local MCP math server behind a pre_mcp_call content-filter guardrail) with a caller-owned lookup_weather function tool, OpenAI chat streamed with usage, a response-cache repeat, and five virtual key generate, use, delete and reuse cycles. OpenAI is reached through a local recorder that forwards to the real API and logs what the proxy sent

Probe	`62a1b7f`	`85baad6`
Router, simple prompts (4)	200, routed to `simple-tier` on Gemini	200, routed to `simple-tier` on Gemini
Router, complex prompt (2)	200, routed to `complex-tier` on OpenAI	200, routed to `complex-tier` on OpenAI
Responses, MCP plus function tool (2, the second served without a new upstream call)	200, upstream got `math-add`, `math-multiply`, `math-fail`, `lookup_weather`; MCP peer got one `add(2, 3)`	200, byte-identical first upstream request; MCP peer got one `add(2, 3)`
Guardrail record for the MCP call	one `mcp-filter` `pre_mcp_call` `success` row	one `mcp-filter` `pre_mcp_call` `success` row
Streaming chat with usage	200, `[DONE]`	200, `[DONE]`
Cache repeat (3)	`cache_hit` False, True, True	`cache_hit` False, True, True
Virtual key cycles (5)	200, 200, 200, then 401 on all 20 reuses	200, 200, 200, then 401 on all 20 reuses
Async success callbacks	26 (6 with `cache_hit=True`)	26 (6 with `cache_hit=True`)
Spend log rows by call type, model and status	identical counts	identical counts

The follow-up model turn after the MCP call ended in a text message at the base and in a lookup_weather call at the head; the proxy's request to OpenAI for that turn differs only by previous_response_id, so the difference is the live model's choice

Changed symbols and every caller, from rg over litellm/ and enterprise/: Logging.handle_sync_success_callbacks_for_async_calls (callers in utils.py, caching/caching_handler.py twice, files/streaming.py, rust_bridge/callbacks_legacy_python.py, plus the LoggingSurface protocol), CustomStreamWrapper._dispatch_provider_chunk (one caller in the same class), _record_mcp_guardrail_evaluations (two callers in mcp_server_manager.py), _get_source_cache_base_spend (one caller in proxy_server.py), LiteLLM_Proxy_MCP_Handler._parse_mcp_tools (local other_tools only) and ComplexityRouter._classify_with_llm (one caller, no subclass). No registry, config key, DB column or wire field changes. litellm.utils.executor is thread_pool_executor.executor is litellm_logging.executor prints True at the head

Live A/B rig: config, probe script and raw output per leg

Each leg runs bash leg.sh <base|head> <checkout>: a fresh Postgres database, an OpenAI recorder that forwards to api.openai.com, a local MCP math peer that logs every tools/call, then litellm --config config.yaml --num_workers 2 and python probes.py. Base checkout is 62a1b7f, head is 85baad6

config.yaml (ports templated per leg)

model_list:
  - model_name: judge
    litellm_params: {model: gemini/gemini-2.5-flash-lite, api_key: os.environ/GEMINI_API_KEY}
  - model_name: simple-tier
    litellm_params: {model: gemini/gemini-2.5-flash-lite, api_key: os.environ/GEMINI_API_KEY}
  - model_name: complex-tier
    litellm_params: {model: openai/gpt-4o-mini, api_key: os.environ/OPENAI_API_KEY, api_base: "http://127.0.0.1:__REC__/v1"}
  - model_name: gpt-4o-mini
    litellm_params: {model: openai/gpt-4o-mini, api_key: os.environ/OPENAI_API_KEY, api_base: "http://127.0.0.1:__REC__/v1"}
  - model_name: smart-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        classifier_type: llm
        classifier_llm_config: {model: judge, timeout_ms: 20000}
        tiers: {SIMPLE: simple-tier, MEDIUM: simple-tier, COMPLEX: complex-tier, REASONING: complex-tier}
mcp_servers:
  math:
    transport: http
    url: "http://127.0.0.1:__MCP__/mcp"
guardrails:
  - guardrail_name: mcp-filter
    litellm_params:
      guardrail: litellm_content_filter
      mode: pre_mcp_call
      default_on: true
      blocked_words:
        - {keyword: forbiddenword, action: BLOCK}
litellm_settings:
  drop_params: true
  cache: true
  cache_params: {type: local}
  callbacks: custom_callbacks.handler
  success_callback: [custom_callbacks.sync_fn]
general_settings:
  master_key: os.environ/QA_MASTER_KEY
  database_url: os.environ/QA_DATABASE_URL

custom_callbacks.py

import json
import os
import time

from litellm.integrations.custom_logger import CustomLogger

MARKS = os.environ["LPR_CALLBACK_LOG"]


def _mark(kind, kwargs):
    with open(MARKS, "a") as f:
        f.write(json.dumps({"ts": time.time(), "pid": os.getpid(), "kind": kind,
                            "model": kwargs.get("model"), "call_type": kwargs.get("call_type"),
                            "cache_hit": kwargs.get("cache_hit")}) + "\n")


class Recorder(CustomLogger):
    def log_success_event(self, kwargs, response_obj, start_time, end_time):
        _mark("sync_success", kwargs)

    async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
        _mark("async_success", kwargs)


def sync_fn(kwargs, completion_response, start_time, end_time):
    _mark("sync_fn_success", kwargs)


handler = Recorder()

probes.py

import json
import sys
import time

import httpx

port, key, out = sys.argv[1], sys.argv[2], sys.argv[3]
base = f"http://127.0.0.1:{port}"
H = {"Authorization": f"Bearer {key}"}
c = httpx.Client(timeout=180)
rows = []


def rec(name, r, extra=None):
    row = {"probe": name, "status": r.status_code,
           "api_base": r.headers.get("x-litellm-model-api-base"),
           "model_group": r.headers.get("x-litellm-model-group")}
    if extra:
        row.update(extra)
    rows.append(row)
    print(json.dumps(row), flush=True)


def chat(model, text, k=key, nocache=True):
    return c.post(base + "/v1/chat/completions", headers={"Authorization": f"Bearer {k}"},
                  json={"model": model, "messages": [{"role": "user", "content": text}], "max_tokens": 64, "temperature": 0, "cache": {"no-cache": nocache}})


prompts = [("simple-hi", "hi"), ("simple-math", "What is 2+2?"),
           ("complex", "Design a Byzantine fault tolerant consensus protocol for a geo-distributed database, "
                       "analyzing liveness and safety under partial synchrony with formal proofs")]
for name, text in prompts:
    for _ in range(2):
        r = chat("smart-router", text)
        rec("router-" + name, r, {"model": r.json().get("model") if r.status_code == 200 else r.text[:200]})

tools = [{"type": "mcp", "server_url": "litellm_proxy", "server_label": "litellm", "require_approval": "never"},
         {"type": "function", "name": "lookup_weather", "description": "Look up the forecast for a city",
          "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}]
for _ in range(2):
    r = c.post(base + "/v1/responses", headers=H, json={
        "model": "gpt-4o-mini", "tools": tools,
        "input": "First use the add tool to add 2 and 3. Then call lookup_weather for Paris."})
    body = r.json() if r.status_code == 200 else {}
    items = [i.get("type") for i in body.get("output", [])]
    fcalls = [(i.get("name"), i.get("arguments")) for i in body.get("output", []) if i.get("type") == "function_call"]
    rec("responses-mcp-mixed", r, {"items": items, "function_calls": fcalls, "err": None if body else r.text[:300]})

r = c.post(base + "/v1/chat/completions", headers=H, json={"model": "gpt-4o-mini", "stream": True,
           "stream_options": {"include_usage": True}, "messages": [{"role": "user", "content": "Count: 1 2 3"}], "max_tokens": 24})
rec("stream-openai", r, {"chunks": r.text.count("data: "), "done": "[DONE]" in r.text})

for i in range(3):
    r = chat("gpt-4o-mini", "Name one primary color in one word", nocache=False)
    rec("cache-repeat", r, {"cache_hit": r.headers.get("x-litellm-cache-key") is not None, "call_id": r.headers.get("x-litellm-call-id")})
time.sleep(3)
cycles = []
for _ in range(5):
    g = c.post(base + "/key/generate", headers=H, json={"models": ["simple-tier"]})
    vk = g.json()["key"]
    first = chat("simple-tier", "Reply OK", vk).status_code
    d = c.post(base + "/key/delete", headers=H, json={"keys": [vk]}).status_code
    reuse = [chat("simple-tier", "Reply OK", vk).status_code for _ in range(4)]
    cycles.append({"generate": g.status_code, "first": first, "delete": d, "reuse": reuse})
print(json.dumps({"key_cycles": cycles}), flush=True)
time.sleep(8)
json.dump({"rows": rows, "key_cycles": cycles}, open(out, "w"), indent=1)

Checkout each leg's proxy imported, printed by leg.sh before starting it

leg=base pwd=/home/ubuntu/any_sweep_20261002/basewt_tests sha=62a1b7fdf315fd3654b14ab5a3a57b0ea2709970 litellm=/home/ubuntu/any_sweep_20261002/basewt_tests/litellm/__init__.py
leg=head pwd=/home/ubuntu/repos/litellm sha=85baad6fa671bc5dc26fcde8acbe6695b07a44dd litellm=/home/ubuntu/repos/litellm/litellm/__init__.py

Every request the proxy sent to OpenAI, from the recorder in front of api.openai.com, compared between legs with response and call ids masked

request 1 /v1/chat/completions: base 257 bytes, head 257 bytes, identical after masking ids: True
request 2 /v1/chat/completions: base 257 bytes, head 257 bytes, identical after masking ids: True
request 3 /v1/responses: base 1052 bytes, head 1052 bytes, identical after masking ids: True
request 4 /v1/responses: base 1632 bytes, head 1632 bytes, identical after masking ids: True
request 5 /v1/chat/completions: base 147 bytes, head 147 bytes, identical after masking ids: True
request 6 /v1/chat/completions: base 131 bytes, head 131 bytes, identical after masking ids: True

base request 3 body (/v1/responses, first MCP turn):
{"model":"gpt-4o-mini","input":"First use the add tool to add 2 and 3. Then call lookup_weather for Paris.","stream":false,"tools":[{"name":"math-add","parameters":{"properties":{"a":{"title":"A","type":"integer"},"b":{"title":"B","type":"integer"}},"required":["a","b"],"type":"object","title":"addArguments","additionalProperties":false},"strict":false,"type":"function","description":"Add two integers"},{"name":"math-multiply","parameters":{"properties":{"a":{"title":"A","type":"integer"},"b":{"title":"B","type":"integer"}},"required":["a","b"],"type":"object","title":"multiplyArguments","additionalProperties":false},"strict":false,"type":"function","description":"Multiply two integers"},{"name":"math-fail","parameters":{"properties":{},"type":"object","title":"failArguments","additionalProperties":false},"strict":false,"type":"function","description":""},{"type":"function","name":"lookup_weather","description":"Look up the forecast for a city","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}]}

base request 4 body (/v1/responses, follow-up turn):
{"model":"gpt-4o-mini","input":[{"type":"message","role":"user","content":"First use the add tool to add 2 and 3. Then call lookup_weather for Paris."},{"type":"function_call","call_id":"call_ceciqVEb8D4y8bYsT4sYxQ0B","name":"math-add","arguments":"{\"a\":2,\"b\":3}"},{"type":"function_call","call_id":"call_Hc4WsQ7CYwxyEGZvHFEGKjuy","name":"lookup_weather","arguments":"{\"city\":\"Paris\"}"},{"type":"function_call_output","call_id":"call_ceciqVEb8D4y8bYsT4sYxQ0B","output":"5"},{"type":"function_call_output","call_id":"call_Hc4WsQ7CYwxyEGZvHFEGKjuy","output":"Error executing tool: 'lookup_weather'"}],"previous_response_id":"resp_0412d166bc73bbc2006abf72e3353087d1b7f12086dc3a7755","stream":false,"tools":[{"name":"math-add","parameters":{"properties":{"a":{"title":"A","type":"integer"},"b":{"title":"B","type":"integer"}},"required":["a","b"],"type":"object","title":"addArguments","additionalProperties":false},"strict":false,"type":"function","description":"Add two integers"},{"name":"math-multiply","parameters":{"properties":{"a":{"title":"A","type":"integer"},"b":{"title":"B","type":"integer"}},"required":["a","b"],"type":"object","title":"multiplyArguments","additionalProperties":false},"strict":false,"type":"function","description":"Multiply two integers"},{"name":"math-fail","parameters":{"properties":{},"type":"object","title":"failArguments","additionalProperties":false},"strict":false,"type":"function","description":""},{"type":"function","name":"lookup_weather","description":"Look up the forecast for a city","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}]}

probes.py stdout at 62a1b7f

{"probe": "router-simple-hi", "status": 200, "api_base": "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash-lite:generateContent", "model_group": "smart-router", "model": "smart-router"}
{"probe": "router-simple-hi", "status": 200, "api_base": "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash-lite:generateContent", "model_group": "smart-router", "model": "smart-router"}
{"probe": "router-simple-math", "status": 200, "api_base": "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash-lite:generateContent", "model_group": "smart-router", "model": "smart-router"}
{"probe": "router-simple-math", "status": 200, "api_base": "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash-lite:generateContent", "model_group": "smart-router", "model": "smart-router"}
{"probe": "router-complex", "status": 200, "api_base": "http://127.0.0.1:50707/v1", "model_group": "smart-router", "model": "smart-router"}
{"probe": "router-complex", "status": 200, "api_base": "http://127.0.0.1:50707/v1", "model_group": "smart-router", "model": "smart-router"}
{"probe": "responses-mcp-mixed", "status": 200, "api_base": "http://127.0.0.1:50707/v1", "model_group": "gpt-4o-mini", "items": ["message", "mcp_tools_fetched", "tool_execution_results"], "function_calls": [], "err": null}
{"probe": "responses-mcp-mixed", "status": 200, "api_base": "http://127.0.0.1:50707/v1", "model_group": "gpt-4o-mini", "items": ["message", "mcp_tools_fetched", "tool_execution_results"], "function_calls": [], "err": null}
{"probe": "stream-openai", "status": 200, "api_base": "http://127.0.0.1:50707/v1", "model_group": "gpt-4o-mini", "chunks": 9, "done": true}
{"probe": "cache-repeat", "status": 200, "api_base": "http://127.0.0.1:50707/v1", "model_group": "gpt-4o-mini", "cache_hit": false, "call_id": "80e7a211-bfbd-443b-a407-8ace648abede"}
{"probe": "cache-repeat", "status": 200, "api_base": "http://127.0.0.1:50707/v1", "model_group": "gpt-4o-mini", "cache_hit": true, "call_id": "c7c49235-479e-47fd-8595-d1863b777a7a"}
{"probe": "cache-repeat", "status": 200, "api_base": "http://127.0.0.1:50707/v1", "model_group": "gpt-4o-mini", "cache_hit": true, "call_id": "331fb54a-f554-4635-8d99-047d180b2d96"}
{"key_cycles": [{"generate": 200, "first": 200, "delete": 200, "reuse": [401, 401, 401, 401]}, {"generate": 200, "first": 200, "delete": 200, "reuse": [401, 401, 401, 401]}, {"generate": 200, "first": 200, "delete": 200, "reuse": [401, 401, 401, 401]}, {"generate": 200, "first": 200, "delete": 200, "reuse": [401, 401, 401, 401]}, {"generate": 200, "first": 200, "delete": 200, "reuse": [401, 401, 401, 401]}]}

probes.py stdout at 85baad6

{"probe": "router-simple-hi", "status": 200, "api_base": "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash-lite:generateContent", "model_group": "smart-router", "model": "smart-router"}
{"probe": "router-simple-hi", "status": 200, "api_base": "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash-lite:generateContent", "model_group": "smart-router", "model": "smart-router"}
{"probe": "router-simple-math", "status": 200, "api_base": "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash-lite:generateContent", "model_group": "smart-router", "model": "smart-router"}
{"probe": "router-simple-math", "status": 200, "api_base": "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash-lite:generateContent", "model_group": "smart-router", "model": "smart-router"}
{"probe": "router-complex", "status": 200, "api_base": "http://127.0.0.1:36869/v1", "model_group": "smart-router", "model": "smart-router"}
{"probe": "router-complex", "status": 200, "api_base": "http://127.0.0.1:36869/v1", "model_group": "smart-router", "model": "smart-router"}
{"probe": "responses-mcp-mixed", "status": 200, "api_base": "http://127.0.0.1:36869/v1", "model_group": "gpt-4o-mini", "items": ["function_call", "mcp_tools_fetched", "tool_execution_results"], "function_calls": [["lookup_weather", "{\"city\":\"Paris\"}"]], "err": null}
{"probe": "responses-mcp-mixed", "status": 200, "api_base": "http://127.0.0.1:36869/v1", "model_group": "gpt-4o-mini", "items": ["function_call", "mcp_tools_fetched", "tool_execution_results"], "function_calls": [["lookup_weather", "{\"city\":\"Paris\"}"]], "err": null}
{"probe": "stream-openai", "status": 200, "api_base": "http://127.0.0.1:36869/v1", "model_group": "gpt-4o-mini", "chunks": 9, "done": true}
{"probe": "cache-repeat", "status": 200, "api_base": "http://127.0.0.1:36869/v1", "model_group": "gpt-4o-mini", "cache_hit": false, "call_id": "cf122621-dcb1-4493-838f-f6767d735a3b"}
{"probe": "cache-repeat", "status": 200, "api_base": "http://127.0.0.1:36869/v1", "model_group": "gpt-4o-mini", "cache_hit": true, "call_id": "2f1d3c17-fce5-49d7-8f27-cfef995590db"}
{"probe": "cache-repeat", "status": 200, "api_base": "http://127.0.0.1:36869/v1", "model_group": "gpt-4o-mini", "cache_hit": true, "call_id": "f59a1824-83d2-44f6-967e-d9696f357115"}
{"key_cycles": [{"generate": 200, "first": 200, "delete": 200, "reuse": [401, 401, 401, 401]}, {"generate": 200, "first": 200, "delete": 200, "reuse": [401, 401, 401, 401]}, {"generate": 200, "first": 200, "delete": 200, "reuse": [401, 401, 401, 401]}, {"generate": 200, "first": 200, "delete": 200, "reuse": [401, 401, 401, 401]}, {"generate": 200, "first": 200, "delete": 200, "reuse": [401, 401, 401, 401]}]}

MCP peer log of tools/call requests, 62a1b7f then 85baad6

{"ts": 1790931684.704444, "params": {"name": "add", "arguments": {"a": 2, "b": 3}, "_meta": {"progressToken": 2}}}
{"ts": 1790931761.2384038, "params": {"name": "add", "arguments": {"a": 2, "b": 3}, "_meta": {"progressToken": 2}}}

LiteLLM_SpendLogs guardrail column at 85baad6 (62a1b7f matches apart from ids and timestamps)

[{"duration": 0.000066, "end_time": 1790931761.218547, "risk_score": 0.0, "start_time": 1790931761.218482, "guardrail_id": "e6fcb2ad-19f3-5b7f-b330-7bebdc2f3a53", "match_details": "REDACTED_BY_LITELM", "guardrail_mode": "pre_mcp_call", "guardrail_name": "mcp-filter", "policy_template": null, "detection_method": null, "guardrail_status": "success", "patterns_checked": 1, "guardrail_provider": "litellm_content_filter", "guardrail_response": "REDACTED_BY_LITELM", "masked_entity_count": {}}]

select call_type, model_group, status, has_guardrail from "LiteLLM_SpendLogs" is byte-identical between the legs (diff exits 0)

acompletion|judge|success|f
acompletion|smart-router|success|f
acompletion|judge|success|f
acompletion|smart-router|success|f
acompletion|judge|success|f
acompletion|smart-router|success|f
acompletion|judge|success|f
acompletion|smart-router|success|f
acompletion|judge|success|f
acompletion|smart-router|success|f
acompletion|judge|success|f
acompletion|smart-router|success|f
list_mcp_tools||success|f
aresponses|gpt-4o-mini|success|f
call_mcp_tool||success|t
||failure|f
list_mcp_tools||success|f
aresponses|gpt-4o-mini|success|f
acompletion|gpt-4o-mini|success|f
acompletion|gpt-4o-mini|success|f
acompletion|gpt-4o-mini|success|f
acompletion|gpt-4o-mini|success|f
acompletion|simple-tier|success|f
acompletion|simple-tier|failure|f
acompletion|simple-tier|failure|f
acompletion|simple-tier|failure|f
acompletion|simple-tier|failure|f
acompletion|simple-tier|success|f
acompletion|simple-tier|failure|f
acompletion|simple-tier|failure|f
acompletion|simple-tier|failure|f
acompletion|simple-tier|failure|f
acompletion|simple-tier|success|f
acompletion|simple-tier|failure|f
acompletion|simple-tier|failure|f
acompletion|simple-tier|failure|f
acompletion|simple-tier|failure|f
acompletion|simple-tier|success|f
acompletion|simple-tier|failure|f
acompletion|simple-tier|failure|f
acompletion|simple-tier|failure|f
acompletion|simple-tier|failure|f
acompletion|simple-tier|success|f
acompletion|simple-tier|failure|f
acompletion|simple-tier|failure|f
acompletion|simple-tier|failure|f
acompletion|simple-tier|failure|f

Sync success callback path on async calls: SDK probe with a real Gemini and OpenAI call per leg

The only runtime change is where litellm_logging.py imports executor from. This probe sets a function in litellm.success_callback and a CustomLogger, wraps handle_sync_success_callbacks_for_async_calls and the sync success handler body to record each hit and its thread, and makes two acompletion calls to gemini/gemini-2.5-flash-lite and one streamed acompletion to gpt-4o-mini. On both legs every call is dispatched with should_run=True and the sync success handler runs on the litellm-logging_0 executor thread, so executor.submit ran live. The plain function in success_callback does not fire on either leg because the handler body skips it for async requests, which is unchanged behavior

import asyncio, json, os, sys, time
import litellm
from litellm.caching.caching import Cache
from litellm.integrations.custom_logger import CustomLogger

OUT = sys.argv[1]
marks = []

def sync_fn(kwargs, completion_response, start_time, end_time):
    marks.append({"kind": "sync_fn", "call_type": kwargs.get("call_type"), "cache_hit": kwargs.get("cache_hit"), "stream": kwargs.get("stream")})

class Recorder(CustomLogger):
    def log_success_event(self, kwargs, response_obj, start_time, end_time):
        marks.append({"kind": "sync_logger", "call_type": kwargs.get("call_type"), "cache_hit": kwargs.get("cache_hit"), "stream": kwargs.get("stream")})
    async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
        marks.append({"kind": "async_logger", "call_type": kwargs.get("call_type"), "cache_hit": kwargs.get("cache_hit"), "stream": kwargs.get("stream")})

from litellm.litellm_core_utils.litellm_logging import Logging as _L
_orig_h = _L.handle_sync_success_callbacks_for_async_calls
def _h(self, *a, **k):
    marks.append({"kind": "dispatch", "call_type": self.call_type, "cache_hit": k.get("cache_hit"), "stream": self.stream, "should_run": self._should_run_sync_callbacks_for_async_calls()})
    return _orig_h(self, *a, **k)
_L.handle_sync_success_callbacks_for_async_calls = _h
_orig_b = _L._success_handler_body
def _b(self, *a, **k):
    marks.append({"kind": "success_handler", "thread": __import__("threading").current_thread().name, "call_type": self.call_type, "cache_hit": k.get("cache_hit"), "stream": self.stream, "should_run": self.should_run_logging(event_type="sync_success")})
    return _orig_b(self, *a, **k)
_L._success_handler_body = _b

async def main():
    litellm.success_callback = [sync_fn]
    litellm.callbacks = [Recorder()]
    litellm.cache = Cache(type="local")
    rows = []
    msgs = [{"role": "user", "content": "Reply with exactly: PONG"}]
    for i in range(2):
        r = await litellm.acompletion(model="gemini/gemini-2.5-flash-lite", messages=msgs, max_tokens=16, temperature=0, caching=True)
        rows.append({"call": f"acompletion-{i}", "content": r.choices[0].message.content, "cache_hit": r._hidden_params.get("cache_hit")})
    s = await litellm.acompletion(model="gpt-4o-mini", messages=[{"role": "user", "content": "Count: 1 2 3"}], max_tokens=24, temperature=0, stream=True, caching=False)
    text = "".join([c.choices[0].delta.content or "" async for c in s if c.choices])
    rows.append({"call": "acompletion-stream", "content": text})
    for _ in range(100):
        if sum(m["kind"] == "sync_fn" for m in marks) >= 3:
            break
        await asyncio.sleep(0.1)
    await asyncio.sleep(2)
    json.dump({"litellm": litellm.__file__, "rows": rows, "marks": marks}, open(OUT, "w"), indent=1)

asyncio.run(main())

Checkout each SDK leg imported

base 62a1b7fdf315fd3654b14ab5a3a57b0ea2709970 litellm=~/any_sweep_20261002/basewt_tests/litellm/__init__.py
head 85baad6fa671bc5dc26fcde8acbe6695b07a44dd litellm=~/repos/litellm/litellm/__init__.py

62a1b7f

{
 "litellm": "~/any_sweep_20261002/basewt_tests/litellm/__init__.py",
 "rows": [
  {
   "call": "acompletion-0",
   "content": "PONG",
   "cache_hit": null
  },
  {
   "call": "acompletion-1",
   "content": "PONG",
   "cache_hit": null
  },
  {
   "call": "acompletion-stream",
   "content": "4 5 6"
  }
 ],
 "marks": [
  {
   "kind": "dispatch",
   "call_type": "acompletion",
   "cache_hit": null,
   "stream": false,
   "should_run": true
  },
  {
   "kind": "success_handler",
   "thread": "litellm-logging_0",
   "call_type": "acompletion",
   "cache_hit": null,
   "stream": false,
   "should_run": true
  },
  {
   "kind": "async_logger",
   "call_type": "acompletion",
   "cache_hit": null,
   "stream": false
  },
  {
   "kind": "dispatch",
   "call_type": "acompletion",
   "cache_hit": null,
   "stream": false,
   "should_run": true
  },
  {
   "kind": "async_logger",
   "call_type": "acompletion",
   "cache_hit": null,
   "stream": false
  },
  {
   "kind": "success_handler",
   "thread": "litellm-logging_0",
   "call_type": "acompletion",
   "cache_hit": null,
   "stream": false,
   "should_run": true
  },
  {
   "kind": "async_logger",
   "call_type": "acompletion",
   "cache_hit": false,
   "stream": true
  },
  {
   "kind": "success_handler",
   "thread": "litellm-logging_0",
   "call_type": "acompletion",
   "cache_hit": false,
   "stream": true,
   "should_run": true
  }
 ]
}

85baad6

{
 "litellm": "~/repos/litellm/litellm/__init__.py",
 "rows": [
  {
   "call": "acompletion-0",
   "content": "PONG",
   "cache_hit": null
  },
  {
   "call": "acompletion-1",
   "content": "PONG",
   "cache_hit": null
  },
  {
   "call": "acompletion-stream",
   "content": "1, 2, 3."
  }
 ],
 "marks": [
  {
   "kind": "dispatch",
   "call_type": "acompletion",
   "cache_hit": null,
   "stream": false,
   "should_run": true
  },
  {
   "kind": "success_handler",
   "thread": "litellm-logging_0",
   "call_type": "acompletion",
   "cache_hit": null,
   "stream": false,
   "should_run": true
  },
  {
   "kind": "async_logger",
   "call_type": "acompletion",
   "cache_hit": null,
   "stream": false
  },
  {
   "kind": "dispatch",
   "call_type": "acompletion",
   "cache_hit": null,
   "stream": false,
   "should_run": true
  },
  {
   "kind": "async_logger",
   "call_type": "acompletion",
   "cache_hit": null,
   "stream": false
  },
  {
   "kind": "success_handler",
   "thread": "litellm-logging_0",
   "call_type": "acompletion",
   "cache_hit": null,
   "stream": false,
   "should_run": true
  },
  {
   "kind": "async_logger",
   "call_type": "acompletion",
   "cache_hit": false,
   "stream": true
  },
  {
   "kind": "success_handler",
   "thread": "litellm-logging_0",
   "call_type": "acompletion",
   "cache_hit": false,
   "stream": true,
   "should_run": true
  }
 ]
}

Fireworks raw response

The qa-fireworks step returns 412 on both legs and the harness prints its own KeyError: 'choices' instead of the body. The same request sent straight to Fireworks with the same key returns

HTTP 412
{"error":{"message":"Account <account> is suspended, possibly due to reaching the monthly spending limit or failure to pay past invoices. Please go to https://fireworks.ai/account/billing for more information.","param":null,"code":"PRECONDITION_FAILED","type":"error"},"request_id":"chatcmpl-5486bdcd90374f8db318575f40d9e53b"}

New integration tests run locally at 85baad6

These run in CircleCI's integration-mcp and integration-providers jobs, which are not GitHub checks on this PR, so they were run locally the way .circleci/scripts/run_integration.sh runs them: a fresh Postgres database, Redis, integration._support.upstream, two integration._support.proxy processes on 4000 and 4001, then tests/integration/run.py

$ python tests/integration/run.py mcp tests/integration/mcp/test_responses_mcp_mixed_tools.py
tests/integration/mcp/test_responses_mcp_mixed_tools.py::test_responses_with_gateway_mcp_and_caller_function_tool_hands_both_to_model_and_returns_the_function_call PASSED
1 passed in 1.33s
$ python tests/integration/run.py providers tests/integration/routing/test_complexity_router_llm_classifier.py
tests/integration/routing/test_complexity_router_llm_classifier.py::test_llm_classifier_verdict_routes_the_request_to_the_classified_tier_model PASSED
1 passed in 17.77s

85baad6 passes /live-pr-risk
85baad6 passes /audit (skipped: no product code on a request path, a refactor: annotations plus a same-object import move)

Type

🧹 Refactoring

Caveats (if any)

Low

No before=fail, after=succeed proof exists, because nothing a user sees changes. The legs above prove parity at the merge base and the tip, which is what a typing refactor can show
SplitTools still types the second list as list[Any], so callers of _parse_mcp_tools still see Any there even though other_tools is now list[ToolParam]. Left as is: narrowing the alias needs a new product commit reaching every consumer of that tuple and a full rerun of every live proof for one alias the next sweep can take
The two new integration tests pass at the merge base too, so they guard the typed paths rather than catch this diff, and they run in CircleCI's integration jobs rather than this PR's GitHub checks (local run at 85baad6 pasted above). Left as is: there is no runtime change for a test to catch
Narrowing cache_hit to bool | None on the LoggingSurface protocol is type-only: every caller passes True, a bool, Literal[True] | None or nothing, and success_handler already takes bool | None
The Fireworks leg proves nothing about this change: the provider account is suspended and both legs get the same 412 (raw body above). Left as is: the account's billing is not this run's to fix, and Gemini and OpenAI cover the same chat path
A plain function in success_callback does not fire on async calls on either leg, because the sync handler skips it for async requests. The changed executor.submit path itself ran on both legs (SDK probe above), so this is unchanged behavior
Seams fed by untyped JSON were left alone instead of annotated without proof

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/aa528093d62148a18c529112c3bc472f
Open in Devin Desktop: https://app.devin.ai/desktop/session/aa528093d62148a18c529112c3bc472f?variant=devin

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration · 2026-09-30T09:38:57Z

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

Disable automatic comment, CI, and merge conflict monitoring

CLAassistant · 2026-09-30T09:39:12Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

greptile-apps · 2026-09-30T09:42:55Z

Confidence Score: 5/5

[Low risk] Type annotations replaced with more specific types.

The PR appears safe to merge based on its current nine-file diff.

Summary

The PR narrows type annotations in seven files, imports the same shared logging executor from its defining module, and adds integration coverage for mixed MCP tools and complexity-router classification.

_{Reviews (5) · Last reviewed commit: "Merge remote-tracking branch 'origin/mai..."}

codspeed · 2026-09-30T09:52:37Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_any_sweep_20260930 (85baad6) with main (62a1b7f)¹}

No successful run was found on main (7be2983) during the generation of this report, so 62a1b7f was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

codecov · 2026-09-30T09:54:18Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

mateo-berri · 2026-09-30T10:11:04Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

…ools, jwt routing and complexity router paths Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

mateo-berri · 2026-09-30T10:29:23Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

mateo-berri · 2026-09-30T10:59:21Z

@greptileai

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

mateo-berri · 2026-10-01T08:42:52Z

bugbot run

mateo-berri · 2026-10-01T08:42:53Z

@greptileai

cursor

Stale Bugbot comment from a previous run.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

mateo-berri · 2026-10-01T09:20:10Z

bugbot run

mateo-berri · 2026-10-01T09:20:11Z

@greptileai

cursor

Stale Bugbot comment from a previous run.

…60930

mateo-berri · 2026-10-02T09:01:18Z

@greptileai

mateo-berri · 2026-10-02T09:06:53Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 85baad6. Configure here.}

mateo-berri

LGTM

refactor(types): replace Any with proven types in 9 files

f0f4f40

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration Bot requested a review from a team September 30, 2026 09:38

devin-ai-integration Bot had a problem deploying to e2e-changed September 30, 2026 09:39 Error

cursor Bot reviewed Sep 30, 2026

View reviewed changes

devin-ai-integration Bot added the run-ci label Sep 30, 2026

test(integration): pin xecguard-adjacent guardrail retry, mcp mixed t…

18181bd

…ools, jwt routing and complexity router paths Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration Bot had a problem deploying to e2e-changed September 30, 2026 10:29 Error

cursor Bot reviewed Sep 30, 2026

View reviewed changes

mateo-berri removed the run-ci label Sep 30, 2026

refactor(types): keep only live-provable Any removals

36a2170

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration Bot had a problem deploying to e2e-changed October 1, 2026 08:42 Error

cursor Bot reviewed Oct 1, 2026

View reviewed changes

devin-ai-integration Bot changed the title ~~refactor(types): replace Any with proven types in 9 files~~ refactor(types): replace Any with proven types in 6 files Oct 1, 2026

refactor(types): type cache_hit as bool | None on the sync success path

b123b8a

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration Bot had a problem deploying to e2e-changed October 1, 2026 09:19 Error

cursor Bot reviewed Oct 1, 2026

View reviewed changes

devin-ai-integration Bot changed the title ~~refactor(types): replace Any with proven types in 6 files~~ refactor(types): replace Any with proven types in 7 files Oct 1, 2026

Merge remote-tracking branch 'origin/main' into litellm_any_sweep_202…

85baad6

…60930

devin-ai-integration Bot requested a deployment to e2e-changed October 2, 2026 08:44 Waiting

cursor Bot reviewed Oct 2, 2026

View reviewed changes

mateo-berri approved these changes Oct 2, 2026

View reviewed changes

mateo-berri merged commit fe9b6fd into main Oct 2, 2026
113 of 115 checks passed

mateo-berri deleted the litellm_any_sweep_20260930 branch October 2, 2026 09:47

fangkangmi mentioned this pull request Oct 2, 2026

Optional pre-push hook rejects the documented internal litellm_ branch convention #44169

Closed

Conversation

devin-ai-integration Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Before (62a1b7f)

Type-check count

Chat completions

Tool calling

Responses API

Embeddings

Virtual key lifecycle, model access control and spend

Guardrails and model list

After (85baad6)

Type-check count

Chat completions

Tool calling

Responses API

Embeddings

Virtual key lifecycle, model access control and spend

Guardrails and model list

Live A/B on changed paths (real Gemini and OpenAI, 2 proxy workers, fresh Postgres per leg)

Type

Caveats (if any)

Low

Final Attestation

Uh oh!

devin-ai-integration Bot commented Sep 30, 2026

Uh oh!

CLAassistant commented Sep 30, 2026

Uh oh!

greptile-apps Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

codspeed Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

codecov Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mateo-berri commented Sep 30, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mateo-berri commented Sep 30, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mateo-berri commented Sep 30, 2026

Uh oh!

mateo-berri commented Oct 1, 2026

Uh oh!

mateo-berri commented Oct 1, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mateo-berri commented Oct 1, 2026

Uh oh!

mateo-berri commented Oct 1, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mateo-berri commented Oct 2, 2026

Uh oh!

devin-ai-integration Bot commented Sep 30, 2026 •

edited

Loading

Before (`62a1b7f`)

After (`85baad6`)

greptile-apps Bot commented Sep 30, 2026 •

edited

Loading

codspeed Bot commented Sep 30, 2026 •

edited

Loading

codecov Bot commented Sep 30, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading