refactor(types): replace Any with proven types in 6 files

[devin-ai-integration]

TLDR

Problem this solves:

• Any silently disables type checking wherever it lands
• 10,759 Any errors across the backend tree
• Reviewers cannot tell which payloads are actually untyped

How it solves it:

• Replaces Any with proven types in 6 files
• Removes 83 Any errors, 83 basedpyright errors total
• 1 files reach zero Any
• Behavior is unchanged: types, protocols and generics only

User Flow

This PR only changes type annotations in six litellm/ modules, so the Before and After flows below are identical step for step; the same live proxy calls returned the same statuses and shapes at the merge base and at the tip.

Before: every call behaves the same way it does today, and the Fireworks call fails with the provider's own account precondition error that has nothing to do with this change.

1. POST http://litellm-domain/v1/chat/completions with model: qa-gemini and a one-word prompt: HTTP 200, content='PONG', finish_reason=stop, usage with prompt and completion tokens above zero.
2. POST http://litellm-domain/v1/chat/completions with model: qa-openai, stream: true and stream_options.include_usage: HTTP 200, a handful of SSE chunks that assemble into a short count (the model varies the wording: 8 chunks spelling 4 5 6 on the Before run, 11 spelling 1, 2, 3. on the After run), then a final usage chunk with prompt tokens above zero.
3. POST http://litellm-domain/v1/embeddings with model: qa-embed: HTTP 200, one 3072-dimension vector, prompt_tokens=5.
4. POST http://litellm-domain/key/generate as the master key, with access to qa-gemini only: HTTP 200, a key shaped like sk-....
5. POST http://litellm-domain/v1/chat/completions with that virtual key and model: qa-gemini: HTTP 200, content='OK'.
6. POST http://litellm-domain/v1/chat/completions with that virtual key and a model it may not use: HTTP 403.
7. GET http://litellm-domain/key/info?key=sk-... as the master key: the key's spend is greater than zero.
8. POST http://litellm-domain/key/delete as the master key: HTTP 200, the deleted key echoed back; reusing the deleted key on /v1/chat/completions right away still returned HTTP 200 in this run on both legs, a two-worker key-cache timing effect noted under Caveats.
9. GET http://litellm-domain/guardrails/list as the master key: HTTP 200, guardrails: [].
10. GET http://litellm-domain/v1/models as the master key: HTTP 200, qa-embed, qa-fireworks, qa-gemini, qa-openai.
11. POST http://litellm-domain/v1/responses with model: qa-gemini, then again with stream: true: HTTP 200 both times, status=completed, text PONG, and for the stream 9 events from response.created to response.completed.
12. POST http://litellm-domain/v1/chat/completions with model: qa-fireworks: HTTP 412 whose JSON body is an error object (no choices) carrying Fireworks' own PRECONDITION_FAILED message that the account is suspended; the gateway relays the provider error unchanged.
13. POST http://litellm-domain/v1/chat/completions with model: qa-gemini and a get_weather JSON-schema tool: HTTP 200, one tool call get_weather with {"city": "Paris"}, finish_reason=tool_calls.

After: every call behaves exactly as before, including the Fireworks account error, because nothing at runtime changed.

1. POST http://litellm-domain/v1/chat/completions with model: qa-gemini and a one-word prompt: HTTP 200, content='PONG', finish_reason=stop, usage with prompt and completion tokens above zero.
2. POST http://litellm-domain/v1/chat/completions with model: qa-openai, stream: true and stream_options.include_usage: HTTP 200, a handful of SSE chunks that assemble into a short count (the model varies the wording: 8 chunks spelling 4 5 6 on the Before run, 11 spelling 1, 2, 3. on the After run), then a final usage chunk with prompt tokens above zero.
3. POST http://litellm-domain/v1/embeddings with model: qa-embed: HTTP 200, one 3072-dimension vector, prompt_tokens=5.
4. POST http://litellm-domain/key/generate as the master key, with access to qa-gemini only: HTTP 200, a key shaped like sk-....
5. POST http://litellm-domain/v1/chat/completions with that virtual key and model: qa-gemini: HTTP 200, content='OK'.
6. POST http://litellm-domain/v1/chat/completions with that virtual key and a model it may not use: HTTP 403.
7. GET http://litellm-domain/key/info?key=sk-... as the master key: the key's spend is greater than zero.
8. POST http://litellm-domain/key/delete as the master key: HTTP 200, the deleted key echoed back; reusing the deleted key on /v1/chat/completions right away still returned HTTP 200 in this run on both legs, a two-worker key-cache timing effect noted under Caveats.
9. GET http://litellm-domain/guardrails/list as the master key: HTTP 200, guardrails: [].
10. GET http://litellm-domain/v1/models as the master key: HTTP 200, qa-embed, qa-fireworks, qa-gemini, qa-openai.
11. POST http://litellm-domain/v1/responses with model: qa-gemini, then again with stream: true: HTTP 200 both times, status=completed, text PONG, and for the stream 9 events from response.created to response.completed.
12. POST http://litellm-domain/v1/chat/completions with model: qa-fireworks: HTTP 412 whose JSON body is an error object (no choices) carrying Fireworks' own PRECONDITION_FAILED message that the account is suspended; the gateway relays the provider error unchanged.
13. POST http://litellm-domain/v1/chat/completions with model: qa-gemini and a get_weather JSON-schema tool: HTTP 200, one tool call get_weather with {"city": "Paris"}, finish_reason=tool_calls.

Relevant issues

Affected release

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup for both runs. Same config, same payloads, same live providers, same Postgres; only the checked-out commit differs

model_list:
  - model_name: qa-gemini
    litellm_params:
      model: gemini/gemini-2.5-flash-lite
      api_key: os.environ/GEMINI_API_KEY
  - model_name: qa-openai
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
  - model_name: qa-fireworks
    litellm_params:
      model: fireworks_ai/accounts/fireworks/models/gpt-oss-120b
      api_key: os.environ/FIREWORKS_AI_API_KEY
  - model_name: qa-embed
    litellm_params:
      model: gemini/gemini-embedding-001
      api_key: os.environ/GEMINI_API_KEY

general_settings:
  master_key: os.environ/QA_MASTER_KEY
  database_url: os.environ/QA_DATABASE_URL

litellm_settings:
  drop_params: true

git checkout <hash>
litellm --config config.yaml --port $PORT --num_workers 2

Before (5fc510a)

Type-check count

1. Run basedpyright over the tree the way scripts/type_check_gate.py does:

   .venv-typecheck/bin/basedpyright --outputjson --pythonpath .venv-typecheck/bin/python
   

2. Observed:

   reportAny             8383
   reportExplicitAny     2376
   Any total            10759
   all errors          138656
   

Chat completions

1. Send a non-streaming chat completion to qa-gemini:

   curl -X POST http://127.0.0.1:$PORT/v1/chat/completions \
     -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
     -d '{"model":"qa-gemini","messages":[{"role":"user","content":"Reply with exactly: PONG"}],"max_tokens":16,"temperature":0}'

2. Observed:

   HTTP 200
   content='PONG'
   finish_reason=stop model=qa-gemini
   usage keys=['completion_tokens', 'completion_tokens_details', 'prompt_tokens'] prompt>0=True completion>0=True
   id_prefix=r1-uarOCG
   

3. Send a streaming chat completion with usage to qa-openai:

   curl -N -X POST http://127.0.0.1:$PORT/v1/chat/completions \
     -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
     -d '{"model":"qa-openai","messages":[{"role":"user","content":"Count: 1 2 3"}],"max_tokens":24,"temperature":0,"stream":true,"stream_options":{"include_usage":true}}'

4. Observed:

   HTTP 200
   n_chunks=8
   assembled='4 5 6'
   final_usage_present=True prompt>0=True
   

5. Send a non-streaming chat completion to qa-fireworks:

   curl -X POST http://127.0.0.1:$PORT/v1/chat/completions \
     -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
     -d '{"model":"qa-fireworks","messages":[{"role":"user","content":"Reply with exactly: PONG"}],"max_tokens":512,"temperature":0}'

6. Observed:

   HTTP 412
   Traceback (most recent call last):
     File "<string>", line 4, in <module>
   KeyError: 'choices'
   

Tool calling

1. Send a chat completion to qa-gemini carrying a get_weather function tool:

   curl -X POST http://127.0.0.1:$PORT/v1/chat/completions \
     -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
     -d '{"model":"qa-gemini","messages":[{"role":"user","content":"What is the weather in Paris? Use the tool."}],"tools":[{"type":"function","function":{"name":"get_weather","description":"Get weather for a city","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],"tool_choice":"auto","max_tokens":64,"temperature":0}'

2. Observed:

   HTTP 200
   n_tool_calls=1
   name=get_weather arguments={"city": "Paris"}
   finish_reason=tool_calls
   

Responses API

1. Send a non-streaming responses request to qa-gemini:

   curl -X POST http://127.0.0.1:$PORT/v1/responses \
     -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
     -d '{"model":"qa-gemini","input":"Reply with exactly: PONG","max_output_tokens":16,"temperature":0}'

2. Observed:

   HTTP 200
   object=response status=completed model=qa-gemini
   text='PONG'
   usage input>0=True output>0=True
   

3. Send a non-streaming responses request to qa-openai:

   curl -X POST http://127.0.0.1:$PORT/v1/responses \
     -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
     -d '{"model":"qa-openai","input":"Reply with exactly: PONG","max_output_tokens":16,"temperature":0}'

4. Observed:

   HTTP 200
   object=response status=completed model=qa-openai
   text='PONG'
   usage input>0=True output>0=True
   

5. Send a streaming responses request to qa-gemini:

   curl -N -X POST http://127.0.0.1:$PORT/v1/responses \
     -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
     -d '{"model":"qa-gemini","input":"Reply with exactly: PONG","max_output_tokens":24,"temperature":0,"stream":true}'

6. Observed:

   HTTP 200
   n_events=9
   first_type=response.created last_type=response.completed
   assembled='PONG'
   completed_event_present=True
   

Embeddings

1. Send an embedding request to qa-embed:

   curl -X POST http://127.0.0.1:$PORT/v1/embeddings \
     -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
     -d '{"model":"qa-embed","input":"litellm any reduction"}'

2. Observed:

   HTTP 200
   object=list n_vectors=1 dim=3072
   first3=[-0.024564, 0.00778, 0.001819]
   prompt_tokens=5
   

Virtual key lifecycle, model access control and spend

1. Generate a key scoped to qa-gemini, use it on both models, read its spend, delete it and reuse it:

   curl -X POST http://127.0.0.1:$PORT/key/generate -H "Authorization: Bearer $QA_MASTER_KEY" \
     -H 'Content-Type: application/json' -d '{"models":["qa-gemini"],"max_budget":1.0,"key_alias":"qa-<label>-<n>"}'
   # then /v1/chat/completions with that key on qa-gemini and on qa-openai,
   # then GET /key/info?key=<key>, POST /key/delete, and one more chat call

2. Observed:

   generated key prefix: sk-W_OU...
   chat-with-virtual-key HTTP 200
   content='OK'
   denied-model HTTP 403
   error_mentions_key_not_allowed=False
   key spend recorded > 0: True
   key-delete HTTP 200
   reuse-deleted-key HTTP 200
   

Guardrails and model list

1. List the guardrails and the models:

   curl http://127.0.0.1:$PORT/guardrails/list -H "Authorization: Bearer $QA_MASTER_KEY"
   curl http://127.0.0.1:$PORT/v1/models -H "Authorization: Bearer $QA_MASTER_KEY"

2. Observed:

   HTTP 200
   guardrails_key=True count=0
   HTTP 200
   models=['qa-embed', 'qa-fireworks', 'qa-gemini', 'qa-openai']
   

After (8ee7591)

Type-check count

1. Run basedpyright over the tree the way scripts/type_check_gate.py does:

   .venv-typecheck/bin/basedpyright --outputjson --pythonpath .venv-typecheck/bin/python
   

2. Observed:

   reportAny             8310
   reportExplicitAny     2366
   Any total            10676
   all errors          138573
   

Chat completions

1. Send a non-streaming chat completion to qa-gemini:

   curl -X POST http://127.0.0.1:$PORT/v1/chat/completions \
     -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
     -d '{"model":"qa-gemini","messages":[{"role":"user","content":"Reply with exactly: PONG"}],"max_tokens":16,"temperature":0}'

2. Observed:

   HTTP 200
   content='PONG'
   finish_reason=stop model=qa-gemini
   usage keys=['completion_tokens', 'completion_tokens_details', 'prompt_tokens'] prompt>0=True completion>0=True
   id_prefix=7l-uate8F
   

3. Send a streaming chat completion with usage to qa-openai:

   curl -N -X POST http://127.0.0.1:$PORT/v1/chat/completions \
     -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
     -d '{"model":"qa-openai","messages":[{"role":"user","content":"Count: 1 2 3"}],"max_tokens":24,"temperature":0,"stream":true,"stream_options":{"include_usage":true}}'

4. Observed:

   HTTP 200
   n_chunks=11
   assembled='1, 2, 3.'
   final_usage_present=True prompt>0=True
   

5. Send a non-streaming chat completion to qa-fireworks:

   curl -X POST http://127.0.0.1:$PORT/v1/chat/completions \
     -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
     -d '{"model":"qa-fireworks","messages":[{"role":"user","content":"Reply with exactly: PONG"}],"max_tokens":512,"temperature":0}'

6. Observed:

   HTTP 412
   Traceback (most recent call last):
     File "<string>", line 4, in <module>
   KeyError: 'choices'
   

Tool calling

1. Send a chat completion to qa-gemini carrying a get_weather function tool:

   curl -X POST http://127.0.0.1:$PORT/v1/chat/completions \
     -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
     -d '{"model":"qa-gemini","messages":[{"role":"user","content":"What is the weather in Paris? Use the tool."}],"tools":[{"type":"function","function":{"name":"get_weather","description":"Get weather for a city","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],"tool_choice":"auto","max_tokens":64,"temperature":0}'

2. Observed:

   HTTP 200
   n_tool_calls=1
   name=get_weather arguments={"city": "Paris"}
   finish_reason=tool_calls
   

Responses API

1. Send a non-streaming responses request to qa-gemini:

   curl -X POST http://127.0.0.1:$PORT/v1/responses \
     -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
     -d '{"model":"qa-gemini","input":"Reply with exactly: PONG","max_output_tokens":16,"temperature":0}'

2. Observed:

   HTTP 200
   object=response status=completed model=qa-gemini
   text='PONG'
   usage input>0=True output>0=True
   

3. Send a non-streaming responses request to qa-openai:

   curl -X POST http://127.0.0.1:$PORT/v1/responses \
     -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
     -d '{"model":"qa-openai","input":"Reply with exactly: PONG","max_output_tokens":16,"temperature":0}'

4. Observed:

   HTTP 200
   object=response status=completed model=qa-openai
   text='PONG'
   usage input>0=True output>0=True
   

5. Send a streaming responses request to qa-gemini:

   curl -N -X POST http://127.0.0.1:$PORT/v1/responses \
     -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
     -d '{"model":"qa-gemini","input":"Reply with exactly: PONG","max_output_tokens":24,"temperature":0,"stream":true}'

6. Observed:

   HTTP 200
   n_events=9
   first_type=response.created last_type=response.completed
   assembled='PONG'
   completed_event_present=True
   

Embeddings

1. Send an embedding request to qa-embed:

   curl -X POST http://127.0.0.1:$PORT/v1/embeddings \
     -H "Authorization: Bearer $QA_MASTER_KEY" -H 'Content-Type: application/json' \
     -d '{"model":"qa-embed","input":"litellm any reduction"}'

2. Observed:

   HTTP 200
   object=list n_vectors=1 dim=3072
   first3=[-0.024564, 0.00778, 0.001819]
   prompt_tokens=5
   

Virtual key lifecycle, model access control and spend

1. Generate a key scoped to qa-gemini, use it on both models, read its spend, delete it and reuse it:

   curl -X POST http://127.0.0.1:$PORT/key/generate -H "Authorization: Bearer $QA_MASTER_KEY" \
     -H 'Content-Type: application/json' -d '{"models":["qa-gemini"],"max_budget":1.0,"key_alias":"qa-<label>-<n>"}'
   # then /v1/chat/completions with that key on qa-gemini and on qa-openai,
   # then GET /key/info?key=<key>, POST /key/delete, and one more chat call

2. Observed:

   generated key prefix: sk-kkpj...
   chat-with-virtual-key HTTP 200
   content='OK'
   denied-model HTTP 403
   error_mentions_key_not_allowed=False
   key spend recorded > 0: True
   key-delete HTTP 200
   reuse-deleted-key HTTP 200
   

Guardrails and model list

1. List the guardrails and the models:

   curl http://127.0.0.1:$PORT/guardrails/list -H "Authorization: Bearer $QA_MASTER_KEY"
   curl http://127.0.0.1:$PORT/v1/models -H "Authorization: Bearer $QA_MASTER_KEY"

2. Observed:

   HTTP 200
   guardrails_key=True count=0
   HTTP 200
   models=['qa-embed', 'qa-fireworks', 'qa-gemini', 'qa-openai']
   

Type

🧹 Refactoring

Caveats (if any)

Low

• No new tests; nothing observable changed to test
• Seams fed by untyped JSON were left alone instead of annotated without proof
• The Fireworks chat case returned HTTP 412 on both legs because the Fireworks account is suspended upstream (PRECONDITION_FAILED in the saved error body, so the harness's summary printer shows KeyError: 'choices'); the status and error message matched Before and After, so it is provider state and that model adds no signal to the proof.
• The reuse-deleted-key case depends on a two-worker key-cache race, not on this change: across three full runs it read 200/401 (mismatch on the unchanged base, rerun), then 401/401, then 200/200 at the current tip, and the two legs of the run pasted here match line for line.
• The live harness exercises the proxy surfaces (chat, streaming, embeddings, virtual keys, guardrails and model lists, Responses API, tool calling); the six edited modules are covered by the 14 mapped test paths (767 tests passing at the tip) and the type-check gate, and a separate live check that booted 5fc510a6fd and 8ee7591fc8 with Alice, GraySwan, PromptGuard and Singulr configured plus the otel callback returned identical statuses and bodies per guarded request (400/400/500/400 against unreachable vendor endpoints) and the same 22 span names on both. Only the Bedrock AgentCore LlmProviders.BEDROCK line is not exercised live and is the one patch line Codecov reports uncovered; the httpx client cache key it builds is byte-identical to the old string ('async_httpx_client' + LlmProviders.BEDROCK == 'async_httpx_clientbedrock').
• The streamed OpenAI step reads differently on the two legs (8 chunks 4 5 6 versus 11 chunks 1, 2, 3.) because the model's wording is nondeterministic; status, chunk shape and the final usage chunk match.
• Three of the four new _CustomGuardrailOptions / _CustomLoggerOptions TypedDicts (OTEL, Alice, GraySwan) declare no fields, only total=False, extra_items=object, so Unpack[...] still accepts any keyword for those constructors; they name the pass-through contract without adding new checking, and only PromptGuard's names a field.
• ci/circleci: litellm_utils_testing is red at the tip and is not a required check; its glob (tests/litellm_utils_tests/**/test_*.py) fails the same 12 credential- and network-dependent tests (NoCredentialsError, api.groq.com unreachable) at the merge base 5fc510a6fd and at the tip, and the same job is red on unrelated open PRs fix(proxy): keep request metadata out of the cost tracking failure alert #41950 and fix(proxy): keep the raw client model out of spend logs for rejections outside the router #41943.

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/cad698a4dd754705a42fd5cf101db571
Open in Devin Desktop: https://app.devin.ai/desktop/session/cad698a4dd754705a42fd5cf101db571?variant=devin

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_any_sweep_20260919 (8ee7591) with main (5fc510a)}

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding correctness, security, or repository-rule issues identified.

Summary

This PR narrows type annotations across six telemetry, Bedrock AgentCore, and guardrail modules without changing their intended runtime behavior.

• Replaces broad Any annotations with TypedDict, Unpack, concrete mappings, message types, and provider enums.
• Preserves caller-supplied PromptGuard event hooks while supplying the existing default when omitted.
• Removes the nonessential TypedDict docstrings identified in the previous review.

_{Reviews (2) · Last reviewed commit: "refactor(types): drop nonessential Typed..."}

[codecov]

Codecov Report

❌ Patch coverage is 96.00000% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...ellm/llms/bedrock/chat/agentcore/transformation.py	50.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 8ee7591. Configure here.}

[mateo-berri]

LGTM