refactor: expose public hidden_params accessors by devin-ai-integration[bot] · Pull Request #44668 · BerriAI/litellm

devin-ai-integration · 2026-10-05T21:56:14Z

TLDR

Problem this solves:

239 of 1,811 reportPrivateUsage hits come from reading response._hidden_params
That blocks driving the private-usage budget to zero

How it solves it:

Add a public hidden_params property on every response type that stores _hidden_params
Move the 242 flagged in-tree reads and writes to hidden_params
_hidden_params stays as storage, so existing user code keeps working
DecisionsResponse is frozen, so it gets a read-only property
Responses whose storage is a HiddenParams model are read and written through HiddenParamsModelView, which only exposes set fields and extras, and transfers between responses move the raw storage object like main did
Duck-typed objects that only carry _hidden_params go through small getattr/setattr helpers in litellm_core_utils/hidden_params.py
First of four PRs taking reportPrivateUsage to zero, the budget ratchet lowers the limit after merge

User Flow

Before: an SDK or proxy user reads response metadata through the underscore attribute

They call litellm.completion(...) or POST http://localhost:4000/v1/chat/completions
They read response._hidden_params["response_cost"], or the proxy returns x-litellm-response-cost and x-litellm-model-id headers

After: the same calls return the same data, and a public name is also available

They make the same call
response._hidden_params and the new response.hidden_params return the same dict, and the proxy headers are unchanged

Pre-Submission checklist

I have added meaningful tests
The handful of test files covering my change pass locally
My PR passes all required CI/CD checks
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Video of the live run is in the Slack thread https://berriaillm.slack.com/archives/C0B302ZJU05/p1791232426675779

Shared setup is two proxies on random free ports, one per worktree, with real OpenAI and Anthropic keys, no DB, and this config

model_list:
  - model_name: qa-chat
    litellm_params: {model: openai/gpt-4o-mini, api_key: os.environ/OPENAI_API_KEY}
  - model_name: qa-responses
    litellm_params: {model: openai/gpt-4o-mini, api_key: os.environ/OPENAI_API_KEY}
  - model_name: qa-anthropic
    litellm_params: {model: anthropic/claude-haiku-4-5-20251001, api_key: os.environ/ANTHROPIC_API_KEY}
  - model_name: qa-embeddings
    litellm_params: {model: openai/text-embedding-3-small, api_key: os.environ/OPENAI_API_KEY}
  - model_name: qa-fallback
    litellm_params: {model: openai/gpt-4o-mini, api_key: deliberately-invalid-qa-key}
router_settings:
  num_retries: 0
  fallbacks: [{qa-fallback: [qa-chat]}]
litellm_settings:
  cache: true
  cache_params: {type: local, ttl: 600}

Before (`2156d6c`)

POST /v1/chat/completions qa-chat returns 200 with x-litellm-response-cost: 3e-06, x-litellm-model-id: qa-chat, x-litellm-model-api-base, and duration headers
Same request with "stream": true streams to [DONE], final usage chunk shows 14 tokens and cost 3e-06
POST /v1/responses qa-responses returns 200 with cost 5.1e-06 and the same metadata headers
POST /v1/messages qa-anthropic returns 200 with cost 5.5e-05
POST /v1/embeddings qa-embeddings returns 200 with cost 8e-08
POST /v1/chat/completions qa-fallback returns 200 from qa-chat with x-litellm-attempted-fallbacks: 1
The same chat request sent twice returns the same response id with x-litellm-cache-key on the hit
SDK litellm.completion(...) gives response._hidden_params["response_cost"] == 3e-06, and response.hidden_params raises AttributeError
SDK call through a registered CustomLLM in litellm.custom_provider_map that calls OpenAI and returns a plain user BaseModel with only _hidden_params keeps response_cost 0.0042 and both additional_headers

After (`9061b14`)

Same, identical x-litellm-* values apart from call ids and durations
Same, 14 tokens and cost 3e-06
Same, cost 5.1e-06
Same, cost 5.5e-05
Same, cost 8e-08
Same, x-litellm-attempted-fallbacks: 1
Same cache key and reused response id
Same private cost, and response.hidden_params is response._hidden_params is True for both reading and assigning
Same, cost 0.0042 and both headers, the user model still has no public accessor

Type

🧹 Refactoring

Caveats (if any)

Low

A real mid-stream fallback was not reproduced live, covered by unit tests only
DB-backed spend logging was not exercised in the live run
Native /v1/completions metadata was not checked live because OpenAI now returns 404 for gpt-3.5-turbo-instruct, davinci-002 and babbage-002 on both commits, unit tests cover that path
ModelResponseBase keeps its shared class-level _hidden_params = {} default, unchanged here

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/32b275f5ec964f1f82cf6192e19a6237
Open in Devin Desktop: https://app.devin.ai/desktop/session/32b275f5ec964f1f82cf6192e19a6237?variant=devin
Requested by: @mateo-berri

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration · 2026-10-05T21:56:17Z

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

Disable automatic comment, CI, and merge conflict monitoring

greptile-apps · 2026-10-05T22:00:21Z

Confidence Score: 5/5

[Medium risk] Refactors internal hidden params access to use public accessors.

The PR appears safe to merge based on the changes since the previous review.

Summary

The PR exposes public hidden_params accessors while retaining _hidden_params as backing storage and migrates in-tree metadata access across response types, routing, caching, and proxy paths.

The changes since the previous review make the async and sync streaming wrappers copy metadata into plain dicts.
Both previous Greptile threads are resolved.

_{Reviews (13) · Last reviewed commit: "fix(router): copy streaming hidden param..." · Reviewed by Greptile}

codspeed · 2026-10-05T22:05:48Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_private_usage_hidden_params (913c0a9) with main (1cdc0ec)¹}

No successful run was found on main (bce020f) during the generation of this report, so 1cdc0ec was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

codecov · 2026-10-05T22:16:45Z

Codecov Report

❌ Patch coverage is 87.72920% with 87 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/router.py	65.21%	8 Missing ⚠️
litellm/llms/openai/responses/transformation.py	25.00%	6 Missing ⚠️
litellm/litellm_core_utils/hidden_params.py	93.58%	5 Missing ⚠️
litellm/types/utils.py	89.58%	5 Missing ⚠️
litellm/llms/manus/responses/transformation.py	0.00%	4 Missing ⚠️
...tellm/llms/stability/image_edit/transformations.py	0.00%	4 Missing ⚠️
litellm/responses/mcp/chat_completions_handler.py	20.00%	4 Missing ⚠️
...litellm_responses_transformation/transformation.py	62.50%	3 Missing ⚠️
litellm/llms/azure_ai/agents/handler.py	25.00%	3 Missing ⚠️
litellm/llms/azure_ai/rerank/transformation.py	0.00%	3 Missing ⚠️
... and 26 more

📢 Thoughts on this report? Let us know!

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

mateo-berri · 2026-10-05T23:17:28Z

@greptileai

mateo-berri · 2026-10-05T23:28:18Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

mateo-berri · 2026-10-06T01:19:25Z

@greptileai

mateo-berri · 2026-10-06T01:19:26Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

…_hidden_params

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

This reverts commit 621e5ff.

mateo-berri · 2026-10-06T02:42:36Z

@greptileai

mateo-berri · 2026-10-06T02:42:37Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

mateo-berri · 2026-10-06T03:44:37Z

@greptileai

mateo-berri · 2026-10-06T03:44:38Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

mateo-berri · 2026-10-07T03:23:09Z

@greptileai review

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

mateo-berri · 2026-10-07T04:18:24Z

@greptileai review

mateo-berri · 2026-10-07T04:27:44Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

mateo-berri · 2026-10-07T06:06:29Z

@greptileai review

…_hidden_params Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> # Conflicts: # tests/unit/proxy/hooks/test_dynamic_rate_limiter.py

mateo-berri · 2026-10-07T06:17:47Z

@greptileai review

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

mateo-berri · 2026-10-07T06:28:03Z

@greptileai review

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

mateo-berri · 2026-10-07T06:39:59Z

@greptileai review

mateo-berri · 2026-10-07T06:49:16Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 913c0a9. Configure here.}

mateo-berri

LGTM

…table/1.104.x The public hidden_params accessor comes from #44668, which is not on stable/1.104.x, so the backported test reads the private attribute like the rest of the file

refactor: expose public hidden_params accessors

1168d3d

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration Bot requested a review from a team October 5, 2026 21:56

devin-ai-integration Bot assigned mateo-berri Oct 5, 2026

greptile-apps Bot reviewed Oct 5, 2026

View reviewed changes

Comment thread litellm/router.py Outdated

mateo-berri and others added 2 commits October 5, 2026 23:00

fix: keep hidden params writes on duck-typed responses

03752c1

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

refactor: move hidden params helpers to core utils

b46587c

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

cursor Bot reviewed Oct 5, 2026

View reviewed changes

mateo-berri and others added 3 commits October 6, 2026 00:08

fix: preserve hidden params in duck-typed cost responses

2d4cb09

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

test: cover hidden params accessors across response types

1a990af

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

fix: preserve hidden params for dynamic responses

aac16d4

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

github-advanced-security Bot found potential problems Oct 6, 2026

View reviewed changes

Comment thread enterprise/litellm_enterprise/proxy/hooks/managed_files.py Fixed

cursor Bot reviewed Oct 6, 2026

View reviewed changes

mateo-berri and others added 3 commits October 6, 2026 02:17

Merge remote-tracking branch 'origin/main' into litellm_private_usage…

2c2711e

…_hidden_params

fix: expose logs through backend component

621e5ff

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Revert "fix: expose logs through backend component"

9598003

This reverts commit 621e5ff.

cursor Bot reviewed Oct 6, 2026

View reviewed changes

Comment thread litellm/litellm_core_utils/hidden_params.py Outdated

mateo-berri and others added 2 commits October 6, 2026 03:29

test: cover dict hidden params helper path

0533184

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

fix: preserve hidden params storage on item writes

9061b14

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

cursor Bot reviewed Oct 7, 2026

View reviewed changes

Comment thread litellm/main.py

fix(litellm): read HiddenParams model storage through get_hidden_params

d054933

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

greptile-apps Bot reviewed Oct 7, 2026

View reviewed changes

Comment thread litellm/responses/litellm_completion_transformation/transformation.py Outdated

fix(litellm): transfer hidden params storage instead of the mapping view

2fca5dd

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

cursor Bot reviewed Oct 7, 2026

View reviewed changes

Comment thread litellm/litellm_core_utils/hidden_params.py Outdated

mateo-berri and others added 2 commits October 7, 2026 04:48

fix(litellm): expose only set HiddenParams keys through the mapping view

7906331

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

perf(litellm): read HiddenParams view keys without dumping the model

e775d5f

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into litellm_private_usage…

34609e3

…_hidden_params Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> # Conflicts: # tests/unit/proxy/hooks/test_dynamic_rate_limiter.py

test(litellm): assert HiddenParams extras through model_extra

c4c050f

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

fix(router): copy streaming hidden params into a plain dict

913c0a9

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

cursor Bot reviewed Oct 7, 2026

View reviewed changes

mateo-berri approved these changes Oct 7, 2026

View reviewed changes

mateo-berri merged commit d364d5e into main Oct 7, 2026
114 of 115 checks passed

mateo-berri deleted the litellm_private_usage_hidden_params branch October 7, 2026 07:01

devin-ai-integration Bot mentioned this pull request Oct 7, 2026

refactor(llms): expose public names for private provider helpers #45037

Merged

3 of 5 tasks

This was referenced Oct 8, 2026

chore: bump litellm-proxy-extras to 0.4.107 and litellm-enterprise to 0.1.75 #45291

Merged

fix(proxy): classify /v1/messages pass-through streams as Anthropic on any host #45602

Merged

yuneng-berri mentioned this pull request Oct 9, 2026

test: fix stale and flaky tests across CircleCI, GHA and Buildkite #45530

Merged

3 of 26 tasks

Conversation

devin-ai-integration Bot commented Oct 5, 2026 • edited by mateo-berri Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Pre-Submission checklist

Screenshots / Proof of Fix

Before (2156d6c)

After (9061b14)

Type

Caveats (if any)

Low

Final Attestation

Uh oh!

devin-ai-integration Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

codspeed Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

codecov Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mateo-berri commented Oct 5, 2026

Uh oh!

mateo-berri commented Oct 5, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mateo-berri commented Oct 6, 2026

Uh oh!

mateo-berri commented Oct 6, 2026

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mateo-berri commented Oct 6, 2026

Uh oh!

mateo-berri commented Oct 6, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mateo-berri commented Oct 6, 2026

Uh oh!

mateo-berri commented Oct 6, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mateo-berri commented Oct 7, 2026

Uh oh!

Uh oh!

mateo-berri commented Oct 7, 2026

Uh oh!

mateo-berri commented Oct 7, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mateo-berri commented Oct 7, 2026

devin-ai-integration Bot commented Oct 5, 2026 •

edited by mateo-berri

Loading

Before (`2156d6c`)

After (`9061b14`)

devin-ai-integration Bot commented Oct 5, 2026 •

edited

Loading

greptile-apps Bot commented Oct 5, 2026 •

edited

Loading

codspeed Bot commented Oct 5, 2026 •

edited

Loading

codecov Bot commented Oct 5, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading