fix(responses): decode JSON-string tool schemas before sending to the provider by tin-berri · Pull Request #39844 · BerriAI/litellm

tin-berri · 2026-09-05T01:31:36Z

TLDR

Problem this solves:

A tool schema sent as a JSON string reached the provider verbatim
Provider 400 named the routed model, not the offending tool
Surfaced through the 1M Context auto-router preset on an Azure GPT tier

How it solves it:

Decode a string parameters that parses to an object before the wire
Reject anything else with a litellm 400 naming tools[i].parameters
One owner for the request and compact-request tool sanitization

User Flow

Before: a developer whose coding agent defines a tool with a JSON-encoded schema gets an opaque provider error blaming the model tier

They send POST https://litellm-domain/v1/messages to their 1M Context auto-router with tools[0].input_schema set to the string "{\"type\":\"object\",...}"
The request classifies SIMPLE and lands on the gpt-5.6-luna tier
They get back HTTP 400: AzureException BadRequestError - Invalid type for 'tools[0].parameters': expected an object, but got a string instead. Received Model Group=gpt-5.6-luna
They read that as a broken preset and file it against the router

After: the same request succeeds, and a genuinely malformed schema gets an error that points at the tool

They send the same POST https://litellm-domain/v1/messages with the same string input_schema
The request classifies SIMPLE and lands on the gpt-5.6-luna tier
They get back HTTP 200 with a tool_use block for the tool they defined
If the string does not decode to an object, they get HTTP 400 litellm.BadRequestError: Invalid type for 'tools[0].parameters': expected an object, but got str instead from the gateway itself, before any provider call

Relevant issues

Linear ticket

Resolves LIT-6991

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Local proxy on :4733 from this worktree, real Postgres, deployments gpt-5.6-luna / gpt-5.6-terra / gpt-5.6-sol / claude-opus-5 registered against a live gateway as upstream, plus my-1m-router, an auto-router with the 1M Context preset's tiers. Every run below is a real paid model call. The customer's Azure resource is not reachable from this box (every deployment 404s), so the upstream here is the same model family behind OpenAI's Responses validator, which emits the byte-identical Invalid type for 'tools[0].parameters' error. The only variable between Before and After is the source tree

Shared payload: a get_weather tool whose schema is the JSON string "{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"]}", and a control request with the same schema as an object. Both arms were captured with the same repro.sh and chatbridge.sh, differing only in the checked-out source

Before (`59d42d3`)

/v1/responses, string parameters

curl -X POST http://127.0.0.1:4733/v1/responses -d '{"model":"gpt-5.6-luna","input":"weather in Paris","tools":[{"type":"function","name":"get_weather","parameters":"<string schema>"}]}'
HTTP 400: Invalid type for 'tools[0].parameters': expected an object, but got a string instead.

/v1/messages, string input_schema

curl -X POST http://127.0.0.1:4733/v1/messages -d '{"model":"gpt-5.6-luna","max_tokens":64,"messages":[{"role":"user","content":"weather in Paris"}],"tools":[{"name":"get_weather","description":"w","input_schema":"<string schema>"}]}'
HTTP 400: Invalid type for 'tools[0].parameters': expected an object, but got a string instead.

/v1/messages through the 1M Context auto-router

Same body as above with "model":"my-1m-router" and content "hi" so it classifies SIMPLE
HTTP 400: Invalid type for 'tools[0].parameters': expected an object, but got a string instead. This is the customer's exact report

/v1/chat/completions bridged to Responses, nested string parameters

curl -X POST http://127.0.0.1:4733/v1/chat/completions -d '{"model":"luna-responses-mode","messages":[{"role":"user","content":"weather in Paris"}],"tools":[{"type":"function","function":{"name":"get_weather","parameters":"<string schema>"}}]}' (deployment declares model_info.mode: responses)
HTTP 400: Invalid type for 'tools[0].parameters': expected an object, but got a string instead. Note the error names the flat tools[0].parameters even though the request nested it under function, which is how the customer's report ended up with that shape

Controls, object schema

Same /v1/responses and /v1/messages requests with the schema as an object
HTTP 200 on both, /v1/messages returns a tool_use block

After (`6140d88`)

/v1/responses, string parameters

Identical curl
HTTP 200, id: resp_84m6-zTsfRUQYDA..., model: gpt-5.6-luna, output: [function_call]

/v1/messages, string input_schema

Identical curl
HTTP 200, model: gpt-5.6-luna, output: [tool_use]

/v1/messages through the 1M Context auto-router

Identical curl
HTTP 200, model: my-1m-router, output: [text]

/v1/chat/completions bridged to Responses, nested string parameters

Identical curl
HTTP 200, id: chatcmpl-a7d710d8-35..., finish_reason: tool_calls, tool call get_weather

Controls, object schema

Identical curls
HTTP 200 on both, unchanged from Before

Type

🐛 Bug Fix

Caveats (if any)

Low

Nested Codex namespace tools arrays are not walked; only top-level entries decode
The Anthropic Messages to chat/completions hop (adapters/transformation.py) copies input_schema the same way and its extra-key merge raises AttributeError on a string schema. Same class, different wire shape, not in this report, left as a follow-up
A non-object schema that used to fail at the provider now fails inside litellm with the same status and wording, but the type name is Python's (str, list) rather than the provider's (a string, an array)
parameters: null and an omitted schema are forwarded untouched, since the API accepts both; a live probe returned 200 for null, null with strict: false, and omitted

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Note

Low Risk
Scoped to Responses API outbound request shaping for tools, with clear validation and unit tests; no auth or persistence changes.

Overview
Fixes Responses API requests where function tools carry parameters as a JSON string (common when agents pass stringified input_schema), which previously caused opaque provider 400s blaming the model.

OpenAI Responses transformation now runs tool prep through a shared _prepared_input_and_tools path for both normal and /compact requests. Before flattening schema combinators, it parses string parameters into objects via safe_json_loads; object, null, and omitted schemas are unchanged. Invalid values raise litellm.BadRequestError naming tools[i].parameters before any upstream call. Tool typing is widened to Sequence for the sanitization helpers.

Tests cover decode/reject/unchanged cases on standard and compact transforms, plus Azure after chat-shaped tool un-nesting. type-discipline-budget.json limits are nudged down slightly.

^{Reviewed by Cursor Bugbot for commit 6140d88. Bugbot is set up for automated code reviews on this repo. Configure here.}

tin-berri · 2026-09-05T01:31:43Z

@greptileai can you reply here with a score and why that score?

tin-berri · 2026-09-05T01:31:44Z

bugbot run

greptile-apps · 2026-09-05T01:32:53Z

Confidence score: 4/5

The fix correctly centralizes tool preparation for both Responses and compact requests, decodes JSON strings only when they produce an object, and raises a targeted litellm.BadRequestError for malformed or non-object schemas. The Azure path flattens nested chat-style tools before invoking the shared sanitizer, and the added tests cover successful decoding, compact requests, invalid values, unchanged object/omitted parameters, and Azure behavior.

I’m not giving 5/5 because the required CI check is still pending, and the PR documents two intentionally unhandled paths: nested Codex namespace tools and the Anthropic Messages-to-chat adapter. Those are reasonable follow-ups if they are later included in scope; for the reported top-level Responses/Azure regression, this is a solid 4/5.

codspeed-hq · 2026-09-05T01:33:44Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_responses_tool_params_string (6140d88) with litellm_internal_staging (7573632)}

greptile-apps · 2026-09-05T01:33:56Z

Greptile Summary

This PR normalizes JSON-encoded Responses API tool schemas into objects before provider dispatch and returns an indexed client error for invalid schemas. The shared preparation path now applies the same behavior to standard and compact requests, with coverage for OpenAI and Azure request shapes, invalid values, null schemas, and omitted schemas

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding correctness or repository-rule findings

The schema conversion accepts objects, decodes only JSON strings containing objects, preserves provider-supported null and absent schemas, and rejects other values before dispatch. Both previous rule findings were manually resolved, and the earlier null-schema concern was withdrawn after the provider compatibility evidence

Important Files Changed

Filename	Overview
litellm/llms/openai/responses/transformation.py	Adds centralized tool-schema decoding and validation shared by standard and compact Responses requests
tests/test_litellm/llms/openai/responses/test_openai_responses_transformation.py	Adds focused regression coverage and completes the typing and null-schema follow-ups from the previous review
type-discipline-budget.json	Lowers the relevant type-discipline limits to match the typed implementation

_{Reviews (2): Last reviewed commit: "test(responses): pin null tool schemas a..." | Re-trigger Greptile}

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-09-05T01:37:38Z

Pushed 66ac753. Declined the null finding with a live probe and a pinning test; fixed both P2s. @greptileai can you reply here with a score and why that score?

tin-berri · 2026-09-05T01:37:39Z

bugbot run

codecov · 2026-09-05T01:38:06Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

greptile-apps · 2026-09-05T01:38:11Z

Score: 5/5 (high confidence).

Commit 66ac753343 addresses the remaining review points:

The null finding is correctly declined: the live probe establishes that explicit parameters: null is accepted, and the regression test pins object, null, omitted, and built-in tool shapes.
Both P2s are fixed: the comments were reduced to necessary rationale, and the parametrized invalid-input test now uses object while the string-only test remains typed as str.
Sanitization is centralized for both normal and compact requests, decodes only JSON strings that produce objects, preserves valid existing shapes, and raises an indexed BadRequestError for invalid schemas.
Coverage includes the compact path and Azure’s chat-shaped tool normalization, with the Buildkite e2e check passing.

No remaining blocking concerns from this review. Confidence score: 5/5.

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-09-05T01:41:08Z

@greptileai review

tin-berri · 2026-09-05T02:39:47Z

The 5 integrations reds are pre-existing, not from this PR. test_datadog_llm_obs.py::test_a_redacted_span_carries_every_declared_guardrail_field fails identically at the merge base 639b3f4: guardrail_cost_by_unit is declared on the type but absent from the redacted record, from 4914914. This PR touches only the OpenAI Responses transform and its test.

tin-berri · 2026-09-05T03:48:09Z

Merged current staging (ab0fc27). The integrations red was staging's own, fixed upstream by d4fd658 after this branch's base; re-ran the live evidence on the merged tree, all legs still 200. @greptileai can you reply here with a score and why that score?

tin-berri · 2026-09-05T03:48:10Z

bugbot run

greptile-apps · 2026-09-05T03:48:54Z

Confidence score: 5/5.

The merged staging tree preserves the fix, and the updated live evidence confirms all reported paths still return 200, including direct Responses requests, Messages input-schema conversion, the 1M Context auto-router/Azure route, the chat-to-Responses bridge, and the object-schema controls.

The implementation remains correctly scoped: standard and compact requests share one preparation path, JSON strings are decoded only when they produce an object, valid object/null/omitted schemas are preserved, and malformed or non-object values fail early with an indexed litellm.BadRequestError. The regression coverage exercises these cases plus Azure’s nested tool shape. The integrations failures were confirmed as staging-only and fixed upstream, so I have no remaining correctness concerns.

cursor

Stale Bugbot comment from a previous run.

… provider A caller that hands a tool schema over already JSON-encoded reached the Responses API with a string `parameters`, and the provider rejected the request with a 400 naming the routed model instead of the offending tool. Decode it at the one place every Responses request converges, and refuse anything that is neither an object nor a string encoding one. Collapses the duplicated input/tool sanitization block shared by the request and compact-request builders into a single owner, so the decode cannot be wired into one path and not the other.

…metrized cases The Responses API serves `parameters: null` and an omitted schema alike, so neither may raise. Pin both against a future tightening, annotate the parametrized inputs, and trim the docstrings back to what the code does not already say.

Lower the type-discipline limits for violations removed by the response transform fix.

tin-berri · 2026-09-05T05:06:55Z

Rebased onto current litellm_internal_staging at 7573632; final head is 6140d88. @greptileai can you reply here with a score and why that score?

tin-berri · 2026-09-05T05:06:56Z

bugbot run

greptile-apps · 2026-09-05T05:07:56Z

Confidence score: 5/5.

The rebased head 6140d881f8 preserves the fix cleanly. Tool preparation is centralized in litellm/llms/openai/responses/transformation.py, so both standard and compact Responses requests decode JSON strings consistently. Existing object, null, omitted, and built-in tool schemas remain unchanged, while malformed or non-object values fail early with an indexed litellm.BadRequestError. Azure’s nested chat-shaped tools are flattened before the shared sanitizer runs.

The focused regression tests cover successful decoding, compact requests, invalid values, accepted edge cases, and Azure behavior. The current Buildkite E2E check is passing, and I found no remaining correctness concerns.

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 6140d88. Configure here.}

tin-berri requested a review from a team September 5, 2026 01:31

greptile-apps Bot reviewed Sep 5, 2026

View reviewed changes

Comment thread litellm/llms/openai/responses/transformation.py

Comment thread litellm/llms/openai/responses/transformation.py Outdated

Comment thread tests/test_litellm/llms/openai/responses/test_openai_responses_transformation.py Outdated

cursor Bot reviewed Sep 5, 2026

View reviewed changes

tin-berri enabled auto-merge (squash) September 5, 2026 01:41

yucheng-berri approved these changes Sep 5, 2026

View reviewed changes

yucheng-berri mentioned this pull request Sep 5, 2026

feat(otel): stamp litellm.request.route on the LLM call span #39698

Merged

6 tasks

tin-berri disabled auto-merge September 5, 2026 03:43

cursor Bot reviewed Sep 5, 2026

View reviewed changes

tin-berri enabled auto-merge (squash) September 5, 2026 03:53

tin-berri added 3 commits September 4, 2026 22:02

chore: ratchet response transform type budgets

6140d88

Lower the type-discipline limits for violations removed by the response transform fix.

tin-berri force-pushed the litellm_responses_tool_params_string branch from ab0fc27 to 6140d88 Compare September 5, 2026 05:05

cursor Bot reviewed Sep 5, 2026

View reviewed changes

tin-berri merged commit 78ad88f into litellm_internal_staging Sep 5, 2026
183 of 184 checks passed

tin-berri deleted the litellm_responses_tool_params_string branch September 5, 2026 05:14

mateo-berri mentioned this pull request Sep 5, 2026

feat(anthropic): workload identity federation, pluggable identity sources, and provider-level setup (internal copy of #38013) #38818

Open

21 tasks

Uh oh!

Uh oh!

Conversation

tin-berri commented Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Relevant issues

Linear ticket

Pre-Submission checklist

Screenshots / Proof of Fix

Before (59d42d3)

/v1/responses, string parameters

/v1/messages, string input_schema

/v1/messages through the 1M Context auto-router

/v1/chat/completions bridged to Responses, nested string parameters

Controls, object schema

After (6140d88)

/v1/responses, string parameters

/v1/messages, string input_schema

/v1/messages through the 1M Context auto-router

/v1/chat/completions bridged to Responses, nested string parameters

Controls, object schema

Type

Caveats (if any)

Low

Final Attestation

Uh oh!

tin-berri commented Sep 5, 2026

Uh oh!

tin-berri commented Sep 5, 2026

Uh oh!

greptile-apps Bot commented Sep 5, 2026

Uh oh!

codspeed-hq Bot commented Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

greptile-apps Bot commented Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tin-berri commented Sep 5, 2026

Uh oh!

tin-berri commented Sep 5, 2026

Uh oh!

codecov Bot commented Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

greptile-apps Bot commented Sep 5, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tin-berri commented Sep 5, 2026

Uh oh!

tin-berri commented Sep 5, 2026

Uh oh!

tin-berri commented Sep 5, 2026

Uh oh!

tin-berri commented Sep 5, 2026

Uh oh!

greptile-apps Bot commented Sep 5, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tin-berri commented Sep 5, 2026

tin-berri commented Sep 5, 2026 •

edited

Loading

Before (`59d42d3`)

After (`6140d88`)

codspeed-hq Bot commented Sep 5, 2026 •

edited

Loading

greptile-apps Bot commented Sep 5, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

codecov Bot commented Sep 5, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading