fix(proxy): make /cursor/chat/completions work with Cursor agent mode

[tin-berri]

TLDR

Problem this solves:

• Cursor BYOK against litellm 400d on real Ask and agent requests
• OpenAI custom (grammar) tools crashed or silently vanished on litellm's chat surface
• gpt-5.4+ function tool chat requests 400d when reasoning defaulted on

How it solves it:

• cursor endpoint normalizes each tool shape level to canonical chat format
• core chat surface gains first-class custom tool call types
• responses bridge auto-fires for gpt-5.4+ function tools with reasoning active

Relevant issues

• /cursor/chat/completions now handles both request shapes Cursor BYOK sends, so Cursor agent mode works against litellm: Responses-shaped agent-mode bodies (input, flat tool defs, custom tools, reasoning) and genuine chat-completions bodies (messages), always answering in chat completions format

• streaming through the endpoint was fully broken (500 on every streaming request) and router-served streams leaked raw Responses API events (including request metadata) instead of chat chunks; both fixed

• custom (grammar) tool calls, e.g. Cursor's ApplyPatch, now surface as chat tool_calls in both streaming and non-streaming responses, and multi-turn tool round trips complete

• Cursor Ask mode sends a genuine chat completions body whose tools array mixes nested function tools with a flat Responses-style custom tool (ApplyPatch); the messages arm forwarded that verbatim so OpenAI 400d (Missing required parameter: 'tools[N].custom'), and even a correctly nested custom tool failed on the way back: litellm's response parsing 500d non-streaming and silently dropped the tool call when streaming. The messages arm now nests flat tool defs, and OpenAI chat custom tool calls are first-class across litellm's chat surface (non-streaming, streaming deltas, chunk rebuild)

• ApplyPatch is grammar-constrained: Cursor ships a Lark grammar in the Responses flat format shape ({"type":"grammar","definition","syntax"}) while chat completions wraps the same fields one level deeper ({"type":"grammar","grammar":{...}}), so the nested envelope alone still 400d (tools[N].custom.format.grammar); format shapes and named tool_choice objects now translate across surfaces in both directions. Live captures further show the dialects mix PER LEVEL (Cursor sends a chat-nested envelope whose inner format is still Responses-flat), so the cursor arm normalizes each level independently and idempotently; a parametrized test pins the full envelope x format matrix. Separately, real Cursor gpt-5.6 requests exposed a routing gap in core litellm: OpenAI enables reasoning by default for gpt-5.4+ and Chat Completions rejects function tools whenever reasoning is on, but the existing responses-bridge heuristic only fired when the caller explicitly sent reasoning_effort, so tools requests without it 400d on chat instead of auto-bridging; the heuristic now treats unset effort as reasoning-active and keeps the documented reasoning_effort "none" escape hatch on chat

Reference behavior studied from the community Cursor-OpenAI-BYOK-Bridge (src/byok): Cursor agent mode POSTs Responses-API-shaped bodies to the chat/completions path while expecting chat-completions SSE back, so the bridge routes those to a Responses upstream and converts the stream back; bodies with messages pass through as genuine chat completions

Linear ticket

Resolves LIT-4395
Resolves LIT-4396

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Live proxy on 127.0.0.1:4396 backed by Postgres, real provider calls. Before runs captured at 214945a223 (origin/litellm_internal_staging), after runs at 90283fab70. The before runs used real OpenAI; the account ran out of quota mid-verification, so the after runs use real Anthropic through the same pipeline (the streaming and conversion defects are provider-independent; the two OpenAI-specific probes are marked)

A. Agent-mode Responses-shaped body with stream_options (Cursor sends stream_options: {"include_usage": true})

curl -s -X POST http://127.0.0.1:4396/cursor/chat/completions -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{
  "model": "gpt-5.6",
  "input": [{"role": "user", "content": "Say OK"}],
  "stream_options": {"include_usage": true}
}'
# BEFORE (real OpenAI): {"error":{"message":"litellm.BadRequestError: OpenAIException - ... \"Unknown parameter: 'stream_options.include_usage'.\" ...","code":"400"}}
# AFTER (claude-sonnet-5): {"finish_reason": "stop", "content": "OK", "object": "chat.completion"}

B. Genuine chat-completions body with tool history (what Cursor sends for models whose BYOK it already fixed)

curl -s -X POST http://127.0.0.1:4396/cursor/chat/completions -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{
  "model": "claude-sonnet-5",
  "messages": [
    {"role": "user", "content": "read a file"},
    {"role": "assistant", "content": null, "tool_calls": [{"id": "call_hist1", "type": "function", "function": {"name": "read_file", "arguments": "{\"path\": \"a.py\"}"}}]},
    {"role": "tool", "tool_call_id": "call_hist1", "content": "print(42)"},
    {"role": "user", "content": "what does it print? answer with just the number"}
  ],
  "tools": [{"type": "function", "function": {"name": "read_file", "parameters": {"type": "object", "properties": {"path": {"type": "string"}}}}}]
}'
# BEFORE (real OpenAI): {"error":{"message":"... \"Invalid type for 'input[1].content': expected one of an array of objects or string, but got null instead.\" ...","code":"400"}}
# AFTER: {"finish_reason": "stop", "content": "42", "model": "claude-sonnet-5", "usage_total": 485}

C. Agent-mode streaming with a function tool

curl -sN -X POST http://127.0.0.1:4396/cursor/chat/completions -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{
  "model": "claude-sonnet-5",
  "input": [{"role": "user", "content": "Read the file a.py using the read_file tool."}],
  "stream": true,
  "stream_options": {"include_usage": true},
  "tools": [{"type": "function", "name": "read_file", "description": "Read a file", "parameters": {"type": "object", "properties": {"path": {"type": "string"}}, "required": ["path"]}}],
  "tool_choice": "required"
}'
# BEFORE: {"error":{"message":"cursor_chat_completions.<locals>.cursor_data_generator() got an unexpected keyword argument 'request'","code":"500"}}
# AFTER: 6 chat.completion.chunk SSE events, then [DONE]:
#   data: {..."choices":[{"index":0,"delta":{"role":"assistant","tool_calls":[{"id":"toolu_01GcJXm9WhT3Y6Tq23KthKAS","function":{"arguments":"","name":"read_file"},"type":"function","index":0}]}...}
#   data: {..."delta":{"tool_calls":[{"function":{"arguments":"{\"path\": \""},"type":"function","index":0}]}...}
#   data: {..."delta":{"tool_calls":[{"function":{"arguments":"a"},"type":"function","index":0}]}...}
#   data: {..."delta":{"tool_calls":[{"function":{"arguments":".py\"}"},"type":"function","index":0}]}...}
#   data: {..."finish_reason":"tool_calls"...}
#   data: [DONE]

The tool call opens at index 0 with the role on the first delta. On the intermediate broken state (generator signature fixed but the class-identity stream gate still in place) this same curl returned raw response.created / response.output_item.added / response.completed Responses events to the client, including litellm request metadata inside response.created

D. Multi-turn tool round trip (Cursor echoes function_call and function_call_output items keyed by call_id)

curl -s -X POST http://127.0.0.1:4396/cursor/chat/completions -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{
  "model": "claude-sonnet-5",
  "input": [
    {"role": "user", "content": "Read the file a.py using the read_file tool, then tell me what it prints. Answer with just the number."},
    {"type": "function_call", "call_id": "toolu_01GcJXm9WhT3Y6Tq23KthKAS", "name": "read_file", "arguments": "{\"path\": \"a.py\"}"},
    {"type": "function_call_output", "call_id": "toolu_01GcJXm9WhT3Y6Tq23KthKAS", "output": "print(42)"}
  ],
  "tools": [{"type": "function", "name": "read_file", "description": "Read a file", "parameters": {"type": "object", "properties": {"path": {"type": "string"}}, "required": ["path"]}}]
}'
# BEFORE: {"error":{"message":"Unknown items in responses API response: [GenericResponseOutputItem(type='message', ..., content=[OutputText(type='output_text', text='42', ...)], ...)]","code":"500"}}
# AFTER: {"finish_reason": "stop", "content": "42", "usage_total": 518}

E. Custom (grammar) tool call, the shape Cursor agent mode uses for ApplyPatch (real OpenAI, non-streaming; captured before quota ran out)

curl -s -X POST http://127.0.0.1:4396/cursor/chat/completions -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{
  "model": "gpt-5.6",
  "input": [{"role": "user", "content": "Create hello.txt containing hello world using the ApplyPatch tool."}],
  "reasoning": {"effort": "low"},
  "tools": [{"type": "custom", "name": "ApplyPatch", "description": "Apply a file patch in V4A diff format"}],
  "tool_choice": "required"
}'
# BEFORE: the model called the tool upstream (real tokens billed) but the client received {"finish_reason": null, "content": null, "tool_calls": null}
# AFTER (real OpenAI, live gateway running this branch): 200, finish_reason "tool_calls", tool call {"function": {"name": "ApplyPatch", "arguments": "*** Begin Patch\n*** Update File: main.py\n@@\n+def hello():\n..."}, "type": "function"}; the owed live re-run of this probe is done

F. Ask-mode messages body with a flat custom tool def (the exact shape a real Cursor Ask request sent through a live gateway running the pre-fix branch; request IDs in the gateway logs)

curl -s -X POST $PROXY/cursor/chat/completions -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{
  "model": "gpt-5.6",
  "messages": [{"role": "user", "content": "use ApplyPatch to add a hello() function to main.py"}],
  "tools": [
    {"type": "function", "function": {"name": "read_file", "description": "Read a file", "parameters": {"type": "object", "properties": {"path": {"type": "string"}}, "required": ["path"]}}},
    {"type": "custom", "name": "ApplyPatch", "description": "Apply a V4A format patch to a file"}
  ],
  "tool_choice": "required"
}'
# BEFORE (real OpenAI): {"error":{"message":"litellm.BadRequestError: OpenAIException - Missing required parameter: 'tools[1].custom'.. ...","param":"tools[1].custom","code":"400"}}
# AFTER: 200, tool_calls: [{"id": "call_...", "type": "custom", "custom": {"name": "ApplyPatch", "input": "*** Begin Patch..."}}], finish_reason "tool_calls"

G. Correctly nested custom tool through plain /v1/chat/completions (proves the response-side break is core litellm, not endpoint-specific)

curl -s -X POST $PROXY/v1/chat/completions -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{
  "model": "gpt-5.6",
  "messages": [{"role": "user", "content": "use ApplyPatch to add a hello() function to main.py"}],
  "tools": [{"type": "custom", "custom": {"name": "ApplyPatch", "description": "Apply a V4A format patch to a file"}}],
  "tool_choice": "required"
}'
# BEFORE (real OpenAI): {"error":{"message":"litellm.InternalServerError: ... TypeError: ChatCompletionMessageToolCall.__init__() missing 1 required positional argument: 'function' (convert_dict_to_response.py:650)"}}
# AFTER: 200 with the custom tool call intact

H. Same body with "stream": true (the worst pre-fix behavior: silent data loss, not an error)

# BEFORE (real OpenAI): 200 SSE stream with finish_reason "tool_calls" and ZERO tool_call deltas; usage reports 68 completion tokens that never reached the client
# AFTER: first delta carries {"id", "type": "custom", "custom": {"name": "ApplyPatch", "input": ""}}, continuation deltas carry {"custom": {"input": ...}} fragments, then finish_reason "tool_calls" and [DONE]

I. Grammar-constrained ApplyPatch (the real Cursor Ask-mode shape: flat grammar format inside the flat custom tool)

curl -s -X POST $PROXY/cursor/chat/completions -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{
  "model": "gpt-5.6",
  "messages": [{"role": "user", "content": "use ApplyPatch to add hello() to main.py"}],
  "tools": [{"type": "custom", "name": "ApplyPatch", "description": "Apply a V4A patch",
             "format": {"type": "grammar", "syntax": "lark", "definition": "start: begin_patch ..."}}],
  "tool_choice": "required"
}'
# BEFORE (real OpenAI, gateway at the previous commit): {"error":{"message":"... Missing required parameter: 'tools[0].custom.format.grammar'...","code":"400"}}
# Contract control (real OpenAI, same gateway): manually moving definition/syntax under format.grammar returns 200 with a proper grammar-constrained ApplyPatch tool call
# AFTER: the gateway performs that exact rewrite; re-run the curl above on the redeployed branch to confirm 200

Named tool_choice objects follow the same surface split (Responses flat {"type":"custom","name"} vs chat nested {"type":"custom","custom":{"name"}}); the cursor arm nests flat ones, and both bridges translate them alongside the tool defs they reference (unit-pinned; live probe rides the same redeploy)

J. The exact shape real Cursor Ask mode sends (live-captured): pre-nested envelope carrying a FLAT grammar format

curl -s -X POST $PROXY/cursor/chat/completions -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{
  "model": "gpt-5.6",
  "messages": [{"role": "user", "content": "use ApplyPatch to add hello() to main.py"}],
  "tools": [{"type": "custom", "custom": {"name": "ApplyPatch", "description": "Apply a V4A patch",
             "format": {"type": "grammar", "definition": "start: begin_patch ...", "syntax": "lark"}}}],
  "tool_choice": "required"
}'
# BEFORE (real OpenAI, gateway at the prior commit): 400 Missing required parameter: 'tools[0].custom.format.grammar' (the prior commit translated format only on the flat-envelope path)
# AFTER: the format level is normalized regardless of which envelope shape it arrived in; re-run on the redeployed branch to confirm 200

The reference community BYOK bridge was checked as prior art and performs no tool normalization on its chat path (messages bodies forward verbatim), so these hybrid shapes have no reference handling to mirror; the per-level matrix test is the contract

K. Function tools on gpt-5.6 without an explicit reasoning_effort (real Cursor agent shape; reproduces on plain /v1/chat/completions, so this was a core litellm gap, not a cursor-endpoint one)

curl -s -X POST $PROXY/v1/chat/completions -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{
  "model": "gpt-5.6",
  "messages": [{"role": "user", "content": "read a.py"}],
  "tools": [{"type": "function", "function": {"name": "read_file", "parameters": {"type": "object", "properties": {"path": {"type": "string"}}}}}]
}'
# BEFORE (real OpenAI): 400 Function tools with reasoning_effort are not supported for gpt-5.6 ... use /v1/responses or set reasoning_effort to 'none'
#   (litellm forwarded to chat completions because the bridge heuristic required an explicit reasoning_effort; OpenAI applies reasoning by default server-side)
# AFTER: the responses bridge auto-activates for gpt-5.4+ with tools unless reasoning_effort is explicitly "none"; the request is served via /v1/responses with reasoning intact and returns chat-shaped output; re-run on the redeployed branch to confirm 200

Control (stays green before and after): history round trip with a nested custom tool_call plus its tool result through /v1/chat/completions returns 200 (litellm forwards custom tool_calls in request history untouched; OpenAI accepts them)

Raw wire shapes these fixes encode were captured live through the /openai passthrough (non-streaming message and the streaming delta sequence) and are pinned verbatim in the regression tests

Negative control: every other /cursor/* path still reaches the Cloud Agents passthrough

curl -s http://127.0.0.1:4396/cursor/v0/models -H "Authorization: Bearer sk-1234"
# BEFORE and AFTER: {"detail":"Cursor API key not found. Add Cursor credentials via the UI ..."}

Spend note: SpendLogs rows land for every request above; the Responses-pipeline rows log zero cost, which is pre-existing and tracked in #30126 with open PR #30277

Type

🐛 Bug Fix

Changes

• route (cursor_chat_completions): bodies with messages run through the standard chat completions pipeline untouched (the old blind messages to input rename broke tool history and chat tool defs); Responses-shaped bodies keep the aresponses pipeline with chat-completions-only stream_options stripped

• cursor_data_generator accepts the request kwarg the processor passes (every streaming request 500d without it) and detects streams by __anext__ instead of a class check that the router's HiddenParamsAsyncIteratorWrapper defeats (which leaked raw Responses events, metadata included, to clients)

• streaming converter: custom_tool_call output items and custom_tool_call_input.delta events become chat tool_calls deltas; tool_call indices are remapped per-stream to 0-based sequential (Responses output_index counts reasoning and message items, so the first tool call of a reasoning model arrived at index 1 or higher and strict SSE accumulators misplace it); response.completed reports finish_reason: "tool_calls" for custom_tool_call items too. The finish check deliberately covers exactly the item types the stream converter emits tool_call deltas for; reporting tool_calls for item types never streamed (e.g. the native apply_patch tool, unhandled in the stream converter before and after this PR) would hand the client a tool_calls finish with an empty tool_calls array

• non-streaming converter: typed ResponseCustomToolCall items and litellm's own GenericResponseOutputItem pydantic items (what the completion bridge emits for non-Responses-native providers behind the router) convert to chat output instead of being dropped; dropping the latter raised "Unknown items in responses API response" on every router-bridged non-streaming completion through this endpoint

• messages arm (cursor_chat_completions): flat Responses-style tool defs in a chat body are nested into the chat completions shape ({"type": "custom", "name", ...} to {"type": "custom", "custom": {...}}, same for flat function defs) and published through _safe_set_request_parsed_body so the delegated chat_completion reads the normalized body from the request-scope cache; bodies whose tools are already chat-shaped skip the cache write entirely

• core chat surface: OpenAI chat completions custom tool calls are now first-class. New ChatCompletionMessageCustomToolCall / ChatCompletionDeltaCustomToolCall sibling types mirror the installed openai SDK union (separate classes so function tool call dumps gain no custom key and custom dumps carry no function key), a single chat_completion_tool_call_from_dict dispatch feeds the three provider-dict sinks (Message.__init__, convert_to_model_response_object, _transform_choices), and Delta.__init__ dispatches streamed dicts on the custom key so both stream re-parse sites (chunk_creator, return_processed_chunk_logic) stop swallowing custom deltas via their except branches

• consumers that assume .function were audited and the unguarded ones now skip custom entries: the json-mode gate _should_convert_tool_call_to_json_mode (single chokepoint for all three json-mode callers), _handle_invalid_parallel_tool_calls, helicone and lunary logging; the chunk builder accumulates custom.input fragments so spend logging and stream_chunk_builder rebuild the full custom call

• chat to responses bridge (_convert_tools_to_responses_format): nested chat custom tools are flattened to the Responses flat shape instead of being forwarded nested (the mirror-image gap, hit when a chat body with custom tools targets a Responses-bridged model)

• the cursor messages arm normalizes tool shapes PER LEVEL, not per whole-tool template: the envelope level (flat def nested under custom/function) and the format level (flat grammar wrapped as format.grammar) are normalized independently and idempotently, so hybrids like a pre-nested envelope with a flat grammar inside land canonical; every envelope x format combination is pinned by a parametrized test

• custom tool format shape translation: shared helper pair in prompt_templates/common_utils (convert_custom_tool_format_to_chat_shape / convert_custom_tool_format_to_responses_shape) wraps or unwraps the grammar object; applied on the cursor messages arm (flat to chat) and the chat to responses bridge (chat to flat); {"type": "text"} is identical on both surfaces and passes through, as does anything unrecognized

• named tool_choice translation follows the tool defs on every surface: the cursor messages arm nests flat Responses-style tool_choice objects, _normalize_tool_choice_for_responses_api flattens chat custom tool_choice exactly like its existing function arm, and _transform_tool_choice on the responses to chat bridge maps custom tool_choice to a function tool_choice because that bridge downgrades custom tools to function tools (a custom choice would reference a tool type absent from the converted request)

• core responses-bridge heuristic (responses_api_bridge_check): for gpt-5.4+ on openai/azure with FUNCTION tools, unset reasoning_effort now bridges to the Responses API (OpenAI defaults reasoning on server-side, so the chat request was unservable) and explicit "none" stays on chat completions (OpenAI's documented escape hatch; the old code over-bridged that cell). The gate requires a function-type tool specifically: custom-only requests are served natively by chat completions with reasoning on (live-proven 200) and keep the native custom tool_call response shape rather than the bridge's function-shaped mapping; the pre-existing arm shared the any-tools coarseness and is fixed by the same leg. Two prior test pins asserting tools-without-reasoning stays chat encoded the unservable behavior and are flipped with the rationale in their docstrings

• cursor input arm gains the mirror of the messages-arm normalization: chat-nested tool envelopes, chat-nested grammar formats, and chat-nested object tool_choice are flattened to the Responses dialect before dispatch (previously an object-form tool_choice in a Responses-shaped body reached OpenAI in the wrong nesting); the mirror envelope x format matrix is pinned parametrically

• the bridge gate resolves the effective OpenAI base exactly as the chat handler does (call arg, then litellm.api_base, then OPENAI_BASE_URL/OPENAI_API_BASE, then the default) through one shared _resolve_openai_api_base, so a custom base configured only via the global or env is no longer invisible to the gate and misread as the default endpoint; only requests that would actually hit the default OpenAI endpoint (or Azure) are auto-bridged

• custom-vs-function classification of a tool-call dict goes through one shared is_custom_tool_call_dict (explicit custom type, or a custom payload present) used by both the non-streaming chat_completion_tool_call_from_dict and the streaming Delta coercion, and the stream assembler classifies from the accumulated custom payload rather than a type field continuation chunks never carry; previously the same payload could parse as a custom tool call mid-stream, raise TypeError on the completed message, and be silently dropped from the assembled message. The tool envelope converter now extracts payload fields from both the nested dict and the top level (nested first) instead of treating a present nested dict as the exclusive source, so an empty nested envelope no longer drops a top-level name, and a nameless envelope passes through unchanged

What does not change: every other /cursor/* path still passes through to the Cursor Cloud Agents API; the /v1/responses endpoints are untouched; the tool-index remap only activates when the per-stream map is passed, so the stateless guardrail caller of the same translate function keeps its existing behavior; function tool call parsing, serialization, and streaming shapes are byte-identical to before (pinned by the pre-existing exact-dict tests, all green); the Responses-arm mapping of custom tool calls to function-shaped chat tool calls is unchanged; the bridge heuristic for pre-5.4 GPT-5 names and the reasoning-summary arm are unchanged. One consequence worth noting: the auto-bridge applies only when the resolved base is the default OpenAI endpoint (or the provider is Azure); OpenAI-compatible backends reached through a custom api_base, whether passed per-call or configured via litellm.api_base or OPENAI_BASE_URL/OPENAI_API_BASE, are never auto-bridged, and reasoning_effort "none" remains the per-request escape hatch on the default endpoint

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
Touches core chat/Responses bridging, proxy routing, and gpt-5.4+ auto-routing; behavior changes are broad but heavily test-covered, with the main regression risk being mis-routing OpenAI-compatible backends or altering tool-call shapes clients depend on.

Overview
Cursor BYOK on /cursor/chat/completions now branches on body shape: real messages bodies delegate to normal chat completions (with per-level normalization of flat Responses-style tools / tool_choice and grammar format into chat shapes); Responses-shaped input bodies still go through aresponses with stream_options stripped without mutating the cached request body. Streaming fixes include accepting the request kwarg, detecting streams via __anext__ (not a brittle class check), and converting Responses events to chat SSE. GET /cursor/models and /cursor/v1/models delegate to the standard model list so BYOK key verification works.

Custom (grammar) tools are first-class on the chat surface: new custom tool call types, chat_completion_tool_call_from_dict / is_custom_tool_call_dict, streaming delta assembly and chunk rebuild for custom.input, and Responses↔chat bridges for custom_tool_call items (including sequential tool_call index remapping and custom_tool_call_input.delta). Helicone/Lunary logging and related paths handle custom payloads.

gpt-5.4+ routing: responses_api_bridge_check shares _resolve_openai_api_base with the chat handler and auto-bridges function tools when reasoning is active (including unset effort on real OpenAI/Azure), while custom-only tool sets stay on chat and reasoning_effort: "none" remains the chat escape hatch.

^{Reviewed by Cursor Bugbot for commit eea9bb1. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Greptile Summary

This PR fixes a cluster of real failures in the /cursor/chat/completions BYOK endpoint and the broader chat surface. Cursor agent mode now works end-to-end: both Responses-shaped bodies (input, flat tool defs) and genuine chat-completions bodies (messages) are handled correctly, streaming no longer 500s, custom (grammar) tool calls are first-class in non-streaming and streaming responses, and gpt-5.4+ with function tools auto-bridges to the Responses API when reasoning is active by default.

• Cursor endpoint (endpoints.py): routes on body shape \u2014 chat bodies normalized and delegated to chat_completion; Responses bodies strip chat-only stream_options and hit aresponses. Per-level tool dialect normalization converts flat\u2194nested tool envelopes and grammar format objects independently (idempotent, matrix-tested). Streaming fixed: generator accepts request kwarg and detects streams via __anext__ instead of class identity.
• Core chat surface (types/utils.py, convert_dict_to_response.py, streaming_chunk_builder_utils.py): new ChatCompletionMessageCustomToolCall / ChatCompletionDeltaCustomToolCall types with shared chat_completion_tool_call_from_dict dispatch; custom tool calls accumulate and rebuild correctly in the chunk builder; consumers that assumed .function (helicone, lunary, json-mode gate, parallel-tool-call handler) now skip custom entries.
• Bridge heuristic (main.py): _resolve_openai_api_base shared between the gate and the chat handler so custom API bases configured via global or env are not misread as the default endpoint; unset reasoning_effort on gpt-5.4+ now bridges (OpenAI applies reasoning by default server-side), while explicit \"none\" keeps chat; custom-only tool sets stay on chat.

Confidence Score: 5/5

Safe to merge. The behavioral changes are narrow: the cursor endpoint was broadly broken before, the bridge routing change only affects the default OpenAI and Azure endpoints (custom API bases are explicitly exempted), and the custom tool call additions are additive to existing union types without touching function tool call serialization.

All changed paths are covered by new mock-based unit tests including the full envelope x format normalization matrix, streaming conversion, and api_base resolution logic with monkeypatching. The two tests whose assertions were inverted correctly encode the new required behavior. No regressions found in function tool call serialization or streaming paths.

Files Needing Attention: The completion_extras/litellm_responses_transformation/transformation.py streaming converter warrants a careful read, particularly the tool_call_index_map per-stream state and the output_item.done split between stateless and per-stream callers.

Important Files Changed

Filename	Overview
litellm/proxy/response_api_endpoints/endpoints.py	Significant refactor: cursor endpoint now branches on body shape (chat vs. Responses-shaped), adds per-level tool dialect normalization helpers, and fixes streaming (generator signature + __anext__ detection). New cursor_model_list routes to the standard model list. The stream_options rebuild (instead of pop) correctly avoids corrupting the request cache.
litellm/main.py	Adds _resolve_openai_api_base shared helper so the bridge heuristic and the chat handler resolve the effective endpoint identically. Bridge now auto-activates for gpt-5.4+ with function tools + unset reasoning_effort on the default OpenAI/Azure endpoint, and correctly exempts custom API bases. reasoning_effort 'none' remains the documented escape hatch.
litellm/completion_extras/litellm_responses_transformation/transformation.py	Complex streaming/non-streaming fixes: custom_tool_call items now produce tool_call deltas; tool-call indices are remapped 0-based per-stream; response.completed fires finish_reason=tool_calls for custom_tool_call; GenericResponseOutputItem (pydantic) no longer raises 'Unknown items'; tool_choice translation extended for custom type.
litellm/types/utils.py	New sibling types ChatCompletionMessageCustomToolCall / ChatCompletionDeltaCustomToolCall with shared dispatch via chat_completion_tool_call_from_dict and is_custom_tool_call_dict; Delta.init now dispatches on the custom key for streaming. Message.tool_calls union type expanded.
litellm/litellm_core_utils/streaming_chunk_builder_utils.py	ChunkProcessor accumulates custom.input fragments alongside function.arguments; finalized custom tool calls build ChatCompletionMessageCustomToolCall. Arguments accumulation converted from list.append to tuple concatenation (immutable).
litellm/litellm_core_utils/llm_response_utils/convert_dict_to_response.py	All three tool-call construction sites now route through chat_completion_tool_call_from_dict. _handle_invalid_parallel_tool_calls and _should_convert_tool_call_to_json_mode guard against custom entries.
litellm/integrations/helicone.py	Tool call logging now falls back to custom.name/custom.input when function is absent. Skips entries without either payload to avoid KeyError.
litellm/integrations/lunary.py	parse_tool_calls filters out entries with neither function nor custom payload; clean_tool_call dispatches on custom attribute.
tests/test_litellm/test_main.py	Two existing bridge-check tests renamed and assertions inverted to reflect the intentional behavior change. Many new parametrized tests added including api_base/env-var scenarios with monkeypatch.
tests/test_litellm/proxy/response_api_endpoints/test_endpoints.py	629 lines of new tests for the cursor endpoint: TestClient-based, mocked dependencies, covers messages vs. input routing, tool normalization matrix, streaming, cache non-corruption, and model list endpoints.

_{Reviews (18): Last reviewed commit: "chore(lint): use a PEP 604 union for the..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 96.06557% with 12 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...litellm_responses_transformation/transformation.py	95.00%	4 Missing ⚠️
litellm/types/utils.py	92.50%	3 Missing ⚠️
litellm/proxy/response_api_endpoints/endpoints.py	97.95%	2 Missing ⚠️
litellm/integrations/helicone.py	80.00%	1 Missing ⚠️
...itellm_core_utils/streaming_chunk_builder_utils.py	96.00%	1 Missing ⚠️
litellm/llms/openai/chat/gpt_transformation.py	50.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[tin-berri]

@greptileai

[tin-berri]

CI note: the osv-scan red on this PR is inherited, not from this diff. GHSA-3jxr-9vmj-r5cp (brace-expansion 5.0.6, npm dev dep of the dashboard) entered the OSV database between this PR's first and second runs; the same lockfile passed at 20:15 and failed at 21:00 with no dependency changes in between. Fix is in #34032, which bumps the single lockfile entry to 5.0.7; osv-scan here will go green once that merges into litellm_internal_staging and this branch rebases or the check re-runs against the updated database expectations

[tin-berri]

@greptileai

[tin-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit def8a2c. Configure here.}

[tin-berri]

@greptileai

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_lit4395_cursor_agent (9eeff06) with litellm_internal_staging (6b3d4f2)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (9eeff06) during the generation of this report, so 6b3d4f2 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[tin-berri]

@greptileai

[tin-berri]

@greptileai

[tin-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 897e1bd. Configure here.}

[tin-berri]

@greptileai

[tin-berri]

@greptileai

[tin-berri]

@greptileai

[tin-berri]

bugbot run

[tin-berri]

@greptileai

[tin-berri]

@greptileai

[tin-berri]

bugbot run

[tin-berri]

@greptileai

[tin-berri]

@greptileai

[tin-berri]

Addressed the 4/5 concern about gpt-5.4+ names on non-OpenAI backends in 2cd909a, with self-scoping instead of a config flag: the unset-effort arm now fires only against endpoints known to enforce the OpenAI constraint (default OpenAI endpoint, or Azure where api_base is always set), so chat-only OpenAI-compatible backends with custom api_base keep their pre-existing behavior exactly (bridge only on an explicit reasoning_effort, escape hatch reasoning_effort none unchanged). Pinned by three new tests: custom api_base with unset effort stays chat, custom api_base with explicit effort still bridges, azure with api_base and unset effort bridges; mutation-checked.

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit eea9bb1. Configure here.}

[mateo-berri]

The version parse predates this PR and degrades gracefully for dated snapshots and future minors. A per-entry json flag would silently miss unlisted deployment names

[mateo-berri]

@greptileai

[mateo-berri]

LGTM. Thanks!

[veria-ai]

High: Per-model budget bypass

user_api_key_auth checks key and end-user model_max_budget against the original request model before this handler runs. An authenticated user can request a name such as claude-opus-5-thinking-high; the budget lookup treats that unconfigured alias as unrestricted, and this code then executes claude-opus-5 even when its model budget is exhausted. Canonicalize the model before the authentication dependency runs, or repeat all model-access and per-model budget checks against variant.base_model before dispatch.

[mateo-berri]

Per-model budget bypass

Already fixed in #35834, which resolves the variant before auth so budget checks see the base model. Leaving open to re-verify later

[veria-ai]

PR overview

This pull request updates the /cursor/chat/completions proxy flow to support Cursor agent mode, including model-variant handling and streamed tool-call assembly.

Two security issues remain open. An authenticated user can bypass per-model budget enforcement by requesting an alias that is normalized only after authorization checks, allowing use of a base model whose budget is exhausted. Streamed tool-call assembly can also consume excessive worker CPU for long outputs due to quadratic concatenation behavior; no issues have yet been addressed.

Open issues (2)

• High: Per-model budget bypass — litellm/proxy/response_api_endpoints/endpoints.py:136
• Low: Quadratic tool-call stream assembly — litellm/litellm_core_utils/streaming_chunk_builder_utils.py:271

Fixed/addressed: 0 · PR risk: 6/10

[veria-ai]

Low: Quadratic tool-call stream assembly

Each tuple concatenation copies all previously collected chunks, so assembling a tool call with many streamed argument deltas takes quadratic time and blocks the worker during end-of-stream processing. An authenticated caller can repeatedly request long tool-call outputs to consume proxy CPU; keep arguments and custom_input as lists, append each delta, and join once after collection.

[mateo-berri]

Quadratic tool-call stream assembly

Already fixed in #35826, which groups fragments once so assembly is linear. Leaving open to re-verify later