fix(litellm): serialize named tool_choice with flat Responses API shape - #2962
fix(litellm): serialize named tool_choice with flat Responses API shape#2962jtstothard wants to merge 1 commit into
Conversation
The LiteLLM provider built named tool_choice using the nested Chat
Completions shape {'type':'function','function':{'name':...}} and referenced
a nonexistent 'selected_function_name' attribute. Both break the Responses
API path used by Copilot/Luna-backed reflect: LiteLLM rejects the nested
shape before the request reaches the provider, and the attribute access
would raise AttributeError if reached.
Use the flat {'type':'function','name':...} shape (matching the Codex
provider in codex_llm.py) and reference the actual dataclass field
'function_name'.
Reflect was timing out / 422-ing end-to-end on any bank routed through a
Copilot/Luna model via LiteLLM. Verified working post-fix: reflect returns
HTTP 200 on a 26k-memory bank.
Refs vectorize-io#2953
Closing this PR — it's based on a false premiseI need to withdraw this. On independent review I got the diagnosis wrong. Claim 1 was false. I asserted Claim 2 is questionable. The only real change here is the shape: nested What actually fixed reflect on our deployment was a broader local container patch that changed Apologies for the noise. I'll re-investigate #2953 properly before submitting anything further. |
Problem
LiteLLMLLM.call_with_toolsbuilt the namedtool_choicewith the nested Chat Completions shape and referenced a field that doesn't exist on theLLMToolChoicedataclass.Two bugs:
selected_function_nameis not an attribute ofLLMToolChoice(the field isfunction_name) — would raiseAttributeErroron any named-tool reflect iteration.{"function": {"name": ...}}shape is rejected by LiteLLM's Responses API path (Copilot/Luna). Reflect routed through a Copilot model via LiteLLM fails end-to-end: LiteLLM raises a validation error before the request reaches the provider, and the reflect run times out / 422s.Fix
Serialize named
tool_choiceto the flat Responses API shape and reference the correct field:This matches the shape already used by the Codex provider (
codex_llm.py).Verification
Before the fix, reflect on a ~27k-memory bank routed through Copilot/Luna via LiteLLM never succeeded — only 422 (validation) and 504 (timeout) responses. After applying the fix locally:
Scope
Single-file, 3-line change. No new dependencies, no API surface changes.
Refs #2953