Skip to content

fix(litellm): serialize named tool_choice with flat Responses API shape - #2962

Closed
jtstothard wants to merge 1 commit into
vectorize-io:mainfrom
jtstothard:fix/litellm-named-tool-choice-responses
Closed

fix(litellm): serialize named tool_choice with flat Responses API shape#2962
jtstothard wants to merge 1 commit into
vectorize-io:mainfrom
jtstothard:fix/litellm-named-tool-choice-responses

Conversation

@jtstothard

Copy link
Copy Markdown

Problem

LiteLLMLLM.call_with_tools built the named tool_choice with the nested Chat Completions shape and referenced a field that doesn't exist on the LLMToolChoice dataclass.

call_kwargs["tool_choice"] = {
    "type": "function",
    "function": {"name": tool_choice.selected_function_name},  # AttributeError — field is 'function_name'
}

Two bugs:

  1. selected_function_name is not an attribute of LLMToolChoice (the field is function_name) — would raise AttributeError on any named-tool reflect iteration.
  2. The nested {"function": {"name": ...}} shape is rejected by LiteLLM's Responses API path (Copilot/Luna). Reflect routed through a Copilot model via LiteLLM fails end-to-end: LiteLLM raises a validation error before the request reaches the provider, and the reflect run times out / 422s.

Fix

Serialize named tool_choice to the flat Responses API shape and reference the correct field:

call_kwargs["tool_choice"] = (
    {"type": "function", "name": tool_choice.function_name}
    if tool_choice.mode is LLMToolChoiceMode.NAMED
    else tool_choice.mode.value
)

This matches the shape already used by the Codex provider (codex_llm.py).

Verification

Before the fix, reflect on a ~27k-memory bank routed through Copilot/Luna via LiteLLM never succeeded — only 422 (validation) and 504 (timeout) responses. After applying the fix locally:

  • Reflect returns HTTP 200 end-to-end.
  • Latency on a 26k-memory bank: ~123s (under the 300s wall-clock limit).

Scope

Single-file, 3-line change. No new dependencies, no API surface changes.

Refs #2953

The LiteLLM provider built named tool_choice using the nested Chat
Completions shape {'type':'function','function':{'name':...}} and referenced
a nonexistent 'selected_function_name' attribute. Both break the Responses
API path used by Copilot/Luna-backed reflect: LiteLLM rejects the nested
shape before the request reaches the provider, and the attribute access
would raise AttributeError if reached.

Use the flat {'type':'function','name':...} shape (matching the Codex
provider in codex_llm.py) and reference the actual dataclass field
'function_name'.

Reflect was timing out / 422-ing end-to-end on any bank routed through a
Copilot/Luna model via LiteLLM. Verified working post-fix: reflect returns
HTTP 200 on a 26k-memory bank.

Refs vectorize-io#2953
@jtstothard

Copy link
Copy Markdown
Author

Closing this PR — it's based on a false premise

I need to withdraw this. On independent review I got the diagnosis wrong.

Claim 1 was false. I asserted selected_function_name is "a nonexistent field" that "would raise AttributeError." It isn't — it's a validated @property on LLMToolChoice (llm_interface.py:44-48) that five other providers use correctly (openai_compatible_llm.py:1197, gemini_llm.py:751, codex_llm.py:829, claude_code_llm.py:509). The original code was idiomatic.

Claim 2 is questionable. The only real change here is the shape: nested {"function":{"name":...}} → flat {"name":...}. I justified this by pointing at codex_llm.py, but Codex talks directly to the Responses API while LiteLLM normalizes provider shapes itself — so matching Codex's shape may be the wrong fix for the LiteLLM path. PR #2957's argument (that this belongs in LiteLLM's Responses bridge) is probably correct.

What actually fixed reflect on our deployment was a broader local container patch that changed call_with_tools' signature, not this minimal diff. I conflated the two when writing this PR and didn't verify the dataclass before opening it.

Apologies for the noise. I'll re-investigate #2953 properly before submitting anything further.

@jtstothard jtstothard closed this Jul 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant