feat(chat_models): record the response id and chain calls with use_previous_response_id

[Akhil Jasson (Akhil-Jasson)]

Fixes #305

Summary

Adds full Responses API response-id chaining for ChatLiteLLM and ChatLiteLLMRouter: Responses API replies now expose LiteLLM’s upstream resp_... id in AIMessage.response_metadata["id"] for sync, async, streaming, and non-streaming calls. Adds use_previous_response_id=True, which finds the most recent response id in message history, sends it as previous_response_id, omits already server-stored history, and forces the request through LiteLLM’s Responses API route; also raises the LiteLLM minimum to 1.102.0, the first version that returns the upstream id.

Verification

• python -m compileall -q langchain_litellm tests/unit_tests/test_litellm.py tests/unit_tests/test_router.py
• git diff --check

Added focused unit coverage for direct and Router chaining, response-id propagation, and streamed response ids.

[Akshay Dongare (Akshay-Dongare)]

Thanks, the id on every path of both classes and the 1.102.0 floor are right: the new id assertions fail on litellm 1.101.0 and pass on 1.102.0. use_previous_response_id needs these before I can merge.

Blocking

1. Chained turns lose the system prompt. litellm's Responses bridge sends a leading system message with str content as instructions, and instructions are not carried over by previous_response_id (see the instructions docstring on openai's ResponseCreateParams). The trim in _last_messages_with_response_id (litellm.py:131) removes it, so from turn 2 on the model runs without its system prompt. ChatOpenAI gets away with the same trim only because it sends system messages as stored input items. Keep every message in the trimmed prefix that _convert_message_to_dict turns into role system with str content (SystemMessage and ChatMessage(role="system")), ahead of the kept turns. List content already goes as a stored input item, so leave that out.
2. Only chain on resp_ ids. This PR puts an id on every reply, so a Chat Completions reply in the history (chatcmpl-..., or '' from an OpenAI-compatible stream) becomes previous_response_id, and everything before it is dropped. Match ChatOpenAI: chain only when isinstance(response_id, str) and response_id.startswith("resp_"), otherwise keep walking back.
3. Refuse Router groups that don't share a response store. Two deployments with different api_base, two Azure regions, or OpenAI plus Azure all pass the Responses check in _route_to_responses_api (litellm_router.py:295), and a chained call can land on a deployment that never stored the response: 12 to 15 of 20 chained calls in an offline repro. litellm's responses_api_deployment_check can't help, since the bridge returns the decoded id. Raise ValueError before any request. _group_endpoint with _reasoning_endpoint already answers "one endpoint and set of credentials?", but evaluate it on params without previous_response_id: that key isn't in _NON_ROUTING_PARAMS, so with it present every call reads as unknown reach and even a single deployment would be refused.

Also needed

4. Refusals name the wrong flag. With only use_previous_response_id set, errors read use_responses_api=True, but ... (litellm.py:1610, litellm_router.py:307, 315, 323, 336). Name the flag the caller set and a fix that keeps chaining; the Router's "without it, ... still reaches the Responses API" hint means giving chaining up.

5. The flag per call, via bind() or in model_kwargs is not consumed. It is never popped, so it lands in the Chat Completions request body and nothing is trimmed. Pop it in _merge_call_params and raise, saying to set it on the model (ChatOpenAI reads the field only). Restore the deleted sentence that use_responses_api is read from the call, model_kwargs or the field, since it's still true, and scope the new sentence (litellm.py:1636) to the field.

6. A caller-supplied previous_response_id doesn't match the trim. A per-call id is sent while the history is trimmed at the history's id, and a model_kwargs id loses to the history's. Pick one rule for all sources: a caller id turns the trim off, or the history's id always wins.

7. Refuse fallbacks when chaining on ChatLiteLLM. With fallbacks in model_kwargs, the fallback model gets only the trimmed turn plus a previous_response_id it can't use. The Router already refuses this through _unknown_reach.

8. Set the stream id once. Drop the four setdefault("id", ...) blocks and their comment (litellm.py:2002, 2106, litellm_router.py:583, 696). litellm keeps one id per stream, so the comment doesn't hold, and after a Router mid-stream fallback the per-chunk ids merge into chatcmpl-PRIMARYresp_BACKUP.... Put the id on the finish_reason chunk so a fallback's id wins, record it only when truthy, and give the reply id a different name from the request's response_id.

9. Chained replies drop their encrypted reasoning items. previous_response_id makes _reasoning_endpoint return None, so _keep_reasoning_items discards them. Leave out only the id the connector set when computing it; adding the key to _NON_ROUTING_PARAMS would start re-sending items for callers who pass their own id with full history.

10. Add use_previous_response_id to _identifying_params next to use_responses_api (litellm.py:2384). Today an LLM cache serves a flag-off reply to a flag-on call with the same history.

11. Docs. The field docstring says it "always routes the call through the Responses API" (litellm.py:1446); on ChatLiteLLMRouter it checks the deployments instead, so say so there as use_responses_api does. Also state: system messages are re-sent, only resp_ ids chain, chaining needs stored responses (store=False breaks it), and Router groups across stores are refused.

12. Keep the 1.101.0 streamed-cost reason in the pyproject.toml comment beside the new one; it still holds.

13. Tests, through serve_http, on both classes:

    • chaining on stream and astream (today, removing the trim from all four stream methods leaves the suite green);
    • payload["instructions"] on a chained turn, including ChatMessage(role="system");
    • a chatcmpl- reply after a resp_ one, an empty-string id, a history with no id, and the flag off;
    • the two-store Router refusal, plus a single-deployment group that still chains;
    • refusal messages that name use_previous_response_id;
    • a stream whose chunks carry two ids, merging to exactly one of them.

    test_stream_keeps_a_responses_api_id mocks one hand-built chunk, so it never merges anything and passes on 1.101.0. Send it through the bridge.

14. Retitle to feat(chat_models): record the response id and chain calls with use_previous_response_id. The title becomes the changelog line, and chat model features are scoped chat_models.

[Akhil Jasson (Akhil-Jasson)]

I’ve updated the Responses API chaining implementation in both ChatLiteLLM and ChatLiteLLMRouter to address the review feedback. The changes preserve string-based system instructions across chained turns, only chain on valid resp_ IDs, and reject Router groups whose deployments may use different response stores. They also handle caller-supplied IDs and unsupported per-call flag usage consistently, prevent incompatible fallbacks, preserve encrypted reasoning items, and record the final streamed response ID without merging IDs from different deployments.
I’ve updated the identifying parameters and documentation, retained the existing LiteLLM 1.101.0 streamed-cost compatibility note, and added regression coverage for sync, async, streaming, and router behavior. I also updated the PR title to match the requested changelog format. Please take another look when you have a chance.

[Akhil Jasson (Akhil-Jasson)]

Hi Akshay Dongare (@Akshay-Dongare), could you go through the PR whenever possible and let me know if there are any other changes to be made?