Repository navigation
fix(responses): keep tool_result next to tool_use on Anthropic previous_response_id continuations - #45322
Merged
Conversation
…us_response_id continuations Replayed history no longer re-adds each stored turn's instructions, and the current request's instructions go before the replayed history instead of between the last tool_use and its tool_result
Contributor
Author
|
bugbot run |
Contributor
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Contributor
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…in the continuation regression test
Contributor
Author
Contributor
Author
|
bugbot run |
Contributor
Author
|
bugbot run |
Contributor
Author
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Carried instructions split tool pairs
- Session replay now prepends only the first stored instructions as a leading system message, so later instructed turns can no longer land a system message between tool_use and tool_result.
Or push these changes by commenting:
@cursor push b85d69c5c8
Preview (b85d69c5c8)
diff --git a/litellm/responses/litellm_completion_transformation/session_handler.py b/litellm/responses/litellm_completion_transformation/session_handler.py
--- a/litellm/responses/litellm_completion_transformation/session_handler.py
+++ b/litellm/responses/litellm_completion_transformation/session_handler.py
@@ -53,6 +53,7 @@
"""
from litellm.responses.litellm_completion_transformation.transformation import (
ChatCompletionSession,
+ LiteLLMCompletionResponsesConfig,
)
verbose_proxy_logger.debug("inside get_chat_completion_message_history_for_previous_response_id")
@@ -72,21 +73,33 @@
| ChatCompletionResponseMessage
| Message
] = []
+ carried_instructions: str | None = None # rebind-ok: first instructed spend log wins
for spend_log in all_spend_logs:
- chat_completion_message_history = (
- await ResponsesSessionHandler.extend_chat_completion_message_with_spend_log_payload(
- spend_log=spend_log,
- chat_completion_message_history=chat_completion_message_history,
- carry_over_instructions=carry_over_instructions,
- )
+ (
+ chat_completion_message_history,
+ log_instructions,
+ ) = await ResponsesSessionHandler.extend_chat_completion_message_with_spend_log_payload(
+ spend_log=spend_log,
+ chat_completion_message_history=chat_completion_message_history,
)
+ if carried_instructions is None and log_instructions:
+ carried_instructions = log_instructions
+ messages_with_carried_instructions: Final = (
+ [
+ LiteLLMCompletionResponsesConfig.transform_instructions_to_system_message(carried_instructions),
+ *chat_completion_message_history,
+ ]
+ if carry_over_instructions and carried_instructions
+ else chat_completion_message_history
+ )
+
verbose_proxy_logger.debug(
"chat_completion_message_history %s",
- json.dumps(chat_completion_message_history, indent=4, default=str),
+ json.dumps(messages_with_carried_instructions, indent=4, default=str),
)
return ChatCompletionSession(
- messages=chat_completion_message_history,
+ messages=messages_with_carried_instructions,
litellm_session_id=litellm_session_id,
)
@@ -100,7 +113,6 @@
| ChatCompletionResponseMessage
| Message
],
- carry_over_instructions: bool,
):
"""
Extend the chat completion message history with the spend log payload
@@ -113,6 +125,10 @@
spend_log=spend_log,
)
response_input_param: str | ResponseInputParam | None = None
+ raw_instructions: Final = (
+ None if proxy_server_request_dict is None else proxy_server_request_dict.get("instructions")
+ )
+ log_instructions: Final = raw_instructions if isinstance(raw_instructions, str) and raw_instructions else None
############################################################
# Add Input messages for this Spend Log
@@ -133,7 +149,7 @@
chat_completion_message_history.extend(
LiteLLMCompletionResponsesConfig.transform_responses_api_input_to_messages(
input=response_input_param,
- responses_api_request=(proxy_server_request_dict or {}) if carry_over_instructions else {},
+ responses_api_request={},
replay_reasoning=True,
)
)
@@ -149,7 +165,7 @@
if hasattr(choice, "message"):
_normalize_redacted_tool_call_arguments(choice.message)
chat_completion_message_history.append(choice.message)
- return chat_completion_message_history
+ return chat_completion_message_history, log_instructions
@staticmethod
async def get_proxy_server_request_from_spend_log(
diff --git a/tests/unit/responses/litellm_completion_transformation/test_anthropic_responses_bridge.py b/tests/unit/responses/litellm_completion_transformation/test_anthropic_responses_bridge.py
--- a/tests/unit/responses/litellm_completion_transformation/test_anthropic_responses_bridge.py
+++ b/tests/unit/responses/litellm_completion_transformation/test_anthropic_responses_bridge.py
@@ -247,3 +247,62 @@
assert _message_shapes(request) == _EXPECTED_MESSAGES
assert [block.text for block in request.system] == [_FIRST_TURN_INSTRUCTIONS]
+
+
+_SECOND_TURN: Final = {
+ "request_id": "chatcmpl-second-turn",
+ "call_type": "aresponses",
+ "session_id": "session-1",
+ "proxy_server_request": {
+ "model": _MODEL,
+ "input": [{"type": "function_call_output", "call_id": "toolu_weather", "output": "47C"}],
+ "instructions": _FIRST_TURN_INSTRUCTIONS,
+ },
+ "response": {
+ "id": "chatcmpl-second-turn",
+ "object": "chat.completion",
+ "created": 0,
+ "model": _MODEL,
+ "choices": [
+ {
+ "index": 0,
+ "finish_reason": "stop",
+ "message": {"role": "assistant", "content": "47C in Tokyo."},
+ }
+ ],
+ },
+}
+
+
+@pytest.mark.asyncio
+async def test_omitted_instructions_after_a_tool_roundtrip_keep_tool_use_next_to_tool_result() -> None:
+ """
+ Replaying two instructed turns (question -> tool_use, then tool_result -> answer) and omitting
+ instructions on the follow-up must not insert a system message between tool_use and tool_result
+ """
+ anthropic: Final = _RecordingAnthropicMessages()
+ with patch("litellm.proxy.proxy_server.prisma_client", _FakePrismaClient([_FIRST_TURN, _SECOND_TURN])):
+ await litellm.aresponses(
+ model="anthropic/claude-haiku-4-5",
+ previous_response_id="chatcmpl-second-turn",
+ input="Thanks. What about Osaka?",
+ tools=[
+ {
+ "type": "function",
+ "name": "get_weather",
+ "parameters": {"type": "object", "properties": {"city": {"type": "string"}}},
+ }
+ ],
+ api_key="sk-ant-fake",
+ client=AsyncHTTPHandler(transport=httpx.MockTransport(anthropic)),
+ )
+
+ assert anthropic.request is not None
+ assert _message_shapes(anthropic.request) == [
+ ("user", [("text", "What is the weather in Tokyo?")]),
+ ("assistant", [("tool_use", "toolu_weather")]),
+ ("user", [("tool_result", "toolu_weather")]),
+ ("assistant", [("text", "47C in Tokyo.")]),
+ ("user", [("text", "Thanks. What about Osaka?")]),
+ ]
+ assert [block.text for block in anthropic.request.system] == [_FIRST_TURN_INSTRUCTIONS]
diff --git a/tests/unit/responses/litellm_completion_transformation/test_session_handler.py b/tests/unit/responses/litellm_completion_transformation/test_session_handler.py
--- a/tests/unit/responses/litellm_completion_transformation/test_session_handler.py
+++ b/tests/unit/responses/litellm_completion_transformation/test_session_handler.py
@@ -787,3 +787,101 @@
tool_message: Final = result["messages"][2]
assert json.loads(assistant_message["tool_calls"][0]["function"]["arguments"]) == function_arguments
assert json.loads(tool_message["content"]) == function_output
+
+
+def _message_role(message: object) -> str | None:
+ if isinstance(message, dict):
+ role: Final = message.get("role")
+ return role if isinstance(role, str) else None
+ role = getattr(message, "role", None)
+ return role if isinstance(role, str) else None
+
+
+def _message_content(message: object) -> object:
+ if isinstance(message, dict):
+ return message.get("content")
+ return getattr(message, "content", None)
+
+
+def _has_tool_calls(message: object) -> bool:
+ if isinstance(message, dict):
+ return bool(message.get("tool_calls"))
+ return bool(getattr(message, "tool_calls", None))
+
+
+@pytest.mark.asyncio
+async def test_carried_instructions_are_a_single_leading_system_message_across_a_tool_roundtrip() -> None:
+ first_turn: Final = {
+ "request_id": "chatcmpl-first-turn",
+ "call_type": "aresponses",
+ "session_id": "session-1",
+ "proxy_server_request": {
+ "input": "What is the weather in Tokyo?",
+ "instructions": "Be terse.",
+ },
+ "response": {
+ "id": "chatcmpl-first-turn",
+ "object": "chat.completion",
+ "created": 0,
+ "model": "claude-haiku-4-5",
+ "choices": [
+ {
+ "index": 0,
+ "finish_reason": "tool_calls",
+ "message": {
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [
+ {
+ "id": "toolu_weather",
+ "type": "function",
+ "function": {"name": "get_weather", "arguments": '{"city": "Tokyo"}'},
+ }
+ ],
+ },
+ }
+ ],
+ },
+ }
+ second_turn: Final = {
+ "request_id": "chatcmpl-second-turn",
+ "call_type": "aresponses",
+ "session_id": "session-1",
+ "proxy_server_request": {
+ "input": [{"type": "function_call_output", "call_id": "toolu_weather", "output": "47C"}],
+ "instructions": "Be terse.",
+ },
+ "response": {
+ "id": "chatcmpl-second-turn",
+ "object": "chat.completion",
+ "created": 0,
+ "model": "claude-haiku-4-5",
+ "choices": [
+ {
+ "index": 0,
+ "finish_reason": "stop",
+ "message": {"role": "assistant", "content": "47C in Tokyo."},
+ }
+ ],
+ },
+ }
+
+ with patch.object( # test-quality-ok: the handler has no DI seam for the spend-log fetch; every test in this file stubs this same boundary
+ ResponsesSessionHandler,
+ "get_all_spend_logs_for_previous_response_id",
+ new_callable=AsyncMock,
+ ) as mock_get_spend_logs:
+ mock_get_spend_logs.return_value = [first_turn, second_turn]
+ result: Final = await ResponsesSessionHandler.get_chat_completion_message_history_for_previous_response_id(
+ "chatcmpl-second-turn",
+ carry_over_instructions=True,
+ )
+
+ messages: Final = result["messages"]
+ roles: Final = [_message_role(message) for message in messages]
+ assert roles[0] == "system"
+ assert _message_content(messages[0]) == "Be terse."
+ assert roles.count("system") == 1
+ tool_use_index: Final = next(index for index, message in enumerate(messages) if _has_tool_calls(message))
+ assert roles[tool_use_index] == "assistant"
+ assert roles[tool_use_index + 1] == "tool"You can send follow-ups to the cloud agent here.
…inuation sends none Replaying each spend log's own instructions put a system message between a replayed tool_use and its tool_result, which Anthropic rejects with a 400
Contributor
Author
|
bugbot run |
Contributor
Author
Contributor
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit bb51cae. Configure here.
yuneng-berri
approved these changes
Oct 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Problem this solves:
/v1/responsesfollow-ups withprevious_response_idfail with a 400instructionsand reaches the proxy worker that served the first turn, so it hits most follow-ups on a multi-worker proxy and every follow-up on a single-worker oneHow it solves it:
instructionsreplace the earlier turns' instructions and go firstinstructionskeeps the latest earlierinstructions, sent once at the top instead of once per replayed turntool_useagainIntentional product change: a follow-up that sends
instructionsnow replaces the earlier turns' instructions instead of adding its own behind them. This follows OpenAI's docs, where instructions from a previous response are not carried over withprevious_response_id. A follow-up that sends none keeps the earlier instructions, where OpenAI drops them: that is the one deliberate difference, so a client that never resendsinstructionskeeps its system promptUser Flow
Before: a developer running a tool-calling agent on Claude through the Responses API cannot send the tool result back
instructions, and aget_weathertoolcompletedwith afunction_calland acall_idliketoolu_01...previous_response_id, afunction_call_outputfor thatcall_id, and the sameinstructionstool_useids were found withouttool_resultblocks immediately after" whenever the follow-up reaches the proxy worker that served the first turnAfter: the same follow-up comes back with the model's answer using the tool result
instructions, and aget_weathertoolcompletedwith afunction_calland acall_idliketoolu_01...previous_response_id, afunction_call_outputfor thatcall_id, and the sameinstructionscompletedresponse whose text uses the tool resultAffected release
Regression since v1.104.0, introduced by #42630, which keeps a later system message inside the Anthropic conversation instead of hoisting it into the top-level system prompt. The continuation put the new
instructionssystem message between the replayedtool_useand its tool result, so it split them apartPre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Both legs ran the same three-turn flow through the OpenAI Python SDK (2.33.0), the client the e2e test that caught this uses, against the real Anthropic API. Each leg booted two proxy instances from its commit, each with
--num_workers 2 --use_v2_migration_resolver, sharing one fresh Postgres database, with this config:sdk_flow.py <turn1_and_2_url> <turn3_url> <model> <trials>is byte-identical in both legs. Turn 1 asks for the weather withinstructionsand a tool. Turn 2 sends the tool result withprevious_response_idand the sameinstructions, which is the flow that broke. Turn 3 asks a follow-up with noinstructions, the case Bugbot flagged on an earlier commit of this PR. Theinstructionsask for every reply to end in BANANA, so a turn 3 ending in BANANA shows the earlierinstructionsstill applyThe bug needs the follow-up to reach the worker that served the tool call, so Cases A and B keep every turn on one instance (2 workers) and Case C sends turn 3 to the other instance.
claude-haiku-4-5is the model the e2e test uses, andclaude-sonnet-5-5is the latest SonnetBefore (057034d, merge base)
After (bb51cae)
Turn 2 failed 11 of 14 trials before and 0 of 14 after. Turn 3 failed 1 of the 3 trials that reached it before and 0 of 14 after, and every turn 3 after still ended in BANANA
Type
Bug Fix
Test
Caveats (if any)
Low
instructionskeeps the latest earlierinstructions, unlike OpenAIinstructionskeeps its system promptinstructions, only the latest applies. Before, every turn's set was replayed in placestore_prompts_in_spend_logsoff, aprevious_response_idfollow-up replays no historylitellm.responses()path still ignoresprevious_response_idQA runbook
previous_response_idand the follow-up uses the tool resultANTHROPIC_API_KEY, a database, andstore_prompts_in_spend_logs: trueso the proxy can rebuild the historyanthropic/claude-haiku-4-5deployment and generate a key for itinstructions, theget_locationstool, andtool_choice: "required"get_locationsfunction call with acall_idand at least one locationprevious_response_id, afunction_call_outputof "Distinctive forecast: 47 degrees Celsius" for thatcall_id, and the sameinstructionstool_useids withouttool_resultblocksFinal Attestation
Note
Medium Risk
Changes how multi-turn Responses API sessions merge instructions and messages before provider conversion; wrong ordering could still break Anthropic tool rounds or alter system-prompt behavior on continuations.
Overview
Fixes Anthropic 400 errors on Responses API follow-ups that use
previous_response_idwith afunction_call_outputand resendinstructions, where a system message was inserted between replayedtool_useandtool_resultblocks.Session replay now recovers
instructionsfrom spend-log proxy requests and passes the current request’sinstructionsintoasync_responses_api_session_handler. When merging history, system prompts are hoisted ahead of replayed turns (not interleaved with tool messages). Newinstructionsreplace prior turns’ system text (OpenAI-style); omittinginstructionsreuses the latest stored ones from the session.ResponsesSessionHandlerprefetches proxy requests per spend log, simplifies input replay (input/messages), and stops re-injecting instructions while rebuilding message history from logs.Adds unit tests that assert Anthropic request shape for tool continuations and re-enables the related e2e Anthropic multi-turn test.
Reviewed by Cursor Bugbot for commit bb51cae. Bugbot is set up for automated code reviews on this repo. Configure here.