fix(bedrock): keep mid-conversation system messages in place on converse path by ege-arhan · Pull Request #41057 · BerriAI/litellm

ege-arhan · 2026-09-14T09:18:18Z

Problem

AmazonConverseConfig._transform_system_message hoists EVERY role:system message to the top-level Bedrock system block, regardless of position. A mid-conversation system reminder appended each turn mutates the system prefix every turn, so Bedrock implicit prompt caching (keyed on exact prompt prefix) never matches — full prefix re-written each request (Fixes #41043).

The Invoke path fixed the same bug class via #32578/#32831 (supports_mid_conversation_system flag, keep in place). The Converse path never got the same treatment.

Solution

Only the leading run of system entries is hoisted to the top-level system block. Mid-conversation system entries are converted to user turns in place (with an operator-note prefix, mirroring AnthropicMessagesTransformation._system_role_message_as_user), because Converse only accepts user/assistant roles in messages[]. This keeps the system prefix and everything before the entry byte-identical, preserving the implicit cache.

Details:

Leading system run behavior unchanged (hoist + cachePoint handling preserved).
Converted entries carry text + per-block cache_control over to the user turn.
Empty system entries (no text) are dropped, matching old extract-nothing behavior.
A system run wedged between an assistant tool-call turn and its tool-result turn is placed after the tool result (same ordering rule as the Invoke fix), so the tool call/result pair is never split.

Impact

bedrock/converse/* requests with per-turn system reminders now reuse the cached prefix instead of re-writing it.
Requests without mid-conversation system messages are unaffected (identical output).

Evidence

tests/test_litellm/llms/bedrock/chat/test_converse_transformation.py: 261 passed.
Behavior checks: leading hoist preserved; mid-conversation entry becomes in-place user turn; system blocks byte-identical across turns T1->T2; downstream _bedrock_converse_messages_pt emits only user/assistant roles; tool-call/result ordering preserved; cache_control carried over.

Fixes #41043.

chatgpt-codex-connector · 2026-09-14T09:18:23Z

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

CLAassistant · 2026-09-14T09:18:26Z

All committers have signed the CLA.

codspeed · 2026-09-14T09:20:25Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing ege-arhan:fix/issue-41043 (3dd2141) with main (071cb49)}

greptile-apps · 2026-09-14T09:22:02Z

Greptile Summary

This PR keeps mid-conversation Bedrock Converse system reminders in the message sequence instead of hoisting them into the cache-sensitive system prefix

Converts later system entries into operator-noted user turns while preserving text-block cache controls
Reorders reminders around assistant tool calls and tool results
Mishandles multiple consecutive tool results by moving the reminder after only the first result
Adds no regression coverage for the changed transformation

Confidence Score: 4/5

This PR is not safe to merge until multi-result tool ordering and the explicit regression-test and source-comment requirements are addressed

The reordering emits a postponed system reminder after the first of several tool results, so downstream conversion combines ordinary text with tool-result content in a Bedrock turn

Files Needing Attention: litellm/llms/bedrock/chat/converse_transformation.py

Important Files Changed

Filename	Overview
litellm/llms/bedrock/chat/converse_transformation.py	Changes system-message placement and cache handling, but breaks ordering when an assistant call has multiple consecutive tool results

_{Reviews (1): Last reviewed commit: "fix(bedrock): keep mid-conversation syst..." | Re-trigger Greptile}

greptile-apps · 2026-09-14T09:22:05Z

+        if self._opens_with_tool_result(message):
+            run: Final = self._system_run_before(messages, index)
+            prev_idx: Final = index - len(run) - 1
+            if run and prev_idx >= 0 and self._assistant_has_tool_calls(messages[prev_idx]):
+                return (message, *run)


Tool Results Are Split With consecutive results, this moves the note after the first, mixing text into Bedrock’s tool-result turn and causing rejection

greptile-apps · 2026-09-14T09:22:05Z

+        }
+        return cast(AllMessageValues, converted)
+
    def _transform_system_message(


Regression Test Is Missing This bug fix adds no regression test, violating the repository requirement to prevent this behavior from recurring unnoticed before merging

Context Used: CLAUDE.md (source)

greptile-apps · 2026-09-14T09:22:05Z

                cache_point["ttl"] = ttl
        return cache_point



Comments Violate Repository Policy This straightforward comment, plus similar comments at lines 1333 and 1366, violates the source-comment requirement and must be removed before merging

Context Used: CLAUDE.md (source)

veria-ai · 2026-09-14T09:24:09Z

+                        text_block["cache_control"] = m["cache_control"]
+                    text_blocks.append(text_block)
+        converted: dict = {
+            "role": "user",


Medium: Mid-conversation system instructions lose authority

Bedrock receives this content with the user role, and the downstream message processor merges it with adjacent user turns. For applications that add authorization or safety policy reminders during a conversation, an end user can now supply conflicting instructions at the same priority; the explanatory text does not restore system-role precedence. Preserve these entries in the top-level system field, reject unsupported mid-conversation system messages, or make this lossy conversion an explicit opt-in rather than changing their trust level by default.

veria-ai · 2026-09-14T09:24:18Z

PR overview

This pull request changes the Bedrock Converse transformation to retain mid-conversation system messages at their original position in the message sequence. It adjusts how these messages are represented when preparing requests for Bedrock.

One security concern remains open: these system instructions are sent with user-level authority and may be merged with nearby user turns. In applications that rely on mid-conversation messages for authorization or safety policy reminders, a user could provide conflicting instructions at the same priority, weakening policy enforcement; no issues have yet been addressed.

Open issues (1)

Medium: Mid-conversation system instructions lose authority — litellm/llms/bedrock/chat/converse_transformation.py:1325

Fixed/addressed: 0 · PR risk: 4/10

codecov · 2026-09-14T09:24:39Z

Codecov Report

❌ Patch coverage is 98.82353% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...tellm/llms/bedrock/chat/converse_transformation.py	98.82%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

…rse path

…C-2026-3865

…pyright budget

- non-dict guard in _opens_with_tool_result - in-place conversion without tool context - str/list cache_control preservation in mid-conversation path - drop unreachable non-system guard in hoisted loop

…edDict

mateo-berri · 2026-09-25T05:02:19Z

#42630 is merged and keeps mid-conversation system messages in place on the Converse path too, so closing this as agreed. Thanks for the fix.

ege-arhan requested a review from a team September 14, 2026 09:18

greptile-apps Bot reviewed Sep 14, 2026

View reviewed changes

veria-ai Bot reviewed Sep 14, 2026

View reviewed changes

ege-arhan force-pushed the fix/issue-41043 branch 5 times, most recently from 8fdd415 to cd51315 Compare September 19, 2026 19:11

Rainmemery mentioned this pull request Sep 21, 2026

fix(bedrock): convert mid-conversation system messages in place on the Converse path #41082

Closed

2 tasks done

ege-arhan added 5 commits September 22, 2026 13:34

fix(bedrock): keep mid-conversation system messages in place on conve…

503fd86

…rse path

fix: ruff format + multi tool_result order + regression test

44e1c22

fix: satisfy type-discipline gate + update osv ignore for mlflow PYSE…

9df6f31

…C-2026-3865

fix(bedrock): restore role narrowing in hoisted system loop for based…

cb4a852

…pyright budget

test(bedrock): cover mid-conversation system conversion branches

efef6f4

- non-dict guard in _opens_with_tool_result - in-place conversion without tool context - str/list cache_control preservation in mid-conversation path - drop unreachable non-system guard in hoisted loop

ege-arhan force-pushed the fix/issue-41043 branch from 5dc43fa to efef6f4 Compare September 22, 2026 13:35

ege-arhan added 2 commits September 22, 2026 13:51

Place type-discipline suppressions on the lines the gate scans

a0e98d8

Narrow hoisted loop to system role so basedpyright sees the right Typ…

3dd2141

…edDict

mateo-berri mentioned this pull request Sep 23, 2026

fix(anthropic): keep the replayed prefix byte-stable for preserved thinking on chat completions #42630

Merged

6 of 20 tasks

mateo-berri closed this Sep 25, 2026

Navigation Menu

Repository navigation

fix(bedrock): keep mid-conversation system messages in place on converse path - #41057

fix(bedrock): keep mid-conversation system messages in place on converse path#41057
ege-arhan wants to merge 7 commits into
BerriAI:mainfrom
ege-arhan:fix/issue-41043

ege-arhan commented Sep 14, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 14, 2026

Uh oh!

CLAassistant commented Sep 14, 2026 •

edited

Loading

Uh oh!

codspeed Bot commented Sep 14, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Sep 14, 2026

Important Files Changed

Uh oh!

greptile-apps Bot Sep 14, 2026

Uh oh!

greptile-apps Bot Sep 14, 2026

Uh oh!

greptile-apps Bot Sep 14, 2026

Uh oh!

veria-ai Bot Sep 14, 2026

Uh oh!

veria-ai Bot commented Sep 14, 2026

Uh oh!

codecov Bot commented Sep 14, 2026 •

edited

Loading

Uh oh!

mateo-berri commented Sep 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ege-arhan commented Sep 14, 2026

Problem

Solution

Impact

Evidence

Uh oh!

chatgpt-codex-connector Bot commented Sep 14, 2026

Uh oh!

CLAassistant commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed Bot commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

greptile-apps Bot commented Sep 14, 2026

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Uh oh!

greptile-apps Bot Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

veria-ai Bot Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

veria-ai Bot commented Sep 14, 2026

PR overview

Open issues (1)

Uh oh!

codecov Bot commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mateo-berri commented Sep 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

CLAassistant commented Sep 14, 2026 •

edited

Loading

codspeed Bot commented Sep 14, 2026 •

edited

Loading

codecov Bot commented Sep 14, 2026 •

edited

Loading