fix(bedrock): trigger Nova Sonic generation on response.create so realtime sessions stop hanging

[mateo-berri]

Relevant issues

Fixes the Nova Sonic realtime hang: on bedrock/amazon.nova-sonic-v1:0 via /v1/realtime, every text session hung forever because response.create was translated to nothing, so the model never generated and Bedrock eventually raised ValidationException ("The following prompts were not closed" or "Timed out waiting for input events") after the client gave up

Linear ticket

LIT-2239

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Run against a live proxy hitting the real Bedrock API in us-east-1 with bedrock/amazon.nova-sonic-v1:0 (mode: realtime); no mocks. The same minimal OpenAI realtime websocket client (session.update with text modality, one user text item, then response.create) was run once against the merge-base and once against this PR, each on a fresh proxy

qa_client.py

import asyncio
import json
import sys

import websockets


async def main(port: str) -> None:
    url = f"ws://localhost:{port}/v1/realtime?model=bedrock-sonic"
    headers = {"Authorization": "Bearer sk-qa-1234"}
    async with websockets.connect(url, additional_headers=headers) as ws:
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "instructions": "You are a helpful assistant.",
                "modalities": ["text"],
            },
        }))
        await ws.send(json.dumps({
            "type": "conversation.item.create",
            "item": {
                "type": "message",
                "role": "user",
                "content": [{"type": "input_text", "text": "Say hello in one sentence."}],
            },
        }))
        await ws.send(json.dumps({"type": "response.create"}))

        received = 0
        text = ""
        while True:
            try:
                raw = await asyncio.wait_for(ws.recv(), timeout=45)
            except asyncio.TimeoutError:
                print(f"TIMED OUT after 45s, received {received} events")
                return
            received += 1
            event = json.loads(raw)
            event_type = event.get("type")
            print(f"event: {event_type}")
            if event_type == "response.text.delta":
                text += event.get("delta", "")
            elif event_type == "response.text.done":
                print(f"  text: {event.get('text', text)}")
            elif event_type == "response.done":
                print(f"final assistant text: {text}")
                return


asyncio.run(main(sys.argv[1]))

The proxy config (qa_config.yaml):

model_list:
  - model_name: "bedrock-sonic"
    litellm_params:
      model: bedrock/amazon.nova-sonic-v1:0
      aws_region_name: us-east-1
    model_info:
      mode: realtime

general_settings:
  master_key: "sk-qa-1234"

Before (merge-base e141596)

set -a; source .env; set +a
.venv/bin/litellm --config qa_config.yaml --port 49781 --detailed_debug > proxy_before.log 2>&1 &
.venv/bin/python qa_client.py 49781

Client output:

TIMED OUT after 45s, received 0 events

About a minute after the client gave up, Bedrock rejected the still-open prompt in proxy_before.log:

17:32:50 - LiteLLM Proxy:DEBUG: handler.py:254 - Bedrock to client forwarding ended: RequestId=c7b0665b-1124-40a3-ae7b-04be45eadee3 : Error(s):
Error 1 : The following prompts were not closed: [1e74e3a4-1f3b-401e-b6bd-18dfa2df7ac3]
Traceback (most recent call last):
  File ".../litellm/llms/bedrock/realtime/handler.py", line 207, in _forward_bedrock_to_client
    result = await output[1].receive()
  File ".../smithy_aws_event_stream/aio/__init__.py", line 154, in receive
    raise value
aws_sdk_bedrock_runtime.models.ValidationException: RequestId=c7b0665b-1124-40a3-ae7b-04be45eadee3 : Error(s):
Error 1 : The following prompts were not closed: [1e74e3a4-1f3b-401e-b6bd-18dfa2df7ac3]

After (merged, staging tip c4f28ce containing PR head 40c8338)

set -a; source .env; set +a
.venv/bin/litellm --config qa_config.yaml --port 53655 --detailed_debug > proxy_after.log 2>&1 &
.venv/bin/python qa_client.py 53655

Client output (74 consecutive event: response.audio.delta lines elided for readability, nothing else changed):

event: response.created
event: response.output_item.added
event: response.content_part.added
event: response.text.delta
event: response.text.done
  text: Hello there! I'm ready to chat with you. How can I assist you today?
event: response.content_part.done
event: response.output_item.done
event: response.created
event: response.output_item.added
event: response.content_part.added
[... 74 response.audio.delta events ...]
event: response.audio.done
event: response.content_part.done
event: response.output_item.done
event: response.done
final assistant text: Hello there! I'm ready to chat with you. How can I assist you today?

After the client disconnected, proxy_after.log shows a clean shutdown with no ValidationException at all: Client to Bedrock forwarding ended: (1000, ''), then the graceful close messages ending in {"event": {"sessionEnd": {}}}, then Bedrock Realtime: Bedrock stream ended

Type

🐛 Bug Fix

Changes

The old transform_response_create_event returned [] with a comment claiming Bedrock starts generating automatically. That is wrong for Nova Sonic v1: the model only starts generating after it detects user speech in an audio content block, so a text-only session (session.update, conversation.item.create, response.create) never produced a response. Verified against a live bidirectional stream: interactive USER text alone, text plus silent audio, and promptEnd all fail to trigger generation on amazon.nova-sonic-v1:0; promptEnd just closes the prompt without a response, and the prompt is rejected outright if it contains no audio content at all

The working pattern (the same one Pipecat's Nova Sonic integration uses for its assistant response trigger) is to speak to the model. response.create now opens the prompt's audio content block and streams a short pre-rendered spoken "ready" utterance (16kHz PCM generated with Amazon Polly, embedded in trigger_audio.py) padded with leading and trailing silence, which makes Nova Sonic run ASR, detect end of turn, and respond to the pending interactive text input. Sessions where the client streams its own audio (input_audio_buffer.append) are untouched; for those, response.create stays a no-op and Nova Sonic's built-in turn detection applies, tracked via a client_audio_streamed flag

On the inbound side, Nova Sonic v1 never emits a promptEnd output event (verified live; the docs' terminal event is completionEnd, which v1 also does not send per turn), so clients previously never received response.done even when generation happened. The transformation now emits response.done and resets per-response state when a contentEnd arrives with stopReason: END_TURN, and the inbound dispatch also accepts completionEnd as an end-of-response signal. The handler additionally closes the session gracefully when the client disconnects (contentEnd for open audio, promptEnd, sessionEnd via the new session_close_messages), which stops the ValidationException noise Bedrock used to raise after every session, and it treats a None read from the Bedrock stream as a normal end of stream instead of crashing on an AttributeError

Following review feedback, the transformation also tracks the sample rate each audio content block was declared with. If the trigger block was opened at 16kHz and the client later streams its own audio on a session configured for a different input rate (e.g. 8kHz G.711), input_audio_buffer.append closes the trigger block, rotates to a new content name, and opens a fresh block declared at the client's configured rate, so client audio is never sent into a block with a mismatched sample rate. In the handler, the disconnect cleanup flushes the graceful close messages and closes the Bedrock input stream in independent contextlib.suppress(Exception) blocks, so a failed flush can never skip the stream close

Regression tests in tests/test_litellm/llms/bedrock/realtime/test_bedrock_realtime_transformation.py cover: response.create emitting the audio contentStart plus the exact trigger PCM, a second response.create reusing the open audio content, response.create staying a no-op before session.update and when the client streams its own audio, session_close_messages ordering, response.done emission on END_TURN (and not on PARTIAL_TURN), and the sample-rate rotation (g711_ulaw reopens the block at 8kHz, pcm16 keeps reusing it). A new tests/test_litellm/llms/bedrock/realtime/test_bedrock_realtime_handler.py covers the handler paths: a client disconnect flushes the graceful close sequence to Bedrock and closes the input stream, nothing is sent when no session was started, the input stream is still closed when the close flush itself fails, and a None read from the Bedrock stream ends the loop and closes the client WebSocket. All of these fail on the previous behavior

---

Note

Medium Risk
Changes bidirectional realtime protocol mapping and synthetic audio injection for Nova Sonic; behavior is nuanced but scoped to Bedrock realtime with dedicated regression tests.

Overview
Fixes Nova Sonic realtime text sessions that hung because response.create previously produced no Bedrock input. For text-only flows (no client mic audio), response.create now streams a short pre-rendered “ready” utterance plus silence via trigger_audio.py, so the model detects speech and answers pending interactive text. If the client already sends input_audio_buffer.append, response.create stays a no-op.

Inbound mapping now emits OpenAI response.done when Bedrock sends contentEnd with stopReason: END_TURN, and treats completionEnd like promptEnd. On disconnect, the handler flushes session_close_messages (contentEnd / promptEnd / sessionEnd), handles a None Bedrock read as normal EOS, and closes the client socket in a finally block. input_audio_buffer.append can close and reopen the audio block when the trigger used 16 kHz but the session’s input format differs (e.g. G.711 8 kHz).

Regression coverage added in test_bedrock_realtime_handler.py and expanded transformation tests for trigger audio, session close, and END_TURN.

^{Reviewed by Cursor Bugbot for commit 828adff. Bugbot is set up for automated code reviews on this repo. Configure here.}

[mateo-berri]

@greptileai

[greptile-apps]

Greptile Summary

This PR fixes a hang in Nova Sonic v1 realtime text sessions by injecting a short pre-rendered "ready" audio utterance on response.create, triggering the model's built-in voice activity detection to start generation. It also adds proper response.done emission on END_TURN contentEnd, graceful session close on client disconnect (contentEnd / promptEnd / sessionEnd), a None-stream guard in the Bedrock reader, and sample-rate rotation when the 16 kHz trigger block must give way to client audio at a different rate (e.g., G.711 8 kHz).

• Trigger audio: transform_response_create_event now opens an audio content block and streams a gzip-compressed, LRU-cached PCM blob padded with leading (0.5 s) and trailing (3 s) silence; the path is gated on prompt_started and client_audio_streamed so audio-streaming clients are unaffected.
• Inbound mapping: response.done is emitted when contentEnd carries stopReason: END_TURN; completionEnd is treated equivalently to promptEnd; a None read from the Bedrock output stream breaks the loop cleanly instead of crashing.
• Disconnect cleanup: the client-to-Bedrock task now sends session_close_messages() via independent contextlib.suppress blocks and unconditionally closes input_stream, eliminating the upstream ValidationException noise that appeared after every session.

Confidence Score: 5/5

Safe to merge; all changes are scoped to the Bedrock realtime handler, the fix is verified against the live API, and the new tests cover the complete set of scenarios introduced.

The trigger-audio injection, END_TURN response.done mapping, graceful session close, and sample-rate rotation are all well-reasoned and match the live-API behaviour documented in the PR. Cleanup paths use independent suppress blocks so a failed flush never prevents stream closure. Tests are mock-only and cover the full matrix of new code paths including the regression scenarios from the bug report.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/llms/bedrock/realtime/transformation.py	Core change: transform_response_create_event now emits a trigger audio block for text-only sessions; END_TURN contentEnd emits response.done; sample-rate rotation for G.711 clients; session_close_messages added. Logic is well-encapsulated and all edge cases are unit-tested.
litellm/llms/bedrock/realtime/handler.py	Extracted send_to_bedrock helper; graceful session close on disconnect using contextlib.suppress; None-stream guard in _forward_bedrock_to_client; client WebSocket closed in finally block. Cleanup ordering is sound.
litellm/llms/bedrock/realtime/trigger_audio.py	New file: gzip+base64 pre-rendered 16 kHz PCM blob with an lru_cache(maxsize=1) decoder. Decompression cost is paid only once per process lifetime. Data is inert audio bytes with no network calls or code execution.
tests/test_litellm/llms/bedrock/realtime/test_bedrock_realtime_transformation.py	Comprehensive mock-only tests: trigger PCM content and byte-exact payload, second response.create reuse, client-audio no-op guard, sample-rate rotation (G.711 and PCM16), session close ordering, END_TURN vs PARTIAL_TURN response.done discrimination.
tests/test_litellm/llms/bedrock/realtime/test_bedrock_realtime_handler.py	New handler-level mock tests using a stub_aws_models monkeypatch fixture: disconnect flushes session close sequence, pre-session disconnect sends nothing, close flush continues after partial send failure, stream close on flush failure, None stream read closes client WebSocket. No real network calls.
ruff-strict-budget.json	Budget file updated (limits adjusted to reflect new code); no functional change.

_{Reviews (6): Last reviewed commit: "fix(bedrock): suppress bedrock close sen..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR fixes a Nova Sonic realtime hang by replacing the no-op transform_response_create_event with an injected spoken "ready" utterance (pre-rendered Polly PCM) that triggers Nova Sonic's VAD and end-of-turn detection in text-only sessions, and by emitting response.done on END_TURN contentEnd events rather than waiting for the never-emitted promptEnd.

• response.create now sends a pcm16 trigger audio clip (0.5 s leading silence + Polly phrase + 3 s trailing silence) when no client audio has been streamed, causing Nova Sonic to detect end of turn and respond to the pending text input.
• session_close_messages was added to gracefully send contentEnd/promptEnd/sessionEnd on client disconnect, stopping ValidationException noise from Bedrock.
• The inbound handler now accepts completionEnd alongside promptEnd, and emits response.done on contentEnd with stopReason: END_TURN.

Confidence Score: 3/5

The trigger audio content block inherits session-negotiated codec fields rather than hardcoding the pcm16 format of the trigger clip, which would send malformed audio declarations to Bedrock for non-pcm16 sessions.

The audioInputConfiguration in transform_response_create_event uses self.input_media_type, self.input_sample_size_bits, and self.input_encoding from the negotiated session codec, while the trigger audio is always 16 kHz 16-bit linear PCM. For g711 sessions, Bedrock receives a content block declaring mulaw encoding but containing pcm16 bytes. All new tests use the default pcm16 config, leaving this path unverified. The handler refactoring and response.done logic are correct and well-covered.

litellm/llws/bedrock/realtime/transformation.py — specifically transform_response_create_event, where the trigger audio contentStart inherits session codec parameters instead of hardcoded pcm16 values.

Important Files Changed

Filename	Overview
litellm/llws/bedrock/realtime/transformation.py	Core logic change: response.create now injects a pcm16 trigger audio clip, but audioInputConfiguration inherits the session-negotiated codec, causing a format mismatch for non-pcm16 sessions.
litellm/llws/bedrock/realtime/handler.py	Clean refactoring: extracts send_to_bedrock closure, adds graceful session_close_messages on disconnect, moves ws.close() to finally block, handles None reads from Bedrock stream.
litellm/llws/bedrock/realtime/trigger_audio.py	New file: embeds a gzip-compressed base64-encoded pcm16 Polly utterance; lru_cache(maxsize=1) ensures single decompression per process.
tests/test_litellm/llws/bedrock/realtime/test_bedrock_realtime_transformation.py	Adds comprehensive mock-only unit tests for the new response.create trigger logic, session_close_messages ordering, and response.done emission.

_{Reviews (2): Last reviewed commit: "fix(bedrock): trigger Nova Sonic generat..." | Re-trigger Greptile}

[mateo-berri]

@greptileai

---

Generated by Claude Code

[mateo-berri]

@greptileai

---

Generated by Claude Code

[mateo-berri]

@greptileai

---

Generated by Claude Code

[mateo-berri]

bugbot run

---

Generated by Claude Code

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Disconnect flush aborts mid-sequence
  • Moved contextlib.suppress inside the for loop so a failed send of one Bedrock close event (e.g. contentEnd or promptEnd) no longer aborts the remaining promptEnd/sessionEnd events, with a new regression test that fails only on promptEnd and asserts sessionEnd still reaches Bedrock.

_{You can send follow-ups to the cloud agent here.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ mateo-berri
❌ cursoragent
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[mateo-berri]

@greptileai

---

Generated by Claude Code

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 40c8338. Configure here.}