Repository navigation
feat(playground): resume calls and replay audio - #3320
Conversation
Realtime voice calls were one-shot: a saved call could not be picked up
again, and assistant speech was played once and discarded.
- Accumulate the assistant's PCM deltas per turn, encode to WAV, and
store them base64 on the transcript entries so each response gets a
Play button live and in history.
- Add "Continue call", which replays a saved call's turns to the model
as conversation items and appends the new turns back onto the same
history row rather than creating a second one.
- Widen PATCH /realtime-history/{id} from rename-only to also append
transcript turns and sum duration/usage server-side.
- Keep the call controls pinned while the transcript scrolls, and
follow the newest turn only when already scrolled to the bottom.
Co-Authored-By: Claude <noreply@anthropic.com>
WalkthroughThe realtime playground now stores assistant WAV audio with transcript turns, supports replaying and continuing saved calls, appends new turns and usage to existing history, and provides per-turn audio playback. The API, database schema, client hooks, UI, and related tests were updated. ChangesRealtime history audio and continuation
Estimated code review effort: 4 (Complex) | ~60 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6ff3a740eb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ...(body.appendTranscript | ||
| ? { transcript: [...existing.transcript, ...body.appendTranscript] } | ||
| : {}), |
There was a problem hiding this comment.
When the same saved call is continued from two tabs or devices and both PATCH requests overlap, each request reads the same original row and then replaces the transcript, duration, and usage using that stale snapshot. The later update therefore discards the other continuation's turns and counters; use atomic JSONB/numeric update expressions or lock the row within a transaction.
Useful? React with 👍 / 👎.
| responseActiveRef.current = false; | ||
| audioChunksRef.current.clear(); | ||
| audioBytesRef.current = 0; |
There was a problem hiding this comment.
When the user ends or navigates away during an assistant response, the PCM collected since the last response.done is cleared here before it is encoded and attached to the transcript. The text is still persisted, but the final assistant turn permanently loses its replay audio; finalize the pending chunks before clearing them during call teardown.
Useful? React with 👍 / 👎.
| for (const entry of seed ?? []) { | ||
| if (entry.status === "partial" || !entry.text.trim()) { | ||
| continue; |
There was a problem hiding this comment.
After barge-in, the original session is truncated to playedMs, but the transcript entry is only marked interrupted and can still contain output that was queued but never heard. This filter skips only partial entries, so a continuation sends the entire interrupted transcript back as output_text, giving the new session context that the user never heard and that the prior session explicitly removed; preserve a truncated version or avoid replaying that unheard tail.
Useful? React with 👍 / 👎.
| reset(); | ||
| seedRef.current = seed?.length ? seed : null; | ||
| if (seed?.length) { | ||
| setTranscript(seed); |
There was a problem hiding this comment.
Each continued session loads all previously saved WAVs into seed while starting its audio counter from zero, allowing another 24 MiB to be appended every time the call is resumed. Repeated continuations can therefore create an unbounded JSONB row and increasingly large history GET responses despite the stated ceiling; initialize the budget from the seeded audio or enforce a cumulative server-side limit.
Useful? React with 👍 / 👎.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
apps/api/src/routes/playground.ts (2)
1145-1152: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy liftUnbounded cumulative growth of the transcript jsonb row.
Per-entry
base64is capped at ~48 MB, but nothing caps the accumulated transcript. Each continuation appends another session's worth of WAV base64 to the same row, and the PATCH handler below reads the whole transcript into memory and rewrites it, so a repeatedly-continued call grows the row plus every read/write payload without limit. Consider a server-side ceiling on total stored audio bytes per call (or offloading audio to blob storage keyed from the transcript entry).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/api/src/routes/playground.ts` around lines 1145 - 1152, The transcript schema’s optional audio field allows unbounded cumulative WAV growth across continuations. Add a server-side per-call limit for total stored audio bytes and enforce it in the PATCH/continuation handler before appending or rewriting the transcript, rejecting updates that exceed the limit while preserving existing per-entry validation.
1435-1487: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winMake realtime-history append updates atomic.
durationSecondsandusageare currently recomputed from the pre-loaded row, so a retried/overlapping patch can lose updates. Use SQL increments/aggregation for these fields and let the.returning()404 confirm the row exists without keeping a separate pre-read.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/api/src/routes/playground.ts` around lines 1435 - 1487, Make the realtime-history update in the PATCH handler atomic by removing the preliminary select and computing durationSeconds and usage from their current database values within the update expression. Preserve transcript/title updates, use SQL increments/aggregation for addDurationSeconds and addUsage, and rely on the update’s returning() result to throw the existing 404 when no authorized row exists.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/api/src/routes/playground-realtime-history.spec.ts`:
- Around line 139-142: Update the test around patchCall so it verifies
ownership: create a playground call under a second user, then attempt to PATCH
that existing call using the first user’s token and assert the 404 response.
Keep the test name aligned with this cross-user ownership scenario and retain
the existing missing-row case separately if needed.
In `@apps/playground/src/hooks/use-realtime-call.ts`:
- Around line 380-408: Add explicit error handling for seed-replay
conversation.item.create events in the continued-call flow around seedRef and
sendEvent. Track replay failures from Realtime error events, log or surface the
failure, and prevent the session from being treated as healthy or transitioning
to "live" after a failed replay; preserve the existing skip rules and successful
replay behavior.
---
Nitpick comments:
In `@apps/api/src/routes/playground.ts`:
- Around line 1145-1152: The transcript schema’s optional audio field allows
unbounded cumulative WAV growth across continuations. Add a server-side per-call
limit for total stored audio bytes and enforce it in the PATCH/continuation
handler before appending or rewriting the transcript, rejecting updates that
exceed the limit while preserving existing per-entry validation.
- Around line 1435-1487: Make the realtime-history update in the PATCH handler
atomic by removing the preliminary select and computing durationSeconds and
usage from their current database values within the update expression. Preserve
transcript/title updates, use SQL increments/aggregation for addDurationSeconds
and addUsage, and rely on the update’s returning() result to throw the existing
404 when no authorized row exists.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: fc8beeac-6dd8-4467-92f7-232a9423867f
📒 Files selected for processing (9)
apps/api/src/routes/playground-realtime-history.spec.tsapps/api/src/routes/playground.tsapps/playground/src/components/playground/call-history-list.tsxapps/playground/src/components/playground/realtime-page-client.tsxapps/playground/src/hooks/use-realtime-call.tsapps/playground/src/hooks/usePlaygroundHistory.tsapps/playground/src/lib/realtime-audio.spec.tsapps/playground/src/lib/realtime-audio.tspackages/db/src/schema.ts
| // Replay a continued call's prior turns before going live, so the | ||
| // model has the earlier conversation in context. Fire-and-forget | ||
| // is safe: microphone chunks are dropped until the status flips | ||
| // to "live" below, so nothing can race ahead of the seed. | ||
| // Only the text is replayed — the realtime API cannot accept | ||
| // assistant audio items — and the content type is the role's | ||
| // side of the GA enum: input_text for user, output_text for | ||
| // assistant. | ||
| const seed = seedRef.current; | ||
| seedRef.current = null; | ||
| for (const entry of seed ?? []) { | ||
| if (entry.status === "partial" || !entry.text.trim()) { | ||
| continue; | ||
| } | ||
| sendEvent({ | ||
| type: "conversation.item.create", | ||
| item: { | ||
| type: "message", | ||
| role: entry.role, | ||
| content: [ | ||
| { | ||
| type: | ||
| entry.role === "user" ? "input_text" : "output_text", | ||
| text: entry.text, | ||
| }, | ||
| ], | ||
| }, | ||
| }); | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
OpenAI Realtime API conversation.item.create assistant message content type output_text
💡 Result:
In the OpenAI Realtime API, when using the conversation.item.create event to add a message, the content type used for text messages differs based on the role of the message sender [1][2]. For conversation.item.create, the valid content types are [1][3]: - For user role messages: Use the input_text content type [4][1][2]. - For system role messages: Use the input_text content type [1][2]. - For assistant role messages: The Realtime API has a limitation regarding conversation.item.create in that it cannot be used to populate assistant audio messages [5][6][7]. While assistant messages generally support text content type in other contexts, the conversation.item.create event is primarily designed for populating history with user, system, or function-related items [6][1][8]. The output_text type is generally associated with model-generated output and is not typically used as a valid content type for a client-sent conversation.item.create event [2][3]. When manually creating a message item, ensure you are using the appropriate input_text type for user/system inputs [4][1].
Citations:
- 1: https://github.com/openai/openai-node/blob/5436f42d/src/resources/beta/realtime/realtime.ts
- 2: https://openai.github.io/openai-agents-js/openai/agents/realtime/type-aliases/realtimemessageitem/
- 3: https://docs.rs/openai-openapi-types/latest/openai_openapi_types/struct.RealtimeConversationItemContent.html
- 4: https://developers.openai.com/api/docs/guides/realtime-conversations
- 5: https://github.com/openai/openai-python/blob/5e8f09c2/src/openai/types/realtime/conversation_item_create_event.py
- 6: https://github.com/openai/openai-python/blob/5e8f09c2/src/openai/resources/realtime/realtime.py
- 7: https://docs.rs/openai-openapi-types/latest/openai_openapi_types/struct.RealtimeClientEventConversationItemCreate.html
- 8: https://github.com/openai/openai-node/blob/e228aaad/src/resources/beta/realtime/realtime.ts
🌐 Web query:
OpenAI Realtime conversation.item.create assistant message output_text documentation
💡 Result:
The conversation.item.create event in the OpenAI Realtime API is used to add items—such as messages, function calls, or function call responses—to a conversation's context [1][2]. Regarding the use of assistant messages with output_text: 1. Limitation on Assistant Messages: Official documentation and SDK definitions state that the conversation.item.create event has a current limitation: it cannot be used to populate assistant audio messages [1][3][4]. 2. Structure for Text Content: When constructing a message item, the content is defined as an array of objects [5]. If you are working with text output, the content object typically includes a type field (e.g., 'output_text') and a text field for the actual string content [5]. 3. Intended Use: While the client can send conversation.item.create to populate conversation history or add mid-stream items, this event is generally intended for user messages or function outputs [1][2]. When the model generates a response (which may include assistant messages with text), it is triggered by the server in response to other events (such as response.create) or user input [6][2]. In summary, while you can structure messages with text content in your application code [5], you cannot use conversation.item.create to "inject" an assistant-role message into the conversation history as if it were a model-generated turn [1][4]. The system is designed for the model to generate assistant responses automatically based on the conversation context [6].
Citations:
- 1: https://github.com/openai/openai-python/blob/5e8f09c2/src/openai/types/realtime/conversation_item_create_event.py
- 2: https://developers.openai.com/api/reference/ruby/resources/realtime
- 3: https://docs.rs/openai-openapi-types/latest/openai_openapi_types/struct.RealtimeClientEventConversationItemCreate.html
- 4: https://github.com/openai/openai-node/blob/5436f42d/src/resources/beta/realtime/realtime.ts
- 5: https://github.com/openai/openai-node/blob/e228aaad/src/resources/realtime/realtime.ts
- 6: https://developers.openai.com/api/docs/guides/realtime-conversations
🌐 Web query:
site:openai.com/developers realtime conversation.item.create assistant output_text
💡 Result:
OpenAI’s Realtime API docs say conversation.item.create can add messages to the conversation, and the created item can be an assistant message with content.type set to output_text when the session is configured for text output. The same docs note it cannot populate assistant audio messages. (platform.openai.com)
If you want, I can also pull the exact JSON shape for a working conversation.item.create assistant text item.
Citations:
Replay error handling for seed-created turns.
The input_text / output_text pairing is the current Realtime shape for created message text content, but seed-replay conversation.item.create events are fire-and-forget; a failed replay still leaves session.updated / live unchanged unless error events are handled. Add logging or explicit handling for failed replay items so session.updated is not treated as a healthy continued call.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/playground/src/hooks/use-realtime-call.ts` around lines 380 - 408, Add
explicit error handling for seed-replay conversation.item.create events in the
continued-call flow around seedRef and sendEvent. Track replay failures from
Realtime error events, log or surface the failure, and prevent the session from
being treated as healthy or transitioning to "live" after a failed replay;
preserve the existing skip rules and successful replay behavior.
Address review findings on call continuation: - Serialize concurrent PATCH continuations of the same saved call with a row lock so neither overwrites the other's appended turns - Finalize buffered assistant audio during call teardown so ending mid-response keeps the final turn's replay clip - Count seeded audio toward the per-call audio budget so repeated continuations cannot grow one saved conversation without bound - Exercise the cross-user ownership path in the PATCH 404 test Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
apps/api/src/routes/playground.ts (1)
1145-1153: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winBound the realtime call request bodies.
transcriptandappendTranscriptonly enforcemin(1), so a single POST/PATCH can contain arbitrarily many entries, each allowing up to 48MB of base64 audio. Add a reasonable max transcription entry count and/or an aggregate audio byte budget in the server schema before storing the joined result.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/api/src/routes/playground.ts` around lines 1145 - 1153, Update the realtime call request schema in playground route to bound transcript and appendTranscript payloads before persistence: add a reasonable maximum entry count and enforce an aggregate audio-size budget across their optional audio.base64 values, while preserving the existing per-entry validation and joined-result behavior.Source: Coding guidelines
🧹 Nitpick comments (2)
apps/api/src/routes/playground.ts (2)
1395-1409: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win
addUsageaccepts negative/fractional counters.
realtimeUsageSchemauses barez.number(), so a client can PATCH negative or fractional token counts that are summed into the stored totals.addDurationSecondsis correctly guarded with.int().min(0); consider a nonnegative-int variant for the additive usage payload.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/api/src/routes/playground.ts` around lines 1395 - 1409, Update the addUsage validation in the PATCH schema around realtimeUsageSchema so every additive usage counter is constrained to nonnegative integers, matching the existing addDurationSeconds validation. Preserve the optional-field behavior and ensure invalid negative or fractional usage values are rejected before totals are updated.
1439-1449: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winLocked read pulls the whole transcript (incl. base64 audio) even when not appending.
tx.select()fetchestranscript— potentially tens of MB of WAV base64 — for every PATCH, including title-only renames and usage/duration-only updates. Narrow the projection and only pulltranscriptwhenbody.appendTranscriptis present (or append in SQL viatranscript || $new::jsonb), which also lets the row-locked read stay cheap.♻️ Sketch: conditional projection
- const [existing] = await tx - .select() + const [existing] = await tx + .select({ + id: tables.playgroundRealtimeHistory.id, + durationSeconds: tables.playgroundRealtimeHistory.durationSeconds, + usage: tables.playgroundRealtimeHistory.usage, + ...(body.appendTranscript + ? { transcript: tables.playgroundRealtimeHistory.transcript } + : {}), + }) .from(tables.playgroundRealtimeHistory)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/api/src/routes/playground.ts` around lines 1439 - 1449, Update the transaction query in the PATCH handler around the playgroundRealtimeHistory lookup to avoid selecting the full transcript for non-append updates: use a narrow projection for metadata and include transcript only when body.appendTranscript is present, or append the new entries in SQL while holding the row lock. Preserve the existing locked-row update behavior for title, usage, and duration changes.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@apps/api/src/routes/playground.ts`:
- Around line 1145-1153: Update the realtime call request schema in playground
route to bound transcript and appendTranscript payloads before persistence: add
a reasonable maximum entry count and enforce an aggregate audio-size budget
across their optional audio.base64 values, while preserving the existing
per-entry validation and joined-result behavior.
---
Nitpick comments:
In `@apps/api/src/routes/playground.ts`:
- Around line 1395-1409: Update the addUsage validation in the PATCH schema
around realtimeUsageSchema so every additive usage counter is constrained to
nonnegative integers, matching the existing addDurationSeconds validation.
Preserve the optional-field behavior and ensure invalid negative or fractional
usage values are rejected before totals are updated.
- Around line 1439-1449: Update the transaction query in the PATCH handler
around the playgroundRealtimeHistory lookup to avoid selecting the full
transcript for non-append updates: use a narrow projection for metadata and
include transcript only when body.appendTranscript is present, or append the new
entries in SQL while holding the row lock. Preserve the existing locked-row
update behavior for title, usage, and duration changes.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 3305284f-22a1-407b-a24f-106eb19f0231
📒 Files selected for processing (3)
apps/api/src/routes/playground-realtime-history.spec.tsapps/api/src/routes/playground.tsapps/playground/src/hooks/use-realtime-call.ts
2 participants
Realtime voice calls were one-shot: a saved call could not be picked up again, and assistant speech was played once and discarded. This adds a Continue call action that replays a saved call's turns to the model as conversation items and appends the new turns back onto the same history row, plus per-turn WAV capture of the assistant's audio so every response gets a Play button both live and in history.
PATCH /realtime-history/{id}is widened from rename-only to also append transcript turns and sum duration/usage server-side, covered by a new API spec; the schema change is type-only, sopnpm migrationsgenerates no SQL. The call controls are now pinned while only the transcript scrolls, following the newest turn only when already scrolled to the bottom.Note: this has not been exercised against a live provider call —
pnpm build,pnpm lintand the unit/API tests all pass, but the seeded-replay item shape and audio playback are verified by tests and code review only.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes
Tests