test(realtime): record and replay websocket traffic in redis vcr cassettes

[mateo-berri]

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

The realtime_translation_testing CI job always hit the live provider realtime APIs because vcrpy only intercepts HTTP; every realtime test printed [VCR NOOP] played=0 entries=0. This adds a WebSocket layer to the existing Redis VCR so realtime sessions record once and then replay deterministically for 24h with zero provider traffic, and re-record live after the TTL lapses to catch provider drift

Both proof runs below were captured at commit 6a78485b41 on branch litellm_ws_vcr_realtime_cassettes, against the live OpenAI realtime API (real spend, no mocks), with a throwaway single-purpose redis for CASSETTE_REDIS_URL. xAI realtime tests skip because no XAI_API_KEY was set

Cold run (empty cassette store, records live)

$ redis-cli -p 56399 flushall
$ CASSETTE_REDIS_URL="redis://127.0.0.1:56399/0" LITELLM_VCR_VERBOSE=1 \
    python -m pytest tests/llm_translation/realtime/ -q --timeout=120

[WS-VCR MISS] recorded sessions=1 frames=42 :: test_realtime_guardrails_openai.py::test_clean_text_message_passes_through_to_openai
[WS-VCR MISS] recorded sessions=1 frames=1  :: test_openai_realtime.py::test_openai_realtime_direct_call_no_intent
[WS-VCR MISS] recorded sessions=1 frames=1  :: test_openai_realtime.py::test_openai_realtime_direct_call_with_intent
[WS-VCR MISS] recorded sessions=1 frames=1  :: test_openai_realtime_simple.py::TestOpenAIRealtime::test_realtime_connection
[WS-VCR MISS] recorded sessions=1 frames=1  :: test_openai_realtime_simple.py::TestOpenAIRealtime::test_realtime_with_query_params
[WS-VCR MISS] recorded sessions=1 frames=39 :: test_openai_realtime_simple.py::TestOpenAIRealtime::test_send_user_message
[WS-VCR MISS] recorded sessions=1 frames=41 :: test_realtime_guardrails_openai.py::test_text_message_blocked_by_guardrail_no_ai_response
13 passed, 3 skipped in 89.24s (0:01:29)

Warm run (same cassettes, replays), executed with a hard DNS block on api.openai.com / api.x.ai to prove zero provider traffic while the API key stays set so no test skips

$ CASSETTE_REDIS_URL="redis://127.0.0.1:56399/0" LITELLM_VCR_VERBOSE=1 \
    python -m pytest tests/llm_translation/realtime/ -q --timeout=120 -p ws_netguard

[WS-VCR HIT] sessions=1 frames=42 :: test_realtime_guardrails_openai.py::test_clean_text_message_passes_through_to_openai
[WS-VCR HIT] sessions=1 frames=1  :: test_openai_realtime.py::test_openai_realtime_direct_call_no_intent
[WS-VCR HIT] sessions=1 frames=1  :: test_openai_realtime.py::test_openai_realtime_direct_call_with_intent
[WS-VCR HIT] sessions=1 frames=1  :: test_openai_realtime_simple.py::TestOpenAIRealtime::test_realtime_connection
[WS-VCR HIT] sessions=1 frames=1  :: test_openai_realtime_simple.py::TestOpenAIRealtime::test_realtime_with_query_params
[WS-VCR HIT] sessions=1 frames=39 :: test_openai_realtime_simple.py::TestOpenAIRealtime::test_send_user_message
[WS-VCR HIT] sessions=1 frames=41 :: test_realtime_guardrails_openai.py::test_text_message_blocked_by_guardrail_no_ai_response
====================== WS-NETGUARD PROVIDER NETWORK PROOF ======================
  0 provider DNS lookups attempted (api.openai.com / api.x.ai never resolved)
13 passed, 3 skipped in 73.52s (0:01:13)

Every recorded frame count is reproduced exactly on replay (42, 1, 1, 1, 1, 39, 41), the run is faster, and the network guard confirms the provider hosts were never resolved. The residual wall time is dominated by fixed asyncio.sleep calls inside the existing test bodies, not provider round-trips

Cassette keys and TTLs after the warm replay, showing the distinct wscassette prefix and that TTL is set on write and deliberately not refreshed on read, so each cassette lapses ~24h after its last recording and the next run past that point re-records live

$ redis-cli -p 56399 keys 'litellm:vcr:wscassette:*' | while read k; do echo "$(redis-cli -p 56399 ttl $k)s  $k"; done
86292s  litellm:vcr:wscassette:tests/llm_translation/realtime/test_realtime_guardrails_openai.py/test_text_message_blocked_by_guardrail_no_ai_response
86207s  litellm:vcr:wscassette:tests/llm_translation/realtime/test_realtime_guardrails_openai.py/test_clean_text_message_passes_through_to_openai
86210s  litellm:vcr:wscassette:tests/llm_translation/realtime/test_openai_realtime.py/test_openai_realtime_direct_call_no_intent
86276s  litellm:vcr:wscassette:tests/llm_translation/realtime/test_openai_realtime_simple.py/TestOpenAIRealtime/test_realtime_connection
86279s  litellm:vcr:wscassette:tests/llm_translation/realtime/test_openai_realtime_simple.py/TestOpenAIRealtime/test_realtime_with_query_params
86287s  litellm:vcr:wscassette:tests/llm_translation/realtime/test_openai_realtime_simple.py/TestOpenAIRealtime/test_send_user_message
86274s  litellm:vcr:wscassette:tests/llm_translation/realtime/test_openai_realtime.py/test_openai_realtime_direct_call_with_intent

The guardrail test_text_message_blocked_by_guardrail_no_ai_response is a known live flake; it passed on this cold run and its cassette was saved, so the warm run replays it deterministically. When a test fails live it saves nothing (the save-on-pass gate), which keeps any prior cassette intact and re-records next time; that path plus the replay timeout, contract-drift, and scrub-on-save behavior are covered by tests/llm_translation/test_ws_vcr.py

The harness's own 17 unit tests, run against fakeredis with no cassette context

$ python -m pytest tests/llm_translation/test_ws_vcr.py -q
17 passed in 0.51s

Type

✅ Test

Changes

tests/_ws_vcr.py intercepts the websockets.connect boundary, which both realtime paths funnel through (the guardrail tests call websockets.connect directly for the backend socket they hand to RealTimeStreaming, and the SDK path connects internally inside OpenAIRealtime.async_realtime), so no litellm/ source change is needed. A per-test autouse fixture in tests/llm_translation/realtime/conftest.py patches websockets.connect for the duration of each test

On a cache miss the wrapper passes through to the real server and logs every frame in order with its direction, a text/binary flag, and, for each server frame, the count of client frames seen before it. On a cache hit a fake connection serves the recorded server frames under that causal gate, releasing each only once the client has sent the recorded number of frames, and matches each client frame against the recording with volatile fields (event/item/response/session ids, timestamps) normalized away. A structurally different client frame raises contract drift with a diff, and every replay wait is bounded by a timeout so a mismatch surfaces loudly instead of hanging CI

Persistence reuses the existing cassette Redis client, 24h TTL, save-on-pass, best-effort degradation counters, and session-end banner from tests/_vcr_redis_persister.py, under a distinct litellm:vcr:wscassette: key keyed by the test node path. Auth material is never stored: connection headers are dropped and stored frame text is scrubbed of bearer tokens and API keys. The layer disables cleanly to live passthrough when LITELLM_VCR_DISABLE=1 or CASSETTE_REDIS_URL is unset. tests/llm_translation/test_ws_vcr.py adds 17 unit tests covering cassette round-trip, the causal replay gate, volatile-tolerant matching that accepts id drift and rejects structural drift, secret scrubbing, TTL-on-save, no-save-on-fail, and replay timeout raising instead of hanging

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR adds a WebSocket-level VCR layer (tests/_ws_vcr.py) so that realtime tests record live provider sessions once and replay them deterministically from Redis, eliminating the live API calls that vcrpy (HTTP-only) was always missing.

• tests/_ws_vcr.py implements recording, causal-gate replay, volatile-field normalization, and secret scrubbing entirely within the test tree, reusing the existing Redis persister's TTL and save-on-pass semantics under a distinct litellm:vcr:wscassette: key prefix.
• tests/llm_translation/realtime/conftest.py wires an autouse _ws_vcr fixture that patches websockets.connect per-test, and tests/llm_translation/conftest.py adds the new unit-test file to the VCR auto-marker skip list.
• tests/llm_translation/test_ws_vcr.py covers the harness with 17 unit tests using fakeredis (no live network calls).

Confidence Score: 5/5

Safe to merge; changes are entirely within the test tree with no production code touched.

All changes live in tests/; no litellm source files are modified. The implementation is well-structured, end-to-end proof runs are provided in the PR description (cold and warm runs with frame-exact counts and a DNS-block network guard), and 17 unit tests cover the harness with fakeredis. The two minor observations do not affect current test correctness.

No files require special attention.

Important Files Changed

Filename	Overview
tests/_ws_vcr.py	New WebSocket VCR harness: record/replay sessions via Redis with causal gating, secret scrubbing, and save-on-pass semantics. Well-structured; no blocking issues found.
tests/llm_translation/realtime/conftest.py	New autouse fixture that patches websockets.connect per-test, loads/saves cassettes from Redis, and re-raises replay errors on pass. Correctly inherits pytest_runtest_makereport from the parent conftest.
tests/llm_translation/test_ws_vcr.py	17 unit tests covering cassette round-trip, causal gate, volatile-tolerant matching, secret scrubbing, TTL-on-save, no-save-on-fail, and replay timeout. All use fakeredis with no live network calls.
tests/llm_translation/conftest.py	Adds test_ws_vcr.py to the VCR auto-marker skip list so the unit tests don't run under a live cassette context; otherwise unchanged.

_{Reviews (4): Last reviewed commit: "fix(realtime): warn instead of silently ..." | Re-trigger Greptile}

[codspeed-hq]

Merging this PR will not alter performance

✅ 30 untouched benchmarks

---

_{Comparing litellm_ws_vcr_realtime_cassettes (5ee826b) with litellm_internal_staging (d6cbf6e)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (bcd5275) during the generation of this report, so d6cbf6e was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[mateo-berri]

the asyncio concern is moot since pyproject sets asyncio_mode = "auto"

[mateo-berri]

@greptileai

[mateo-berri]

a symmetric normalization quirk in the ID regex (which does not break matching)

fixing this -> the reward is epsilon (symmetric normalization already neutralizes over-matching) while any tightening carries real replay-flake risk, e.g., all-letter ids

a theoretical multi-waiter fragility that cannot be triggered by the existing single-consumer usage

rewriting proven concurrency plumbing for an untriggerable case invites real regressions

[mateo-berri]

@greptileai