test(realtime): assert guardrail block on backend wire traffic instead of model refusal wording

[mateo-berri]

Relevant issues

No GitHub issue; fixes the recurring realtime_translation_testing CI flake that turns unrelated PRs red, e.g. job 2009316 on #32380

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

All runs below hit the real OpenAI realtime API (wss://api.openai.com/v1/realtime?model=gpt-realtime) with no mocks, billing real tokens

Before (CI): 11 of the 13 failed realtime_translation_testing runs between 2026-06-24 and 2026-07-07 fail on this exact assertion, each with a differently worded refusal that the safe_markers list does not contain (jobs 1930076, 1937431, 1951307, 1954778, 1956799, 1959775, 1965566, 1972283, 1978635, 2001790, 2009316). From job 2009316, PR #32380 at commit 64c4cfe:

AssertionError: AI responded with non-guardrail content even though message was blocked:
"Sorry, I can't comply with that request. Let me know if there's something else I can help you with."

The list had already been extended for the same reason in #28191, #28200 and #29477; the model's live phrasing keeps drifting past it

Before (local, old test at litellm_internal_staging commit 46d9742): the flake tracks the model's current phrasing distribution and did not reproduce tonight, so the CI numbers above are the reproduction evidence

$ for i in $(seq 1 25); do .venv/bin/python -m pytest "tests/llm_translation/realtime/test_realtime_guardrails_openai.py::test_text_message_blocked_by_guardrail_no_ai_response" -q --timeout=120 >/dev/null 2>&1 && echo "run $i: PASS" || echo "run $i: FAIL"; done
run 1: PASS ... run 25: PASS
25 passed, 0 failed of 25

After (branch litellm_fix_realtime_guardrail_flake, commit a2b68b7): 25 consecutive green runs of the previously flaky test, then 20 consecutive green runs of the whole file

$ for i in $(seq 1 25); do .venv/bin/python -m pytest tests/llm_translation/realtime/test_realtime_guardrails_openai.py::test_text_message_blocked_by_guardrail_no_ai_response -q --timeout=120 >/dev/null 2>&1 && echo "run $i: PASS" || echo "run $i: FAIL"; done
run 1: PASS ... run 25: PASS
AFTER-SINGLE: 25 passed, 0 failed of 25

$ for i in $(seq 1 20); do .venv/bin/python -m pytest tests/llm_translation/realtime/test_realtime_guardrails_openai.py -q --timeout=120 >/dev/null 2>&1 && echo "run $i: PASS" || echo "run $i: FAIL"; done
run 1: PASS ... run 20: PASS
AFTER-FILE: 20 passed, 0 failed of 20

The new assertions were mutation-checked against the live API at commit a2b68b7 to prove they still guard the feature. Disabling the response.create drop in client_ack_messages fails with

AssertionError: Expected only the guardrail's response.create to reach OpenAI, got backend frames:
['response.cancel', 'conversation.item.create', 'response.create', 'response.create']

and disabling the guardrail entirely fails with

AssertionError: Expected at least one guardrail_violation error but got: ['invalid_request_error']

Type

✅ Test

Changes

test_text_message_blocked_by_guardrail_no_ai_response used to decide whether the guardrail worked by classifying the model's reply against a safe_markers keyword list. gpt-realtime words its refusal of the guardrail's "say exactly" voice prompt nondeterministically, so any phrasing outside the list turned CI red even though the guardrail had blocked correctly; in every recent CI failure the model's reply was a refusal of the guardrail's own prompt, which is only possible when the blocked message never reached OpenAI

The test now asserts the invariant the product actually guarantees, on the wire instead of on wording. A RecordingBackendWebSocket wrapper records every frame the proxy sends to the backend, and the test asserts the blocked phrase never reaches OpenAI, that only the guardrail's own conversation.item.create and response.create are forwarded (the client's reflexive response.create is dropped), and that the blocked phrase never appears in AI output. The fixed 0.3s/3.0s sleeps are replaced with an event-driven wait for response.done, and the two client frames are now sent back-to-back, which additionally pins the sequential ordering guarantee of client_ack_messages

No product change: client_ack_messages processes client frames sequentially, so the guardrail verdict always lands before the follow-up response.create is read; the enforcement itself was never racy

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This is a test-only fix for a recurring CI flake in realtime_translation_testing. The previously failing test test_text_message_blocked_by_guardrail_no_ai_response asserted that the model's audio reply contained safe-marker keywords, but gpt-realtime phrases its refusals nondeterministically, making that keyword list permanently stale.

• Introduces RecordingBackendWebSocket, a thin wrapper that intercepts every frame LiteLLM sends to the OpenAI backend so assertions can target wire traffic instead of model wording.
• Replaces the safe_markers keyword check and fixed asyncio.sleep calls with exact frame-count assertions (response.create == 1, conversation.item.create == 1) and an event-driven _wait_for_event("response.done").
• The new assertions are strictly stronger: they verify the blocked phrase never leaves LiteLLM toward OpenAI and that the client's reflexive response.create is dropped, both of which are the actual product guarantees.

Confidence Score: 5/5

Safe to merge; the change is confined to a single test file, touches no production code, and the new assertions are demonstrably stricter than the ones they replace.

No production code is changed. The new RecordingBackendWebSocket wrapper is a simple pass-through that only records outgoing frames. The replaced safe_markers keyword check was the documented root cause of 11 CI failures; the new wire-traffic assertions are both more deterministic and more semantically correct. The PR includes 25 consecutive green runs of the fixed test and 20 green runs of the full file as reproduction evidence.

No files require special attention. The only changed file is the test itself, and the logic change is straightforward and well-motivated.

Important Files Changed

Filename	Overview
tests/llm_translation/realtime/test_realtime_guardrails_openai.py	Replaces flaky wording-based guardrail assertions with deterministic wire-traffic recording via new RecordingBackendWebSocket; removes fixed sleeps in favour of event-driven waits; no logic regressions found.

_{Reviews (1): Last reviewed commit: "test(realtime): assert guardrail block o..." | Re-trigger Greptile}

[codspeed-hq]

Merging this PR will not alter performance

✅ 30 untouched benchmarks

---

_{Comparing litellm_fix_realtime_guardrail_flake (a2b68b7) with litellm_internal_staging (ff6dc33)}