Skip to content

test(e2e): drop vertex from pipecat tool smoke and remove key models dropdown suite #32925

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mubashir1osmani merged 2 commits into from
Jul 11, 2026
Merged

test(e2e): drop vertex from pipecat tool smoke and remove key models dropdown suite #32925

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9e2ce09
test(e2e): drop vertex from pipecat tool smoke
mubashir1osmani Jul 10, 2026
fe2fcb2
test(e2e): restore key models dropdown Playwright suite
mubashir1osmani Jul 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 9 additions & 8 deletions tests/e2e/llm_translation/realtime/REALTIME_COVERAGE_MATRIX.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,15 @@ transcript, and that `response.done` carries normalized usage.
normalized `response.function_call_arguments.done` with valid JSON arguments and
a matching `function_call` output item, the test sends a `function_call_output`
back, and the follow-up response incorporates the result (the temperature 72
appears).
appears). That raw-websocket tool path is the source of truth for tool calling.

`test_realtime_pipecat_e2e` is a realism layer that drives the same providers
through pipecat's GA `OpenAIRealtimeLLMService` (base_url pointed at the proxy)
rather than speaking the protocol by hand. Its assertions are coarse (the tool
callback fired, assistant text was produced); the raw-websocket suite is the
source of truth. It skips unless `pipecat-ai` is installed
(`uv pip install "pipecat-ai[openai]"`).
`test_pipecat_tool_smoke` is a realism layer through pipecat for openai, azure,
and gemini only (not vertex_ai: native-audio live is flaky under pipecat tool
calling while raw-ws tools pass; see pipecat-ai/pipecat#2544). Assertions are
coarse; raw-ws remains authoritative. Requires `pipecat-ai`.

Pipecat audio coverage lives in `test_realtime_pipecat_audio_e2e.py` (VAD / audio
I/O).

## Provisioning

Expand All @@ -43,7 +44,7 @@ at call time. The provider table below is the source of truth; edit `PROVIDERS`
| gemini | `gemini-realtime` | `gemini/gemini-3.1-flash-live-preview` |
| vertex_ai | `vertex-realtime` | `vertex_ai/gemini-live-2.5-flash-preview-native-audio-09-2025` |

Bedrock and xai (`xai/grok-4-1-fast`) are supported by the proxy but
Bedrock and xai (`xai/grok-4-1-fast-non-reasoning`) are supported by the proxy but
kept commented out in `PROVIDERS` until they pass end-to-end here; re-enable them by
uncommenting their entry.

Expand Down
7 changes: 6 additions & 1 deletion tests/e2e/llm_translation/realtime/test_realtime_pipecat_e2e.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,12 @@

from pipecat_service import LiteLLMRealtimeLLMService # noqa: E402

PROVIDER_PARAMS = [pytest.param(p, id=p.id) for p in PROVIDERS]
# Vertex native-audio live is flaky through pipecat tool calling (upstream
# pipecat-ai/pipecat#2544); raw-ws tool_call_round_trip[vertex_ai] is the
# source of truth for that provider. Keep openai/azure/gemini here.
PROVIDER_PARAMS = [
pytest.param(p, id=p.id) for p in PROVIDERS if p.id != "vertex_ai"
]

WEATHER_TOOL = ToolsSchema(
standard_tools=[
Expand Down
Loading