diff --git a/AGENTS.md b/AGENTS.md index 5e6ebbfc..014cdd26 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -192,14 +192,14 @@ NAT's built-in LangChain-backed agent types; applications install The public **native voice runtime** lives in `xr-ai-voice` (it depends on pipecat internally): -- **Voice session** — `VoiceSession.run(handler)` privately assembles - `input → VadStt → VoiceGate → handler → StreamingTts → output`, owns model - readiness and ready-file semantics, installs signal handlers, and closes the - transport and model clients. It touches the ready file only after the input - transport has entered its hub IPC receive loop. -- **Native handler** — `xr_ai_nat.adapters.as_voice_handler` maps a typed NAT - function onto `VoiceSession`; `TextMessageInput` routes participant text - through the same turn path as speech. +- **Voice agent** — `VoiceAgent` owns `VoiceSession`, publishes accepted speech + and typed text as its `UserQuery` schema on a sample-named topic, publishes + participant and interruption events on sample-named topics, and subscribes + to `voice.output`. Application agents subscribe to lifecycle events and own + their cleanup; the application entry point only composes them. Runtime + publication provides acknowledged delivery through the same output path. The + private session owns model readiness, ready-file semantics, signals, the + media pipeline, and cleanup. - **Wake word / speech gate** — `xr-ai-voicegate` (the `VoiceGate` state machine) wired in as `VoiceGateProcessor`; per-sample config in `yaml/voice_gate.yaml` (`magic_phrases: ["hey agent"]`, or `[]` for @@ -210,8 +210,8 @@ pipecat internally): `run_voice_pipeline(worker, transport, on_ready=ready_file.touch)` so they use the same IPC-start readiness boundary. -A native voice sample adapts its NAT function to `VoiceSession`; wake-word -behavior comes from config alone. +A native voice sample registers `VoiceAgent` and application agents on the same +runtime; wake-word behavior comes from config alone. ### Scope decision and named follow-ups diff --git a/DEPENDENCIES.md b/DEPENDENCIES.md index bc91723b..c0dc0ecb 100644 --- a/DEPENDENCIES.md +++ b/DEPENDENCIES.md @@ -38,13 +38,15 @@ CI matrices: ``` xr-ai-agent-runtime (agent-sdk/xr-ai-agent-runtime/) + └── nemo-relay >=0.7.2,<0.8 └── pydantic >=2.10 └── xr-ai-tools [editable: ../xr-ai-tools] - In-process runtime for agent resource lifetimes, runtime-owned background - tasks, and typed ``publish`` fan-out. Agents expose ordinary ``Tool`` and - ``AsyncTool`` instances from ``xr-ai-tools`` and own their synchronization. - Tool execution, model clients, tool loops, planning, memory, and raw media - transport are not runtime responsibilities. + In-process typed ``publish`` fan-out for agents that expose ordinary + ``Tool`` and ``AsyncTool`` instances from ``xr-ai-tools``. Agents own their + resources, background tasks, lifecycle, and synchronization. Tool + execution, model clients, tool loops, planning, memory, and raw media + transport are not runtime responsibilities. Relay scopes record runtime + publications and receiving-agent subscription callbacks. xr-ai-hub-client (agent-sdk/xr-ai-hub-client/) └── pyzmq >=27.0 @@ -76,6 +78,9 @@ xr-ai-pipecat (agent-sdk/xr-ai-pipecat/) Not a dep of xr-ai-hub-client itself — import only in workers that use Pipecat. xr-ai-voice (agent-sdk/xr-ai-voice/) + └── nemo-relay >=0.7.2,<0.8 + └── pydantic >=2.10 + └── xr-ai-agent-runtime [editable: ../xr-ai-agent-runtime] └── xr-ai-hub-client [editable: ../xr-ai-hub-client] └── xr-ai-logging [editable: ../../utils/xr-ai-logging] └── xr-ai-models [editable: ../xr-ai-models] @@ -85,13 +90,14 @@ xr-ai-voice (agent-sdk/xr-ai-voice/) └── nltk !=3.10.1 (3.10.1 rejects deps in in-project venvs) └── numpy >=1.24 └── scipy >=1.11 - Native voice runtime used by simple-vlm-example. Exposes the - ``VoiceSession`` public API plus the ``VoiceHandler`` / ``VoiceQuery`` / - ``VoiceResponse`` / ``VoiceTurn`` handler surface, ``HubVoiceTransport``, - ``VadConfig``, and ``TextMessageInput``; Pipecat, audio framing, and - pipeline processors are implementation details. Service health gates - transport construction, while ``VoiceSession.run`` touches its ready file - only after the input transport starts its hub IPC receive loop. The + Native voice runtime used by simple-vlm-example. Exposes ``VoiceAgent``, + its ``UserQuery`` / ``VoiceOutput`` / participant-lifecycle schemas, + ``VoiceSession``, ``HubVoiceTransport``, and + ``VadConfig``. Voice lifecycle events enter application-named topics so + application agents own their cleanup. Pipecat, audio framing, and pipeline + processors are implementation details. Service health gates transport + construction, while the session touches its ready file only after the input + transport starts its hub IPC receive loop. The readiness contract is split across the ``_readiness`` / ``_session`` modules. Not a dep of xr-ai-hub-client itself — import only in workers that opt into the voice runtime. @@ -566,16 +572,18 @@ the latest video frame via streaming VLM and replies with both | Sub-project | Package | Internal deps | External deps | |---|---|---|---| | Orchestrator | `simple-vlm-example` | `xr-ai-launcher` | — | -| Worker | `simple-vlm-example-worker` | `xr-ai-hub-client [editable]`, `xr-ai-logging [editable]`, `xr-ai-models [editable]`, `xr-ai-tools[live-vision] [editable]`, `xr-ai-voice [editable]`, `xr-ai-voicegate [editable]` | loguru >=0.7, pyyaml >=6.0 (`xr-ai-voice` pulls in VAD, pipecat-ai, numpy, and scipy; `xr-ai-tools[live-vision]` pulls in numpy and Pillow) | - -The packaged worker constructs a transport-independent `StreamingVisionTool` -and adapts its typed async chunks to `VoiceSession` locally. The tool owns -current-frame acquisition through `xr-ai-hub-client`, has no voice dependency, -and uses NeMo Relay's managed streaming LLM path. Camera bytes are redacted from -Relay telemetry while the provider receives the original frame. -`VoiceSession` owns readiness, hub transport, signals, the -private Pipecat pipeline, and cleanup; `TextMessageInput` routes `"ping"` and -ad-hoc text through the same participant-aware path as speech. Voice-gate +| Worker | `simple-vlm-example-worker` | `xr-ai-agent-runtime [editable]`, `xr-ai-hub-client [editable]`, `xr-ai-logging [editable]`, `xr-ai-models [editable]`, `xr-ai-tools[live-vision] [editable]`, `xr-ai-voice [editable]`, `xr-ai-voicegate [editable]` | nemo-relay >=0.7.2,<0.8, loguru >=0.7, pyyaml >=6.0 (`xr-ai-voice` pulls in VAD, pipecat-ai, numpy, and scipy; `xr-ai-tools[live-vision]` pulls in numpy and Pillow) | + +The packaged worker runs a transport-independent `StreamingVisionTool` inside +`SimpleVlmAgent` and publishes its typed async chunks to `VoiceAgent`. The tool +owns current-frame acquisition through `xr-ai-hub-client`, has no voice +dependency, and uses NeMo Relay's managed streaming LLM path. Camera bytes are +redacted from Relay telemetry while the provider receives the original frame. +`VoiceAgent` owns `VoiceSession`, readiness, hub transport, signals, and the +private Pipecat pipeline; it routes `"ping"` and ad-hoc text through the same +sample-named `UserQuery` topic as speech and publishes lifecycle events on +sample-named topics. `SimpleVlmAgent` handles cancellation and frame cleanup +inside its own subscriber methods. Voice-gate behavior (magic phrases, follow-up grace, listening chime, stop acknowledgement), VAD/STT, and sentence-batched TTS remain provided by the shared voice runtime. The sample has no direct `xr-ai-pipecat` or MCP dependency and selects no legacy diff --git a/README.md b/README.md index 138abcfc..df7cd0a1 100644 --- a/README.md +++ b/README.md @@ -117,7 +117,7 @@ frames are dropped if it is closed. | Hub service | `services/xr-media-hub/` | XR-Media-Hub + LiveKit internal transport | | Launcher | `utils/xr-ai-launcher/` | stdlib-only process manager used by samples | | Logging | `utils/xr-ai-logging/` | shared loguru sink + stdlib bridge for every process | -| Agent runtime | `agent-sdk/xr-ai-agent-runtime/` | Agent resource lifetimes, background tasks, existing native tools, and pub/sub | +| Agent runtime | `agent-sdk/xr-ai-agent-runtime/` | Agents with existing native tools and typed pub/sub routing | | Agent tools | `agent-sdk/xr-ai-tools/` | Toolkit-independent Relay-managed native tools | | Legacy NAT | `agent-sdk/xr-ai-nat/` | NeMo Agent Toolkit compatibility during migration | | Reusable services | `services/` | Model-serving and typed capability processes | @@ -197,9 +197,9 @@ channel, or send the literal text `"ping"` — all routes go through the same VLM pipeline against the latest video frame. Replies arrive as streaming Piper TTS audio plus a `vlm.response` text message. -The packaged worker adapts the transport-independent `StreamingVisionTool` to -`xr-ai-voice`'s `VoiceSession`; Pipecat remains private to that runtime and no -MCP client is involved. See the +The packaged worker runs `StreamingVisionTool` inside `SimpleVlmAgent` and +publishes its chunks to `VoiceAgent`; Pipecat remains private to the voice +runtime and no MCP client is involved. See the [sample README](agent-samples/simple-vlm-example/README.md) for the worker layout and configuration boundaries. diff --git a/agent-samples/simple-vlm-example/README.md b/agent-samples/simple-vlm-example/README.md index c4606b85..d2b4ed51 100644 --- a/agent-samples/simple-vlm-example/README.md +++ b/agent-samples/simple-vlm-example/README.md @@ -13,18 +13,23 @@ topic. Sending the literal text `ping` uses the configured default question, The worker is a package under `worker/simple_vlm_example_worker/`: - `__main__.py` parses launcher arguments. +- `agent.py` owns participant-scoped vision turns and cancellation. - `config.py` resolves worker, model-profile, voice-gate, and prompt settings. - `app.py` composes the native runtime. - `prompts/system.txt` owns the VLM system prompt. -`VoiceSession` owns STT/TTS/VLM readiness, the hub voice transport, voice-gate -processing, streaming TTS, signals, and cleanup. The application constructs a -transport-independent `StreamingVisionTool` and adapts its async chunks to the -voice handler locally. The tool has no voice dependency and sends its provider -stream through Relay's managed LLM path. The camera frame is redacted from -Relay telemetry. Typed text uses the same -participant-aware turn path as speech. Participant leave events release cached -live-frame state, and a newer turn cancels and interrupts a superseded response. +`VoiceAgent` owns `VoiceSession`, which provides STT/TTS/VLM readiness, the hub +voice transport, voice-gate processing, streaming TTS, signals, and cleanup. +It publishes accepted speech and typed text as `UserQuery` on this sample's +topic. `SimpleVlmAgent` subscribes to that topic, owns participant-scoped +streaming and cancellation around the transport-independent +`StreamingVisionTool`, and publishes chunks to `voice.output`. The tool has no +voice dependency and sends its provider stream through Relay's managed LLM +path. The camera frame is redacted from Relay telemetry. `VoiceAgent` publishes +participant departure and interruption on sample-named topics; +`SimpleVlmAgent` subscribes and releases its own cached frames and tasks. A +newer turn cancels and interrupts a superseded response. `app.py` only composes +the two agents and their dependencies. No MCP client or MCP tool invocation is part of this sample. @@ -50,6 +55,28 @@ The worker and orchestrator consume the deployment profile selected by The same profile owns model behavior, endpoints, credentials, readiness, and launcher process ownership. +## Relay visibility + +The worker writes a compact Relay lifecycle stream to `relay-events.jsonl` +beside `worker.log` in the per-run log directory printed at startup. The JSONL +records include runtime publications, receiving-agent callbacks, the complete +`simple-vlm.turn` lifetime, and nested vision tool and VLM calls. Per-token +`llm.chunk` marks, incremental `voice.output` fragments, and empty stream +terminators are omitted. `VoiceAgent` emits one `voice.response` scope containing +the complete text and timing for both non-streamed and aggregated incremental +output. Each real STT request is a `voice.stt` scope with a transcript result +mark, and each sentence synthesis is a `voice.tts` scope. Raw audio is summarized +by byte count, duration, and sample rate; TTS records synthesis rather than +client playback. The completed LLM and turn records remain available alongside +them. No telemetry server or network exporter is required. Live camera bytes are +replaced with ``; prompts, questions, responses, +participant IDs, and correlation metadata remain visible and may contain +sensitive data. + +```bash +tail -F /tmp/log_simple-vlm-example_*/relay-events.jsonl +``` + Voice-gate behavior remains in `yaml/voice_gate.yaml`. Worker timing, frame freshness, the default `ping` question, and optional prompt overrides are in `yaml/simple_vlm_example_worker.yaml`; the default prompt ships inside the diff --git a/agent-samples/simple-vlm-example/worker/pyproject.toml b/agent-samples/simple-vlm-example/worker/pyproject.toml index dc490054..8ebd2a50 100644 --- a/agent-samples/simple-vlm-example/worker/pyproject.toml +++ b/agent-samples/simple-vlm-example/worker/pyproject.toml @@ -10,6 +10,8 @@ name = "simple-vlm-example-worker" version = "0.1.0" requires-python = ">=3.11,<3.13" dependencies = [ + "nemo-relay>=0.7.2,<0.8", + "xr-ai-agent-runtime", "xr-ai-hub-client", "xr-ai-logging", "xr-ai-models", @@ -21,6 +23,7 @@ dependencies = [ ] [tool.uv.sources] +xr-ai-agent-runtime = { path = "../../../agent-sdk/xr-ai-agent-runtime", editable = true } xr-ai-hub-client = { path = "../../../agent-sdk/xr-ai-hub-client", editable = true } xr-ai-logging = { path = "../../../utils/xr-ai-logging", editable = true } xr-ai-models = { path = "../../../agent-sdk/xr-ai-models", editable = true } diff --git a/agent-samples/simple-vlm-example/worker/simple_vlm_example_worker/agent.py b/agent-samples/simple-vlm-example/worker/simple_vlm_example_worker/agent.py new file mode 100644 index 00000000..3cbc7143 --- /dev/null +++ b/agent-samples/simple-vlm-example/worker/simple_vlm_example_worker/agent.py @@ -0,0 +1,187 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Participant-scoped streaming orchestration for the simple VLM sample.""" + +from __future__ import annotations + +import asyncio +from collections.abc import Callable +from contextlib import suppress + +import nemo_relay +from xr_ai_runtime import ( + Agent, + RuntimeClosedError, + RuntimeContext, + Topic, + subscribe, +) +from xr_ai_tools.streaming_vision import StreamingVisionTool, VisionRequest +from xr_ai_voice import ( + VOICE_OUTPUT_TOPIC, + UserQuery, + VoiceInterrupted, + VoiceOutput, + VoiceParticipantLeft, +) + +USER_QUERY_TOPIC = Topic("simple-vlm.user-query", UserQuery) +PARTICIPANT_LEFT_TOPIC = Topic( + "simple-vlm.participant-left", + VoiceParticipantLeft, +) +INTERRUPTED_TOPIC = Topic("simple-vlm.interrupted", VoiceInterrupted) + + +class SimpleVlmAgent(Agent): + """Own streamed user turns and cancellation around a vision tool.""" + + def __init__(self, vision_factory: Callable[[], StreamingVisionTool]) -> None: + super().__init__() + self._vision_factory = vision_factory + self._vision: StreamingVisionTool | None = None + self._tasks: dict[str, asyncio.Task[None]] = {} + + @subscribe(USER_QUERY_TOPIC) + async def answer_user(self, request: UserQuery, ctx: RuntimeContext) -> None: + """Supersede and start one participant's streamed response.""" + + participant_id = ctx.metadata.participant_id + if participant_id is None: + raise ValueError("simple VLM queries require a participant") + await self._cancel(participant_id) + task = asyncio.create_task( + self._stream(request, ctx), + name=f"simple-vlm-query:{participant_id}", + context=nemo_relay.fork_asyncio_context(), + ) + self._tasks[participant_id] = task + task.add_done_callback( + lambda completed, pid=participant_id: self._discard(pid, completed) + ) + + @subscribe(PARTICIPANT_LEFT_TOPIC) + async def participant_left( + self, + _event: VoiceParticipantLeft, + ctx: RuntimeContext, + ) -> None: + """Release this agent's work and frame state for a departed participant.""" + + participant_id = ctx.metadata.participant_id + if participant_id is None: + raise ValueError("participant-left events require a participant") + await self._cancel(participant_id) + if self._vision is not None: + self._vision.release(participant_id) + + @subscribe(INTERRUPTED_TOPIC) + async def interrupted( + self, + _event: VoiceInterrupted, + ctx: RuntimeContext, + ) -> None: + """Cancel participant-scoped or global vision work.""" + + participant_id = ctx.metadata.participant_id + if participant_id is None: + await self._cancel_all() + return + await self._cancel(participant_id) + + async def _stream(self, request: UserQuery, ctx: RuntimeContext) -> None: + with nemo_relay.use_scope_stack(nemo_relay.create_scope_stack()): + with nemo_relay.scope.scope( + "simple-vlm.turn", + nemo_relay.ScopeType.Agent, + input=request.model_dump(mode="json"), + metadata={ + "agent": ctx.agent_name, + "message_id": ctx.metadata.message_id, + "correlation_id": ctx.metadata.correlation_id, + "participant_id": ctx.metadata.participant_id, + }, + ): + await self._stream_response(request, ctx) + + async def _stream_response( + self, + request: UserQuery, + ctx: RuntimeContext, + ) -> None: + response_id = ctx.metadata.message_id + participant_id = ctx.metadata.participant_id + if participant_id is None: + raise ValueError("simple VLM queries require a participant") + first = True + opened = False + if self._vision is None: + self._vision = self._vision_factory() + stream = self._vision.stream( + VisionRequest(participant_id=participant_id, query=request.text) + ) + cancelled = False + try: + try: + async for chunk in stream: + await ctx.publish( + VOICE_OUTPUT_TOPIC, + VoiceOutput( + text=chunk.text, + response_id=response_id, + final=False, + interrupt=first, + timestamp_us=request.timestamp_us, + ), + ) + first = False + opened = True + finally: + close = getattr(stream, "aclose", None) + if close is not None: + await close() + except asyncio.CancelledError: + cancelled = True + raise + finally: + if opened and not cancelled: + with suppress(RuntimeClosedError): + await ctx.publish( + VOICE_OUTPUT_TOPIC, + VoiceOutput( + response_id=response_id, + timestamp_us=request.timestamp_us, + ), + ) + async def _cancel(self, participant_id: str) -> None: + task = self._tasks.pop(participant_id, None) + if task is None: + return + task.cancel() + await asyncio.gather(task, return_exceptions=True) + + async def _cancel_all(self) -> None: + tasks = tuple(self._tasks.values()) + self._tasks.clear() + for task in tasks: + task.cancel() + if tasks: + await asyncio.gather(*tasks, return_exceptions=True) + + async def stop(self) -> None: + """Cancel all vision turns owned by this agent.""" + + await self._cancel_all() + + def _discard(self, participant_id: str, task: asyncio.Task[None]) -> None: + if self._tasks.get(participant_id) is task: + self._tasks.pop(participant_id, None) + + +__all__ = [ + "INTERRUPTED_TOPIC", + "PARTICIPANT_LEFT_TOPIC", + "SimpleVlmAgent", + "USER_QUERY_TOPIC", +] diff --git a/agent-samples/simple-vlm-example/worker/simple_vlm_example_worker/app.py b/agent-samples/simple-vlm-example/worker/simple_vlm_example_worker/app.py index 8dd6d512..71794586 100644 --- a/agent-samples/simple-vlm-example/worker/simple_vlm_example_worker/app.py +++ b/agent-samples/simple-vlm-example/worker/simple_vlm_example_worker/app.py @@ -5,44 +5,61 @@ from __future__ import annotations -from collections.abc import Callable +from collections.abc import AsyncIterator, Callable +from contextlib import asynccontextmanager from pathlib import Path +from threading import Lock +import nemo_relay from loguru import logger from xr_ai_logging import setup_logging from xr_ai_models import load_models_config, make_stt, make_tts, make_vlm -from xr_ai_tools.streaming_vision import StreamingVisionTool, VisionRequest -from xr_ai_voice import TextMessageInput, VadConfig, VoiceHandler, VoiceSession +from xr_ai_runtime import AgentRuntime +from xr_ai_tools.streaming_vision import StreamingVisionTool +from xr_ai_voice import VadConfig, VoiceAgent, VoiceSession from xr_ai_voicegate import load_voice_gate_config +from .agent import ( + INTERRUPTED_TOPIC, + PARTICIPANT_LEFT_TOPIC, + USER_QUERY_TOPIC, + SimpleVlmAgent, +) from .config import WorkerConfig -def _make_vision_handler(vision: StreamingVisionTool) -> VoiceHandler: - async def handle(turn): - async def response(): - request = VisionRequest( - participant_id=turn.participant_id, - query=turn.text, - ) - stream = vision.stream(request) - try: - async for chunk in stream: - yield chunk.text - finally: - close = getattr(stream, "aclose", None) - if close is not None: - await close() - - return response() - - return handle - - def _text_transform(default_prompt: str) -> Callable[[str], str]: return lambda text: default_prompt if text.lower() == "ping" else text +@asynccontextmanager +async def _relay_event_log(log_file: Path) -> AsyncIterator[Path]: + event_path = log_file.parent / "relay-events.jsonl" + sink = event_path.open("w", encoding="utf-8") + lock = Lock() + subscriber = "simple-vlm-compact-event-log" + + def write_event(event: nemo_relay.Event) -> None: + if event.kind == "mark" and event.name == "llm.chunk": + return + with lock: + sink.write(event.to_json()) + sink.write("\n") + sink.flush() + + try: + nemo_relay.subscribers.register(subscriber, write_event) + except Exception: + sink.close() + raise + try: + yield event_path + finally: + await nemo_relay.subscribers.flush_async() + nemo_relay.subscribers.deregister(subscriber) + sink.close() + + async def run_app( config: WorkerConfig, *, @@ -50,7 +67,7 @@ async def run_app( ) -> None: """Run the worker until the voice session shuts down.""" - setup_logging("worker") + log_file = setup_logging("worker") models = load_models_config(config.models_config) voice_gate = load_voice_gate_config(config.voice_gate_yaml) stt = make_stt(models, "stt") @@ -73,24 +90,36 @@ async def run_app( idle_timeout_secs=config.idle_timeout_secs, ) - async with session: - vision = StreamingVisionTool( - endpoint=session.transport.endpoint, - vlm=vlm, - system_prompt=config.system_prompt, - frame_max_age_s=config.frame_max_age_s, - frame_timeout_s=config.frame_timeout_s, - ) - TextMessageInput( - session=session, - transform=_text_transform(config.default_prompt), - fresh_match=True, - ) - - logger.info("simple-vlm-example starting") - await session.run( - _make_vision_handler(vision), - on_participant_left=vision.release, - interrupt_on_supersede=True, - ) - logger.info("simple-vlm-example stopped") + runtime = AgentRuntime() + simple_vlm = runtime.register( + "simple-vlm", + SimpleVlmAgent( + lambda: StreamingVisionTool( + endpoint=session.endpoint, + vlm=vlm, + system_prompt=config.system_prompt, + frame_max_age_s=config.frame_max_age_s, + frame_timeout_s=config.frame_timeout_s, + ) + ), + ) + + voice = VoiceAgent( + session, + query_topic=USER_QUERY_TOPIC, + text_transform=_text_transform(config.default_prompt), + participant_left_topic=PARTICIPANT_LEFT_TOPIC, + interrupted_topic=INTERRUPTED_TOPIC, + interrupt_on_supersede=True, + ) + runtime.register("voice", voice) + + logger.info("Relay events → {}", log_file.parent / "relay-events.jsonl") + logger.info("simple-vlm-example starting") + async with _relay_event_log(log_file): + async with runtime: + try: + await voice.run(runtime) + finally: + await simple_vlm.stop() + logger.info("simple-vlm-example stopped") diff --git a/agent-sdk/xr-ai-agent-runtime/README.md b/agent-sdk/xr-ai-agent-runtime/README.md index 3ec41383..67a248ee 100644 --- a/agent-sdk/xr-ai-agent-runtime/README.md +++ b/agent-sdk/xr-ai-agent-runtime/README.md @@ -73,15 +73,27 @@ the workflow boundary with `ToolSet.namespaced({"vision": vision.tools, "planner": planner.tools})`. This remaps only model-visible catalog names; the agents and underlying tools remain unchanged. Participant identity needed by a direct tool belongs in that -tool's typed request. Relay supplies nested execution tracing, while runtime -message metadata applies only to pub/sub. +tool's typed request. Relay records each publication as a function scope +and each subscription delivery as an agent scope, carrying topic, participant, +message, correlation, parent-message, source, and subscriber metadata. Tool and +model scopes invoked by a callback nest under that delivery. Agent-owned +detached tasks must open a fresh Relay scope stack when their lifetime extends +beyond the callback, then add an agent scope and preserve logical correlation in +metadata. This prevents a detached operation from becoming the child of a scope +that has already ended. `publish(topic, event)` is the separate asynchronous fan-out operation for -events. An agent that owns resources or background work is responsible for -controlling them, including creating, cancelling, and awaiting its own tasks. -The runtime neither knows nor controls whether an agent's internal work is -running. `publish()` waits for every fan-out delivery to settle before -propagating any subscriber failures. +participant-scoped or global events. An agent that owns resources or background +work is responsible for controlling them, including creating, cancelling, and +awaiting its own tasks. The runtime neither knows nor controls whether an +agent's internal work is running. `publish()` waits for every fan-out delivery +to settle before propagating any subscriber failures. + +Topics default to `telemetry="full"`. High-cardinality transport topics use +`"none"` when their consumer aggregates fragments and records one semantic +operation scope. Delivery and failure behavior remains unchanged. Keeping the +policy on the topic declaration gives every producer and consumer the same +cardinality behavior. Tools and subscription callbacks may run concurrently. An agent whose mutable state is shared between them owns the appropriate synchronization, such as an diff --git a/agent-sdk/xr-ai-agent-runtime/pyproject.toml b/agent-sdk/xr-ai-agent-runtime/pyproject.toml index c38bbb6e..74cd9aaa 100644 --- a/agent-sdk/xr-ai-agent-runtime/pyproject.toml +++ b/agent-sdk/xr-ai-agent-runtime/pyproject.toml @@ -11,6 +11,7 @@ version = "0.1.0" description = "Typed pub/sub runtime for XR AI agents." requires-python = ">=3.11,<3.13" dependencies = [ + "nemo-relay>=0.7.2,<0.8", "pydantic>=2.10", "xr-ai-tools", ] diff --git a/agent-sdk/xr-ai-agent-runtime/xr_ai_runtime/events.py b/agent-sdk/xr-ai-agent-runtime/xr_ai_runtime/events.py index 66fbb2bd..ffe4e46b 100644 --- a/agent-sdk/xr-ai-agent-runtime/xr_ai_runtime/events.py +++ b/agent-sdk/xr-ai-agent-runtime/xr_ai_runtime/events.py @@ -7,7 +7,7 @@ from collections.abc import Awaitable, Callable from dataclasses import dataclass -from typing import Any, Generic, TypeVar +from typing import Any, Generic, Literal, TypeVar from pydantic import BaseModel @@ -20,12 +20,15 @@ class Topic(Generic[MessageT]): name: str message_type: type[MessageT] + telemetry: Literal["full", "none"] = "full" def __post_init__(self) -> None: if not self.name.strip(): raise ValueError("topic name must not be empty") if not issubclass(self.message_type, BaseModel): raise TypeError("topic messages must be Pydantic models") + if self.telemetry not in ("full", "none"): + raise ValueError("topic telemetry must be 'full' or 'none'") def validate(self, message: MessageT | dict[str, Any]) -> MessageT: """Validate a message before delivery.""" @@ -39,7 +42,7 @@ class MessageMetadata: message_id: str correlation_id: str - participant_id: str + participant_id: str | None source: str parent_message_id: str | None timestamp_us: int diff --git a/agent-sdk/xr-ai-agent-runtime/xr_ai_runtime/runtime.py b/agent-sdk/xr-ai-agent-runtime/xr_ai_runtime/runtime.py index a8420c4b..4393c706 100644 --- a/agent-sdk/xr-ai-agent-runtime/xr_ai_runtime/runtime.py +++ b/agent-sdk/xr-ai-agent-runtime/xr_ai_runtime/runtime.py @@ -11,9 +11,11 @@ import uuid from builtins import BaseExceptionGroup from collections.abc import Awaitable, Callable +from contextlib import nullcontext from dataclasses import dataclass, field from typing import Any, TypeAlias, TypeVar, cast, get_type_hints +import nemo_relay from pydantic import BaseModel from .agent import Agent @@ -34,7 +36,6 @@ class _AgentState: name: str deliveries: set[asyncio.Task[None]] = field(default_factory=set) - class RuntimeContext: """Runtime operations available during a subscription delivery.""" @@ -69,7 +70,7 @@ async def publish( *, participant_id: str | None = None, ) -> None: - """Publish an event while preserving current trace context.""" + """Publish an event, preserving delivery scope when one exists.""" await self._runtime._publish( topic, @@ -80,7 +81,7 @@ async def publish( parent_message_id=self._metadata.message_id, ) - def _resolve_participant(self, participant_id: str | None) -> str: + def _resolve_participant(self, participant_id: str | None) -> str | None: if participant_id is not None: return participant_id return self._metadata.participant_id @@ -146,7 +147,7 @@ async def publish( topic: Topic[MessageT], message: MessageT | dict[str, Any], *, - participant_id: str, + participant_id: str | None = None, source: str = "application", ) -> None: """Validate and deliver one event to every topic subscriber.""" @@ -185,7 +186,7 @@ async def _publish( topic: Topic[MessageT], message: MessageT | dict[str, Any], *, - participant_id: str, + participant_id: str | None, source: str, correlation_id: str | None = None, parent_message_id: str | None = None, @@ -199,34 +200,68 @@ async def _publish( correlation_id=correlation_id, parent_message_id=parent_message_id, ) - deliveries: list[asyncio.Task[None]] = [] - for state, method in tuple(self._subscribers.get(topic.name, ())): - task = asyncio.create_task( - self._deliver( - state, - method, - value.model_copy(deep=True), - metadata, - ), - name=f"agent:{state.name}:subscription:{topic.name}", + traced = topic.telemetry == "full" + publication_scope = ( + nemo_relay.scope.scope( + f"publish:{topic.name}", + nemo_relay.ScopeType.Function, + input=value.model_dump(mode="json"), + metadata=self._relay_metadata(metadata, topic=topic.name), ) - state.deliveries.add(task) - task.add_done_callback(state.deliveries.discard) - deliveries.append(task) - if deliveries: - results = await asyncio.gather(*deliveries, return_exceptions=True) - errors = [result for result in results if isinstance(result, BaseException)] - if errors: - raise BaseExceptionGroup("errors during event publication", errors) + if traced + else nullcontext() + ) + with publication_scope: + deliveries: list[asyncio.Task[None]] = [] + for state, method in tuple(self._subscribers.get(topic.name, ())): + task = asyncio.create_task( + self._deliver( + state, + method, + topic.name, + value.model_copy(deep=True), + metadata, + traced=traced, + ), + name=f"agent:{state.name}:subscription:{topic.name}", + context=nemo_relay.fork_asyncio_context(), + ) + state.deliveries.add(task) + task.add_done_callback(state.deliveries.discard) + deliveries.append(task) + if deliveries: + results = await asyncio.gather(*deliveries, return_exceptions=True) + errors = [ + result for result in results if isinstance(result, BaseException) + ] + if errors: + raise BaseExceptionGroup("errors during event publication", errors) async def _deliver( self, state: _AgentState, method: BoundSubscriber, + topic_name: str, message: BaseModel, metadata: MessageMetadata, + *, + traced: bool, ) -> None: - await method(message, RuntimeContext(self, state.name, metadata)) + if not traced: + await method(message, RuntimeContext(self, state.name, metadata)) + return + with nemo_relay.scope.scope( + f"agent:{state.name}", + nemo_relay.ScopeType.Agent, + input=message.model_dump(mode="json"), + metadata=self._relay_metadata( + metadata, + topic=topic_name, + agent=state.name, + subscriber=method.__name__, + ), + ): + await method(message, RuntimeContext(self, state.name, metadata)) def _discover_subscriptions( self, @@ -271,12 +306,12 @@ def _ensure_running(self) -> None: @staticmethod def _metadata( *, - participant_id: str, + participant_id: str | None, source: str, correlation_id: str | None, parent_message_id: str | None, ) -> MessageMetadata: - if not participant_id.strip(): + if participant_id is not None and not participant_id.strip(): raise ValueError("participant_id must not be empty") if not source.strip(): raise ValueError("message source must not be empty") @@ -290,6 +325,26 @@ def _metadata( timestamp_us=time.time_ns() // 1_000, ) + @staticmethod + def _relay_metadata( + metadata: MessageMetadata, + *, + topic: str, + agent: str | None = None, + subscriber: str | None = None, + ) -> dict[str, str | int | None]: + return { + "topic": topic, + "agent": agent, + "subscriber": subscriber, + "message_id": metadata.message_id, + "correlation_id": metadata.correlation_id, + "parent_message_id": metadata.parent_message_id, + "participant_id": metadata.participant_id, + "source": metadata.source, + "timestamp_us": metadata.timestamp_us, + } + __all__ = [ "AgentRuntime", diff --git a/agent-sdk/xr-ai-hub-client/README.md b/agent-sdk/xr-ai-hub-client/README.md index 30e28b4e..8bc7f60c 100644 --- a/agent-sdk/xr-ai-hub-client/README.md +++ b/agent-sdk/xr-ai-hub-client/README.md @@ -20,8 +20,10 @@ endpoint = ProcessorEndpoint( async def on_data(message: DataMessage) -> None: print(message.participant_id, message.topic) -endpoint.on_data(on_data) +unsubscribe = endpoint.on_data(on_data) await endpoint.run() + +unsubscribe() # Remove the callback when its owner stops. ``` `LiveFrameSource` adds raw frame acquisition without adding image conversion or diff --git a/agent-sdk/xr-ai-hub-client/xr_ai_hub/_processor.py b/agent-sdk/xr-ai-hub-client/xr_ai_hub/_processor.py index 53ac7012..b4d6a608 100644 --- a/agent-sdk/xr-ai-hub-client/xr_ai_hub/_processor.py +++ b/agent-sdk/xr-ai-hub-client/xr_ai_hub/_processor.py @@ -74,6 +74,7 @@ AudioCallback = Callable[[AudioChunk], Awaitable[None]] DataCallback = Callable[[DataMessage], Awaitable[None]] ParticipantCallback = Callable[[ParticipantEvent], Awaitable[None]] +CallbackUnsubscribe = Callable[[], None] # Reserved topic for internal SDK status messages — not forwarded to app callbacks. AGENT_STATUS_TOPIC = "_agent.status" @@ -358,7 +359,16 @@ async def _probe(self, timeout: float) -> bool: def on_frame(self, cb: FrameSignalCallback) -> None: self._frame_cbs.append(cb) def on_frame_data(self, cb: FrameDataCallback) -> None: self._frame_data_cbs.append(cb) def on_audio(self, cb: AudioCallback) -> None: self._audio_cbs.append(cb) - def on_data(self, cb: DataCallback) -> None: self._data_cbs.append(cb) + def on_data(self, cb: DataCallback) -> CallbackUnsubscribe: + """Register a data callback and return an idempotent unsubscriber.""" + + self._data_cbs.append(cb) + + def unsubscribe() -> None: + if cb in self._data_cbs: + self._data_cbs.remove(cb) + + return unsubscribe def on_participant(self, cb: ParticipantCallback) -> None: self._participant_cbs.append(cb) # ── return path ─────────────────────────────────────────────────────────── diff --git a/agent-sdk/xr-ai-nat/README.md b/agent-sdk/xr-ai-nat/README.md index 538b4719..7b2086cc 100644 --- a/agent-sdk/xr-ai-nat/README.md +++ b/agent-sdk/xr-ai-nat/README.md @@ -109,34 +109,6 @@ observer below: it stores each turn under the role-scoped source `{participant_id}:{role}`, exactly the pair `recall_conversation` reads. Without it wired up, recall is empty. -## Voice adapters - -Install `xr-ai-nat[voice]` to drive a native function from a voice session. Both -adapters are exported from `xr_ai_nat.adapters` (resolved lazily, so importing -that package without the extra still works): - -```python -from xr_ai_nat.adapters import as_voice_handler, record_voice_transcripts - -handler = as_voice_handler( - some_function, - request=lambda query: MyRequest(text=query.text), - response=str, -) -observer = record_voice_transcripts(add_transcript) -``` - -- `as_voice_handler(function, *, request, response, streaming=False)` wraps a - native function as a voice handler: it maps a `VoiceQuery` onto the function's - request model and maps the result back to text. With `streaming=True` it - forwards the function's `astream` output chunk by chunk for incremental - speech. -- `record_voice_transcripts(add_transcript)` returns a turn observer that - persists each completed turn under `{participant_id}:{role}`, feeding - `recall_conversation` above. Recording is an observer rather than a side - effect of invoking a function, so a session records turns even when the agent - did not handle them. - ## Vision Install `xr-ai-nat[vision]` to use the `xr_vision_tools` function group. The diff --git a/agent-sdk/xr-ai-nat/xr_ai_nat/adapters/__init__.py b/agent-sdk/xr-ai-nat/xr_ai_nat/adapters/__init__.py index 39edeaf9..44bc8683 100644 --- a/agent-sdk/xr-ai-nat/xr_ai_nat/adapters/__init__.py +++ b/agent-sdk/xr-ai-nat/xr_ai_nat/adapters/__init__.py @@ -1,43 +1,6 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -"""Optional transport and framework adapters for native XR functions. +"""Deprecated adapter namespace retained only for explicit submodule imports.""" -The voice adapters are the public way to drive a native function from a voice -session: - - from xr_ai_nat.adapters import as_voice_handler, record_voice_transcripts - -They are re-exported lazily because they need the optional ``[voice]`` extra -(``xr-ai-voice``). Importing this package without that extra stays cheap and -succeeds; only touching an attribute raises, and the error names the extra to -install. Lazy access also keeps the deprecated ``adapters.mcp`` alias from -emitting its warning on an unrelated import of this package. -""" - -from typing import TYPE_CHECKING - -if TYPE_CHECKING: # pragma: no cover - typing-only import - from .voice import as_voice_handler, record_voice_transcripts - -_VOICE_EXPORTS = frozenset({"as_voice_handler", "record_voice_transcripts"}) - -__all__ = ["as_voice_handler", "record_voice_transcripts"] - - -def __getattr__(name: str): - """Resolve the voice adapters on first access (PEP 562).""" - if name in _VOICE_EXPORTS: - try: - from . import voice - except ImportError as exc: # pragma: no cover - depends on install extras - raise ImportError( - f"xr_ai_nat.adapters.{name} requires the optional 'voice' extra; " - "install xr-ai-nat[voice]." - ) from exc - return getattr(voice, name) - raise AttributeError(f"module {__name__!r} has no attribute {name!r}") - - -def __dir__() -> list[str]: - return sorted(__all__) +__all__: list[str] = [] diff --git a/agent-sdk/xr-ai-nat/xr_ai_nat/adapters/voice.py b/agent-sdk/xr-ai-nat/xr_ai_nat/adapters/voice.py deleted file mode 100644 index 6744d28a..00000000 --- a/agent-sdk/xr-ai-nat/xr_ai_nat/adapters/voice.py +++ /dev/null @@ -1,62 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -"""Adapt native NAT functions and transcript storage to XR voice sessions.""" - -from __future__ import annotations - -from collections.abc import AsyncIterator, Awaitable, Callable -from typing import Any - -from nat.plugin_api import Function -from xr_ai_voice import VoiceHandler, VoiceQuery, VoiceResponse, VoiceTurn - -from ..functions.text_memory import AddTranscriptRequest - -_RequestMapper = Callable[[VoiceQuery], Any] -_ResponseMapper = Callable[[Any], str] - - -def as_voice_handler( - function: Function, - *, - request: _RequestMapper, - response: _ResponseMapper, - streaming: bool = False, -) -> VoiceHandler: - """Map voice turns to one native NAT function and its textual response.""" - - async def handle(query: VoiceQuery) -> VoiceResponse: - function_request = request(query) - if not streaming: - return response(await function.ainvoke(function_request)) - - async def stream() -> AsyncIterator[str]: - async for chunk in function.astream(function_request): # pyright: ignore[reportGeneralTypeIssues] - if text := response(chunk): - yield text - - return stream() - - return handle - - -def record_voice_transcripts( - add_transcript: Function, -) -> Callable[[VoiceTurn], Awaitable[None]]: - """Return a voice-session observer that stores completed user and agent turns.""" - - async def record(turn: VoiceTurn) -> None: - if turn.text.strip(): - await add_transcript.ainvoke( - AddTranscriptRequest( - source_id=f"{turn.participant_id}:{turn.role}", - timestamp_us=turn.timestamp_us, - text=turn.text, - ) - ) - - return record - - -__all__ = ["as_voice_handler", "record_voice_transcripts"] diff --git a/agent-sdk/xr-ai-nat/xr_ai_nat/functions/text_memory/functions.py b/agent-sdk/xr-ai-nat/xr_ai_nat/functions/text_memory/functions.py index de352b7e..e3e97574 100644 --- a/agent-sdk/xr-ai-nat/xr_ai_nat/functions/text_memory/functions.py +++ b/agent-sdk/xr-ai-nat/xr_ai_nat/functions/text_memory/functions.py @@ -284,7 +284,7 @@ async def conversation_memory_functions(config: ConversationMemoryFunctionsConfi """Build participant conversation recall over the transcript store. Reads the ``{participant_id}:user`` and ``{participant_id}:agent`` transcript - sources produced by ``xr_ai_nat.adapters.voice.record_voice_transcripts``. + role-scoped sources written by the application. """ text_memory = await builder.get_function_group(config.text_memory) diff --git a/agent-sdk/xr-ai-voice/README.md b/agent-sdk/xr-ai-voice/README.md index 1053a9d9..37475e16 100644 --- a/agent-sdk/xr-ai-voice/README.md +++ b/agent-sdk/xr-ai-voice/README.md @@ -5,54 +5,97 @@ # xr-ai-voice -The public voice-session API for XR agents. Pipecat implements the pipeline, -but applications work with XR concepts rather than Pipecat modules: +The voice runtime for XR agents. Pipecat implements the media pipeline, but +applications work with XR concepts rather than Pipecat modules: +- `VoiceAgent` publishes accepted speech, text, participant departure, and + interruption as voice-owned schemas on application-named topics; it + subscribes to `voice.output`. - `VoiceSession` owns readiness, hub transport, private pipeline assembly, - signals, execution, and cleanup. Its default hub transport opens only after - service probes succeed. -- `VoiceHandler` is an async callable from `VoiceQuery` to text or a text stream. -- `TextMessageInput` routes typed participant messages through the same turn path as speech. + signals, execution, and cleanup behind `VoiceAgent`. - `HubVoiceTransport` is available when an application needs to share one transport explicitly. ## Usage +Applications register one `VoiceAgent` with the shared runtime: + ```python -from xr_ai_voice import VadConfig, VoiceQuery, VoiceSession +from xr_ai_runtime import Topic +from xr_ai_voice import ( + UserQuery, + VadConfig, + VoiceAgent, + VoiceInterrupted, + VoiceSession, +) from xr_ai_voicegate import VoiceGateConfig -async def handle(query: VoiceQuery) -> str: - # query.participant_id / .text / .fresh_match / .timestamp_us - # timestamp_us is Unix-epoch µs anchored to when the user spoke. - return f"You said: {query.text}" - session = VoiceSession( - stt=stt, tts=tts, vad=VadConfig(), voice_gate=VoiceGateConfig(), + stt=stt, + tts=tts, + vad=VadConfig(), + voice_gate=VoiceGateConfig(), +) +queries = Topic("my-sample.user-query", UserQuery) +interruptions = Topic("my-sample.interrupted", VoiceInterrupted) +voice = VoiceAgent( + session, + query_topic=queries, + interrupted_topic=interruptions, ) -async with session: # awaits STT/TTS readiness - await session.run(handle) # starts hub IPC, touches ready_file, then runs +runtime.register("voice", voice) + +async with runtime: + await voice.run(runtime) ``` -A handler may also return an `AsyncIterator[str]` to stream the reply token by -token. Typed messages route through the same path via `TextMessageInput`; data -received outside an active `run()` is ignored. - -`VoiceSession.run()` accepts participant lifecycle callbacks, a turn observer, -and explicit follow-up policies. `queue_queries` preserves FIFO execution per -participant instead of cancelling the active handler. With -`interrupt_on_supersede`, each queued turn flushes speech left from the -preceding response when it starts. Explicit interruption frames such as stop -cancel the active turn and clear its participant queue. The `on_query_superseded` -callback fires only when a new query actually replaces a still-in-flight turn — -a follow-up that arrives after the previous turn finished, or that is queued, is -not a supersede. - -All per-turn state — pending TTS text, the synthesis/order queue, interruption, -and hub flush — is keyed by participant id, so concurrent participants on one -hub never share a buffer or misroute each other's audio; a departing -participant's transport sender is released on leave. NAT applications create -the callable with `xr_ai_nat.adapters.as_voice_handler`; transcript recording -is a separate observer rather than a side effect of function invocation. +`VoiceAgent` publishes accepted speech, typed text, participant departure, and +interruption on application-named topics. Application agents subscribe to the +events they own, perform cleanup in their own subscriber methods, and may +publish finite or incremental `VoiceOutput` messages: + +```python +from xr_ai_voice import VOICE_OUTPUT_TOPIC, VoiceOutput + +await runtime.publish( + VOICE_OUTPUT_TOPIC, + VoiceOutput(text="Move your hand away.", interrupt=True), + participant_id="alice", + source="safety-monitor", +) +``` + +Incremental producers reuse a `response_id`, set `final=False` while more +chunks remain, and end with `final=True`. Aggregation is private to voice/TTS, +and producer identity is part of the response key so independent agents cannot +merge output accidentally. Output is serialized per participant; urgent output +sets `interrupt=True` to replace active and queued speech. Producers may copy +the originating query's `timestamp_us` into `VoiceOutput` so the TTS response +preserves the input timestamp. + +Relay telemetry treats `voice.output` as a high-cardinality transport topic and +does not emit runtime scopes per fragment. `VoiceAgent` instead emits one +`voice.response` agent scope for each finite response or completed incremental +stream. Its input contains the combined text, streaming flag, fragment count, +and interrupt flag; metadata identifies the participant, producer, response, +timestamp, and completion status. + +The media pipeline also emits one `voice.stt` function scope per final or +bounded partial-probe transcription and one `voice.tts` function scope per +sentence synthesis. STT inputs contain only byte count, duration, and sample +rate; a nested `voice.stt.result` mark carries the transcript. TTS inputs carry +the sentence being synthesized. Raw audio is never written to Relay events. +These scopes measure provider work and downstream handoff, not client playback. + +`VoiceSession` is the media engine owned by `VoiceAgent`. It manages +readiness, hub transport, VAD/STT, voice gating, TTS, signals, and cleanup. It +The lower-level `VoiceSession.run()`, `enqueue_query()`, and +`enqueue_response()` methods are public for runtime integrations. +`VoiceSession.endpoint` is available only after entering the session, so model +health probes complete before the default hub transport opens its sockets. + +does not execute application handlers. Typed-text ingress is also internal to +`VoiceAgent`. When wake phrases and the listening chime are enabled, the VAD/STT stage probes the opening audio while the user is still speaking. A recognized phrase emits diff --git a/agent-sdk/xr-ai-voice/pyproject.toml b/agent-sdk/xr-ai-voice/pyproject.toml index 3f9b6e5e..b5d26bab 100644 --- a/agent-sdk/xr-ai-voice/pyproject.toml +++ b/agent-sdk/xr-ai-voice/pyproject.toml @@ -10,6 +10,9 @@ name = "xr-ai-voice" version = "0.1.0" requires-python = ">=3.11,<3.13" dependencies = [ + "nemo-relay>=0.7.2,<0.8", + "pydantic>=2.10", + "xr-ai-agent-runtime", "xr-ai-hub-client", "xr-ai-logging", "xr-ai-models", @@ -23,6 +26,7 @@ dependencies = [ ] [tool.uv.sources] +xr-ai-agent-runtime = { path = "../xr-ai-agent-runtime", editable = true } xr-ai-hub-client = { path = "../xr-ai-hub-client", editable = true } xr-ai-logging = { path = "../../utils/xr-ai-logging", editable = true } xr-ai-models = { path = "../xr-ai-models", editable = true } diff --git a/agent-sdk/xr-ai-voice/xr_ai_voice/__init__.py b/agent-sdk/xr-ai-voice/xr_ai_voice/__init__.py index 6ce54358..78b793f2 100644 --- a/agent-sdk/xr-ai-voice/xr_ai_voice/__init__.py +++ b/agent-sdk/xr-ai-voice/xr_ai_voice/__init__.py @@ -1,26 +1,33 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -"""Public voice-session API for XR agents. +"""Public voice runtime for XR agents. Pipecat, audio framing, and pipeline processors are implementation details. -Applications create a :class:`VoiceSession` and supply an async -:class:`VoiceHandler` callable. +Applications register :class:`VoiceAgent`; :class:`VoiceSession` owns its media +pipeline and service lifecycle. """ -from ._handler import VoiceHandler, VoiceQuery, VoiceResponse, VoiceTurn from ._processors import VadConfig +from ._runtime import ( + VOICE_OUTPUT_TOPIC, + UserQuery, + VoiceAgent, + VoiceInterrupted, + VoiceOutput, + VoiceParticipantLeft, +) from ._session import VoiceSession -from ._text_input import TextMessageInput from ._transport import HubVoiceTransport __all__ = [ "HubVoiceTransport", - "TextMessageInput", "VadConfig", - "VoiceHandler", - "VoiceQuery", - "VoiceResponse", + "VOICE_OUTPUT_TOPIC", + "UserQuery", + "VoiceAgent", + "VoiceInterrupted", + "VoiceOutput", + "VoiceParticipantLeft", "VoiceSession", - "VoiceTurn", ] diff --git a/agent-sdk/xr-ai-voice/xr_ai_voice/_handler.py b/agent-sdk/xr-ai-voice/xr_ai_voice/_handler.py deleted file mode 100644 index 3ac61668..00000000 --- a/agent-sdk/xr-ai-voice/xr_ai_voice/_handler.py +++ /dev/null @@ -1,44 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -"""Application-facing contracts for participant-aware voice turns.""" - -from __future__ import annotations - -from collections.abc import AsyncIterator, Awaitable, Callable -from dataclasses import dataclass -from typing import Literal, TypeAlias - - -@dataclass(frozen=True, slots=True) -class VoiceQuery: - """One participant query produced by speech or typed input.""" - - participant_id: str - text: str - fresh_match: bool - #: Unix-epoch microseconds anchoring the query — the utterance PTS carried - #: from the hub for spoken input, or ``time.time_ns() // 1_000`` at submit - #: time for typed input. Use it to anchor time-relative tool calls (e.g. a - #: "what did I just show you" recorded-frame lookup) to when the user spoke. - timestamp_us: int - - -VoiceResponse: TypeAlias = str | AsyncIterator[str] -VoiceHandler: TypeAlias = Callable[[VoiceQuery], Awaitable[VoiceResponse]] - - -@dataclass(frozen=True, slots=True) -class VoiceTurn: - """One completed user or agent turn observed by a voice session.""" - - participant_id: str - role: Literal["user", "agent"] - #: Unix-epoch microseconds for the turn — the originating query's - #: ``timestamp_us`` (both the user and the agent turn of one exchange share - #: it, so a transcript orders the pair deterministically). - timestamp_us: int - text: str - - -__all__ = ["VoiceHandler", "VoiceQuery", "VoiceResponse", "VoiceTurn"] diff --git a/agent-sdk/xr-ai-voice/xr_ai_voice/_pipeline.py b/agent-sdk/xr-ai-voice/xr_ai_voice/_pipeline.py index ff095f35..022be2cb 100644 --- a/agent-sdk/xr-ai-voice/xr_ai_voice/_pipeline.py +++ b/agent-sdk/xr-ai-voice/xr_ai_voice/_pipeline.py @@ -5,7 +5,7 @@ One call composes: - input → VadStt → VoiceGate → handler → StreamingTts → output + input → VadStt → VoiceGate → runtime I/O → StreamingTts → output and returns the assembled :class:`Pipeline` plus a :class:`PipelineWorker` ready for :meth:`WorkerRunner.run`. :class:`VoiceSession` owns this private @@ -19,7 +19,7 @@ from xr_ai_models import STTService, TTSService from xr_ai_voicegate import VoiceGateConfig -from ._processors.handler import _VoiceHandlerProcessor +from ._processors.io import _VoiceIOProcessor from ._processors.streaming_tts import StreamingTtsProcessor from ._processors.vad_stt import VadConfig, VadSttProcessor from ._processors.voice_gate import VoiceGateProcessor @@ -31,7 +31,7 @@ def _build_voice_pipeline( transport: HubVoiceTransport, stt: STTService, tts: TTSService, - handler_processor: _VoiceHandlerProcessor, + io_processor: _VoiceIOProcessor, vad_cfg: VadConfig, voice_gate_cfg: VoiceGateConfig, text_topic: str = "agent.response", @@ -81,7 +81,7 @@ def _build_voice_pipeline( transport.input(), vad_stt, voice_gate_proc, - handler_processor, + io_processor, streaming_tts, transport.output(), ]) diff --git a/agent-sdk/xr-ai-voice/xr_ai_voice/_processors/__init__.py b/agent-sdk/xr-ai-voice/xr_ai_voice/_processors/__init__.py index e3a5ac62..bc03c51e 100644 --- a/agent-sdk/xr-ai-voice/xr_ai_voice/_processors/__init__.py +++ b/agent-sdk/xr-ai-voice/xr_ai_voice/_processors/__init__.py @@ -4,13 +4,13 @@ """Private FrameProcessors that compose the unified voice pipeline.""" from __future__ import annotations -from .handler import _VoiceHandlerProcessor +from .io import _VoiceIOProcessor from .streaming_tts import StreamingTtsProcessor from .vad_stt import VadConfig, VadSttProcessor from .voice_gate import VoiceGateProcessor __all__ = [ - "_VoiceHandlerProcessor", + "_VoiceIOProcessor", "StreamingTtsProcessor", "VadConfig", "VadSttProcessor", diff --git a/agent-sdk/xr-ai-voice/xr_ai_voice/_processors/handler.py b/agent-sdk/xr-ai-voice/xr_ai_voice/_processors/handler.py deleted file mode 100644 index 15901bc8..00000000 --- a/agent-sdk/xr-ai-voice/xr_ai_voice/_processors/handler.py +++ /dev/null @@ -1,338 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -"""Private Pipecat processor that executes a public voice handler.""" - -from __future__ import annotations - -import asyncio -import inspect -import time -from collections import deque -from collections.abc import Awaitable, Callable -from typing import TYPE_CHECKING - -from loguru import logger -from pipecat.frames.frames import ( - CancelFrame, - EndFrame, - Frame, - InterruptionFrame, - TextFrame, - UserStartedSpeakingFrame, -) -from pipecat.processors.frame_processor import FrameDirection, FrameProcessor - -from .._frames import ( - AssistantResponseEndFrame, - GatedQueryFrame, - ParticipantJoinedFrame, - ParticipantLeftFrame, -) -from .._handler import VoiceHandler, VoiceQuery, VoiceTurn - -if TYPE_CHECKING: - from .._transport import HubVoiceTransport - - -class _VoiceHandlerProcessor(FrameProcessor): - """Run a handler while preserving participant-aware pipeline semantics.""" - - def __init__( - self, - handler: VoiceHandler, - *, - transport: "HubVoiceTransport | None" = None, - observer: Callable[[VoiceTurn], Awaitable[None]] | None = None, - on_participant_joined: Callable[[str], Awaitable[None] | None] | None = None, - on_participant_left: Callable[[str], Awaitable[None] | None] | None = None, - on_user_started_speaking: Callable[[str], Awaitable[None] | None] | None = None, - on_query_superseded: Callable[[str], Awaitable[None] | None] | None = None, - interrupt_on_supersede: bool = False, - queue_queries: bool = False, - ) -> None: - super().__init__() - self._handler = handler - self._turn_observer = observer - self._on_participant_joined = on_participant_joined - self._on_participant_left = on_participant_left - self._on_user_started_speaking = on_user_started_speaking - self._on_query_superseded = on_query_superseded - self._interrupt_on_supersede = interrupt_on_supersede - self._queue_queries = queue_queries - self._inflight: dict[str, asyncio.Task[None]] = {} - self._queued: dict[str, deque[GatedQueryFrame]] = {} - self._turn_tokens: dict[str, object] = {} - # "Has this participant spoken a turn before." A finished turn may still - # have TTS draining downstream, so a follow-up turn interrupts to clear - # that lingering audio — tracked by history, not live task state. NOTE: - # this drives the downstream interrupt only; the on_query_superseded - # callback fires solely on an actual in-flight replacement (see - # _spawn_query), never merely because a turn was seen before. - self._seen_query: set[str] = set() - # Joined participants receive speech hooks even before their first turn. - self._joined: set[str] = set() - # A supplied transport enables automatic single-participant routing. - self._transport = transport - - async def enqueue_query( - self, - participant_id: str, - text: str, - *, - fresh_match: bool = False, - pts_us: int | None = None, - ) -> None: - """Submit text through the same participant-aware path as voice input.""" - await self._spawn_query( - GatedQueryFrame( - participant_id=participant_id, - text=text, - fresh_match=fresh_match, - pts_us=pts_us if pts_us is not None else time.time_ns() // 1_000, - ) - ) - - async def process_frame(self, frame: Frame, direction: FrameDirection) -> None: - # Pipecat owns interruption metrics, but cannot cancel handler tasks. - await super().process_frame(frame, direction) - - if isinstance(frame, UserStartedSpeakingFrame): - # Speech onset may be acoustic echo; only InterruptionFrame cancels. - await self._dispatch_user_started_speaking(frame.transport_source) - await self.push_frame(frame, direction) - return - - if isinstance(frame, InterruptionFrame): - pid = frame.transport_source - if pid: - logger.info("voice handler cancel pid={!r} reason=interruption", pid) - self._cancel_pid(pid) - else: - if self._inflight: - for p in list(self._inflight): - logger.info("voice handler cancel pid={!r} reason=interruption", p) - self._cancel_all_inflight() - await self.push_frame(frame, direction) - return - - if isinstance(frame, (EndFrame, CancelFrame)): - # Pipeline shutdown: cancel and await every in-flight handler task so - # a turn cannot keep emitting text — or writing transcripts through a - # turn observer — after the session has ended. - await self._shutdown_inflight() - await self.push_frame(frame, direction) - return - - if isinstance(frame, GatedQueryFrame): - await self._spawn_query(frame) - return - - if isinstance(frame, ParticipantJoinedFrame): - self._joined.add(frame.participant_id) - logger.info("voice participant joined pid={!r}", frame.participant_id) - if self._transport is not None: - self._transport.set_target_participant(frame.participant_id) - await self._notify(self._on_participant_joined, frame.participant_id) - await self.push_frame(frame, direction) - return - - if isinstance(frame, ParticipantLeftFrame): - self._joined.discard(frame.participant_id) - self._seen_query.discard(frame.participant_id) - logger.info("voice participant left pid={!r}", frame.participant_id) - if self._transport is not None: - self._transport.cleanup_participant(frame.participant_id) - await self._notify(self._on_participant_left, frame.participant_id) - self._cancel_pid(frame.participant_id) - await self.push_frame(frame, direction) - return - - await self.push_frame(frame, direction) - - async def _dispatch_user_started_speaking(self, pid: str | None) -> None: - # Third-party frames may omit transport_source; notify all joined users. - targets = [pid] if (pid and pid in self._joined) else list(self._joined) - for p in targets: - await self._notify(self._on_user_started_speaking, p) - - async def _notify(self, callback: Callable[[str], Awaitable[None] | None] | None, pid: str) -> None: - if callback is None: - return None - try: - result = callback(pid) - if inspect.isawaitable(result): - await result - except Exception: - logger.exception("voice session callback raised pid={!r}", pid) - - async def _observe(self, turn: VoiceTurn) -> None: - if self._turn_observer is None: - return - try: - await self._turn_observer(turn) - except Exception: - logger.exception("voice session observer raised pid={!r} role={}", turn.participant_id, turn.role) - - async def _spawn_query(self, frame: GatedQueryFrame) -> None: - pid = frame.participant_id - logger.info( - "voice handler dispatch pid={!r} fresh_match={}", - pid, - frame.fresh_match, - ) - current = self._inflight.get(pid) - is_active = current is not None and not current.done() - - if is_active and self._queue_queries: - # A turn is still running: queue this one as a follow-up. This is - # NOT a supersede — the running turn completes and this runs after - # it — so on_query_superseded does not fire. - pending = self._queued.setdefault(pid, deque()) - pending.append(frame) - logger.info("voice handler queued pid={!r} depth={}", pid, len(pending)) - return - - if is_active: - # Non-queue mode: the new query actually replaces the in-flight turn. - # That replacement is the only real supersede. - logger.info("voice handler superseded pid={!r}", pid) - await self._notify(self._on_query_superseded, pid) - self._cancel_pid(pid) - - # A prior turn — even a finished one whose TTS may still be draining — - # means the new turn interrupts downstream audio so it starts clean. - had_prior = pid in self._seen_query - self._seen_query.add(pid) - await self._start_query(frame, interrupt=had_prior) - - async def _start_query(self, frame: GatedQueryFrame, *, interrupt: bool) -> None: - pid = frame.participant_id - if interrupt and self._interrupt_on_supersede: - # Tag the pid so the downstream TTS drain/flush scopes to this - # participant instead of every participant's audio. - interruption = InterruptionFrame() - interruption.transport_source = pid - await self.push_frame(interruption) - token = object() - self._turn_tokens[pid] = token - task = asyncio.create_task( - self._run_query(frame, token), - name=f"voice-query-{pid}", - ) - self._inflight[pid] = task - - async def _run_query(self, frame: GatedQueryFrame, token: object) -> None: - pid = frame.participant_id - # The end frame carries one assembled data-channel response per turn. - accumulated: list[str] = [] - cancelled = False - try: - query = VoiceQuery( - participant_id=pid, - text=frame.text, - fresh_match=frame.fresh_match, - timestamp_us=frame.pts_us, - ) - await self._observe( - VoiceTurn(participant_id=pid, role="user", timestamp_us=frame.pts_us, text=frame.text) - ) - result = await self._handler(query) - if isinstance(result, str): - if result and self._turn_tokens.get(pid) is token: - accumulated.append(result) - await self._push_text(result, pid=pid) - return - try: - async for chunk in result: - if not chunk or self._turn_tokens.get(pid) is not token: - continue - accumulated.append(chunk) - await self._push_text(chunk, pid=pid) - finally: - close = getattr(result, "aclose", None) - if close is not None: - await close() - except asyncio.CancelledError: - cancelled = True - raise - except Exception: - logger.exception("voice handler raised pid={!r}", pid) - finally: - # A cancelled turn must not emit a partial response as final data. - is_current = self._turn_tokens.get(pid) is token - if not cancelled and is_current: - logger.info("voice handler query complete pid={!r}", pid) - try: - response = "".join(accumulated) - await self._observe( - VoiceTurn(participant_id=pid, role="agent", timestamp_us=frame.pts_us, text=response) - ) - await self.push_frame( - AssistantResponseEndFrame( - pid=pid, - text=response, - pts_us=frame.pts_us, - ) - ) - except Exception: - logger.exception("emit AssistantResponseEndFrame failed pid={!r}", pid) - - if is_current: - self._turn_tokens.pop(pid, None) - self._inflight.pop(pid, None) - pending = self._queued.get(pid) - if pending: - next_frame = pending.popleft() - if not pending: - self._queued.pop(pid, None) - await self._start_query(next_frame, interrupt=True) - - async def _push_text(self, text: str, *, pid: str) -> None: - """Push a ``TextFrame`` tagged with the participant id. - - ``transport_destination`` flows through the pipeline to the - ``StreamingTtsProcessor``, which copies it onto the - ``OutputAudioRawFrame``s it emits so the output transport knows - which participant to address. Without this tag, the empty - string ends up on every downstream send and the hub drops the - audio. - """ - f = TextFrame(text=text) - f.transport_destination = pid - await self.push_frame(f) - - def _cancel_pid(self, pid: str) -> None: - self._turn_tokens.pop(pid, None) - self._queued.pop(pid, None) - task = self._inflight.pop(pid, None) - if task is not None and not task.done(): - task.cancel() - - async def _shutdown_inflight(self) -> None: - """Cancel every in-flight handler task and await its teardown. - - Awaiting matters: a cancelled turn still runs its ``finally`` (turn - observer, end frame), so returning before that lands would let a - transcript write outlive the session. - """ - tasks = [t for t in self._inflight.values() if not t.done()] - self._cancel_all_inflight() - for task in tasks: - try: - await task - except asyncio.CancelledError: - pass # expected — we cancelled it ourselves - except Exception: - logger.exception("voice handler task raised during shutdown") - - def _cancel_all_inflight(self) -> None: - for task in self._inflight.values(): - if not task.done(): - task.cancel() - self._queued.clear() - self._inflight.clear() - self._turn_tokens.clear() - - -__all__ = ["_VoiceHandlerProcessor"] diff --git a/agent-sdk/xr-ai-voice/xr_ai_voice/_processors/io.py b/agent-sdk/xr-ai-voice/xr_ai_voice/_processors/io.py new file mode 100644 index 00000000..4874bc54 --- /dev/null +++ b/agent-sdk/xr-ai-voice/xr_ai_voice/_processors/io.py @@ -0,0 +1,348 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Private Pipecat processor for runtime voice input and output.""" + +from __future__ import annotations + +import asyncio +import inspect +import time +from collections import deque +from collections.abc import Awaitable, Callable +from dataclasses import dataclass +from typing import TYPE_CHECKING + +from loguru import logger +from pipecat.frames.frames import ( + CancelFrame, + EndFrame, + Frame, + InterruptionFrame, + TextFrame, +) +from pipecat.processors.frame_processor import FrameDirection, FrameProcessor + +from .._frames import ( + AssistantResponseEndFrame, + GatedQueryFrame, + ParticipantJoinedFrame, + ParticipantLeftFrame, +) +from .._types import VoiceInputSink, VoiceQuery, VoiceResponse + +if TYPE_CHECKING: + from .._transport import HubVoiceTransport + + +@dataclass(frozen=True, slots=True) +class _QueuedResponse: + participant_id: str + response: VoiceResponse + pts_us: int + interrupt: bool = False + + +class _VoiceIOProcessor(FrameProcessor): + """Publish accepted input and serialize participant-scoped voice output.""" + + def __init__( + self, + input_sink: VoiceInputSink, + *, + transport: HubVoiceTransport | None = None, + on_participant_left: Callable[[str], Awaitable[None] | None] | None = None, + on_interrupted: Callable[[str | None], Awaitable[None] | None] | None = None, + interrupt_on_supersede: bool = False, + ) -> None: + super().__init__() + self._input_sink = input_sink + self._transport = transport + self._on_participant_left = on_participant_left + self._on_interrupted = on_interrupted + self._interrupt_on_supersede = interrupt_on_supersede + self._inflight: dict[str, asyncio.Task[None]] = {} + self._input_tasks: set[asyncio.Task[None]] = set() + self._queued: dict[str, deque[_QueuedResponse]] = {} + self._turn_tokens: dict[str, object] = {} + # Completed output may still be draining through TTS when new input arrives. + self._seen_output: set[str] = set() + + async def enqueue_query( + self, + participant_id: str, + text: str, + *, + pts_us: int | None = None, + ) -> None: + """Submit typed text through the same path as accepted speech.""" + + await self._spawn_query( + GatedQueryFrame( + participant_id=participant_id, + text=text, + fresh_match=False, + pts_us=pts_us if pts_us is not None else time.time_ns() // 1_000, + ) + ) + + async def enqueue_response( + self, + participant_id: str, + response: VoiceResponse, + *, + interrupt: bool = False, + pts_us: int | None = None, + ) -> None: + """Submit finite or incremental assistant output.""" + + await self._spawn_response( + _QueuedResponse( + participant_id=participant_id, + response=response, + interrupt=interrupt, + pts_us=pts_us if pts_us is not None else time.time_ns() // 1_000, + ) + ) + + async def process_frame(self, frame: Frame, direction: FrameDirection) -> None: + await super().process_frame(frame, direction) + + if isinstance(frame, InterruptionFrame): + pid = frame.transport_source + if pid: + logger.info("voice cancel pid={!r} reason=interruption", pid) + await self._cancel_pid(pid) + else: + await self._cancel_all() + await self.push_frame(frame, direction) + await self._notify_interrupted(pid) + return + + if isinstance(frame, (EndFrame, CancelFrame)): + await self._shutdown() + await self.push_frame(frame, direction) + return + + if isinstance(frame, GatedQueryFrame): + await self._spawn_query(frame) + return + + if isinstance(frame, ParticipantJoinedFrame): + logger.info("voice participant joined pid={!r}", frame.participant_id) + if self._transport is not None: + self._transport.set_target_participant(frame.participant_id) + await self.push_frame(frame, direction) + return + + if isinstance(frame, ParticipantLeftFrame): + self._seen_output.discard(frame.participant_id) + logger.info("voice participant left pid={!r}", frame.participant_id) + if self._transport is not None: + self._transport.cleanup_participant(frame.participant_id) + await self._notify_left(frame.participant_id) + await self._cancel_pid(frame.participant_id) + await self.push_frame(frame, direction) + return + + await self.push_frame(frame, direction) + + async def _spawn_query(self, frame: GatedQueryFrame) -> None: + pid = frame.participant_id + logger.info("voice input pid={!r}", pid) + if current := self._inflight.get(pid): + if not current.done(): + await self._cancel_pid(pid) + interrupt = self._interrupt_on_supersede and pid in self._seen_output + if interrupt: + frame_to_push = InterruptionFrame() + frame_to_push.transport_source = pid + await self.push_frame(frame_to_push) + task = asyncio.create_task( + self._run_query(frame), + name=f"voice-input-{pid}", + ) + self._input_tasks.add(task) + task.add_done_callback(self._input_tasks.discard) + + async def _run_query(self, frame: GatedQueryFrame) -> None: + pid = frame.participant_id + try: + await self._input_sink( + VoiceQuery( + participant_id=pid, + text=frame.text, + timestamp_us=frame.pts_us, + ) + ) + except asyncio.CancelledError: + raise + except Exception: + logger.exception("voice input sink raised pid={!r}", pid) + + async def _spawn_response(self, item: _QueuedResponse) -> None: + pid = item.participant_id + if isinstance(item.response, str) and not item.response.strip(): + return + current = self._inflight.get(pid) + if not item.interrupt and ( + (current is not None and not current.done()) or self._queued.get(pid) + ): + pending = self._queued.setdefault(pid, deque()) + pending.append(item) + logger.info("voice response queued pid={!r} depth={}", pid, len(pending)) + return + if item.interrupt: + await self._cancel_pid(pid) + interruption = InterruptionFrame() + interruption.transport_source = pid + await self.push_frame(interruption) + await self._start_response(item) + + async def _start_response(self, item: _QueuedResponse) -> None: + pid = item.participant_id + self._seen_output.add(pid) + token = object() + self._turn_tokens[pid] = token + task = asyncio.create_task( + self._run_response(item, token), + name=f"voice-response-{pid}", + ) + self._inflight[pid] = task + + async def _run_response(self, item: _QueuedResponse, token: object) -> None: + pid = item.participant_id + chunks: list[str] = [] + cancelled = False + try: + await self._consume_response( + item.response, + pid=pid, + token=token, + accumulated=chunks, + ) + except asyncio.CancelledError: + cancelled = True + raise + except Exception: + logger.exception("voice response failed pid={!r}", pid) + finally: + await self._close_response(item.response) + is_current = self._turn_tokens.get(pid) is token + if not cancelled and is_current: + try: + await self.push_frame( + AssistantResponseEndFrame( + pid=pid, + text="".join(chunks), + pts_us=item.pts_us, + ) + ) + except Exception: + logger.exception("emit voice response end failed pid={!r}", pid) + if is_current: + self._turn_tokens.pop(pid, None) + self._inflight.pop(pid, None) + await self._start_next(pid) + + async def _consume_response( + self, + response: VoiceResponse, + *, + pid: str, + token: object, + accumulated: list[str], + ) -> None: + if isinstance(response, str): + if response and self._turn_tokens.get(pid) is token: + accumulated.append(response) + await self._push_text(response, pid=pid) + return + async for chunk in response: + if self._turn_tokens.get(pid) is not token: + return + if not chunk: + continue + accumulated.append(chunk) + await self._push_text(chunk, pid=pid) + + async def _start_next(self, pid: str) -> None: + pending = self._queued.get(pid) + if not pending: + return + item = pending.popleft() + if not pending: + self._queued.pop(pid, None) + await self._start_response(item) + + async def _push_text(self, text: str, *, pid: str) -> None: + frame = TextFrame(text=text) + frame.transport_destination = pid + await self.push_frame(frame) + + async def _notify_left(self, pid: str) -> None: + if self._on_participant_left is None: + return + try: + result = self._on_participant_left(pid) + if inspect.isawaitable(result): + await result + except Exception: + logger.exception("voice participant-left callback raised pid={!r}", pid) + + async def _notify_interrupted(self, pid: str | None) -> None: + if self._on_interrupted is None: + return + try: + result = self._on_interrupted(pid) + if inspect.isawaitable(result): + await result + except Exception: + logger.exception("voice interruption callback raised pid={!r}", pid) + + async def _cancel_pid(self, pid: str) -> None: + self._turn_tokens.pop(pid, None) + for item in self._queued.pop(pid, ()): + await self._close_response(item.response) + task = self._inflight.pop(pid, None) + if task is not None and not task.done(): + task.cancel() + + @staticmethod + async def _close_response(response: VoiceResponse) -> None: + if isinstance(response, str): + return + close = getattr(response, "aclose", None) + if close is None: + return + result = close() + if inspect.isawaitable(result): + _ = await result + + async def _shutdown(self) -> None: + tasks = [ + task + for task in (*self._inflight.values(), *self._input_tasks) + if not task.done() + ] + await self._cancel_all() + if tasks: + await asyncio.gather(*tasks, return_exceptions=True) + + async def _cancel_all(self) -> None: + for task in self._inflight.values(): + if not task.done(): + task.cancel() + for pending in self._queued.values(): + for item in pending: + await self._close_response(item.response) + self._queued.clear() + self._inflight.clear() + self._turn_tokens.clear() + for task in self._input_tasks: + if not task.done(): + task.cancel() + self._input_tasks.clear() + + +__all__ = ["_VoiceIOProcessor"] diff --git a/agent-sdk/xr-ai-voice/xr_ai_voice/_processors/streaming_tts.py b/agent-sdk/xr-ai-voice/xr_ai_voice/_processors/streaming_tts.py index 04cb0221..44605163 100644 --- a/agent-sdk/xr-ai-voice/xr_ai_voice/_processors/streaming_tts.py +++ b/agent-sdk/xr-ai-voice/xr_ai_voice/_processors/streaming_tts.py @@ -30,6 +30,7 @@ import re from typing import TYPE_CHECKING +import nemo_relay from loguru import logger from pipecat.frames.frames import ( CancelFrame, @@ -234,11 +235,21 @@ async def _dispatch_sentence(self, sentence: str, *, pid: str) -> None: st = self._state(pid) st.synth_seq += 1 task = asyncio.create_task( - self._tts.synthesize(sentence), + self._synthesize(sentence, pid=pid), name=f"tts-synth-{pid}-{st.synth_seq}", + context=nemo_relay.fork_asyncio_context(), ) await queue.put((task, pid)) + async def _synthesize(self, text: str, *, pid: str) -> bytes: + with nemo_relay.scope.scope( + "voice.tts", + nemo_relay.ScopeType.Function, + input={"text": text}, + metadata={"participant_id": pid or None}, + ): + return await self._tts.synthesize(text) + async def _sender_loop(self, queue: asyncio.Queue) -> None: """Await each synth task in FIFO order, observe the WAV, and push the decoded audio downstream as ``OutputAudioRawFrame``s.""" diff --git a/agent-sdk/xr-ai-voice/xr_ai_voice/_processors/vad_stt.py b/agent-sdk/xr-ai-voice/xr_ai_voice/_processors/vad_stt.py index 30188ab7..7673fe43 100644 --- a/agent-sdk/xr-ai-voice/xr_ai_voice/_processors/vad_stt.py +++ b/agent-sdk/xr-ai-voice/xr_ai_voice/_processors/vad_stt.py @@ -24,6 +24,7 @@ from dataclasses import dataclass from typing import Awaitable, Callable +import nemo_relay from loguru import logger from pipecat.frames.frames import ( CancelFrame, @@ -225,7 +226,12 @@ async def on_utterance(audio_bytes: bytes, sample_rate: int) -> None: f.transport_source = pid await self.push_frame(f) try: - text = await self._stt.transcribe(audio_bytes, sample_rate=sample_rate) + text = await self._transcribe( + audio_bytes, + sample_rate=sample_rate, + participant_id=pid, + mode="final", + ) except Exception: logger.exception("stt transcribe failed pid={!r}", pid) return @@ -304,7 +310,13 @@ async def _run_partial_probes(self, pid: str) -> None: return try: - text = await self._stt.transcribe(bytes(buf), sample_rate=sr) + text = await self._transcribe( + bytes(buf), + sample_rate=sr, + participant_id=pid, + mode="partial-probe", + attempt=attempt, + ) except asyncio.CancelledError: return except Exception: @@ -334,6 +346,42 @@ async def _run_partial_probes(self, pid: str) -> None: if decision is None: return + async def _transcribe( + self, + audio: bytes, + *, + sample_rate: int, + participant_id: str, + mode: str, + attempt: int | None = None, + ) -> str: + metadata: dict[str, object] = { + "participant_id": participant_id, + "mode": mode, + } + if attempt is not None: + metadata["attempt"] = attempt + with nemo_relay.scope.scope( + "voice.stt", + nemo_relay.ScopeType.Function, + input={ + "audio_bytes": len(audio), + "audio_duration_ms": round( + (len(audio) // 2) * 1_000 / max(sample_rate, 1), + 3, + ), + "sample_rate": sample_rate, + }, + metadata=metadata, + ): + text = await self._stt.transcribe(audio, sample_rate=sample_rate) + nemo_relay.scope.event( + "voice.stt.result", + data={"text": text}, + metadata={"status": "completed" if text else "empty"}, + ) + return text + async def _emit_early_stop(self, pid: str, text: str) -> None: """Emit the interrupt sequence for a STOP matched by a partial probe.""" diff --git a/agent-sdk/xr-ai-voice/xr_ai_voice/_runtime.py b/agent-sdk/xr-ai-voice/xr_ai_voice/_runtime.py new file mode 100644 index 00000000..5729bb10 --- /dev/null +++ b/agent-sdk/xr-ai-voice/xr_ai_voice/_runtime.py @@ -0,0 +1,479 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Bidirectional agent-runtime boundary for participant-aware voice.""" + +from __future__ import annotations + +import asyncio +from collections.abc import AsyncIterator, Callable, Iterable +from dataclasses import dataclass, field +from datetime import UTC, datetime + +import nemo_relay +from loguru import logger +from pydantic import BaseModel, ConfigDict, Field, model_validator +from xr_ai_hub import DataMessage +from xr_ai_runtime import Agent, AgentRuntime, RuntimeContext, Topic, subscribe + +from ._session import VoiceSession +from ._types import VoiceQuery + +QueryTransform = Callable[[str], str] + +_OPEN_STREAM_CAPACITY = 1024 +_CLOSED_STREAM_CAPACITY = 1024 + + +class UserQuery(BaseModel): + """One accepted user query emitted by the voice input boundary.""" + + model_config = ConfigDict(extra="forbid") + + text: str = Field(min_length=1) + timestamp_us: int = Field(ge=0) + + +class VoiceParticipantLeft(BaseModel): + """Notification that one participant left the voice transport.""" + + model_config = ConfigDict(extra="forbid") + + +class VoiceInterrupted(BaseModel): + """Notification that participant-scoped or global voice work was interrupted.""" + + model_config = ConfigDict(extra="forbid") + + +class VoiceOutput(BaseModel): + """One complete response or one chunk of an incremental voice response.""" + + model_config = ConfigDict(extra="forbid") + + text: str = "" + response_id: str | None = Field(default=None, min_length=1) + final: bool = True + interrupt: bool = False + timestamp_us: int | None = Field(default=None, ge=0) + + @model_validator(mode="after") + def validate_boundary(self) -> VoiceOutput: + """Require identity for incremental output and text for finite output.""" + + if not self.final and self.response_id is None: + raise ValueError("non-final voice output needs a response_id") + if self.response_id is None and not self.text.strip(): + raise ValueError("complete voice output must contain text") + if self.interrupt and self.response_id is not None and not self.text.strip(): + raise ValueError("an empty stream terminator cannot interrupt output") + return self + + +VOICE_OUTPUT_TOPIC = Topic("voice.output", VoiceOutput, telemetry="none") + + +@dataclass(slots=True) +class _ResponseTrace: + participant_id: str + source: str + response_id: str + correlation_id: str + timestamp_us: int + interrupt: bool + started_at: datetime = field(default_factory=lambda: datetime.now(UTC)) + fragments: list[str] = field(default_factory=list) + + +class _ResponseStream(AsyncIterator[str]): + def __init__( + self, + capacity: int, + on_close: Callable[[_ResponseStream], None], + ) -> None: + self.queue: asyncio.Queue[str] = asyncio.Queue(maxsize=capacity) + self.closed = asyncio.Event() + self._on_close = on_close + + async def send(self, text: str) -> None: + if not text or self.closed.is_set(): + return + queued = asyncio.create_task(self.queue.put(text)) + closed = asyncio.create_task(self.closed.wait()) + try: + done, _ = await asyncio.wait( + (queued, closed), + return_when=asyncio.FIRST_COMPLETED, + ) + if closed in done and not queued.done(): + queued.cancel() + finally: + if not queued.done(): + queued.cancel() + if not closed.done(): + closed.cancel() + await asyncio.gather(queued, closed, return_exceptions=True) + + def __aiter__(self) -> AsyncIterator[str]: + return self + + async def __anext__(self) -> str: + if not self.queue.empty(): + return self.queue.get_nowait() + if self.closed.is_set(): + raise StopAsyncIteration + queued = asyncio.create_task(self.queue.get()) + closed = asyncio.create_task(self.closed.wait()) + try: + done, _ = await asyncio.wait( + (queued, closed), + return_when=asyncio.FIRST_COMPLETED, + ) + if queued in done: + return queued.result() + raise StopAsyncIteration + except asyncio.CancelledError: + await self.aclose() + raise + finally: + if not queued.done(): + queued.cancel() + if not closed.done(): + closed.cancel() + await asyncio.gather(queued, closed, return_exceptions=True) + + async def aclose(self) -> None: + """Release blocked producers and evict the stream from its owner.""" + + if self.closed.is_set(): + return + self.closed.set() + self._on_close(self) + + +class VoiceAgent(Agent): + """Own voice media lifecycle and bridge runtime input and output topics.""" + + def __init__( + self, + session: VoiceSession, + *, + query_topic: Topic[UserQuery], + response_capacity: int = 32, + text_input: bool = True, + text_ignore_topics: Iterable[str] | None = None, + text_transform: QueryTransform | None = None, + participant_left_topic: Topic[VoiceParticipantLeft] | None = None, + interrupted_topic: Topic[VoiceInterrupted] | None = None, + interrupt_on_supersede: bool = False, + ) -> None: + if response_capacity <= 0: + raise ValueError("voice response capacity must be positive") + super().__init__() + self.session = session + self.query_topic = query_topic + self.response_capacity = response_capacity + self.text_input = text_input + self.text_ignore_topics = ( + tuple(text_ignore_topics) if text_ignore_topics is not None else (session.text_topic,) + ) + self.text_transform = text_transform + self.participant_left_topic = participant_left_topic + self.interrupted_topic = interrupted_topic + self.interrupt_on_supersede = interrupt_on_supersede + self._runtime: AgentRuntime | None = None + self._source = "voice" + self._output_lock = asyncio.Lock() + self._streams: dict[tuple[str, str, str], _ResponseStream] = {} + self._response_traces: dict[tuple[str, str, str], _ResponseTrace] = {} + self._closed_streams: dict[tuple[str, str, str], None] = {} + + async def run(self, runtime: AgentRuntime, *, source: str = "voice") -> None: + """Run the owned voice session and bridge it to a running runtime.""" + + if self._runtime is not None: + raise RuntimeError("voice agent is already running") + if not runtime.running: + raise RuntimeError("agent runtime must be running") + if not source.strip(): + raise ValueError("voice agent source must not be empty") + self._runtime = runtime + self._source = source + unsubscribe: Callable[[], None] | None = None + try: + await self.session.__aenter__() + try: + if self.text_input: + unsubscribe = self.session.endpoint.on_data(self._on_data) + await self.session.run( + self._publish_input, + on_participant_left=( + self._publish_participant_left + if self.participant_left_topic is not None + else None + ), + on_interrupted=( + self._publish_interrupted + if self.interrupted_topic is not None + else None + ), + interrupt_on_supersede=self.interrupt_on_supersede, + ) + finally: + if unsubscribe is not None: + unsubscribe() + await asyncio.gather( + *(stream.aclose() for stream in tuple(self._streams.values())) + ) + self._closed_streams.clear() + await self.session.close() + finally: + self._runtime = None + self._source = "voice" + + @subscribe(VOICE_OUTPUT_TOPIC) + async def output(self, output: VoiceOutput, ctx: RuntimeContext) -> None: + """Send one voice message using participant and producer metadata.""" + + metadata = ctx.metadata + participant_id = metadata.participant_id + if participant_id is None: + raise ValueError("voice output requires a participant") + timestamp_us = ( + output.timestamp_us + if output.timestamp_us is not None + else metadata.timestamp_us + ) + if output.response_id is None: + async with self._output_lock: + with self._response_scope( + participant_id=participant_id, + source=metadata.source, + response_id=None, + correlation_id=metadata.correlation_id, + text=output.text, + fragment_count=1, + interrupt=output.interrupt, + timestamp_us=timestamp_us, + streaming=False, + status="completed", + ): + await self.session.enqueue_response( + participant_id, + output.text, + interrupt=output.interrupt, + pts_us=timestamp_us, + ) + return + + key = (participant_id, metadata.source, output.response_id) + stream: _ResponseStream + async with self._output_lock: + if key in self._closed_streams: + return + existing = self._streams.get(key) + if existing is None: + if output.final: + if not output.text.strip(): + raise ValueError("voice stream terminator has no open response") + with self._response_scope( + participant_id=participant_id, + source=metadata.source, + response_id=output.response_id, + correlation_id=metadata.correlation_id, + text=output.text, + fragment_count=1, + interrupt=output.interrupt, + timestamp_us=timestamp_us, + streaming=False, + status="completed", + ): + await self.session.enqueue_response( + participant_id, + output.text, + interrupt=output.interrupt, + pts_us=timestamp_us, + ) + return + if len(self._streams) >= _OPEN_STREAM_CAPACITY: + oldest_key = next(iter(self._streams)) + await self._streams[oldest_key].aclose() + stream = _ResponseStream( + self.response_capacity, + lambda closed: self._discard_stream(key, closed), + ) + trace = _ResponseTrace( + participant_id=participant_id, + source=metadata.source, + response_id=output.response_id, + correlation_id=metadata.correlation_id, + timestamp_us=timestamp_us, + interrupt=output.interrupt, + ) + await self.session.enqueue_response( + participant_id, + stream, + interrupt=output.interrupt, + pts_us=timestamp_us, + ) + if stream.closed.is_set(): + self._remember_closed_stream(key) + return + self._streams[key] = stream + self._response_traces[key] = trace + else: + stream = existing + if output.interrupt: + raise ValueError("only the first chunk of a voice stream may interrupt") + + trace = self._response_traces[key] + trace.fragments.append(output.text) + trace.interrupt = trace.interrupt or output.interrupt + + await stream.send(output.text) + if output.final: + async with self._output_lock: + if self._streams.get(key) is stream: + self._finish_response_trace(key, status="completed") + await stream.aclose() + + async def _publish_input(self, query: VoiceQuery) -> None: + runtime = self._running_runtime() + await runtime.publish( + self.query_topic, + UserQuery(text=query.text, timestamp_us=query.timestamp_us), + participant_id=query.participant_id, + source=self._source, + ) + + async def _publish_participant_left(self, participant_id: str) -> None: + runtime = self._running_runtime() + topic = self.participant_left_topic + assert topic is not None + await runtime.publish( + topic, + VoiceParticipantLeft(), + participant_id=participant_id, + source=self._source, + ) + + async def _publish_interrupted(self, participant_id: str | None) -> None: + runtime = self._running_runtime() + topic = self.interrupted_topic + assert topic is not None + await runtime.publish( + topic, + VoiceInterrupted(), + participant_id=participant_id, + source=self._source, + ) + + def _running_runtime(self) -> AgentRuntime: + if self._runtime is None: + raise RuntimeError("voice agent is not running") + return self._runtime + + def _discard_stream( + self, + key: tuple[str, str, str], + stream: _ResponseStream, + ) -> None: + self._remember_closed_stream(key) + if self._streams.get(key) is stream: + self._streams.pop(key, None) + self._finish_response_trace(key, status="closed") + + def _remember_closed_stream(self, key: tuple[str, str, str]) -> None: + self._closed_streams.pop(key, None) + self._closed_streams[key] = None + if len(self._closed_streams) > _CLOSED_STREAM_CAPACITY: + oldest = next(iter(self._closed_streams)) + self._closed_streams.pop(oldest, None) + + def _finish_response_trace( + self, + key: tuple[str, str, str], + *, + status: str, + ) -> None: + trace = self._response_traces.pop(key, None) + if trace is None: + return + with self._response_scope( + participant_id=trace.participant_id, + source=trace.source, + response_id=trace.response_id, + correlation_id=trace.correlation_id, + text="".join(trace.fragments), + fragment_count=len(trace.fragments), + interrupt=trace.interrupt, + timestamp_us=trace.timestamp_us, + streaming=True, + status=status, + started_at=trace.started_at, + ): + pass + + @staticmethod + def _response_scope( + *, + participant_id: str, + source: str, + response_id: str | None, + correlation_id: str, + text: str, + fragment_count: int, + interrupt: bool, + timestamp_us: int, + streaming: bool, + status: str, + started_at: datetime | None = None, + ): + return nemo_relay.scope.scope( + "voice.response", + nemo_relay.ScopeType.Agent, + input={ + "text": text, + "streaming": streaming, + "fragment_count": fragment_count, + "interrupt": interrupt, + }, + metadata={ + "participant_id": participant_id, + "source": source, + "response_id": response_id, + "correlation_id": correlation_id, + "timestamp_us": timestamp_us, + "status": status, + }, + timestamp=started_at, + end_timestamp=datetime.now(UTC) if started_at is not None else None, + ) + async def _on_data(self, message: DataMessage) -> None: + if message.topic in self.text_ignore_topics: + return + text = (message.data or b"").decode("utf-8", errors="replace").strip() + if not text or not self.session.is_running: + return + if not self.session.transport.target_participant: + self.session.transport.set_target_participant(message.participant_id) + if self.text_transform is not None: + text = self.text_transform(text) + text = text.strip() + if not text: + return + logger.info("text input pid={!r} {!r}", message.participant_id, text[:80]) + await self.session.enqueue_query( + message.participant_id, + text, + pts_us=message.pts_us, + ) + +__all__ = [ + "VOICE_OUTPUT_TOPIC", + "UserQuery", + "VoiceAgent", + "VoiceInterrupted", + "VoiceOutput", + "VoiceParticipantLeft", +] diff --git a/agent-sdk/xr-ai-voice/xr_ai_voice/_session.py b/agent-sdk/xr-ai-voice/xr_ai_voice/_session.py index d16ca53f..56fa9013 100644 --- a/agent-sdk/xr-ai-voice/xr_ai_voice/_session.py +++ b/agent-sdk/xr-ai-voice/xr_ai_voice/_session.py @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -"""Shared lifecycle host for participant-aware voice handlers.""" +"""Media lifecycle for the runtime voice agent.""" from __future__ import annotations import asyncio @@ -13,15 +13,16 @@ from loguru import logger from pipecat.pipeline.runner import PipelineRunner +from xr_ai_hub import ProcessorEndpoint from xr_ai_models import STTService, TTSService from xr_ai_voicegate import VoiceGateConfig -from ._handler import VoiceHandler, VoiceTurn from ._pipeline import _build_voice_pipeline -from ._processors.handler import _VoiceHandlerProcessor +from ._processors.io import _VoiceIOProcessor from ._processors.vad_stt import VadConfig from ._readiness import ProbeFn, wait_for_services from ._transport import HubVoiceTransport +from ._types import VoiceInputSink, VoiceResponse _STATUS_REANNOUNCE_INTERVAL_S = 2.0 @@ -63,7 +64,8 @@ def __init__( self.text_topic = text_topic self.idle_timeout_secs = idle_timeout_secs self._transport = transport - self._handler_processor: _VoiceHandlerProcessor | None = None + self._io_processor: _VoiceIOProcessor | None = None + self._closed = False @property def transport(self) -> HubVoiceTransport: @@ -72,12 +74,22 @@ def transport(self) -> HubVoiceTransport: self._transport = HubVoiceTransport() return self._transport + @property + def endpoint(self) -> ProcessorEndpoint: + """Return the hub endpoint after readiness has initialized the transport.""" + + if self._transport is None: + raise RuntimeError("voice session endpoint is not ready") + return self._transport.endpoint + @property def is_running(self) -> bool: """Whether the session currently accepts voice or text queries.""" - return self._handler_processor is not None + return self._io_processor is not None async def __aenter__(self) -> "VoiceSession": + if self._closed: + raise RuntimeError("voice session is closed") probes = { "stt": self.stt.health, "tts": self.tts.health, @@ -93,42 +105,28 @@ async def __aenter__(self) -> "VoiceSession": async def run( self, - handler: VoiceHandler, + input_sink: VoiceInputSink, *, - observer: Callable[[VoiceTurn], Awaitable[None]] | None = None, - on_participant_joined: Callable[[str], Awaitable[None] | None] | None = None, on_participant_left: Callable[[str], Awaitable[None] | None] | None = None, - on_user_started_speaking: Callable[[str], Awaitable[None] | None] | None = None, - on_query_superseded: Callable[[str], Awaitable[None] | None] | None = None, + on_interrupted: Callable[[str | None], Awaitable[None] | None] | None = None, interrupt_on_supersede: bool = False, - queue_queries: bool = False, ) -> None: - """Run a voice handler with explicit turn and participant callbacks. - - ``queue_queries`` runs participant queries sequentially instead of - cancelling the active query. With ``interrupt_on_supersede``, the next - queued query flushes speech left from the preceding response as it - starts. - """ - if self._handler_processor is not None: + """Run media input/output until the pipeline exits.""" + if self._io_processor is not None: raise RuntimeError("voice session is already running") - handler_processor = _VoiceHandlerProcessor( - handler, + io_processor = _VoiceIOProcessor( + input_sink, transport=self.transport, - observer=observer, - on_participant_joined=on_participant_joined, on_participant_left=on_participant_left, - on_user_started_speaking=on_user_started_speaking, - on_query_superseded=on_query_superseded, + on_interrupted=on_interrupted, interrupt_on_supersede=interrupt_on_supersede, - queue_queries=queue_queries, ) - self._handler_processor = handler_processor + self._io_processor = io_processor _, task = _build_voice_pipeline( transport=self.transport, stt=self.stt, tts=self.tts, - handler_processor=handler_processor, + io_processor=io_processor, vad_cfg=self.vad, voice_gate_cfg=self.voice_gate, text_topic=self.text_topic, @@ -192,7 +190,7 @@ def request_cancel() -> None: started_task.cancel() with contextlib.suppress(asyncio.CancelledError): _ = await started_task - self._handler_processor = None + self._io_processor = None for sig in installed: loop.remove_signal_handler(sig) @@ -201,16 +199,33 @@ async def enqueue_query( participant_id: str, text: str, *, - fresh_match: bool = False, pts_us: int | None = None, ) -> None: """Submit typed text through the active participant-aware voice path.""" - if self._handler_processor is None: + if self._io_processor is None: raise RuntimeError("voice session is not running") - await self._handler_processor.enqueue_query( + await self._io_processor.enqueue_query( participant_id, text, - fresh_match=fresh_match, + pts_us=pts_us, + ) + + async def enqueue_response( + self, + participant_id: str, + response: VoiceResponse, + *, + interrupt: bool = False, + pts_us: int | None = None, + ) -> None: + """Queue finite or incremental output on the active participant voice path.""" + + if self._io_processor is None: + raise RuntimeError("voice session is not running") + await self._io_processor.enqueue_response( + participant_id, + response, + interrupt=interrupt, pts_us=pts_us, ) @@ -219,6 +234,9 @@ async def __aexit__(self, *_exc: Any) -> None: async def close(self) -> None: """Release transport and model clients; safe to call without a context manager.""" + if self._closed: + return + self._closed = True if self._transport is not None: self._transport.shutdown() seen: set[int] = set() diff --git a/agent-sdk/xr-ai-voice/xr_ai_voice/_text_input.py b/agent-sdk/xr-ai-voice/xr_ai_voice/_text_input.py deleted file mode 100644 index 7f253036..00000000 --- a/agent-sdk/xr-ai-voice/xr_ai_voice/_text_input.py +++ /dev/null @@ -1,49 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -"""Typed-data ingress for participant-aware assistants.""" -from __future__ import annotations - -from collections.abc import Callable, Iterable - -from loguru import logger -from xr_ai_hub import DataMessage - -from ._session import VoiceSession - -QueryTransform = Callable[[str], str] - - -class TextMessageInput: - """Route hub data messages through an assistant's normal query path.""" - - def __init__( - self, - *, - session: VoiceSession, - ignore_topics: Iterable[str] = (), - transform: QueryTransform | None = None, - fresh_match: bool = False, - ) -> None: - self._session = session - self._ignore_topics = frozenset(ignore_topics) - self._transform = transform or (lambda text: text) - self._fresh_match = fresh_match - session.transport.endpoint.on_data(self._on_data) - - async def _on_data(self, message: DataMessage) -> None: - if message.topic in self._ignore_topics: - return - text = (message.data or b"").decode("utf-8", errors="replace").strip() - if not text or not self._session.is_running: - return - if not self._session.transport.target_participant: - self._session.transport.set_target_participant(message.participant_id) - text = self._transform(text) - logger.info("text input pid={!r} {!r}", message.participant_id, text[:80]) - await self._session.enqueue_query( - message.participant_id, - text, - fresh_match=self._fresh_match, - pts_us=message.pts_us, - ) diff --git a/agent-sdk/xr-ai-voice/xr_ai_voice/_types.py b/agent-sdk/xr-ai-voice/xr_ai_voice/_types.py new file mode 100644 index 00000000..40f104c1 --- /dev/null +++ b/agent-sdk/xr-ai-voice/xr_ai_voice/_types.py @@ -0,0 +1,27 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Internal contracts for participant-aware voice I/O.""" + +from __future__ import annotations + +from collections.abc import AsyncIterator, Awaitable, Callable +from dataclasses import dataclass +from typing import TypeAlias + + +@dataclass(frozen=True, slots=True) +class VoiceQuery: + """One participant query produced by speech or typed input.""" + + participant_id: str + text: str + #: Unix-epoch microseconds anchoring the query to when the user spoke or typed. + timestamp_us: int + + +VoiceInputSink: TypeAlias = Callable[[VoiceQuery], Awaitable[None]] +VoiceResponse: TypeAlias = str | AsyncIterator[str] + + +__all__ = ["VoiceInputSink", "VoiceQuery", "VoiceResponse"] diff --git a/docs/architecture.md b/docs/architecture.md index 4ea4eeee..4c3071f4 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -37,13 +37,12 @@ docs/ # Design docs and topic deep-dives - **`agent-sdk/xr-ai-hub-client`** contains only the agent-facing IPC layer. Its sole runtime dependencies are `pyzmq` and `msgpack` — no LiveKit, FastAPI, or uvicorn. -- **`agent-sdk/xr-ai-agent-runtime`** owns agent resource lifetimes, - runtime-owned background tasks, and typed publish/subscribe. Agents expose - existing `Tool` and `AsyncTool` instances from `xr-ai-tools`; direct callers - and model loops use those tools without a second runtime dispatch API. Each - agent owns its concurrency policy, including any locks or private queues - needed to coordinate tools and subscriptions. Model loops, planning, memory, - and raw media transport remain outside the runtime. +- **`agent-sdk/xr-ai-agent-runtime`** provides typed publish/subscribe routing. + Agents expose existing `Tool` and `AsyncTool` instances from `xr-ai-tools`; + direct callers and model loops use those tools without a second runtime + dispatch API. Each agent owns its resources, background tasks, lifecycle, + and concurrency policy. Model loops, planning, memory, and raw media + transport remain outside the runtime. - **Native agents compose typed tools in process.** Model-backed tools call typed capability services, while deterministic tools run locally. MCP adapters only republish selected tools for MCP consumers. diff --git a/docs/changelog.md b/docs/changelog.md index 2b457cca..b90faee1 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -9,6 +9,36 @@ Significant decisions, in reverse-chronological order. Update this whenever a non-trivial architectural or design decision is made so the rationale is preserved and not re-litigated. +### 2026-08-12 — Runtime and agent telemetry uses Relay's local event stream + +`AgentRuntime` records every typed publication as a Relay function scope and +every receiving subscription callback as a nested agent scope. The scopes carry +the runtime topic and existing participant, message, correlation, parent, +source, and subscriber metadata without changing delivery or failure +semantics. Each delivery receives a forked Relay context so concurrent fan-out +cannot mutate the publisher's scope stack. Agent-owned work remains agent-owned; +a task that outlives its callback starts a fresh scope stack and adds an agent +scope for its active lifetime, as the simple VLM turn does around its streamed +response. Logical correlation remains in metadata without parenting the turn to +an already-ended subscription scope. + +The simple VLM worker writes compact ATOF JSONL to `relay-events.jsonl` beside +`worker.log`. Its subscriber omits per-token `llm.chunk` marks. +`Topic.telemetry` is the reusable runtime cardinality policy. The +`voice.output` transport topic uses `"none"`, and `VoiceAgent` emits one +`voice.response` scope after aggregating either a finite response or a completed +stream. That scope carries the combined text, fragment count, response and +participant identity, producer, timing, interrupt flag, and completion status. +Final and bounded partial-probe STT calls use `voice.stt` function scopes with +audio summaries and nested transcript result marks. Sentence-level TTS calls use +`voice.tts` function scopes. This shows provider latency and failures without +recording raw audio or creating per-audio-frame events; remote client playback +remains outside the worker's observable boundary. +Commands, lifecycle messages, tools, models, and turns remain fully traced. The +worker does not start a collector or emit telemetry over the network. The +existing live-frame sanitizer remains in the tool scope, while other event +fields stay observable for local diagnosis. + ### 2026-08-12 — Streaming tools isolate Relay scopes in producer tasks `AsyncTool` runs each handler in a forked producer task that exclusively owns @@ -29,6 +59,20 @@ chunk is consumed. Producer cleanup remains unbounded: a timeout and detached cleanup policy requires a separate decision because abandoning cleanup could leave Relay state or participant status unfinished. +### 2026-08-12 — Voice output is a subscriber, not a privileged side channel + +`VoiceAgent` owns `VoiceSession`, subscribes to the typed `voice.output` topic, +and serializes output through its participant-aware delivery path. Producers +may publish one finite message or a sequence of chunks sharing a response ID; +both use the same TTS aggregation, data echo, turn observation, and interruption +path. Voice delivery is one runtime agent among the other application tasks +rather than a privileged dispatcher side channel. + +Participant departure and interruption follow the same rule: `VoiceAgent` +publishes voice-owned schemas on application-named topics, and each application +agent handles its own tasks and resources. `app.py` remains +the composition root and contains no transport callbacks or resource logic. + ### 2026-08-12 — Agents own existing tools and their concurrency policy `xr-ai-agent-runtime` defines an `Agent` as a plain object containing private diff --git a/docs/nemo-agent-toolkit-migration.md b/docs/nemo-agent-toolkit-migration.md index 88fc7be1..d49827e3 100644 --- a/docs/nemo-agent-toolkit-migration.md +++ b/docs/nemo-agent-toolkit-migration.md @@ -14,12 +14,12 @@ for all model HTTP, and reach clients only through the Hub IPC SDK. NeMo Relay is the local execution boundary. The XR-owned `xr-ai-tools` package is the toolkit-independent tools layer: Pydantic tool schemas, trigger dispatch, and small helpers that adapt those tools to `xr-ai-models` tool-call -types. `xr-ai-agent-runtime` separately owns agent resource lifetimes, -runtime-owned background tasks, and fan-out `publish()`. Agents expose the -existing `Tool` and `AsyncTool` objects directly and own their synchronization. -Applications own their model calls, history, and loop policy. Relay owns tool -lifecycles, middleware, guardrails, and telemetry. Existing NeMo Agent Toolkit -function groups remain compatibility extras until their concrete tools migrate. +types. `xr-ai-agent-runtime` separately provides typed fan-out `publish()`. +Agents expose the existing `Tool` and `AsyncTool` objects directly and own +their resources, background tasks, lifecycle, and synchronization. Applications +own their model calls, history, and loop policy. Relay owns tool lifecycles, +middleware, guardrails, and telemetry. Existing NeMo Agent Toolkit function +groups remain compatibility extras until their concrete tools migrate. NeMo Platform and NeMo Fabric are deployment and evaluation integrations, not worker dependencies. Platform currently requires Python 3.12 or 3.13 and owns @@ -30,7 +30,7 @@ remain optional launch targets after the local runtime has migrated. ```text XR worker - -> xr-ai-agent-runtime: agent lifetimes, background tasks, and publish + -> xr-ai-agent-runtime: agent definitions and typed publish/subscribe -> xr-ai-tools: typed tools and trigger dispatch -> NeMo Relay: managed tool execution, guardrails, telemetry -> xr-ai-models: private model boundary used by model-backed tools @@ -61,9 +61,9 @@ acceptance behavior rather than an implementation dependency. 3. **Native event dispatcher** — port tea-making's typed participant-scoped subscriptions and periodic background sources so voice and autonomous work invoke the same registered tools. -4. **Agent runtime** — add runtime-owned agent resource lifetimes, background - tasks, typed `publish()`, and agents that expose existing unary and streaming - tools directly while owning their concurrency policy. +4. **Agent runtime** — add typed `publish()` and agents that expose existing + unary and streaming tools directly while owning their resources, tasks, + lifecycle, and concurrency policy. 5. **Deterministic and service capabilities** — port spatial math, text memory, RAG, vision, XR tracking, and video memory to the native tool surface; keep MCP adapters as explicit compatibility publishers. diff --git a/docs/process-model.md b/docs/process-model.md index d336d36a..9df6c43d 100644 --- a/docs/process-model.md +++ b/docs/process-model.md @@ -60,8 +60,9 @@ def run() -> None: process can serve requests; the client is told the room is ready only once the hub sees every attached agent available and each agent's subscription for that client confirmed. See `docs/source/components/agent-sdk.md`. -- **Native voice workers** pass the ready file to `VoiceSession`; `run()` - touches it only after the input transport's hub IPC receive loop has started. +- **Native voice workers** pass the ready file to the `VoiceSession` owned by + `VoiceAgent`; the session touches it only after the input transport's hub IPC + receive loop has started. - **Pipecat workers** build their voice pipeline, then call `run_voice_pipeline(worker, transport, on_ready=ready_file.touch)`. The callback runs only after the input transport's hub IPC receive loop has diff --git a/docs/source/components/agent-sdk.md b/docs/source/components/agent-sdk.md index 0f69ed05..b89fa294 100644 --- a/docs/source/components/agent-sdk.md +++ b/docs/source/components/agent-sdk.md @@ -14,9 +14,10 @@ from: `STTService`, `TTSService`, `EmbeddingService`) plus OpenAI-compatible HTTP clients, driven by a structured model deployment profile. Swapping a backend is a configuration edit, not a code edit. -- **`xr-ai-voice`** — the native voice runtime. `VoiceSession` owns readiness, - hub transport, voice gating, streaming responses, signals, and cleanup while - applications provide a `VoiceHandler`. +- **`xr-ai-voice`** — the native voice runtime. `VoiceAgent` publishes + `UserQuery` and lifecycle events to application-named topics and consumes + `voice.output`. `VoiceSession` owns readiness, hub transport, voice gating, + streaming responses, signals, and cleanup. - **`xr-ai-pipecat`** — the direct Pipecat surface retained for unmigrated consumers such as `xr-render-demo`. Its `run_voice_pipeline` helper exposes the same IPC-start request-readiness boundary as `VoiceSession`. @@ -53,6 +54,23 @@ correlation metadata on `RuntimeContext` applies to pub/sub. Agents create, cancel, and await their own background tasks. `publish()` settles all fan-out deliveries before propagating subscriber failures. +Relay records each publication and receiving-agent callback as nested runtime +and agent scopes. These scopes include the topic plus message, correlation, +participant, source, and subscriber metadata. Agent-owned work that outlives a +subscription callback starts a fresh Relay scope stack and adds its own agent +scope, preserving logical correlation in metadata without retaining an ended +callback as its parent. + +`Topic.telemetry` controls runtime cardinality without changing delivery. Keep +`"full"` for commands, state changes, and lifecycle events. High-volume +transport streams use `"none"`; their receiving agent records one semantic +scope after aggregation. The voice output topic follows this pattern, producing +one `voice.response` scope for either a finite response or a completed stream. +Voice records provider work separately: `voice.stt` covers final and bounded +partial-probe transcription, while `voice.tts` covers each sentence synthesis. +Audio is summarized rather than stored, and playback on the remote client is +outside these spans. + --- ## xr-ai-models @@ -283,11 +301,20 @@ lifecycle ownership. ## xr-ai-voice -Native voice applications work with participant-aware turns rather than -Pipecat processors. A handler returns a string or an async stream of strings: +Native voice applications work with participant-aware runtime topics rather +than Pipecat processors. `VoiceAgent` owns `VoiceSession`, publishes accepted +input and lifecycle events with voice-owned schemas on application-named +topics, and subscribes to `voice.output`: ```python -from xr_ai_voice import VadConfig, VoiceSession +from xr_ai_runtime import Topic +from xr_ai_voice import ( + UserQuery, + VadConfig, + VoiceAgent, + VoiceParticipantLeft, + VoiceSession, +) session = VoiceSession( stt=stt, @@ -298,19 +325,26 @@ session = VoiceSession( ready_file=ready_file, closeables=(vlm,), ) -async with session: - await session.run(handler, on_participant_left=release_participant) +queries = Topic("my-sample.user-query", UserQuery) +participant_left = Topic("my-sample.participant-left", VoiceParticipantLeft) +voice = VoiceAgent( + session, + query_topic=queries, + participant_left_topic=participant_left, +) +runtime.register("voice", voice) +async with runtime: + await voice.run(runtime) ``` -`TextMessageInput` routes typed messages through the same turn path as speech -and ignores data received outside an active `run()`. The default hub transport -is opened only after readiness probes succeed; failed readiness closes the -session's model clients without opening hub sockets. The ready file is touched -from `run()` only after the input transport enters its hub IPC receive loop. -NAT applications create handlers with -`xr_ai_nat.adapters.as_voice_handler`. `VoiceSession` preserves participant -routing, cancels superseded or interrupted turns, installs signal handlers, -and closes its transport and model clients. +The default hub transport is opened only after readiness probes succeed; failed +readiness closes the session's model clients without opening hub sockets. The +ready file is touched only after the input transport enters its hub IPC receive +loop. `VoiceSession` preserves participant routing, cancels superseded or +interrupted output, installs signal handlers, and closes its transport and +model clients. `VoiceAgent` turns transport lifecycle callbacks into typed +runtime events; application agents subscribe and clean up their own state. A +pid-less interruption is a global event. ## xr-ai-pipecat diff --git a/docs/source/components/launcher-and-process-model.md b/docs/source/components/launcher-and-process-model.md index daf25389..1d583507 100644 --- a/docs/source/components/launcher-and-process-model.md +++ b/docs/source/components/launcher-and-process-model.md @@ -56,8 +56,9 @@ def run() -> None: before workers, cloudxr before MCP servers that open OpenXR sessions, etc.). - **Every process accepts `--ready-file `** and must `Path(path).touch()` when it is fully initialized and ready to serve requests. -- **Native voice workers** pass the ready file to `VoiceSession`; `run()` - touches it only after the input transport's hub IPC receive loop has started. +- **Native voice workers** pass the ready file to the `VoiceSession` owned by + `VoiceAgent`; the session touches it only after the input transport's hub IPC + receive loop has started. - **Direct Pipecat workers** call `run_voice_pipeline(worker, transport, on_ready=ready_file.touch)` to use the same IPC-start readiness boundary. diff --git a/tests/test_adapters_voice.py b/tests/test_adapters_voice.py deleted file mode 100644 index d59c7ef7..00000000 --- a/tests/test_adapters_voice.py +++ /dev/null @@ -1,175 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -"""NAT↔voice adapters, and the conversation-recall producer/consumer loop. - -`record_voice_transcripts` is the producer that writes ``{pid}:user`` / -``{pid}:agent`` transcript sources; `xr_conversation_memory.recall_conversation` -is the consumer that reads them back. This exercises both ends end-to-end. -""" -from __future__ import annotations - -from collections.abc import AsyncIterator - -import pytest -from nat.builder.workflow_builder import WorkflowBuilder -from xr_ai_nat.adapters import as_voice_handler, record_voice_transcripts -from xr_ai_nat.functions.text_memory import ( - ConversationMemoryFunctionsConfig, - RecallConversationRequest, - TextMemoryFunctionsConfig, -) -from xr_ai_voice import VoiceQuery, VoiceTurn - - -class _EchoFunction: - """Duck-typed NAT ``Function`` stand-in for the adapter unit tests.""" - - async def ainvoke(self, request: object) -> str: - return f"answer:{request}" - - async def astream(self, request: object) -> AsyncIterator[str]: - del request - for part in ("one ", "two"): - yield part - - -async def test_as_voice_handler_maps_request_and_response() -> None: - handler = as_voice_handler( - _EchoFunction(), - request=lambda query: query.text.upper(), - response=str, - ) - response = await handler( - VoiceQuery(participant_id="alice", text="hi", fresh_match=True, timestamp_us=1) - ) - assert response == "answer:HI" - - -async def test_as_voice_handler_streams_and_drops_empty_chunks() -> None: - handler = as_voice_handler( - _EchoFunction(), - request=lambda query: query.text, - response=lambda chunk: str(chunk).strip(), # "one " -> "one", "two" -> "two" - streaming=True, - ) - stream = await handler( - VoiceQuery(participant_id="alice", text="go", fresh_match=True, timestamp_us=1) - ) - assert [chunk async for chunk in stream] == ["one", "two"] - - -async def test_record_voice_transcripts_then_recall_conversation(tmp_path) -> None: - async with WorkflowBuilder() as builder: - await builder.add_function_group( - "text_memory", TextMemoryFunctionsConfig(directory=tmp_path) - ) - await builder.add_function_group( - "conversation_memory", ConversationMemoryFunctionsConfig() - ) - text_memory = await builder.get_function_group("text_memory") - conversation = await builder.get_function_group("conversation_memory") - add_transcript = (await text_memory.get_all_functions())["text_memory__add_transcript"] - recall = (await conversation.get_all_functions())["conversation_memory__recall_conversation"] - - record = record_voice_transcripts(add_transcript) - # A real exchange gives the user turn and the agent turn the SAME - # timestamp — both carry the originating query's time — so recall has to - # order the tie user-before-agent rather than relying on distinct stamps. - await record(VoiceTurn(participant_id="alice", role="user", timestamp_us=10, text="hello")) - await record(VoiceTurn(participant_id="alice", role="agent", timestamp_us=10, text="hi there")) - await record(VoiceTurn(participant_id="alice", role="user", timestamp_us=30, text="how are you")) - # Whitespace-only turns are not persisted. - await record(VoiceTurn(participant_id="alice", role="agent", timestamp_us=30, text=" ")) - # A different participant must not leak into alice's recall. - await record(VoiceTurn(participant_id="bob", role="user", timestamp_us=10, text="not alice")) - - result = await recall.ainvoke(RecallConversationRequest(participant_id="alice")) - - assert [(entry.timestamp_us, entry.role, entry.text) for entry in result.entries] == [ - (10, "user", "hello"), - (10, "agent", "hi there"), - (30, "user", "how are you"), - ] - # The producer stored role-scoped sources under the participant id. - assert (tmp_path / "alice_user.identity").read_text() == "alice:user" - assert (tmp_path / "alice_agent.identity").read_text() == "alice:agent" - - -def test_voice_adapters_are_reachable_from_the_public_adapters_namespace() -> None: - """Applications are told to use ``xr_ai_nat.adapters.as_voice_handler``; the - package must actually export both adapters (they resolve lazily because they - need the optional ``[voice]`` extra).""" - from xr_ai_nat import adapters - from xr_ai_nat.adapters import voice as voice_module - - assert adapters.as_voice_handler is voice_module.as_voice_handler - assert adapters.record_voice_transcripts is voice_module.record_voice_transcripts - assert sorted(adapters.__all__) == ["as_voice_handler", "record_voice_transcripts"] - assert "as_voice_handler" in dir(adapters) - with pytest.raises(AttributeError): - adapters.not_an_adapter - - -async def test_recall_conversation_respects_time_window(tmp_path) -> None: - async with WorkflowBuilder() as builder: - await builder.add_function_group( - "text_memory", TextMemoryFunctionsConfig(directory=tmp_path) - ) - await builder.add_function_group( - "conversation_memory", ConversationMemoryFunctionsConfig() - ) - add_transcript = (await (await builder.get_function_group("text_memory")).get_all_functions())[ - "text_memory__add_transcript" - ] - recall = (await (await builder.get_function_group("conversation_memory")).get_all_functions())[ - "conversation_memory__recall_conversation" - ] - record = record_voice_transcripts(add_transcript) - await record(VoiceTurn(participant_id="alice", role="user", timestamp_us=10, text="early")) - await record(VoiceTurn(participant_id="alice", role="user", timestamp_us=100, text="late")) - - result = await recall.ainvoke( - RecallConversationRequest(participant_id="alice", start_us=50, end_us=200) - ) - - assert [entry.text for entry in result.entries] == ["late"] - - -async def test_recall_conversation_generated_contract_is_fully_described(tmp_path) -> None: - """The recall surface is consumed by agents, so its generated schemas must - describe every field and constrain ``role`` to the two roles that exist. - - An undescribed field gives the model nothing to reason about, and an - unconstrained ``role`` invites it to invent a third value. - """ - async with WorkflowBuilder() as builder: - await builder.add_function_group( - "text_memory", TextMemoryFunctionsConfig(directory=tmp_path) - ) - await builder.add_function_group( - "conversation_memory", ConversationMemoryFunctionsConfig() - ) - conversation = await builder.get_function_group("conversation_memory") - recall = (await conversation.get_all_functions())["conversation_memory__recall_conversation"] - - request = recall.input_schema.model_json_schema() - result = recall.single_output_schema.model_json_schema() - - # Every request field is described. - for name, prop in request["properties"].items(): - assert prop.get("description"), f"request field {name} has no description" - - # The result and its nested entry model are fully described. - entry = result["$defs"]["ConversationEntry"] - assert result["properties"]["entries"].get("description") - for name in ("timestamp_us", "role", "text"): - assert entry["properties"][name].get("description"), f"{name} has no description" - - # role is constrained to exactly the two roles the producer writes. - assert entry["properties"]["role"]["enum"] == ["user", "agent"] - - -def test_conversation_memory_config_documents_its_text_memory_reference() -> None: - field = ConversationMemoryFunctionsConfig.model_fields["text_memory"] - assert field.description and "xr_text_memory" in field.description diff --git a/tests/test_agent_runtime.py b/tests/test_agent_runtime.py index d7a44ef6..7ac9aa8d 100644 --- a/tests/test_agent_runtime.py +++ b/tests/test_agent_runtime.py @@ -10,6 +10,7 @@ from collections.abc import AsyncIterator from typing import assert_type +import nemo_relay import pytest from pydantic import BaseModel from xr_ai_models import ChatMessage, ToolCall @@ -165,7 +166,7 @@ class _VoiceOutput(Agent): def __init__(self, *, mutate: bool = False) -> None: super().__init__() self.mutate = mutate - self.received: list[tuple[str, str, list[str]]] = [] + self.received: list[tuple[str | None, str, list[str]]] = [] @subscribe(OBSERVATIONS) async def observe(self, event: _Observation, ctx: RuntimeContext) -> None: @@ -198,6 +199,78 @@ async def test_publish_fans_out_isolated_typed_payloads() -> None: assert second.received == [("alice", "camera", ["kettle"])] +async def test_publish_records_runtime_and_receiving_agent_scopes() -> None: + agent = _VoiceOutput() + runtime = AgentRuntime() + runtime.register("observer", agent) + events = [] + subscriber = "xr-ai-agent-runtime-scopes" + nemo_relay.subscribers.register(subscriber, events.append) + try: + async with runtime: + await runtime.publish( + OBSERVATIONS, + _Observation(labels=["kettle"]), + participant_id="alice", + source="camera", + ) + await nemo_relay.subscribers.flush_async() + finally: + nemo_relay.subscribers.deregister(subscriber) + + starts = { + event.name: event.to_dict() + for event in events + if event.kind == "scope" and event.to_dict()["scope_category"] == "start" + } + publication = starts["publish:vision.observation"] + delivery = starts["agent:observer"] + assert publication["category"] == "function" + assert delivery["category"] == "agent" + assert delivery["parent_uuid"] == publication["uuid"] + expected_metadata = { + "topic": "vision.observation", + "agent": "observer", + "subscriber": "observe", + "participant_id": "alice", + "source": "camera", + } + assert { + key: delivery["metadata"][key] + for key in expected_metadata + } == expected_metadata + + +async def test_untraced_topic_delivers_without_runtime_or_agent_scopes() -> None: + quiet_topic = Topic("stream.chunk", _Echo, telemetry="none") + + class Receiver(Agent): + def __init__(self) -> None: + super().__init__() + self.received: list[str] = [] + + @subscribe(quiet_topic) + async def receive(self, event: _Echo, _ctx: RuntimeContext) -> None: + self.received.append(event.text) + + receiver = Receiver() + runtime = AgentRuntime() + runtime.register("receiver", receiver) + events = [] + subscriber = "xr-ai-agent-runtime-untraced-topic" + nemo_relay.subscribers.register(subscriber, events.append) + try: + async with runtime: + await runtime.publish(quiet_topic, _Echo(text="fragment")) + await nemo_relay.subscribers.flush_async() + finally: + nemo_relay.subscribers.deregister(subscriber) + + assert receiver.received == ["fragment"] + assert "publish:stream.chunk" not in {event.name for event in events} + assert "agent:receiver" not in {event.name for event in events} + + class _FailingSubscriber(Agent): def __init__(self, *, failure: Exception | None = None) -> None: super().__init__() @@ -276,6 +349,19 @@ async def test_nested_publish_preserves_participant_and_trace_context() -> None: assert speaker.metadata.correlation_id == forwarder.input_message_id +async def test_publish_can_broadcast_without_participant_scope() -> None: + speaker = _Speaker() + runtime = AgentRuntime() + runtime.register("speaker", speaker) + + async with runtime: + await runtime.publish(VOICE_OUTPUT, _Speak(text="Stop all.")) + + assert speaker.metadata is not None + assert speaker.metadata.participant_id is None + assert speaker.metadata.source == "application" + + class _SerializedAgent(Agent): def __init__(self) -> None: self._lock = asyncio.Lock() diff --git a/tests/test_simple_vlm_example_worker.py b/tests/test_simple_vlm_example_worker.py index f09e6e43..7a59d5a2 100644 --- a/tests/test_simple_vlm_example_worker.py +++ b/tests/test_simple_vlm_example_worker.py @@ -8,6 +8,7 @@ import asyncio import sys import time +from collections.abc import AsyncIterator from pathlib import Path from types import SimpleNamespace from typing import cast @@ -18,7 +19,8 @@ import yaml from xr_ai_hub import FrameData, FrameSignal, PixelFormat, ProcessorEndpoint from xr_ai_models import ChatResponse, VLMService -from xr_ai_voice import VoiceQuery, VoiceSession +from xr_ai_runtime import AgentRuntime +from xr_ai_voice import UserQuery, VoiceAgent, VoiceInterrupted, VoiceOutput, VoiceSession from xr_ai_voicegate import VoiceGateConfig _REPO_ROOT = Path(__file__).resolve().parents[1] @@ -28,6 +30,11 @@ from simple_vlm_example_worker import __main__ as worker_main # noqa: E402 # pyright: ignore[reportMissingImports] from simple_vlm_example_worker import app # noqa: E402 # pyright: ignore[reportMissingImports] +from simple_vlm_example_worker.agent import ( # noqa: E402 # pyright: ignore[reportMissingImports] + INTERRUPTED_TOPIC, + USER_QUERY_TOPIC, + SimpleVlmAgent, +) from simple_vlm_example_worker.config import load_config # noqa: E402 # pyright: ignore[reportMissingImports] from xr_ai_tools.live_vision import ( # noqa: E402 LiveVisionTool, @@ -52,13 +59,18 @@ async def close(self) -> None: class _Transport: def __init__(self) -> None: - self.endpoint = object() + self.endpoint = _DataEndpoint() self.shutdown_calls = 0 def shutdown(self) -> None: self.shutdown_calls += 1 +class _DataEndpoint: + def on_data(self, callback) -> None: + self.data_callback = callback + + class _StreamingVisionTool: instances: list["_StreamingVisionTool"] = [] @@ -142,6 +154,10 @@ def test_worker_is_a_package_with_module_and_console_entry_points() -> None: "simple_vlm_example_worker.__main__:run" ) assert "xr-ai-hub-client" in dependencies + assert "xr-ai-agent-runtime" in dependencies + assert project["tool"]["uv"]["sources"]["xr-ai-agent-runtime"]["path"] == ( + "../../../agent-sdk/xr-ai-agent-runtime" + ) assert project["tool"]["uv"]["sources"]["xr-ai-hub-client"]["path"] == ( "../../../agent-sdk/xr-ai-hub-client" ) @@ -156,6 +172,7 @@ def test_worker_is_a_package_with_module_and_console_entry_points() -> None: assert { "__init__.py", "__main__.py", + "agent.py", "app.py", "config.py", "prompts/system.txt", @@ -286,34 +303,108 @@ def test_config_rejects_a_non_mapping_yaml_document(tmp_path) -> None: load_config(config_path) -async def test_vision_handler_closes_nested_tool_stream() -> None: +async def test_simple_vlm_agent_closes_tool_stream_when_publication_fails() -> None: closed = asyncio.Event() - blocked = asyncio.Event() + + class Stream: + def __init__(self) -> None: + self.sent = False + + def __aiter__(self): + return self + + async def __anext__(self): + if self.sent: + await asyncio.Event().wait() + self.sent = True + return SimpleNamespace(text="first") + + async def aclose(self) -> None: + closed.set() class Vision: - async def stream(self, _request): - try: - yield SimpleNamespace(text="first") - await blocked.wait() - finally: - closed.set() - - handler = app._make_vision_handler(Vision()) # pyright: ignore[reportArgumentType] - response = await handler( - VoiceQuery( + def stream(self, _request): + return Stream() + + class Context: + agent_name = "simple-vlm" + metadata = SimpleNamespace( + message_id="turn-1", + correlation_id="turn-1", participant_id="alice", - text="What is shown?", - fresh_match=True, - timestamp_us=123, ) + + async def publish(self, *_args, **_kwargs) -> None: + raise RuntimeError("runtime stopped") + + agent = SimpleVlmAgent(lambda: Vision()) # type: ignore[return-value] + with pytest.raises(RuntimeError, match="runtime stopped"): + await agent._stream( # noqa: SLF001 + UserQuery(text="What is shown?", timestamp_us=123), + Context(), # type: ignore[arg-type] + ) + + assert closed.is_set() + + +async def test_cancelled_vlm_turn_does_not_publish_stream_terminator() -> None: + waiting = asyncio.Event() + closed = asyncio.Event() + published: list[VoiceOutput] = [] + + class Stream: + def __init__(self) -> None: + self.sent = False + + def __aiter__(self): + return self + + async def __anext__(self): + if not self.sent: + self.sent = True + return SimpleNamespace(text="first") + waiting.set() + await asyncio.Event().wait() + raise StopAsyncIteration + + async def aclose(self) -> None: + closed.set() + + class Vision: + def stream(self, _request): + return Stream() + + class Context: + agent_name = "simple-vlm" + metadata = SimpleNamespace( + message_id="turn-1", + correlation_id="turn-1", + participant_id="alice", + ) + + async def publish(self, _topic, output) -> None: + published.append(output) + + factory_calls: list[None] = [] + agent = SimpleVlmAgent( + lambda: factory_calls.append(None) or Vision() # type: ignore[return-value] ) - assert not isinstance(response, str) + assert factory_calls == [] - assert await anext(response) == "first" - close = getattr(response, "aclose") - await close() + task = asyncio.create_task( + agent._stream( # noqa: SLF001 + UserQuery(text="What is shown?", timestamp_us=123), + Context(), # type: ignore[arg-type] + ) + ) + await asyncio.wait_for(waiting.wait(), 1.0) + task.cancel() + await asyncio.gather(task, return_exceptions=True) + assert factory_calls == [None] assert closed.is_set() + assert len(published) == 1 + assert published[0].final is False async def test_app_wires_text_voice_cleanup_readiness_and_shutdown( @@ -327,11 +418,14 @@ async def test_app_wires_text_voice_cleanup_readiness_and_shutdown( tts = _Service() transport = _Transport() sessions: list[VoiceSession] = [] - text_inputs = [] run_options = {} - streamed = [] + responses = [] + response_tasks: list[asyncio.Task[None]] = [] + response_complete = asyncio.Event() - monkeypatch.setattr(app, "setup_logging", lambda _name: None) + worker_log = tmp_path / "logs" / "worker.log" + worker_log.parent.mkdir() + monkeypatch.setattr(app, "setup_logging", lambda _name: worker_log) monkeypatch.setattr(app, "load_models_config", lambda path: path) monkeypatch.setattr(app, "load_voice_gate_config", lambda _path: VoiceGateConfig()) monkeypatch.setattr(app, "make_stt", lambda _models, _name: stt) @@ -347,31 +441,63 @@ async def run(handler, **options) -> None: if session.ready_file: session.ready_file.touch() run_options.update(options) - response = await handler( - VoiceQuery( + assert await handler( + SimpleNamespace( participant_id="alice", text="What is in front of me?", - fresh_match=True, timestamp_us=123, ) - ) - streamed.extend([chunk async for chunk in response]) - options["on_participant_left"]("alice") + ) is None + await asyncio.wait_for(response_complete.wait(), 1.0) + await options["on_participant_left"]("alice") + async def wait_until_released() -> None: + while not _StreamingVisionTool.instances[0].released: + await asyncio.sleep(0) + + await asyncio.wait_for(wait_until_released(), 1.0) + + async def enqueue_response( + participant_id: str, + response: str | AsyncIterator[str], + *, + interrupt: bool = False, + pts_us: int | None = None, + ) -> None: + async def consume() -> None: + text = ( + response + if isinstance(response, str) + else "".join([chunk async for chunk in response]) + ) + responses.append((participant_id, text, interrupt, pts_us)) + response_complete.set() + + response_tasks.append(asyncio.create_task(consume())) session.run = run # type: ignore[method-assign] + session.enqueue_response = enqueue_response # type: ignore[method-assign] return session - class CaptureTextInput: - def __init__(self, **kwargs) -> None: - text_inputs.append(kwargs) - monkeypatch.setattr(app, "VoiceSession", make_session) - monkeypatch.setattr(app, "TextMessageInput", CaptureTextInput) _StreamingVisionTool.instances.clear() await app.run_app(config, ready_file=ready_file) assert ready_file.exists() + relay_log = worker_log.parent / "relay-events.jsonl" + assert relay_log.exists() + relay_events = [ + yaml.safe_load(line) + for line in relay_log.read_text().splitlines() + ] + assert {event["name"] for event in relay_events} >= { + "publish:simple-vlm.user-query", + "agent:simple-vlm", + "simple-vlm.turn", + "voice.response", + } + assert "publish:voice.output" not in {event["name"] for event in relay_events} + assert "agent:voice" not in {event["name"] for event in relay_events} assert stt.health_calls == tts.health_calls == vlm.health_calls == 1 assert stt.close_calls == tts.close_calls == vlm.close_calls == 1 assert transport.shutdown_calls == 1 @@ -387,12 +513,30 @@ def __init__(self, **kwargs) -> None: assert _StreamingVisionTool.instances[0].released == ["alice"] assert _StreamingVisionTool.instances[0].requests[0].participant_id == "alice" assert _StreamingVisionTool.instances[0].requests[0].query == "What is in front of me?" - assert streamed == ["a ", "blue square"] + assert responses == [("alice", "a blue square", True, 123)] + assert all(task.done() for task in response_tasks) assert run_options["interrupt_on_supersede"] is True - assert text_inputs[0]["session"] is sessions[0] - assert text_inputs[0]["fresh_match"] is True - assert text_inputs[0]["transform"]("PING") == config.default_prompt - assert text_inputs[0]["transform"]("What is this?") == "What is this?" + assert callable(run_options["on_interrupted"]) + assert app._text_transform(config.default_prompt)("PING") == config.default_prompt + assert app._text_transform(config.default_prompt)("What is this?") == "What is this?" + + +async def test_relay_event_log_excludes_stream_chunks(tmp_path) -> None: + worker_log = tmp_path / "worker.log" + + async with app._relay_event_log(worker_log): # noqa: SLF001 + with nemo_relay.scope.scope("test-turn", nemo_relay.ScopeType.Agent): + nemo_relay.scope.event("llm.chunk", data={"text": "fragment"}) + nemo_relay.scope.event("turn.summary", data={"text": "complete"}) + + events = [ + yaml.safe_load(line) + for line in (tmp_path / "relay-events.jsonl").read_text().splitlines() + ] + names = [event["name"] for event in events] + assert "llm.chunk" not in names + assert "turn.summary" in names + assert names.count("test-turn") == 2 async def test_live_vision_tool_returns_a_complete_agent_observation() -> None: @@ -616,3 +760,178 @@ async def malformed_frame(_participant_id: str) -> str: with pytest.raises(RuntimeError, match="malformed pixels"): await finite.execute(request) + + +async def test_sample_runtime_streams_vision_through_voice_agent() -> None: + endpoint = _LiveEndpoint() + vlm = _StreamingVlm() + vision = StreamingVisionTool( + endpoint=cast(ProcessorEndpoint, endpoint), + vlm=cast(VLMService, vlm), + system_prompt="Answer briefly.", + ) + + class Session: + def __init__(self) -> None: + self.text = "" + self.complete = asyncio.Event() + self.started = asyncio.Event() + self.text_topic = "agent.response" + + async def __aenter__(self): + return self + + async def run(self, _handler, **_options) -> None: + self.started.set() + await asyncio.Event().wait() + + async def enqueue_response( + self, + participant_id: str, + response: str | AsyncIterator[str], + *, + interrupt: bool = False, + pts_us: int | None = None, + ) -> None: + assert participant_id == "alice" + assert interrupt is True + assert pts_us == 123 + + async def consume() -> None: + self.text = ( + response + if isinstance(response, str) + else "".join([chunk async for chunk in response]) + ) + self.complete.set() + + asyncio.create_task(consume()) + + async def close(self) -> None: + pass + + session = Session() + runtime = AgentRuntime() + runtime.register("simple-vlm", SimpleVlmAgent(lambda: vision)) + runtime.register( + "voice", + VoiceAgent( # type: ignore[arg-type] + session, + query_topic=USER_QUERY_TOPIC, + text_input=False, + ), + ) + assert endpoint.frame_callback is not None + await endpoint.frame_callback( + FrameSignal( + slot=0, + seq=1, + pts_us=time.time_ns() // 1_000, + width=2, + height=2, + fmt=PixelFormat.RGB24, + data_sz=12, + participant_id="alice", + track_id="camera", + ) + ) + events = [] + subscriber = "simple-vlm-agent-vision" + intercept = "simple-vlm-agent-vision-header" + + def add_header(_name, request, annotated): + headers = dict(request.headers) + headers["X-Relay-Session"] = "turn-9" + return nemo_relay.LLMRequestInterceptOutcome( + nemo_relay.LLMRequest(headers, request.content), + annotated, + ) + + nemo_relay.subscribers.register(subscriber, events.append) + nemo_relay.intercepts.register_llm_request(intercept, 0, False, add_header) + try: + async with runtime: + await runtime.publish( + USER_QUERY_TOPIC, + UserQuery(text="What is shown?", timestamp_us=123), + participant_id="alice", + source="test-input", + ) + await asyncio.wait_for(session.complete.wait(), 1.0) + await nemo_relay.subscribers.flush_async() + finally: + nemo_relay.intercepts.deregister_llm_request(intercept) + nemo_relay.subscribers.deregister(subscriber) + + assert session.text == "a blue square" + image, question, system_prompt, headers = vlm.calls[0] + assert image.startswith("data:image/jpeg;base64,") + assert question == "What is shown?" + assert system_prompt == "Answer briefly." + assert headers["X-Relay-Session"] == "turn-9" + assert endpoint.statuses == [("processing", "alice"), ("idle", "alice")] + assert {"tool", "llm"} <= { + getattr(event, "category", None) for event in events + } + starts = { + event.name: event.to_dict() + for event in events + if event.kind == "scope" + and event.to_dict().get("scope_category") == "start" + } + assert starts["simple-vlm.turn"]["parent_uuid"] != ( + starts["agent:simple-vlm"]["uuid"] + ) + tool_starts = [ + event.to_dict() + for event in events + if event.kind == "scope" + and event.to_dict().get("scope_category") == "start" + and event.to_dict().get("category") == "tool" + ] + assert tool_starts + assert tool_starts[0]["parent_uuid"] == starts["simple-vlm.turn"]["uuid"] + llm_events = [ + event.to_json() + for event in events + if getattr(event, "category", None) == "llm" + ] + assert llm_events + assert all(image not in event for event in llm_events) + assert any("" in event for event in llm_events) + + +async def test_simple_vlm_agent_handles_global_interruption_event() -> None: + started = asyncio.Event() + cancelled = asyncio.Event() + + class BlockingVision: + async def stream(self, _request): + started.set() + try: + await asyncio.Event().wait() + except asyncio.CancelledError: + cancelled.set() + raise + yield SimpleNamespace(text="unreachable") + + runtime = AgentRuntime() + runtime.register( + "simple-vlm", + SimpleVlmAgent(lambda: BlockingVision()), # type: ignore[return-value] + ) + + async with runtime: + await runtime.publish( + USER_QUERY_TOPIC, + UserQuery(text="What is shown?", timestamp_us=123), + participant_id="alice", + source="test-input", + ) + await asyncio.wait_for(started.wait(), 1.0) + await runtime.publish( + INTERRUPTED_TOPIC, + VoiceInterrupted(), + source="voice.interruption", + ) + await asyncio.wait_for(cancelled.wait(), 1.0) diff --git a/tests/test_subscriptions.py b/tests/test_subscriptions.py index 7e874876..89f39362 100644 --- a/tests/test_subscriptions.py +++ b/tests/test_subscriptions.py @@ -71,6 +71,20 @@ async def test_stop_clears_endpoint_running_barrier(make_processor): await asyncio.gather(running_wait, return_exceptions=True) + +async def test_data_callback_registration_can_be_removed(make_processor): + agent = make_processor(auto_subscribe=False) + + async def callback(_message) -> None: + return None + + unsubscribe = agent.on_data(callback) + assert callback in agent._data_cbs # noqa: SLF001 + + unsubscribe() + unsubscribe() + assert callback not in agent._data_cbs # noqa: SLF001 + # ── auto_subscribe=False ──────────────────────────────────────────────────── diff --git a/tests/test_voice_pipeline.py b/tests/test_voice_pipeline.py index 4053a091..8706526b 100644 --- a/tests/test_voice_pipeline.py +++ b/tests/test_voice_pipeline.py @@ -23,6 +23,7 @@ import warnings from typing import Any, AsyncIterator, Sequence +import nemo_relay import numpy as np import pytest from pipecat.frames.frames import ( @@ -41,7 +42,8 @@ from pipecat.processors.frame_processor import FrameDirection, FrameProcessor from pipecat.workers.runner import WorkerRunner -from xr_ai_voice import VadConfig, VoiceQuery, VoiceTurn +from xr_ai_voice import VadConfig +from xr_ai_voice._types import VoiceQuery from xr_ai_voice._pipeline import _build_voice_pipeline from xr_ai_voice._frames import ( AssistantResponseEndFrame, @@ -50,7 +52,7 @@ ParticipantLeftFrame, ) from xr_ai_voice._processors import ( - _VoiceHandlerProcessor, + _VoiceIOProcessor, StreamingTtsProcessor, VadSttProcessor, VoiceGateProcessor, @@ -240,7 +242,14 @@ async def feed(self, pcm_int16: bytes, sample_rate: int) -> None: frame = InputAudioRawFrame(audio=b"\x00\x00" * 320, sample_rate=16000, num_channels=1) frame.transport_source = "web-client" - sink = await _run_chain(proc, sends=[frame]) + events = [] + subscriber = "xr-ai-voice-stt-scopes" + nemo_relay.subscribers.register(subscriber, events.append) + try: + sink = await _run_chain(proc, sends=[frame]) + await nemo_relay.subscribers.flush_async() + finally: + nemo_relay.subscribers.deregister(subscriber) kinds = [type(f).__name__ for f in sink.frames] assert "UserStartedSpeakingFrame" in kinds @@ -253,6 +262,27 @@ async def feed(self, pcm_int16: bytes, sample_rate: int) -> None: assert transcripts[0].user_id == "web-client" assert transcripts[0].transport_source == "web-client" assert stt.calls and stt.calls[0][1] == 16000 + stt_start = next( + event.to_dict() + for event in events + if event.name == "voice.stt" + and event.to_dict().get("scope_category") == "start" + ) + stt_result = next( + event.to_dict() for event in events if event.name == "voice.stt.result" + ) + assert stt_start["category"] == "function" + assert stt_start["data"] == { + "audio_bytes": 640, + "audio_duration_ms": 20.0, + "sample_rate": 16000, + } + assert stt_start["metadata"] | { + "participant_id": "web-client", + "mode": "final", + } == stt_start["metadata"] + assert stt_result["parent_uuid"] == stt_start["uuid"] + assert stt_result["data"] == {"text": "hello agent"} @pytest.mark.asyncio @@ -1037,23 +1067,27 @@ async def test_voice_gate_processor_phrase_only_falls_back_to_final_chime(): # ════════════════════════════════════════════════════════════════════════════ -class _StringAssistant(_VoiceHandlerProcessor): +class _StringAssistant(_VoiceIOProcessor): def __init__(self, **callbacks: Any) -> None: super().__init__(self.handle, **callbacks) - self.handle_calls: list[tuple[str, str, bool]] = [] - - async def handle(self, query: VoiceQuery): - self.handle_calls.append((query.participant_id, query.text, query.fresh_match)) - return f"answer: {query.text}" + self.handle_calls: list[tuple[str, str]] = [] + + async def handle(self, query: VoiceQuery) -> None: + self.handle_calls.append((query.participant_id, query.text)) + await self.enqueue_response( + query.participant_id, + f"answer: {query.text}", + pts_us=query.timestamp_us, + ) -class _IterAssistant(_VoiceHandlerProcessor): +class _IterAssistant(_VoiceIOProcessor): def __init__(self, chunks: list[str], **callbacks: Any) -> None: super().__init__(self.handle, **callbacks) self._chunks = chunks self.cancelled = False - async def handle(self, _query: VoiceQuery) -> AsyncIterator[str]: + async def handle(self, query: VoiceQuery) -> None: async def _gen(): try: for c in self._chunks: @@ -1062,36 +1096,39 @@ async def _gen(): except asyncio.CancelledError: self.cancelled = True raise - return _gen() + + await self.enqueue_response( + query.participant_id, + _gen(), + pts_us=query.timestamp_us, + ) -class _LifecycleAssistant(_VoiceHandlerProcessor): +class _LifecycleAssistant(_VoiceIOProcessor): def __init__(self) -> None: - self.joined: list[str] = [] self.left: list[str] = [] - self.started_speaking: list[str] = [] super().__init__( self.handle, - on_participant_joined=self.on_participant_joined, on_participant_left=self.on_participant_left, - on_user_started_speaking=self.on_user_started_speaking, ) - async def handle(self, _query: VoiceQuery): - return "" - - async def on_participant_joined(self, pid: str) -> None: - self.joined.append(pid) + async def handle(self, _query: VoiceQuery) -> None: + pass async def on_participant_left(self, pid: str) -> None: self.left.append(pid) - async def on_user_started_speaking(self, pid: str) -> None: - self.started_speaking.append(pid) +class _InputOnlyAssistant(_VoiceIOProcessor): + def __init__(self) -> None: + self.queries: list[VoiceQuery] = [] + super().__init__(self.handle) + + async def handle(self, query: VoiceQuery) -> None: + self.queries.append(query) @pytest.mark.asyncio -async def test_assistant_string_return_pushes_single_text_frame(): +async def test_runtime_input_can_enqueue_finite_voice_output(): assistant = _StringAssistant() sink = await _run_chain( assistant, @@ -1100,30 +1137,11 @@ async def test_assistant_string_return_pushes_single_text_frame(): texts = [f for f in sink.frames if isinstance(f, TextFrame)] assert [t.text for t in texts] == ["answer: hi"] - assert assistant.handle_calls == [("pid-1", "hi", True)] - - -@pytest.mark.asyncio -async def test_voice_turn_observer_receives_user_and_completed_agent_text(): - turns: list[VoiceTurn] = [] - - async def observe(turn: VoiceTurn) -> None: - turns.append(turn) - - assistant = _StringAssistant(observer=observe) - await _run_chain( - assistant, - sends=[GatedQueryFrame(participant_id="pid-1", text="hi", fresh_match=True, pts_us=42)], - ) - - assert [(turn.role, turn.text, turn.timestamp_us) for turn in turns] == [ - ("user", "hi", 42), - ("agent", "answer: hi", 42), - ] + assert assistant.handle_calls == [("pid-1", "hi")] @pytest.mark.asyncio -async def test_assistant_async_iter_return_pushes_text_frame_per_chunk(): +async def test_runtime_input_can_enqueue_incremental_voice_output(): assistant = _IterAssistant(chunks=["alpha ", "beta ", "gamma."]) sink = await _run_chain( assistant, @@ -1134,6 +1152,167 @@ async def test_assistant_async_iter_return_pushes_text_frame_per_chunk(): assert texts == ["alpha ", "beta ", "gamma."] +@pytest.mark.asyncio +@pytest.mark.asyncio +async def test_interrupting_response_does_not_cancel_query_delivery() -> None: + query_started = asyncio.Event() + release_query = asyncio.Event() + + async def input_sink(_query: VoiceQuery) -> None: + query_started.set() + await release_query.wait() + + assistant = _VoiceIOProcessor(input_sink) + + async def capture( + _frame: Frame, + _direction: FrameDirection = FrameDirection.DOWNSTREAM, + ) -> None: + return None + + assistant.push_frame = capture # type: ignore[method-assign] + await assistant.enqueue_query("pid-1", "look") + await asyncio.wait_for(query_started.wait(), 1.0) + input_task = next(iter(assistant._input_tasks)) # noqa: SLF001 + + await assistant.enqueue_response("pid-1", "answer", interrupt=True) + await asyncio.wait_for(assistant._inflight["pid-1"], 1.0) # noqa: SLF001 + + assert not input_task.cancelled() + release_query.set() + await asyncio.wait_for(input_task, 1.0) + + +@pytest.mark.asyncio +async def test_failed_response_iterator_is_closed() -> None: + class FailingStream: + def __init__(self) -> None: + self.closed = False + + def __aiter__(self): + return self + + async def __anext__(self) -> str: + raise RuntimeError("producer failed") + + async def aclose(self) -> None: + self.closed = True + + async def input_sink(_query: VoiceQuery) -> None: + return None + + stream = FailingStream() + assistant = _VoiceIOProcessor(input_sink) + + async def capture( + _frame: Frame, + _direction: FrameDirection = FrameDirection.DOWNSTREAM, + ) -> None: + return None + + assistant.push_frame = capture # type: ignore[method-assign] + await assistant.enqueue_response("pid-1", stream) + await asyncio.wait_for(assistant._inflight["pid-1"], 1.0) # noqa: SLF001 + + assert stream.closed is True + + +async def test_external_response_stream_uses_normal_assistant_output_frames() -> None: + frames: list[Frame] = [] + + async def input_sink(_query: VoiceQuery) -> None: + pass + + async def chunks() -> AsyncIterator[str]: + yield "The kettle " + yield "is boiling." + + assistant = _VoiceIOProcessor(input_sink) + + async def capture(frame: Frame, _direction: FrameDirection = FrameDirection.DOWNSTREAM) -> None: + frames.append(frame) + + assistant.push_frame = capture # type: ignore[method-assign] + await assistant.enqueue_response("pid-1", chunks(), pts_us=42) + task = assistant._inflight["pid-1"] # noqa: SLF001 + _ = await task + + assert [frame.text for frame in frames if isinstance(frame, TextFrame)] == [ + "The kettle ", + "is boiling.", + ] + end = next(frame for frame in frames if isinstance(frame, AssistantResponseEndFrame)) + assert (end.pid, end.text, end.pts_us) == ("pid-1", "The kettle is boiling.", 42) + + +@pytest.mark.asyncio +async def test_external_response_preserves_chunks_before_iterator_failure() -> None: + frames: list[Frame] = [] + + async def input_sink(_query: VoiceQuery) -> None: + pass + + async def chunks() -> AsyncIterator[str]: + yield "Partial response." + raise RuntimeError("producer failed") + + assistant = _VoiceIOProcessor(input_sink) + + async def capture( + frame: Frame, + _direction: FrameDirection = FrameDirection.DOWNSTREAM, + ) -> None: + frames.append(frame) + + assistant.push_frame = capture # type: ignore[method-assign] + await assistant.enqueue_response("pid-1", chunks(), pts_us=42) + await assistant._inflight["pid-1"] # noqa: SLF001 + + end = next(frame for frame in frames if isinstance(frame, AssistantResponseEndFrame)) + assert end.text == "Partial response." + + +@pytest.mark.asyncio +async def test_external_responses_preserve_participant_fifo() -> None: + frames: list[Frame] = [] + release_first = asyncio.Event() + + async def input_sink(_query: VoiceQuery) -> None: + pass + + async def first_response() -> AsyncIterator[str]: + yield "first " + await release_first.wait() + yield "done" + + assistant = _VoiceIOProcessor(input_sink) + + async def capture( + frame: Frame, + _direction: FrameDirection = FrameDirection.DOWNSTREAM, + ) -> None: + frames.append(frame) + + assistant.push_frame = capture # type: ignore[method-assign] + await assistant.enqueue_response("pid-1", first_response()) + await asyncio.sleep(0) + await assistant.enqueue_response("pid-1", "second") + + assert len(assistant._queued["pid-1"]) == 1 # noqa: SLF001 + release_first.set() + async def wait_until_idle() -> None: + while assistant._inflight: # noqa: SLF001 + await asyncio.gather(*tuple(assistant._inflight.values())) # noqa: SLF001 + + await asyncio.wait_for(wait_until_idle(), 1.0) + + assert [frame.text for frame in frames if isinstance(frame, TextFrame)] == [ + "first ", + "done", + "second", + ] + + @pytest.mark.asyncio async def test_assistant_does_not_cancel_on_user_started_speaking(): """Regression guard: ``UserStartedSpeakingFrame`` is a hook, not a @@ -1178,57 +1357,6 @@ async def test_assistant_cancels_inflight_on_new_query_for_same_pid(): assert assistant.cancelled is True -@pytest.mark.asyncio -async def test_assistant_queues_queries_and_interrupts_audio_when_next_turn_starts(): - order: list[str] = [] - cancelled: list[str] = [] - release_first = asyncio.Event() - - async def handle(query: VoiceQuery) -> str: - order.append(f"start:{query.text}") - try: - if query.text == "first": - await release_first.wait() - order.append(f"finish:{query.text}") - return f"answer: {query.text}" - except asyncio.CancelledError: - cancelled.append(query.text) - raise - - async def release_first_later() -> None: - await asyncio.sleep(0.12) - release_first.set() - - assistant = _VoiceHandlerProcessor( - handle, - interrupt_on_supersede=True, - queue_queries=True, - ) - release = asyncio.create_task(release_first_later()) - sink = await _run_chain( - assistant, - sends=[ - GatedQueryFrame(participant_id="pid-1", text="first", fresh_match=True, pts_us=0), - GatedQueryFrame(participant_id="pid-1", text="second", fresh_match=True, pts_us=1), - ], - settle_s=0.3, - per_send_delay_s=0.03, - ) - await release - - assert order == ["start:first", "finish:first", "start:second", "finish:second"] - assert cancelled == [] - assert [frame.text for frame in sink.frames if isinstance(frame, TextFrame)] == [ - "answer: first", - "answer: second", - ] - text_indices = [index for index, frame in enumerate(sink.frames) if isinstance(frame, TextFrame)] - interrupt_index = next( - index for index, frame in enumerate(sink.frames) if isinstance(frame, InterruptionFrame) - ) - assert text_indices[0] < interrupt_index < text_indices[1] - - @pytest.mark.asyncio async def test_assistant_cancels_inflight_on_interruption_frame(): assistant = _IterAssistant(chunks=[f"chunk{i} " for i in range(200)]) @@ -1245,7 +1373,7 @@ async def test_assistant_cancels_inflight_on_interruption_frame(): @pytest.mark.asyncio -async def test_assistant_closes_stream_on_interruption() -> None: +async def test_voice_io_closes_cancelled_response_stream() -> None: started = asyncio.Event() blocked = asyncio.Event() @@ -1266,154 +1394,67 @@ async def aclose(self) -> None: stream = HeldStream() - async def handle(_query: VoiceQuery) -> AsyncIterator[str]: - return stream - - assistant = _VoiceHandlerProcessor(handle) - await _run_chain( - assistant, - sends=[ - GatedQueryFrame( - participant_id="pid-1", - text="hi", - fresh_match=True, - pts_us=0, - ), - InterruptionFrame(), - ], - settle_s=0.2, - per_send_delay_s=0.05, - ) - - assert started.is_set() - assert stream.closed is True + async def handle(_query: VoiceQuery) -> None: + return None + assistant = _VoiceIOProcessor(handle) -@pytest.mark.asyncio -async def test_interruption_cancels_active_task_and_clears_queued_queries(): - started: list[str] = [] - cancelled: list[str] = [] + async def capture( + _frame: Frame, + _direction: FrameDirection = FrameDirection.DOWNSTREAM, + ) -> None: + return None - async def handle(query: VoiceQuery) -> str: - started.append(query.text) - try: - await asyncio.Event().wait() - except asyncio.CancelledError: - cancelled.append(query.text) - raise - return "unreachable" + assistant.push_frame = capture # type: ignore[method-assign] + await assistant.enqueue_response("pid-1", stream) + task = assistant._inflight["pid-1"] # noqa: SLF001 + await asyncio.wait_for(started.wait(), 1.0) + await assistant._cancel_pid("pid-1") # noqa: SLF001 + await asyncio.gather(task, return_exceptions=True) - assistant = _VoiceHandlerProcessor(handle, queue_queries=True) - await _run_chain( - assistant, - sends=[ - GatedQueryFrame(participant_id="pid-1", text="first", fresh_match=True, pts_us=0), - GatedQueryFrame(participant_id="pid-1", text="second", fresh_match=True, pts_us=1), - InterruptionFrame(), - ], - settle_s=0.2, - per_send_delay_s=0.05, - ) - - assert started == ["first"] - assert cancelled == ["first"] - assert assistant._queued == {} # noqa: SLF001 + assert stream.closed is True @pytest.mark.asyncio -async def test_assistant_on_query_superseded_fires_on_second_query_while_inflight(): - """When a fresh ``GatedQueryFrame`` arrives while a prior query's - task is still in-flight, ``on_query_superseded`` must fire so an - agent can decide what to do with the previous response's queued - downstream state (e.g. push an InterruptionFrame to drain queued - TTS audio). The library default is a no-op; this test only checks - the hook is invoked.""" - class _SupersedeRecorder(_IterAssistant): - def __init__(self) -> None: - self.supersede_calls: list[str] = [] - super().__init__( - chunks=[f"chunk{i} " for i in range(200)], - on_query_superseded=self.on_query_superseded, - ) +@pytest.mark.asyncio +async def test_participant_left_clears_seen_output_state() -> None: + async def input_sink(_query: VoiceQuery) -> None: + return None - async def on_query_superseded(self, pid: str) -> None: - self.supersede_calls.append(pid) + assistant = _VoiceIOProcessor(input_sink) + assistant._seen_output.add("pid-1") # noqa: SLF001 - assistant = _SupersedeRecorder() - await _run_chain( - assistant, - sends=[ - GatedQueryFrame(participant_id="pid-1", text="first", fresh_match=True, pts_us=0), - GatedQueryFrame(participant_id="pid-1", text="second", fresh_match=True, pts_us=1), - ], - settle_s=0.2, - per_send_delay_s=0.05, - ) - assert assistant.supersede_calls == ["pid-1"] - assert assistant.cancelled is True + async def capture( + _frame: Frame, + _direction: FrameDirection = FrameDirection.DOWNSTREAM, + ) -> None: + return None + assistant.push_frame = capture # type: ignore[method-assign] + await assistant.process_frame( + ParticipantLeftFrame(participant_id="pid-1"), + FrameDirection.DOWNSTREAM, + ) -@pytest.mark.asyncio -async def test_assistant_on_query_superseded_not_fired_when_prior_task_done(): - """The supersede hook fires only for an ACTUAL replacement — a new query - that lands while the prior turn is still in flight. When the prior assistant - task has already completed before the next query arrives, there is nothing - to supersede, so the hook must NOT fire even though the participant has - spoken before. (Clearing any lingering TTS is handled by the downstream - interrupt, not by this callback.)""" - class _SupersedeRecorder(_StringAssistant): - def __init__(self) -> None: - self.supersede_calls: list[str] = [] - super().__init__(on_query_superseded=self.on_query_superseded) + assert "pid-1" not in assistant._seen_output # noqa: SLF001 - async def on_query_superseded(self, pid: str) -> None: - self.supersede_calls.append(pid) - assistant = _SupersedeRecorder() - await _run_chain( - assistant, - sends=[ - GatedQueryFrame(participant_id="pid-1", text="first", fresh_match=True, pts_us=0), - GatedQueryFrame(participant_id="pid-1", text="second", fresh_match=True, pts_us=1), - ], - # Long enough that the one-shot ``_StringAssistant`` response for the - # first query has fully run before the second query is queued. - settle_s=0.3, - per_send_delay_s=0.2, - ) - assert assistant.supersede_calls == [] - # Both queries still ran end-to-end. - handled = [t for _pid, t, _fm in assistant.handle_calls] - assert "first" in handled - assert "second" in handled +async def test_assistant_notifies_external_runtime_on_interruption_frame(): + interrupted: list[str | None] = [] + async def handle(_query: VoiceQuery) -> str: + return "unused" -@pytest.mark.asyncio -async def test_assistant_on_query_superseded_not_fired_on_serial_queries(): - """Serial queries — each completing before the next arrives — are not - supersedes. Four serial queries fire the hook zero times; it fires only when - a new query replaces a still-in-flight turn.""" - class _SupersedeRecorder(_StringAssistant): - def __init__(self) -> None: - self.supersede_calls: list[str] = [] - super().__init__(on_query_superseded=self.on_query_superseded) + async def on_interrupted(pid: str | None) -> None: + interrupted.append(pid) - async def on_query_superseded(self, pid: str) -> None: - self.supersede_calls.append(pid) + assistant = _VoiceIOProcessor(handle, on_interrupted=on_interrupted) + participant_frame = InterruptionFrame() + participant_frame.transport_source = "pid-1" + global_frame = InterruptionFrame() + await _run_chain(assistant, sends=[participant_frame, global_frame]) - assistant = _SupersedeRecorder() - await _run_chain( - assistant, - sends=[ - GatedQueryFrame(participant_id="pid-1", text="first", fresh_match=True, pts_us=0), - GatedQueryFrame(participant_id="pid-1", text="second", fresh_match=True, pts_us=1), - GatedQueryFrame(participant_id="pid-1", text="third", fresh_match=True, pts_us=2), - GatedQueryFrame(participant_id="pid-1", text="fourth", fresh_match=True, pts_us=3), - ], - settle_s=0.3, - per_send_delay_s=0.1, - ) - assert assistant.supersede_calls == [] + assert interrupted == ["pid-1", None] @pytest.mark.asyncio @@ -1437,59 +1478,6 @@ def __init__(self) -> None: assert assistant.cancelled, "the in-flight handler task must be cancelled" -@pytest.mark.asyncio -async def test_assistant_on_query_superseded_seen_state_cleared_on_participant_left(): - """``_seen_query`` is per-pid and must be cleared on - ``ParticipantLeftFrame`` so a rejoin's first query is treated - as cold (no supersede) rather than as a follow-up. Without - this, an override that pushes ``InterruptionFrame`` on - supersede would flush unrelated audio on every fresh session.""" - class _SupersedeRecorder(_StringAssistant): - def __init__(self) -> None: - self.supersede_calls: list[str] = [] - super().__init__(on_query_superseded=self.on_query_superseded) - - async def on_query_superseded(self, pid: str) -> None: - self.supersede_calls.append(pid) - - assistant = _SupersedeRecorder() - await _run_chain( - assistant, - sends=[ - GatedQueryFrame(participant_id="pid-1", text="first", fresh_match=True, pts_us=0), - ParticipantLeftFrame(participant_id="pid-1"), - # Same pid rejoins (or different session): the first - # query after the left frame must NOT fire the hook. - GatedQueryFrame(participant_id="pid-1", text="second", fresh_match=True, pts_us=1), - ], - settle_s=0.3, - per_send_delay_s=0.1, - ) - assert assistant.supersede_calls == [] - - -@pytest.mark.asyncio -async def test_assistant_on_query_superseded_not_called_on_cold_path_first_query(): - """The cold path — first query, no in-flight task — must NOT call - ``on_query_superseded``. There is nothing to supersede, and agents - that override to push an InterruptionFrame would otherwise flush - unrelated in-flight audio (e.g. a voice-gate chime).""" - class _SupersedeRecorder(_StringAssistant): - def __init__(self) -> None: - self.supersede_calls: list[str] = [] - super().__init__(on_query_superseded=self.on_query_superseded) - - async def on_query_superseded(self, pid: str) -> None: - self.supersede_calls.append(pid) - - assistant = _SupersedeRecorder() - await _run_chain( - assistant, - sends=[GatedQueryFrame(participant_id="pid-1", text="hi", fresh_match=True, pts_us=0)], - ) - assert assistant.supersede_calls == [] - - @pytest.mark.asyncio async def test_handler_can_request_audio_interruption_when_superseded(): """The explicit supersede option drains queued TTS audio for the prior response.""" @@ -1515,45 +1503,6 @@ def __init__(self) -> None: ) -@pytest.mark.asyncio -async def test_assistant_on_query_superseded_exception_is_swallowed_and_spawn_proceeds(): - """A misbehaving override must not break the supersede contract: - the previous task is still cancelled and the new query still - spawns. The exception is logged at the library boundary.""" - class _RaisingAssistant(_IterAssistant): - def __init__(self) -> None: - self.handle_calls: list[str] = [] - self.supersede_calls: list[str] = [] - super().__init__( - chunks=[f"chunk{i} " for i in range(200)], - on_query_superseded=self.on_query_superseded, - ) - - async def handle(self, query: VoiceQuery): - self.handle_calls.append(query.text) - return await super().handle(query) - - async def on_query_superseded(self, pid: str) -> None: - self.supersede_calls.append(pid) - raise RuntimeError("boom") - - assistant = _RaisingAssistant() - await _run_chain( - assistant, - sends=[ - GatedQueryFrame(participant_id="pid-1", text="first", fresh_match=True, pts_us=0), - GatedQueryFrame(participant_id="pid-1", text="second", fresh_match=True, pts_us=1), - ], - settle_s=0.2, - per_send_delay_s=0.05, - ) - assert assistant.supersede_calls == ["pid-1"] - # Previous task still cancelled; new query still ran. - assert assistant.cancelled is True - assert "first" in assistant.handle_calls - assert "second" in assistant.handle_calls - - @pytest.mark.asyncio async def test_assistant_steers_transport_target_on_participant_joined(): """Single-participant routing default: when a assistant is constructed @@ -1604,7 +1553,7 @@ async def test_assistant_no_transport_steering_when_not_configured(): @pytest.mark.asyncio -async def test_assistant_participant_lifecycle_hooks_fire(): +async def test_assistant_participant_left_callback_fires(): assistant = _LifecycleAssistant() sink = await _run_chain( assistant, @@ -1614,61 +1563,12 @@ async def test_assistant_participant_lifecycle_hooks_fire(): ], ) - assert assistant.joined == ["p1"] assert assistant.left == ["p1"] kinds = [type(f).__name__ for f in sink.frames] assert "ParticipantJoinedFrame" in kinds assert "ParticipantLeftFrame" in kinds -@pytest.mark.asyncio -async def test_assistant_user_started_speaking_hook_fires_for_joined_pids(): - """on_user_started_speaking fires for every joined pid (NOT just the - in-flight ones), so the cold path — first utterance, nothing in - flight yet — still gets the speculative-warmup hook. Tracking - in-flight tasks here would mean the very first turn never sees - camera warmup, which is precisely the case it was designed for.""" - started_for: list[str] = [] - - async def speech_hook(pid: str) -> None: - started_for.append(pid) - - assistant = _IterAssistant(chunks=[], on_user_started_speaking=speech_hook) - - await _run_chain( - assistant, - sends=[ - ParticipantJoinedFrame(participant_id="pid-1"), - UserStartedSpeakingFrame(), - ], - settle_s=0.1, - per_send_delay_s=0.05, - ) - assert started_for == ["pid-1"] - - -@pytest.mark.asyncio -async def test_assistant_user_started_speaking_hook_skipped_after_leave(): - started_for: list[str] = [] - - async def speech_hook(pid: str) -> None: - started_for.append(pid) - - assistant = _IterAssistant(chunks=[], on_user_started_speaking=speech_hook) - - await _run_chain( - assistant, - sends=[ - ParticipantJoinedFrame(participant_id="pid-1"), - ParticipantLeftFrame(participant_id="pid-1"), - UserStartedSpeakingFrame(), - ], - settle_s=0.1, - per_send_delay_s=0.05, - ) - assert started_for == [] - - # ════════════════════════════════════════════════════════════════════════════ # StreamingTtsProcessor # ════════════════════════════════════════════════════════════════════════════ @@ -1680,13 +1580,29 @@ async def test_streaming_tts_sentence_boundary_triggers_synth(): gate = VoiceGate(VoiceGateConfig(), audio_sink=_NullSink(), tts=tts) proc = StreamingTtsProcessor(tts=tts, voice_gate=gate) - sink = await _run_chain( - proc, - sends=[TextFrame(text="hello"), TextFrame(text=" world. ")], - ) + events = [] + subscriber = "xr-ai-voice-tts-scopes" + nemo_relay.subscribers.register(subscriber, events.append) + try: + sink = await _run_chain( + proc, + sends=[TextFrame(text="hello"), TextFrame(text=" world. ")], + ) + await nemo_relay.subscribers.flush_async() + finally: + nemo_relay.subscribers.deregister(subscriber) assert tts.calls == ["hello world."] audio = [f for f in sink.frames if isinstance(f, OutputAudioRawFrame)] assert audio, "synth produced no audio frames downstream" + tts_start = next( + event.to_dict() + for event in events + if event.name == "voice.tts" + and event.to_dict().get("scope_category") == "start" + ) + assert tts_start["category"] == "function" + assert tts_start["data"] == {"text": "hello world."} + assert tts_start["metadata"]["participant_id"] is None @pytest.mark.asyncio @@ -2090,12 +2006,16 @@ async def capture(frame, direction=FrameDirection.DOWNSTREAM): # ════════════════════════════════════════════════════════════════════════════ -class _EchoAssistant(_VoiceHandlerProcessor): +class _EchoAssistant(_VoiceIOProcessor): def __init__(self) -> None: super().__init__(self.handle) - async def handle(self, query: VoiceQuery) -> str: - return f"echo {query.text}." + async def handle(self, query: VoiceQuery) -> None: + await self.enqueue_response( + query.participant_id, + f"echo {query.text}.", + pts_us=query.timestamp_us, + ) @pytest.mark.asyncio @@ -2129,7 +2049,7 @@ async def feed(self, pcm_int16: bytes, sample_rate: int) -> None: transport = transport, stt = stt, tts = tts, - handler_processor = _EchoAssistant(), + io_processor = _EchoAssistant(), vad_cfg = VadConfig(), voice_gate_cfg = VoiceGateConfig(), ) @@ -2183,7 +2103,7 @@ def test_private_pipeline_assembly_wires_early_wake_ack_for_chime_config(): transport=transport, stt=_FakeStt(), tts=_FakeTts(), - handler_processor=_EchoAssistant(), + io_processor=_EchoAssistant(), vad_cfg=VadConfig(), voice_gate_cfg=VoiceGateConfig( magic_phrases=("hey agent",), @@ -2216,7 +2136,7 @@ def test_private_pipeline_assembly_disables_idle_timeout_by_default(): transport = transport, stt = _FakeStt(), tts = _FakeTts(), - handler_processor = _EchoAssistant(), + io_processor = _EchoAssistant(), vad_cfg = VadConfig(), voice_gate_cfg = VoiceGateConfig(), ) @@ -2237,7 +2157,7 @@ def test_private_pipeline_assembly_accepts_idle_timeout(): transport = transport, stt = _FakeStt(), tts = _FakeTts(), - handler_processor = _EchoAssistant(), + io_processor = _EchoAssistant(), vad_cfg = VadConfig(), voice_gate_cfg = VoiceGateConfig(), idle_timeout_secs = 300.0, @@ -2254,7 +2174,7 @@ def test_private_pipeline_assembly_accepts_idle_timeout(): @pytest.mark.asyncio -async def test_assistant_tags_text_frame_with_pid_for_string_return(): +async def test_assistant_tags_finite_output_with_pid(): """The assistant MUST set ``transport_destination`` on every TextFrame. Downstream ``StreamingTtsProcessor`` reads @@ -2275,7 +2195,7 @@ async def test_assistant_tags_text_frame_with_pid_for_string_return(): @pytest.mark.asyncio -async def test_assistant_tags_text_frame_with_pid_for_async_iter_return(): +async def test_assistant_tags_incremental_output_with_pid(): assistant = _IterAssistant(chunks=["alpha ", "beta."]) sink = await _run_chain( assistant, @@ -2293,7 +2213,7 @@ async def test_assistant_tags_text_frame_with_pid_for_async_iter_return(): @pytest.mark.asyncio -async def test_assistant_emits_response_end_after_string_turn(): +async def test_assistant_emits_response_end_after_finite_output(): """One ``AssistantResponseEndFrame`` per completed turn carries the full assembled text and pid — the downstream data-channel echo keys off this marker.""" @@ -2310,7 +2230,7 @@ async def test_assistant_emits_response_end_after_string_turn(): @pytest.mark.asyncio -async def test_assistant_emits_response_end_after_streamed_turn(): +async def test_assistant_emits_response_end_after_streamed_output(): assistant = _IterAssistant(chunks=["one ", "two ", "three."]) sink = await _run_chain( assistant, @@ -2323,6 +2243,26 @@ async def test_assistant_emits_response_end_after_streamed_turn(): assert ends[0].pid == "pid-1" +@pytest.mark.asyncio +async def test_input_only_handler_does_not_emit_an_empty_agent_response(): + assistant = _InputOnlyAssistant() + sink = await _run_chain( + assistant, + sends=[ + GatedQueryFrame( + participant_id="pid-1", + text="publish this", + fresh_match=True, + pts_us=7, + ) + ], + ) + + assert [query.text for query in assistant.queries] == ["publish this"] + assert not any(isinstance(frame, TextFrame) for frame in sink.frames) + assert not any(isinstance(frame, AssistantResponseEndFrame) for frame in sink.frames) + + @pytest.mark.asyncio async def test_assistant_does_not_emit_response_end_on_cancel(): """Cancellation (new query or InterruptionFrame) supersedes the @@ -2350,44 +2290,6 @@ async def test_assistant_does_not_emit_response_end_on_cancel(): # ════════════════════════════════════════════════════════════════════════════ -class _StreamMethodAssistant(_VoiceHandlerProcessor): - """Lock in a handler method returning an async iterator: - - async def handle(...) -> AsyncIterator[str]: - return self._stream(...) - - where ``_stream`` is itself an async-generator function. Awaiting - ``handle`` resolves to the async-generator object the processor consumes. - """ - - def __init__(self, chunks: list[str]) -> None: - super().__init__(self.handle) - self._chunks = chunks - - async def handle(self, query: VoiceQuery) -> AsyncIterator[str]: - return self._stream(query.participant_id, query.text) - - async def _stream(self, pid: str, text: str) -> AsyncIterator[str]: - for c in self._chunks: - yield c - await asyncio.sleep(0.001) - - -@pytest.mark.asyncio -async def test_assistant_supports_handler_returning_async_generator_method(): - assistant = _StreamMethodAssistant(chunks=["foo ", "bar."]) - sink = await _run_chain( - assistant, - sends=[GatedQueryFrame(participant_id="pid-1", text="hi", fresh_match=True, pts_us=0)], - settle_s=0.15, - ) - texts = [f.text for f in sink.frames if isinstance(f, TextFrame)] - assert texts == ["foo ", "bar."] - ends = [f for f in sink.frames if isinstance(f, AssistantResponseEndFrame)] - assert len(ends) == 1 - assert ends[0].text == "foo bar." - - # ════════════════════════════════════════════════════════════════════════════ # Regression: StreamingTts data-channel echo (Bug #4) # ════════════════════════════════════════════════════════════════════════════ @@ -2951,7 +2853,7 @@ async def test_private_pipeline_assembly_routes_text_through_streaming_tts(monke transport = transport, stt = _FakeStt(), tts = _FakeTts(), - handler_processor = _StringAssistant(), + io_processor = _StringAssistant(), vad_cfg = VadConfig(), voice_gate_cfg = VoiceGateConfig(), text_topic = "vlm.response", diff --git a/tests/test_voice_runtime.py b/tests/test_voice_runtime.py new file mode 100644 index 00000000..5f9656d9 --- /dev/null +++ b/tests/test_voice_runtime.py @@ -0,0 +1,686 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Contract tests for the bidirectional voice runtime agent.""" + +from __future__ import annotations + +import asyncio +from builtins import ExceptionGroup +from collections.abc import AsyncIterator +from contextlib import asynccontextmanager + +import nemo_relay +import pytest +from pydantic import ValidationError +from xr_ai_hub import DataMessage +from xr_ai_runtime import Agent, AgentRuntime, RuntimeContext, Topic, subscribe +from xr_ai_voice import ( + VOICE_OUTPUT_TOPIC, + UserQuery, + VoiceAgent, + VoiceInterrupted, + VoiceOutput, + VoiceParticipantLeft, +) +from xr_ai_voice import _runtime as voice_runtime_module +from xr_ai_voice._types import VoiceQuery + +QUERY_TOPIC = Topic("test.user-query", UserQuery) +PARTICIPANT_LEFT_TOPIC = Topic("test.participant-left", VoiceParticipantLeft) +INTERRUPTED_TOPIC = Topic("test.interrupted", VoiceInterrupted) + + +class _Endpoint: + def on_data(self, callback): + self.data_callback = callback + + def unsubscribe() -> None: + self.data_callback = None + + return unsubscribe + + +class _Transport: + def __init__(self) -> None: + self.endpoint = _Endpoint() + self.target_participant = "" + + def set_target_participant(self, participant_id: str) -> None: + self.target_participant = participant_id + + +class _Session: + def __init__(self) -> None: + self.transport = _Transport() + self.text_topic = "agent.response" + self.responses: list[tuple[str, str, bool, int | None]] = [] + self.response_tasks: list[asyncio.Task[None]] = [] + self.run_options = {} + self.handler = None + self.started = asyncio.Event() + self.changed = asyncio.Event() + self.closed = False + self.queries: list[tuple[str, str, int | None]] = [] + + @property + def is_running(self) -> bool: + return self.handler is not None + + @property + def endpoint(self): + return self.transport.endpoint + + async def __aenter__(self): + return self + + async def run(self, handler, **options) -> None: + self.handler = handler + self.run_options = options + self.started.set() + await asyncio.Event().wait() + + async def enqueue_response( + self, + participant_id: str, + response: str | AsyncIterator[str], + *, + interrupt: bool = False, + pts_us: int | None = None, + ) -> None: + if isinstance(response, str): + self.responses.append((participant_id, response, interrupt, pts_us)) + self.changed.set() + return + + async def consume() -> None: + chunks = [chunk async for chunk in response] + self.responses.append((participant_id, "".join(chunks), interrupt, pts_us)) + self.changed.set() + + self.response_tasks.append(asyncio.create_task(consume())) + + async def enqueue_query( + self, + participant_id: str, + text: str, + *, + pts_us: int | None = None, + ) -> None: + self.queries.append((participant_id, text, pts_us)) + + async def wait_for(self, count: int) -> None: + while len(self.responses) < count: + self.changed.clear() + await self.changed.wait() + + async def close(self) -> None: + self.closed = True + if self.response_tasks: + await asyncio.gather(*self.response_tasks, return_exceptions=True) + + +class _InputRecorder(Agent): + def __init__(self) -> None: + super().__init__() + self.messages: list[tuple[str | None, str, UserQuery]] = [] + self.changed = asyncio.Event() + + @subscribe(QUERY_TOPIC) + async def record(self, query: UserQuery, ctx: RuntimeContext) -> None: + self.messages.append((ctx.metadata.participant_id, ctx.metadata.source, query)) + self.changed.set() + + +class _LifecycleRecorder(Agent): + def __init__(self) -> None: + super().__init__() + self.events: list[tuple[str, str | None]] = [] + self.changed = asyncio.Event() + + @subscribe(PARTICIPANT_LEFT_TOPIC) + async def participant_left( + self, + _event: VoiceParticipantLeft, + ctx: RuntimeContext, + ) -> None: + self.events.append(("participant-left", ctx.metadata.participant_id)) + self.changed.set() + + @subscribe(INTERRUPTED_TOPIC) + async def interrupted( + self, + _event: VoiceInterrupted, + ctx: RuntimeContext, + ) -> None: + self.events.append(("interrupted", ctx.metadata.participant_id)) + self.changed.set() + + async def wait_for(self, count: int) -> None: + while len(self.events) < count: + self.changed.clear() + await self.changed.wait() + + +@asynccontextmanager +async def _running_voice( + runtime: AgentRuntime, + voice: VoiceAgent, + session: _Session, +) -> AsyncIterator[None]: + async with runtime: + task = asyncio.create_task(voice.run(runtime)) + await asyncio.wait_for(session.started.wait(), 1.0) + try: + yield + finally: + task.cancel() + await asyncio.gather(task, return_exceptions=True) + + +async def test_voice_agent_publishes_to_configured_query_topic() -> None: + session = _Session() + recorder = _InputRecorder() + runtime = AgentRuntime() + runtime.register("recorder", recorder) + voice = VoiceAgent( # type: ignore[arg-type] + session, + query_topic=QUERY_TOPIC, + text_input=False, + ) + runtime.register("voice", voice) + + async with _running_voice(runtime, voice, session): + assert session.handler is not None + assert await session.handler( + VoiceQuery( + participant_id="alice", + text="start monitoring", + timestamp_us=7, + ) + ) is None + await asyncio.wait_for(recorder.changed.wait(), 1.0) + + assert recorder.messages == [ + ( + "alice", + "voice", + UserQuery(text="start monitoring", timestamp_us=7), + ) + ] + assert session.closed is True + + +async def test_voice_agent_publishes_configured_lifecycle_topics() -> None: + session = _Session() + recorder = _LifecycleRecorder() + runtime = AgentRuntime() + runtime.register("recorder", recorder) + voice = VoiceAgent( # type: ignore[arg-type] + session, + query_topic=QUERY_TOPIC, + participant_left_topic=PARTICIPANT_LEFT_TOPIC, + interrupted_topic=INTERRUPTED_TOPIC, + text_input=False, + ) + runtime.register("voice", voice) + + async with _running_voice(runtime, voice, session): + await session.run_options["on_participant_left"]("alice") + await session.run_options["on_interrupted"](None) + await asyncio.wait_for(recorder.wait_for(2), 1.0) + + assert recorder.events == [ + ("participant-left", "alice"), + ("interrupted", None), + ] + + +async def test_voice_agent_accepts_output_from_multiple_publishers() -> None: + session = _Session() + runtime = AgentRuntime() + voice = VoiceAgent( # type: ignore[arg-type] + session, + query_topic=QUERY_TOPIC, + text_input=False, + ) + runtime.register("voice", voice) + + async with _running_voice(runtime, voice, session): + await runtime.publish( + VOICE_OUTPUT_TOPIC, + VoiceOutput(text="Careful.", interrupt=True), + participant_id="alice", + source="safety-monitor", + ) + await runtime.publish( + VOICE_OUTPUT_TOPIC, + VoiceOutput(text="The timer is done."), + participant_id="alice", + source="tea-timer", + ) + + assert [(pid, text, interrupt) for pid, text, interrupt, _ in session.responses] == [ + ("alice", "Careful.", True), + ("alice", "The timer is done.", False), + ] + + +async def test_voice_agent_records_one_summary_for_finite_and_streamed_output() -> None: + session = _Session() + runtime = AgentRuntime() + voice = VoiceAgent( # type: ignore[arg-type] + session, + query_topic=QUERY_TOPIC, + text_input=False, + ) + runtime.register("voice", voice) + events = [] + subscriber = "xr-ai-voice-response-summary" + nemo_relay.subscribers.register(subscriber, events.append) + try: + async with _running_voice(runtime, voice, session): + await runtime.publish( + VOICE_OUTPUT_TOPIC, + VoiceOutput(text="Finite answer."), + participant_id="alice", + source="finite-agent", + ) + await runtime.publish( + VOICE_OUTPUT_TOPIC, + VoiceOutput(text="Streamed ", response_id="turn-1", final=False), + participant_id="alice", + source="stream-agent", + ) + await runtime.publish( + VOICE_OUTPUT_TOPIC, + VoiceOutput(text="answer.", response_id="turn-1"), + participant_id="alice", + source="stream-agent", + ) + await asyncio.wait_for(session.wait_for(2), 1.0) + await nemo_relay.subscribers.flush_async() + finally: + nemo_relay.subscribers.deregister(subscriber) + + starts = [ + event.to_dict() + for event in events + if event.name == "voice.response" + and event.to_dict().get("scope_category") == "start" + ] + assert len(starts) == 2 + by_text = {event["data"]["text"]: event for event in starts} + assert by_text["Finite answer."]["data"] | { + "streaming": False, + "fragment_count": 1, + } == by_text["Finite answer."]["data"] + assert by_text["Streamed answer."]["data"] | { + "streaming": True, + "fragment_count": 2, + } == by_text["Streamed answer."]["data"] + assert by_text["Streamed answer."]["metadata"] | { + "participant_id": "alice", + "source": "stream-agent", + "response_id": "turn-1", + "status": "completed", + } == by_text["Streamed answer."]["metadata"] + assert by_text["Finite answer."]["metadata"]["correlation_id"] + assert by_text["Streamed answer."]["metadata"]["correlation_id"] + assert "publish:voice.output" not in {event.name for event in events} + assert "agent:voice" not in {event.name for event in events} + + +async def test_voice_agent_routes_typed_input() -> None: + session = _Session() + runtime = AgentRuntime() + voice = VoiceAgent( + session, # type: ignore[arg-type] + query_topic=QUERY_TOPIC, + text_ignore_topics={"control"}, + text_transform=str.upper, + ) + runtime.register("voice", voice) + + async with _running_voice(runtime, voice, session): + await session.transport.endpoint.data_callback( + DataMessage( + participant_id="alice", + topic="control", + pts_us=1, + data=b"ignored", + ) + ) + await session.transport.endpoint.data_callback( + DataMessage( + participant_id="alice", + topic="", + pts_us=2, + data=b"hello", + ) + ) + + assert session.transport.target_participant == "alice" + assert session.queries == [("alice", "HELLO", 2)] + + +async def test_voice_agent_drops_inactive_ignored_and_empty_transformed_text() -> None: + session = _Session() + runtime = AgentRuntime() + voice = VoiceAgent( + session, # type: ignore[arg-type] + query_topic=QUERY_TOPIC, + text_transform=lambda _text: " ", + ) + runtime.register("voice", voice) + message = DataMessage( + participant_id="alice", + topic="request", + pts_us=3, + data=b"hello", + ) + + await voice._on_data(message) # noqa: SLF001 + async with _running_voice(runtime, voice, session): + await session.transport.endpoint.data_callback( + DataMessage( + participant_id="alice", + topic=session.text_topic, + pts_us=4, + data=b"ignore output loop", + ) + ) + await session.transport.endpoint.data_callback(message) + + assert session.queries == [] + + +async def test_voice_agent_unregisters_typed_input_callback() -> None: + session = _Session() + runtime = AgentRuntime() + voice = VoiceAgent(session, query_topic=QUERY_TOPIC) # type: ignore[arg-type] + runtime.register("voice", voice) + + async with _running_voice(runtime, voice, session): + assert session.transport.endpoint.data_callback is not None + + assert session.transport.endpoint.data_callback is None + + +async def test_incremental_responses_are_isolated_by_participant_and_source() -> None: + session = _Session() + runtime = AgentRuntime() + voice = VoiceAgent( # type: ignore[arg-type] + session, + query_topic=QUERY_TOPIC, + text_input=False, + ) + runtime.register("voice", voice) + + async with _running_voice(runtime, voice, session): + for source, text in (("observer-a", "alpha "), ("observer-b", "beta ")): + await runtime.publish( + VOICE_OUTPUT_TOPIC, + VoiceOutput(text=text, response_id="shared", final=False), + participant_id="alice", + source=source, + ) + for source, text in (("observer-b", "two"), ("observer-a", "one")): + await runtime.publish( + VOICE_OUTPUT_TOPIC, + VoiceOutput(text=text, response_id="shared"), + participant_id="alice", + source=source, + ) + await asyncio.wait_for(session.wait_for(2), 1.0) + + assert sorted(text for _pid, text, _interrupt, _pts in session.responses) == [ + "alpha one", + "beta two", + ] + + +async def test_cancelled_response_stream_releases_blocked_publishers() -> None: + session = _Session() + runtime = AgentRuntime() + agent = VoiceAgent( # type: ignore[arg-type] + session, + query_topic=QUERY_TOPIC, + response_capacity=1, + text_input=False, + ) + runtime.register("voice", agent) + + async with _running_voice(runtime, agent, session): + await runtime.publish( + VOICE_OUTPUT_TOPIC, + VoiceOutput(text="one", response_id="turn", final=False), + participant_id="alice", + source="observer", + ) + await asyncio.sleep(0) + assert session.response_tasks + session.response_tasks[0].cancel() + await asyncio.gather(*session.response_tasks, return_exceptions=True) + await asyncio.wait_for( + runtime.publish( + VOICE_OUTPUT_TOPIC, + VoiceOutput(text="two", response_id="turn", final=False), + participant_id="alice", + source="observer", + ), + 1.0, + ) + await runtime.publish( + VOICE_OUTPUT_TOPIC, + VoiceOutput(response_id="turn"), + participant_id="alice", + source="observer", + ) + assert len(session.response_tasks) == 1 + + assert agent._closed_streams # noqa: SLF001 + assert agent._streams == {} # noqa: SLF001 + + +async def test_voice_output_preserves_originating_query_timestamp() -> None: + session = _Session() + runtime = AgentRuntime() + voice = VoiceAgent( # type: ignore[arg-type] + session, + query_topic=QUERY_TOPIC, + text_input=False, + ) + runtime.register("voice", voice) + + async with _running_voice(runtime, voice, session): + await runtime.publish( + VOICE_OUTPUT_TOPIC, + VoiceOutput(text="answer", timestamp_us=123), + participant_id="alice", + source="observer", + ) + + assert session.responses == [("alice", "answer", False, 123)] + + +async def test_blocked_stream_does_not_block_unrelated_output() -> None: + class HeldSession(_Session): + def __init__(self) -> None: + super().__init__() + self.held: list[AsyncIterator[str]] = [] + + async def enqueue_response( + self, + participant_id: str, + response: str | AsyncIterator[str], + *, + interrupt: bool = False, + pts_us: int | None = None, + ) -> None: + if isinstance(response, str): + await super().enqueue_response( + participant_id, + response, + interrupt=interrupt, + pts_us=pts_us, + ) + else: + self.held.append(response) + + session = HeldSession() + runtime = AgentRuntime() + voice = VoiceAgent( + session, # type: ignore[arg-type] + query_topic=QUERY_TOPIC, + response_capacity=1, + text_input=False, + ) + runtime.register("voice", voice) + + async with _running_voice(runtime, voice, session): + await runtime.publish( + VOICE_OUTPUT_TOPIC, + VoiceOutput(text="one", response_id="held", final=False), + participant_id="alice", + source="observer", + ) + blocked = asyncio.create_task( + runtime.publish( + VOICE_OUTPUT_TOPIC, + VoiceOutput(text="two", response_id="held", final=False), + participant_id="alice", + source="observer", + ) + ) + await asyncio.sleep(0) + assert not blocked.done() + await asyncio.wait_for( + runtime.publish( + VOICE_OUTPUT_TOPIC, + VoiceOutput(text="independent"), + participant_id="bob", + source="other", + ), + 1.0, + ) + await session.held[0].aclose() # type: ignore[attr-defined] + await asyncio.wait_for(blocked, 1.0) + + assert [(pid, text, interrupt) for pid, text, interrupt, _ in session.responses] == [ + ("bob", "independent", False) + ] + + +async def test_open_response_streams_are_bounded(monkeypatch) -> None: + class HeldSession(_Session): + def __init__(self) -> None: + super().__init__() + self.held: list[AsyncIterator[str]] = [] + + async def enqueue_response(self, _participant_id, response, **_kwargs) -> None: + self.held.append(response) + + monkeypatch.setattr(voice_runtime_module, "_OPEN_STREAM_CAPACITY", 1) + session = HeldSession() + runtime = AgentRuntime() + voice = VoiceAgent(session, query_topic=QUERY_TOPIC, text_input=False) # type: ignore[arg-type] + runtime.register("voice", voice) + + async with _running_voice(runtime, voice, session): + for response_id in ("first", "second"): + await runtime.publish( + VOICE_OUTPUT_TOPIC, + VoiceOutput(text=response_id, response_id=response_id, final=False), + participant_id="alice", + source="observer", + ) + + assert len(voice._streams) == 1 # noqa: SLF001 + assert ("alice", "observer", "second") in voice._streams # noqa: SLF001 + assert session.held[0].closed.is_set() # type: ignore[attr-defined] + + +async def test_failed_stream_enqueue_does_not_register_response() -> None: + class FailingSession(_Session): + async def enqueue_response(self, *_args, **_kwargs) -> None: + raise RuntimeError("session stopped") + + session = FailingSession() + runtime = AgentRuntime() + voice = VoiceAgent(session, query_topic=QUERY_TOPIC, text_input=False) # type: ignore[arg-type] + runtime.register("voice", voice) + + async with _running_voice(runtime, voice, session): + with pytest.raises(ExceptionGroup, match="event publication"): + await runtime.publish( + VOICE_OUTPUT_TOPIC, + VoiceOutput(text="first", response_id="turn", final=False), + participant_id="alice", + source="observer", + ) + + assert voice._streams == {} # noqa: SLF001 + assert voice._response_traces == {} # noqa: SLF001 + + +async def test_midstream_interrupt_is_rejected() -> None: + session = _Session() + runtime = AgentRuntime() + voice = VoiceAgent(session, query_topic=QUERY_TOPIC, text_input=False) # type: ignore[arg-type] + runtime.register("voice", voice) + + async with _running_voice(runtime, voice, session): + await runtime.publish( + VOICE_OUTPUT_TOPIC, + VoiceOutput(text="first", response_id="turn", final=False), + participant_id="alice", + source="observer", + ) + with pytest.raises(ExceptionGroup, match="event publication") as raised: + await runtime.publish( + VOICE_OUTPUT_TOPIC, + VoiceOutput( + text="second", + response_id="turn", + final=False, + interrupt=True, + ), + participant_id="alice", + source="observer", + ) + + assert "only the first chunk" in str(raised.value.exceptions[0]) + + +async def test_unknown_empty_stream_terminator_is_rejected() -> None: + session = _Session() + runtime = AgentRuntime() + voice = VoiceAgent( # type: ignore[arg-type] + session, + query_topic=QUERY_TOPIC, + text_input=False, + ) + runtime.register("voice", voice) + + async with _running_voice(runtime, voice, session): + with pytest.raises(ExceptionGroup, match="event publication") as raised: + await runtime.publish( + VOICE_OUTPUT_TOPIC, + VoiceOutput(response_id="missing"), + participant_id="alice", + source="observer", + ) + + assert len(raised.value.exceptions) == 1 + assert isinstance(raised.value.exceptions[0], ValueError) + assert "no open response" in str(raised.value.exceptions[0]) + + +def test_voice_output_rejects_ambiguous_empty_messages() -> None: + with pytest.raises(ValidationError, match="response_id"): + VoiceOutput(final=False) + with pytest.raises(ValidationError, match="contain text"): + VoiceOutput() + with pytest.raises(ValidationError, match="cannot interrupt"): + VoiceOutput(response_id="turn", interrupt=True) diff --git a/tests/test_voice_session.py b/tests/test_voice_session.py index 951866bd..f566ee6a 100644 --- a/tests/test_voice_session.py +++ b/tests/test_voice_session.py @@ -7,8 +7,7 @@ import asyncio import pytest -from xr_ai_hub import DataMessage -from xr_ai_voice import TextMessageInput, VadConfig, VoiceSession +from xr_ai_voice import VadConfig, VoiceSession from xr_ai_voice import _session as session_module from xr_ai_voicegate import VoiceGateConfig @@ -48,21 +47,19 @@ def shutdown(self) -> None: self.shutdown_called = True -class _Session: +class _HandlerProcessor: def __init__(self) -> None: - self.transport = _Transport() - self.queries: list[tuple[str, str, bool, int | None]] = [] - self.is_running = True + self.responses: list[tuple[str, object, bool, int | None]] = [] - async def enqueue_query( + async def enqueue_response( self, participant_id: str, - text: str, + response: object, *, - fresh_match: bool = False, + interrupt: bool = False, pts_us: int | None = None, ) -> None: - self.queries.append((participant_id, text, fresh_match, pts_us)) + self.responses.append((participant_id, response, interrupt, pts_us)) class _Service: @@ -76,46 +73,25 @@ async def close(self) -> None: self.closed += 1 -async def test_data_query_adapter_routes_text_and_ignores_control_topics() -> None: - session = _Session() - TextMessageInput( - session=session, # type: ignore[arg-type] - ignore_topics={"control"}, - transform=str.upper, - fresh_match=True, +async def test_voice_session_queues_external_responses_through_active_processor() -> None: + service = _Service() + session = VoiceSession( + stt=service, # type: ignore[arg-type] + tts=service, # type: ignore[arg-type] + vad=VadConfig(), + voice_gate=VoiceGateConfig(), + ) + processor = _HandlerProcessor() + session._io_processor = processor # type: ignore[assignment] # noqa: SLF001 + + await session.enqueue_response( + "alice", + "Careful.", + interrupt=True, + pts_us=12, ) - await session.transport.endpoint.callback(DataMessage( - participant_id="alice", - topic="control", - pts_us=1, - data=b"ignored", - )) - await session.transport.endpoint.callback(DataMessage( - participant_id="alice", - topic="", - pts_us=2, - data=b"hello", - )) - - assert session.transport.target_participant == "alice" - assert session.queries == [("alice", "HELLO", True, 2)] - - -async def test_data_query_adapter_drops_text_while_session_is_stopped() -> None: - session = _Session() - session.is_running = False - TextMessageInput(session=session) # type: ignore[arg-type] - - await session.transport.endpoint.callback(DataMessage( - participant_id="alice", - topic="", - pts_us=2, - data=b"hello", - )) - - assert session.transport.target_participant == "" - assert session.queries == [] + assert processor.responses == [("alice", "Careful.", True, 12)] async def test_voice_session_owns_readiness_ready_file_and_cleanup( @@ -157,12 +133,12 @@ async def run(self, _worker) -> None: transport=transport, # type: ignore[arg-type] ) - async def handler(_query) -> str: - return "unused" + async def input_sink(_query) -> None: + pass async with session: assert not ready_file.exists() - run_task = asyncio.create_task(session.run(handler)) + run_task = asyncio.create_task(session.run(input_sink)) await runner_started.wait() await asyncio.sleep(0) assert not ready_file.exists() @@ -207,6 +183,8 @@ def make_transport() -> _Transport: ) assert transports == [] + with pytest.raises(RuntimeError, match="not ready"): + _ = session.endpoint async with session: assert session.transport is transports[0] @@ -236,6 +214,7 @@ async def health(self) -> bool: with pytest.raises(RuntimeError, match="unavailable"): async with session: pass + await session.close() assert transports == [] assert stt.closed == 1