refactor(voice): unify runtime input and output

AGENTS.md

@@ -192,14 +192,14 @@ NAT's built-in LangChain-backed agent types; applications install
 The public **native voice runtime** lives in `xr-ai-voice` (it depends on
 pipecat internally):
 
-- **Voice session** — `VoiceSession.run(handler)` privately assembles
-  `input → VadStt → VoiceGate → handler → StreamingTts → output`, owns model
-  readiness and ready-file semantics, installs signal handlers, and closes the
-  transport and model clients. It touches the ready file only after the input
-  transport has entered its hub IPC receive loop.
-- **Native handler** — `xr_ai_nat.adapters.as_voice_handler` maps a typed NAT
-  function onto `VoiceSession`; `TextMessageInput` routes participant text
-  through the same turn path as speech.
+- **Voice agent** — `VoiceAgent` owns `VoiceSession`, publishes accepted speech
+  and typed text as its `UserQuery` schema on a sample-named topic, publishes
+  participant and interruption events on sample-named topics, and subscribes
+  to `voice.output`. Application agents subscribe to lifecycle events and own
+  their cleanup; the application entry point only composes them. Runtime
+  publication provides acknowledged delivery through the same output path. The
+  private session owns model readiness, ready-file semantics, signals, the
+  media pipeline, and cleanup.
 - **Wake word / speech gate** — `xr-ai-voicegate` (the `VoiceGate` state
   machine) wired in as `VoiceGateProcessor`; per-sample config in
   `yaml/voice_gate.yaml` (`magic_phrases: ["hey agent"]`, or `[]` for
@@ -210,8 +210,8 @@ pipecat internally):
 `run_voice_pipeline(worker, transport, on_ready=ready_file.touch)` so they use
 the same IPC-start readiness boundary.
 
-A native voice sample adapts its NAT function to `VoiceSession`; wake-word
-behavior comes from config alone.
+A native voice sample registers `VoiceAgent` and application agents on the same
+runtime; wake-word behavior comes from config alone.
 
 ### Scope decision and named follow-ups
 

DEPENDENCIES.md

@@ -38,13 +38,15 @@ CI matrices:
 
 ```
 xr-ai-agent-runtime  (agent-sdk/xr-ai-agent-runtime/)
+    └── nemo-relay >=0.7.2,<0.8
     └── pydantic >=2.10
     └── xr-ai-tools [editable: ../xr-ai-tools]
-    In-process runtime for agent resource lifetimes, runtime-owned background
-    tasks, and typed ``publish`` fan-out. Agents expose ordinary ``Tool`` and
-    ``AsyncTool`` instances from ``xr-ai-tools`` and own their synchronization.
-    Tool execution, model clients, tool loops, planning, memory, and raw media
-    transport are not runtime responsibilities.
+    In-process typed ``publish`` fan-out for agents that expose ordinary
+    ``Tool`` and ``AsyncTool`` instances from ``xr-ai-tools``. Agents own their
+    resources, background tasks, lifecycle, and synchronization. Tool
+    execution, model clients, tool loops, planning, memory, and raw media
+    transport are not runtime responsibilities. Relay scopes record runtime
+    publications and receiving-agent subscription callbacks.
 
 xr-ai-hub-client  (agent-sdk/xr-ai-hub-client/)
     └── pyzmq >=27.0
@@ -76,6 +78,9 @@ xr-ai-pipecat  (agent-sdk/xr-ai-pipecat/)
     Not a dep of xr-ai-hub-client itself — import only in workers that use Pipecat.
 
 xr-ai-voice  (agent-sdk/xr-ai-voice/)
+    └── nemo-relay >=0.7.2,<0.8
+    └── pydantic >=2.10
+    └── xr-ai-agent-runtime [editable: ../xr-ai-agent-runtime]
     └── xr-ai-hub-client [editable: ../xr-ai-hub-client]
     └── xr-ai-logging   [editable: ../../utils/xr-ai-logging]
     └── xr-ai-models    [editable: ../xr-ai-models]
@@ -85,13 +90,14 @@ xr-ai-voice  (agent-sdk/xr-ai-voice/)
     └── nltk !=3.10.1       (3.10.1 rejects deps in in-project venvs)
     └── numpy >=1.24
     └── scipy >=1.11
-    Native voice runtime used by simple-vlm-example. Exposes the
-    ``VoiceSession`` public API plus the ``VoiceHandler`` / ``VoiceQuery`` /
-    ``VoiceResponse`` / ``VoiceTurn`` handler surface, ``HubVoiceTransport``,
-    ``VadConfig``, and ``TextMessageInput``; Pipecat, audio framing, and
-    pipeline processors are implementation details. Service health gates
-    transport construction, while ``VoiceSession.run`` touches its ready file
-    only after the input transport starts its hub IPC receive loop. The
+    Native voice runtime used by simple-vlm-example. Exposes ``VoiceAgent``,
+    its ``UserQuery`` / ``VoiceOutput`` / participant-lifecycle schemas,
+    ``VoiceSession``, ``HubVoiceTransport``, and
+    ``VadConfig``. Voice lifecycle events enter application-named topics so
+    application agents own their cleanup. Pipecat, audio framing, and pipeline
+    processors are implementation details. Service health gates transport
+    construction, while the session touches its ready file only after the input
+    transport starts its hub IPC receive loop. The
     readiness contract is split across the ``_readiness`` / ``_session``
     modules. Not a dep of xr-ai-hub-client itself — import only in workers that
     opt into the voice runtime.
@@ -566,16 +572,18 @@ the latest video frame via streaming VLM and replies with both
 | Sub-project | Package | Internal deps | External deps |
 |---|---|---|---|
 | Orchestrator | `simple-vlm-example` | `xr-ai-launcher` | — |
-| Worker | `simple-vlm-example-worker` | `xr-ai-hub-client [editable]`, `xr-ai-logging [editable]`, `xr-ai-models [editable]`, `xr-ai-tools[live-vision] [editable]`, `xr-ai-voice [editable]`, `xr-ai-voicegate [editable]` | loguru >=0.7, pyyaml >=6.0 (`xr-ai-voice` pulls in VAD, pipecat-ai, numpy, and scipy; `xr-ai-tools[live-vision]` pulls in numpy and Pillow) |
-
-The packaged worker constructs a transport-independent `StreamingVisionTool`
-and adapts its typed async chunks to `VoiceSession` locally. The tool owns
-current-frame acquisition through `xr-ai-hub-client`, has no voice dependency,
-and uses NeMo Relay's managed streaming LLM path. Camera bytes are redacted from
-Relay telemetry while the provider receives the original frame.
-`VoiceSession` owns readiness, hub transport, signals, the
-private Pipecat pipeline, and cleanup; `TextMessageInput` routes `"ping"` and
-ad-hoc text through the same participant-aware path as speech. Voice-gate
+| Worker | `simple-vlm-example-worker` | `xr-ai-agent-runtime [editable]`, `xr-ai-hub-client [editable]`, `xr-ai-logging [editable]`, `xr-ai-models [editable]`, `xr-ai-tools[live-vision] [editable]`, `xr-ai-voice [editable]`, `xr-ai-voicegate [editable]` | nemo-relay >=0.7.2,<0.8, loguru >=0.7, pyyaml >=6.0 (`xr-ai-voice` pulls in VAD, pipecat-ai, numpy, and scipy; `xr-ai-tools[live-vision]` pulls in numpy and Pillow) |
+
+The packaged worker runs a transport-independent `StreamingVisionTool` inside
+`SimpleVlmAgent` and publishes its typed async chunks to `VoiceAgent`. The tool
+owns current-frame acquisition through `xr-ai-hub-client`, has no voice
+dependency, and uses NeMo Relay's managed streaming LLM path. Camera bytes are
+redacted from Relay telemetry while the provider receives the original frame.
+`VoiceAgent` owns `VoiceSession`, readiness, hub transport, signals, and the
+private Pipecat pipeline; it routes `"ping"` and ad-hoc text through the same
+sample-named `UserQuery` topic as speech and publishes lifecycle events on
+sample-named topics. `SimpleVlmAgent` handles cancellation and frame cleanup
+inside its own subscriber methods. Voice-gate
 behavior (magic phrases, follow-up grace, listening chime, stop acknowledgement),
 VAD/STT, and sentence-batched TTS remain provided by the shared voice runtime.
 The sample has no direct `xr-ai-pipecat` or MCP dependency and selects no legacy

README.md

@@ -117,7 +117,7 @@ frames are dropped if it is closed.
 | Hub service | `services/xr-media-hub/` | XR-Media-Hub + LiveKit internal transport |
 | Launcher | `utils/xr-ai-launcher/` | stdlib-only process manager used by samples |
 | Logging | `utils/xr-ai-logging/` | shared loguru sink + stdlib bridge for every process |
-| Agent runtime | `agent-sdk/xr-ai-agent-runtime/` | Agent resource lifetimes, background tasks, existing native tools, and pub/sub |
+| Agent runtime | `agent-sdk/xr-ai-agent-runtime/` | Agents with existing native tools and typed pub/sub routing |
 | Agent tools | `agent-sdk/xr-ai-tools/` | Toolkit-independent Relay-managed native tools |
 | Legacy NAT | `agent-sdk/xr-ai-nat/` | NeMo Agent Toolkit compatibility during migration |
 | Reusable services | `services/` | Model-serving and typed capability processes |
@@ -197,9 +197,9 @@ channel, or send the literal text `"ping"` — all routes go through the
 same VLM pipeline against the latest video frame.  Replies arrive as
 streaming Piper TTS audio plus a `vlm.response` text message.
 
-The packaged worker adapts the transport-independent `StreamingVisionTool` to
-`xr-ai-voice`'s `VoiceSession`; Pipecat remains private to that runtime and no
-MCP client is involved. See the
+The packaged worker runs `StreamingVisionTool` inside `SimpleVlmAgent` and
+publishes its chunks to `VoiceAgent`; Pipecat remains private to the voice
+runtime and no MCP client is involved. See the
 [sample README](agent-samples/simple-vlm-example/README.md) for the worker
 layout and configuration boundaries.
 

agent-samples/simple-vlm-example/README.md

@@ -13,18 +13,23 @@ topic. Sending the literal text `ping` uses the configured default question,
 The worker is a package under `worker/simple_vlm_example_worker/`:
 
 - `__main__.py` parses launcher arguments.
+- `agent.py` owns participant-scoped vision turns and cancellation.
 - `config.py` resolves worker, model-profile, voice-gate, and prompt settings.
 - `app.py` composes the native runtime.
 - `prompts/system.txt` owns the VLM system prompt.
 
-`VoiceSession` owns STT/TTS/VLM readiness, the hub voice transport, voice-gate
-processing, streaming TTS, signals, and cleanup. The application constructs a
-transport-independent `StreamingVisionTool` and adapts its async chunks to the
-voice handler locally. The tool has no voice dependency and sends its provider
-stream through Relay's managed LLM path. The camera frame is redacted from
-Relay telemetry. Typed text uses the same
-participant-aware turn path as speech. Participant leave events release cached
-live-frame state, and a newer turn cancels and interrupts a superseded response.
+`VoiceAgent` owns `VoiceSession`, which provides STT/TTS/VLM readiness, the hub
+voice transport, voice-gate processing, streaming TTS, signals, and cleanup.
+It publishes accepted speech and typed text as `UserQuery` on this sample's
+topic. `SimpleVlmAgent` subscribes to that topic, owns participant-scoped
+streaming and cancellation around the transport-independent
+`StreamingVisionTool`, and publishes chunks to `voice.output`. The tool has no
+voice dependency and sends its provider stream through Relay's managed LLM
+path. The camera frame is redacted from Relay telemetry. `VoiceAgent` publishes
+participant departure and interruption on sample-named topics;
+`SimpleVlmAgent` subscribes and releases its own cached frames and tasks. A
+newer turn cancels and interrupts a superseded response. `app.py` only composes
+the two agents and their dependencies.
 
 No MCP client or MCP tool invocation is part of this sample.
 
@@ -50,6 +55,28 @@ The worker and orchestrator consume the deployment profile selected by
 The same profile owns model behavior, endpoints, credentials, readiness, and
 launcher process ownership.
 
+## Relay visibility
+
+The worker writes a compact Relay lifecycle stream to `relay-events.jsonl`
+beside `worker.log` in the per-run log directory printed at startup. The JSONL
+records include runtime publications, receiving-agent callbacks, the complete
+`simple-vlm.turn` lifetime, and nested vision tool and VLM calls. Per-token
+`llm.chunk` marks, incremental `voice.output` fragments, and empty stream
+terminators are omitted. `VoiceAgent` emits one `voice.response` scope containing
+the complete text and timing for both non-streamed and aggregated incremental
+output. Each real STT request is a `voice.stt` scope with a transcript result
+mark, and each sentence synthesis is a `voice.tts` scope. Raw audio is summarized
+by byte count, duration, and sample rate; TTS records synthesis rather than
+client playback. The completed LLM and turn records remain available alongside
+them. No telemetry server or network exporter is required. Live camera bytes are
+replaced with `<redacted:live-camera-frame>`; prompts, questions, responses,
+participant IDs, and correlation metadata remain visible and may contain
+sensitive data.
+
+```bash
+tail -F /tmp/log_simple-vlm-example_*/relay-events.jsonl
+```
+
 Voice-gate behavior remains in `yaml/voice_gate.yaml`. Worker timing, frame
 freshness, the default `ping` question, and optional prompt overrides are in
 `yaml/simple_vlm_example_worker.yaml`; the default prompt ships inside the

agent-samples/simple-vlm-example/worker/pyproject.toml

@@ -10,6 +10,8 @@ name = "simple-vlm-example-worker"
 version = "0.1.0"
 requires-python = ">=3.11,<3.13"
 dependencies = [
+    "nemo-relay>=0.7.2,<0.8",
+    "xr-ai-agent-runtime",
     "xr-ai-hub-client",
     "xr-ai-logging",
     "xr-ai-models",
@@ -21,6 +23,7 @@ dependencies = [
 ]
 
 [tool.uv.sources]
+xr-ai-agent-runtime = { path = "../../../agent-sdk/xr-ai-agent-runtime", editable = true }
 xr-ai-hub-client = { path = "../../../agent-sdk/xr-ai-hub-client", editable = true }
 xr-ai-logging   = { path = "../../../utils/xr-ai-logging", editable = true }
 xr-ai-models    = { path = "../../../agent-sdk/xr-ai-models", editable = true }