refactor(tools): move native tools out of NAT

AGENTS.md

@@ -12,12 +12,13 @@ historical decisions in `docs/changelog.md`.
 
 ```
 client-samples/     # Platform clients (Android, iOS/visionOS, Web)
-agent-sdk/          # Five packages:
+agent-sdk/          # Six packages:
                     #   xr-ai-hub-client   — IPC client library (pyzmq + msgpack only)
                     #   xr-ai-models       — LLM/VLM/STT/TTS service protocols + OpenAI-compat clients
                     #   xr-ai-pipecat      — optional Pipecat transport bridge (heavier deps)
                     #   xr-ai-voice        — voice runtime (VoiceSession); introduced alongside xr-ai-pipecat
-                    #   xr-ai-nat          — native Relay-managed tools; legacy NAT compatibility during migration
+                    #   xr-ai-tools        — toolkit-independent Relay-managed tools
+                    #   xr-ai-nat          — legacy NeMo Agent Toolkit compatibility during migration
 utils/              # Shared infra: launcher, logging, vad, vllm, voicegate
 services/           # XR hub, CloudXR, model-serving, and typed capability services
 agent-mcp-servers/  # MCP adapters: oxr, render, transcript, vec, video, vlm
@@ -58,12 +59,12 @@ deps/               # Gitignored downloaded binaries (e.g. LOVR AppImage)
   `anthropic`, no `litellm`); all in-tree backends speak
   OpenAI-compatible HTTP.
 - **Workers never import from `xr_media_hub` or `xr_ai_launcher`.** Use the
-  public `xr_ai_hub`, `xr_ai_models`, `xr_ai_nat`, and `xr_ai_voice` SDK
-  surfaces plus task-specific libraries (numpy, torch, …).
+  public `xr_ai_hub`, `xr_ai_models`, `xr_ai_tools`, `xr_ai_nat`, and
+  `xr_ai_voice` SDK surfaces plus task-specific libraries (numpy, torch, …).
 - **Agentic functions are native and in-process.** New and migrated tools live
-  in `xr-ai-nat`; every tool and tool-driven agent lifecycle passes through
-  NeMo Relay, and all model I/O remains in `xr-ai-models`. Its legacy extras
-  retain NeMo Agent Toolkit compatibility only while existing function groups
+  in `xr-ai-tools`; every tool execution passes through NeMo Relay, and all
+  model I/O remains in `xr-ai-models`. `xr-ai-nat` retains
+  NeMo Agent Toolkit compatibility only while existing function groups
   migrate. Existing MCP servers remain compatibility surfaces while their
   capabilities migrate.
 - **RAG is a native typed capability.** `rag-service` owns document chunking,
@@ -172,9 +173,8 @@ itself. `XRTrackingFunctionsConfig` exposes the current user frame through
 the typed OpenXR service without routing native agents through MCP.
 `VideoMemoryFunctionsConfig` exposes recorded-video discovery, queries, and
 frame extraction through a typed service while keeping MCP optional; callers
-obtain current frames through the hub client. `StreamingVisionConfig` composes
-raw frame acquisition with VLM streaming behind one native function for voice
-workflows. `ModelsLLMConfig` adapts the `xr-ai-models` service boundary to
+obtain current frames through the hub client. `StreamingVisionTool` in `xr-ai-tools` composes raw frame
+acquisition with VLM streaming and stays independent of voice. `ModelsLLMConfig` adapts the `xr-ai-models` service boundary to
 NAT's built-in LangChain-backed agent types; applications install
 `xr-ai-nat[agents]` rather than calling LangChain model clients directly.
 

DEPENDENCIES.md

@@ -119,23 +119,25 @@ xr-ai-models  (agent-sdk/xr-ai-models/)
     own httpx wrappers. Profiles may separate adapter, endpoint, and deployment
     metadata while the existing flat YAML schema remains valid.
 
-xr-ai-nat  (agent-sdk/xr-ai-nat/)
+xr-ai-tools  (agent-sdk/xr-ai-tools/)
     └── nemo-relay >=0.7.2,<0.8
     └── pydantic >=2.10
     └── [relay] xr-ai-models [editable: ../xr-ai-models]
     └── [live-vision] numpy >=1.24, Pillow >=10.0, xr-ai-hub-client [editable: ../xr-ai-hub-client], xr-ai-models [editable: ../xr-ai-models]
-    └── [agents] nvidia-nat-core ==1.8.0, nvidia-nat-langchain ==1.8.0, xr-ai-models [editable: ../xr-ai-models]
-    └── [mcp] nvidia-nat-core ==1.8.0, fastmcp >=3.4,<4
-    └── [services] nvidia-nat-core ==1.8.0, msgpack >=1.0, pyzmq >=27.0
-    └── [vision] nvidia-nat-core ==1.8.0, httpx >=0.27, numpy >=1.24, Pillow >=10.0, xr-ai-hub-client [editable: ../xr-ai-hub-client], xr-ai-models [editable: ../xr-ai-models]
-    └── [voice] nvidia-nat-core ==1.8.0, xr-ai-voice [editable: ../xr-ai-voice]
-    The base package is the toolkit-independent native tools layer: Pydantic
-    request and response models, Relay-managed execution, the generic
-    ``AgentRunner`` protocol, and a bounded default tool loop over
-    `xr-ai-models`. The ``[relay]`` and
-    ``[live-vision]`` extras add model-backed tools without selecting NeMo Agent
-    Toolkit. The existing function groups remain behind legacy compatibility
-    extras while they migrate. The ``xr_spatial_math`` function group accepts
+    Toolkit-independent native tools: Pydantic request and response models,
+    Relay-managed finite and async execution, model tool-call workflow helpers,
+    and participant-scoped live vision.
+
+xr-ai-nat  (agent-sdk/xr-ai-nat/)
+    └── nvidia-nat-core ==1.8.0
+    └── pydantic >=2.10
+    └── [agents] nvidia-nat-langchain ==1.8.0, xr-ai-models [editable: ../xr-ai-models]
+    └── [mcp] fastmcp >=3.4,<4
+    └── [services] msgpack >=1.0, pyzmq >=27.0
+    └── [vision] httpx >=0.27, numpy >=1.24, Pillow >=10.0, xr-ai-hub-client [editable: ../xr-ai-hub-client], xr-ai-models [editable: ../xr-ai-models]
+    └── [voice] xr-ai-voice [editable: ../xr-ai-voice]
+    Typed, in-process NeMo Agent Toolkit functions retained while their
+    concrete capabilities migrate. The ``xr_spatial_math`` function group accepts
     explicit coordinate frames and
     performs deterministic spatial calculations without OpenXR, model, or MCP
     dependencies. ``xr_text_memory`` owns persistent per-source JSONL text
@@ -153,9 +155,8 @@ xr-ai-nat  (agent-sdk/xr-ai-nat/)
     ``xr_vision_tools`` exposes ``look_at_current_frame`` and
     ``look_at_past_frame`` over the always-on live-frame source, acquiring the
     frame itself and calling an injected xr-ai-models VLM; recorded lookups
-    resolve through the ``xr_video_memory`` group. A separate
-    ``xr_streaming_vision`` function composes current-frame acquisition with
-    complete or streaming VLM invocation. ``xr_tracking`` calls
+    resolve through the ``xr_video_memory`` group. The replaced NAT
+    streaming function has moved to ``xr-ai-tools``. ``xr_tracking`` calls
     the typed OpenXR service and returns a complete user coordinate frame.
     ``xr_video_memory`` calls the typed video-memory service for recorded-video
     discovery, queries, and frame extraction. ``xr_rag`` calls the typed RAG
@@ -334,7 +335,8 @@ vec-mcp-server  (agent-mcp-servers/vec-mcp/)
 xr-ai-tests  (tests/)
     └── xr-ai-hub-client             [editable: ../agent-sdk/xr-ai-hub-client]
     └── xr-ai-models            [editable: ../agent-sdk/xr-ai-models]
-    └── xr-ai-nat[agents,relay,services,vision] [editable: ../agent-sdk/xr-ai-nat]
+    └── xr-ai-nat[agents,services,vision] [editable: ../agent-sdk/xr-ai-nat]
+    └── xr-ai-tools[live-vision] [editable: ../agent-sdk/xr-ai-tools]
     └── xr-rag-service [editable: ../services/rag-service]
     └── xr-ai-pipecat           [editable: ../agent-sdk/xr-ai-pipecat]
     └── xr-ai-voice             [editable: ../agent-sdk/xr-ai-voice]
@@ -554,14 +556,13 @@ the latest video frame via streaming VLM and replies with both
 | Sub-project | Package | Internal deps | External deps |
 |---|---|---|---|
 | Orchestrator | `simple-vlm-example` | `xr-ai-launcher` | — |
-| Worker | `simple-vlm-example-worker` | `xr-ai-hub-client [editable]`, `xr-ai-logging [editable]`, `xr-ai-models [editable]`, `xr-ai-nat[relay,live-vision] [editable]`, `xr-ai-voice [editable]`, `xr-ai-voicegate [editable]` | loguru >=0.7, pyyaml >=6.0 (`xr-ai-voice` pulls in VAD, pipecat-ai, numpy, and scipy; `xr-ai-nat[live-vision]` pulls in numpy and Pillow) |
-
-The packaged worker constructs the finite `LiveVisionTool` used by agentic
-flows, then maps its separate direct-voice `LiveVisionResponder` to
-`VoiceSession`. They share current-frame acquisition through
-`xr-ai-hub-client`; only the direct voice response uses NeMo Relay's managed
-streaming LLM path under an Agent scope. Camera bytes are redacted from Relay
-telemetry while the provider receives the original frame.
+| Worker | `simple-vlm-example-worker` | `xr-ai-hub-client [editable]`, `xr-ai-logging [editable]`, `xr-ai-models [editable]`, `xr-ai-tools[live-vision] [editable]`, `xr-ai-voice [editable]`, `xr-ai-voicegate [editable]` | loguru >=0.7, pyyaml >=6.0 (`xr-ai-voice` pulls in VAD, pipecat-ai, numpy, and scipy; `xr-ai-tools[live-vision]` pulls in numpy and Pillow) |
+
+The packaged worker constructs a transport-independent `StreamingVisionTool`
+and adapts its typed async chunks to `VoiceSession` locally. The tool owns
+current-frame acquisition through `xr-ai-hub-client`, has no voice dependency,
+and uses NeMo Relay's managed streaming LLM path. Camera bytes are redacted from
+Relay telemetry while the provider receives the original frame.
 `VoiceSession` owns readiness, hub transport, signals, the
 private Pipecat pipeline, and cleanup; `TextMessageInput` routes `"ping"` and
 ad-hoc text through the same participant-aware path as speech. Voice-gate

README.md

@@ -117,7 +117,8 @@ frames are dropped if it is closed.
 | Hub service | `services/xr-media-hub/` | XR-Media-Hub + LiveKit internal transport |
 | Launcher | `utils/xr-ai-launcher/` | stdlib-only process manager used by samples |
 | Logging | `utils/xr-ai-logging/` | shared loguru sink + stdlib bridge for every process |
-| Agent tools | `agent-sdk/xr-ai-nat/` | Relay-managed native tools and legacy NAT compatibility during migration |
+| Agent tools | `agent-sdk/xr-ai-tools/` | Toolkit-independent Relay-managed native tools |
+| Legacy NAT | `agent-sdk/xr-ai-nat/` | NeMo Agent Toolkit compatibility during migration |
 | Reusable services | `services/` | Model-serving and typed capability processes |
 | Agent interfaces | `agent-mcp-servers/` | MCP compatibility processes for XR data & rendering |
 | Agent demos | `agent-samples/` | End-to-end agent pipelines |
@@ -195,7 +196,7 @@ channel, or send the literal text `"ping"` — all routes go through the
 same VLM pipeline against the latest video frame.  Replies arrive as
 streaming Piper TTS audio plus a `vlm.response` text message.
 
-The packaged worker composes the NAT-native streaming vision function with
+The packaged worker adapts the transport-independent `StreamingVisionTool` to
 `xr-ai-voice`'s `VoiceSession`; Pipecat remains private to that runtime and no
 MCP client is involved. See the
 [sample README](agent-samples/simple-vlm-example/README.md) for the worker

agent-samples/simple-vlm-example/README.md

@@ -19,9 +19,9 @@ The worker is a package under `worker/simple_vlm_example_worker/`:
 
 `VoiceSession` owns STT/TTS/VLM readiness, the hub voice transport, voice-gate
 processing, streaming TTS, signals, and cleanup. The application constructs a
-finite `LiveVisionTool` and maps its separate `LiveVisionResponder` to voice;
-both share participant frame acquisition while only the direct voice response
-streams through Relay's managed LLM path. The camera frame is redacted from
+transport-independent `StreamingVisionTool` and adapts its async chunks to the
+voice handler locally. The tool has no voice dependency and sends its provider
+stream through Relay's managed LLM path. The camera frame is redacted from
 Relay telemetry. Typed text uses the same
 participant-aware turn path as speech. Participant leave events release cached
 live-frame state, and a newer turn cancels and interrupts a superseded response.

agent-samples/simple-vlm-example/worker/pyproject.toml

@@ -13,7 +13,7 @@ dependencies = [
     "xr-ai-hub-client",
     "xr-ai-logging",
     "xr-ai-models",
-    "xr-ai-nat[relay,live-vision]",
+    "xr-ai-tools[live-vision]",
     "xr-ai-voice",
     "xr-ai-voicegate",
     "loguru>=0.7",
@@ -24,7 +24,7 @@ dependencies = [
 xr-ai-hub-client = { path = "../../../agent-sdk/xr-ai-hub-client", editable = true }
 xr-ai-logging   = { path = "../../../utils/xr-ai-logging", editable = true }
 xr-ai-models    = { path = "../../../agent-sdk/xr-ai-models", editable = true }
-xr-ai-nat       = { path = "../../../agent-sdk/xr-ai-nat", editable = true }
+xr-ai-tools     = { path = "../../../agent-sdk/xr-ai-tools", editable = true }
 xr-ai-voice     = { path = "../../../agent-sdk/xr-ai-voice", editable = true }
 xr-ai-voicegate = { path = "../../../utils/xr-ai-voicegate", editable = true }
 

agent-samples/simple-vlm-example/worker/simple_vlm_example_worker/app.py

@@ -11,14 +11,14 @@
 from loguru import logger
 from xr_ai_logging import setup_logging
 from xr_ai_models import load_models_config, make_stt, make_tts, make_vlm
-from xr_ai_nat.live_vision import LiveVisionResponder, LiveVisionTool, VisionRequest
+from xr_ai_tools.streaming_vision import StreamingVisionTool, VisionRequest
 from xr_ai_voice import TextMessageInput, VadConfig, VoiceHandler, VoiceSession
 from xr_ai_voicegate import load_voice_gate_config
 
 from .config import WorkerConfig
 
 
-def _make_vision_handler(vision: LiveVisionResponder) -> VoiceHandler:
+def _make_vision_handler(vision: StreamingVisionTool) -> VoiceHandler:
     async def handle(turn):
         async def response():
             async for chunk in vision.stream(
@@ -69,14 +69,13 @@ async def run_app(
     )
 
     async with session:
-        vision = LiveVisionTool(
+        vision = StreamingVisionTool(
             endpoint=session.transport.endpoint,
             vlm=vlm,
             system_prompt=config.system_prompt,
             frame_max_age_s=config.frame_max_age_s,
             frame_timeout_s=config.frame_timeout_s,
         )
-        voice = LiveVisionResponder(vision)
         TextMessageInput(
             session=session,
             transform=_text_transform(config.default_prompt),
@@ -85,7 +84,7 @@ async def run_app(
 
         logger.info("simple-vlm-example starting")
         await session.run(
-            _make_vision_handler(voice),
+            _make_vision_handler(vision),
             on_participant_left=vision.release,
             interrupt_on_supersede=True,
         )

agent-sdk/xr-ai-hub-client/README.md

@@ -25,8 +25,8 @@ await endpoint.run()
 ```
 
 `LiveFrameSource` adds raw frame acquisition without adding image conversion or
-model dependencies. Use `xr_ai_nat.functions.vision.StreamingVisionConfig` when the desired
-interface is a model-facing vision function rather than raw pixels.
+model dependencies. Use `xr_ai_tools.streaming_vision.StreamingVisionTool` when the
+desired interface is a model-facing async vision tool rather than raw pixels.
 
 ## Migrating from `xr-ai-agent`
 

agent-sdk/xr-ai-nat/README.md

@@ -3,80 +3,11 @@
   SPDX-License-Identifier: Apache-2.0
 -->
 
-# XR AI native tools
+# XR AI functions for NeMo Agent Toolkit
 
-`xr-ai-nat` is the toolkit-independent native tools layer for XR AI.
-`Tool` gives voice, background triggers, and model-driven agents one typed
-Pydantic invocation interface. NeMo Relay manages every new tool execution;
-model-backed tools use injected `xr-ai-models` services rather than exposing a
-model client to an application trigger.
-
-The existing NeMo Agent Toolkit function groups remain available through legacy
-extras while their concrete capabilities migrate. They are compatibility code,
-not the destination for new tools.
-
-## Native tools and tool-driven agents
-
-The base install supplies `Tool`, `AgentRunner`, and `as_agent_tool`. Install
-`xr-ai-nat[relay]` for the bundled bounded tool-driven `Agent`:
-
-```python
-from pydantic import BaseModel
-from xr_ai_nat import Tool
-from xr_ai_nat.agents import Agent
-
-
-class LookupRequest(BaseModel):
-    query: str
-
-
-class LookupResult(BaseModel):
-    answer: str
-
-
-async def lookup(request: LookupRequest) -> LookupResult:
-    return LookupResult(answer=request.query)
-
-
-lookup_tool = Tool(
-    "lookup",
-    "Look up one answer.",
-    LookupRequest,
-    LookupResult,
-    lookup,
-)
-agent = Agent(
-    name="assistant",
-    llm=llm,
-    system_prompt="Use the available tools.",
-    tools=(lookup_tool,),
-)
-```
-
-`AgentRunner` is the small async turn protocol behind `as_agent_tool(...)`.
-The bundled `Agent` is the basic stateless tool loop; applications can expose a
-custom, Fabric-backed, or framework-backed runner through the same registered
-`Tool`. That keeps voice, text, and autonomous background work on one
-invocation path. Relay observes model calls inside a tool-backed runner; the
-application never calls an LLM client as a separate control path.
-
-## Live vision tool and direct voice responder
-
-Install `xr-ai-nat[relay,live-vision]` for `LiveVisionTool`. The finite
-`look_at_current_frame` tool acquires a participant's current frame and returns
-one complete `VisionResponse` for agentic planning. `LiveVisionResponder`
-shares that tool's frame source and streams only the direct voice path. Both
-call an injected `VLMService` through Relay's matching managed LLM boundary,
-forward controlled Relay headers, and redact the inline camera frame from
-events while the provider receives the original. `LiveVisionTool.release()`
-clears participant frame state.
-
-Relay's managed tool API accepts completed JSON results, while its managed LLM
-API supports streaming. Agentic vision therefore uses `Tool.execute()` and a
-complete result; direct voice remains an application-owned response stream
-with Relay managing the nested model call.
-
-## Legacy NAT compatibility
+`xr-ai-nat` provides typed, in-process XR functions for NVIDIA NeMo Agent
+Toolkit (NAT). Applications compose these functions directly; process-backed
+or MCP compatibility adapters remain separate boundaries.
 
 ## Shared value models and the service boundary
 
@@ -238,15 +169,6 @@ The `video_memory` reference is resolved lazily — only on the first
 `look_at_past_frame` call. A **live-only** consumer may omit `video_memory` (and
 need not register that group) as long as it never calls `look_at_past_frame`.
 
-For live voice workflows, `StreamingVisionConfig` (`xr_streaming_vision`) accepts
-a hub `ProcessorEndpoint` and exposes one native function with complete and
-streaming invocation modes. It owns fresh-frame acquisition and VLM invocation;
-Pipecat continues to own audio framing, interruption, and TTS.
-
-Its complete invocation returns a `VisionResult` with `status` set to `ok` or
-`unavailable`; callers must handle an unavailable result without treating its
-text as an answer about the scene.
-
 MCP-only agents that already hold a local image path can still reach the legacy
 file-path `ask_image` tool through the vlm-mcp compatibility server
 (`agent-mcp-servers/vlm-mcp/`), which now owns that path-based surface directly.

agent-sdk/xr-ai-nat/pyproject.toml

@@ -8,21 +8,19 @@ build-backend = "hatchling.build"
 [project]
 name = "xr-ai-nat"
 version = "0.1.0"
-description = "Native Relay-managed tools and legacy NeMo Agent Toolkit compatibility for XR AI."
+description = "NVIDIA NeMo Agent Toolkit functions for XR AI."
 requires-python = ">=3.11,<3.13"
 dependencies = [
-    "nemo-relay>=0.7.2,<0.8",
+    "nvidia-nat-core==1.8.0",
     "pydantic>=2.10",
 ]
 
 [project.optional-dependencies]
-relay = ["xr-ai-models"]
-live-vision = ["numpy>=1.24", "Pillow>=10.0", "xr-ai-hub-client", "xr-ai-models"]
-agents = ["nvidia-nat-core==1.8.0", "nvidia-nat-langchain==1.8.0", "xr-ai-models"]
-mcp = ["nvidia-nat-core==1.8.0", "fastmcp>=3.4,<4"]
-services = ["nvidia-nat-core==1.8.0", "msgpack>=1.0", "pyzmq>=27.0"]
-vision = ["nvidia-nat-core==1.8.0", "httpx>=0.27", "numpy>=1.24", "Pillow>=10.0", "xr-ai-hub-client", "xr-ai-models"]
-voice = ["nvidia-nat-core==1.8.0", "xr-ai-voice"]
+agents = ["nvidia-nat-langchain==1.8.0", "xr-ai-models"]
+mcp = ["fastmcp>=3.4,<4"]
+services = ["msgpack>=1.0", "pyzmq>=27.0"]
+vision = ["httpx>=0.27", "numpy>=1.24", "Pillow>=10.0", "xr-ai-hub-client", "xr-ai-models"]
+voice = ["xr-ai-voice"]
 
 [project.entry-points."nat.plugins"]
 xr_ai_nat_llm = "xr_ai_nat.llm.config"