refactor(native-tools): add Relay-managed tool layer

AGENTS.md

@@ -17,7 +17,7 @@ agent-sdk/          # Five packages:
                     #   xr-ai-models       — LLM/VLM/STT/TTS service protocols + OpenAI-compat clients
                     #   xr-ai-pipecat      — optional Pipecat transport bridge (heavier deps)
                     #   xr-ai-voice        — voice runtime (VoiceSession); introduced alongside xr-ai-pipecat
-                    #   xr-ai-nat          — typed, in-process NAT functions for XR capabilities
+                    #   xr-ai-nat          — native Relay-managed tools; legacy NAT compatibility during migration
 utils/              # Shared infra: launcher, logging, vad, vllm, voicegate
 services/           # XR hub, CloudXR, model-serving, and typed capability services
 agent-mcp-servers/  # MCP adapters: oxr, render, transcript, vec, video, vlm
@@ -60,9 +60,12 @@ deps/               # Gitignored downloaded binaries (e.g. LOVR AppImage)
 - **Workers never import from `xr_media_hub` or `xr_ai_launcher`.** Use the
   public `xr_ai_hub`, `xr_ai_models`, `xr_ai_nat`, and `xr_ai_voice` SDK
   surfaces plus task-specific libraries (numpy, torch, …).
-- **Agentic functions are NAT-first and in-process.** Reusable deterministic
-  functions live in `xr-ai-nat` as typed NAT function groups. Existing MCP
-  servers remain compatibility surfaces while their capabilities migrate.
+- **Agentic functions are native and in-process.** New and migrated tools live
+  in `xr-ai-nat`; every tool and tool-driven agent lifecycle passes through
+  NeMo Relay, and all model I/O remains in `xr-ai-models`. Its legacy extras
+  retain NeMo Agent Toolkit compatibility only while existing function groups
+  migrate. Existing MCP servers remain compatibility surfaces while their
+  capabilities migrate.
 - **RAG is a native typed capability.** `rag-service` owns document chunking,
   embedding caches, and dense retrieval behind private msgpack/ZMQ;
   `RAGFunctionsConfig` exposes it as the `xr_rag` NAT function group.

DEPENDENCIES.md

@@ -107,7 +107,9 @@ xr-ai-models  (agent-sdk/xr-ai-models/)
     and OpenAI-compatible HTTP clients that cover every in-tree model backend
     (vLLM-served VLM/LLMs, NeMo Parakeet STT, Piper/Magpie TTS).  Per-model
     profiles separate adapter behavior, endpoint connectivity/readiness, and
-    launcher-facing deployment ownership. Per-model quirks remain behind one
+    launcher-facing deployment ownership. Relay may pass controlled per-call
+    context headers; configured model credentials remain non-overridable.
+    Per-model quirks remain behind one
     seam: reasoning-field aliasing (nano_v3 →
     `reasoning`, nemotron_v3 → `reasoning_content`), `chat_template_kwargs`
     plumbing for `enable_thinking` / `thinking_budget`, and built-in presets
@@ -118,15 +120,23 @@ xr-ai-models  (agent-sdk/xr-ai-models/)
     metadata while the existing flat YAML schema remains valid.
 
 xr-ai-nat  (agent-sdk/xr-ai-nat/)
-    └── nvidia-nat-core ==1.8.0
+    └── nemo-relay >=0.7.2,<0.8
     └── pydantic >=2.10
-    └── [agents] nvidia-nat-langchain ==1.8.0, xr-ai-models [editable: ../xr-ai-models]
-    └── [mcp] fastmcp >=3.4,<4
-    └── [services] msgpack >=1.0, pyzmq >=27.0
-    └── [vision] httpx >=0.27, numpy >=1.24, Pillow >=10.0, xr-ai-hub-client [editable: ../xr-ai-hub-client], xr-ai-models [editable: ../xr-ai-models]
-    └── [voice] xr-ai-voice [editable: ../xr-ai-voice]
-    Typed, in-process NeMo Agent Toolkit functions for XR capabilities. The
-    ``xr_spatial_math`` function group accepts explicit coordinate frames and
+    └── [relay] xr-ai-models [editable: ../xr-ai-models]
+    └── [live-vision] numpy >=1.24, Pillow >=10.0, xr-ai-hub-client [editable: ../xr-ai-hub-client], xr-ai-models [editable: ../xr-ai-models]
+    └── [agents] nvidia-nat-core ==1.8.0, nvidia-nat-langchain ==1.8.0, xr-ai-models [editable: ../xr-ai-models]
+    └── [mcp] nvidia-nat-core ==1.8.0, fastmcp >=3.4,<4
+    └── [services] nvidia-nat-core ==1.8.0, msgpack >=1.0, pyzmq >=27.0
+    └── [vision] nvidia-nat-core ==1.8.0, httpx >=0.27, numpy >=1.24, Pillow >=10.0, xr-ai-hub-client [editable: ../xr-ai-hub-client], xr-ai-models [editable: ../xr-ai-models]
+    └── [voice] nvidia-nat-core ==1.8.0, xr-ai-voice [editable: ../xr-ai-voice]
+    The base package is the toolkit-independent native tools layer: Pydantic
+    request and response models, Relay-managed execution, the generic
+    ``AgentRunner`` protocol, and a bounded default tool loop over
+    `xr-ai-models`. The ``[relay]`` and
+    ``[live-vision]`` extras add model-backed tools without selecting NeMo Agent
+    Toolkit. The existing function groups remain behind legacy compatibility
+    extras while they migrate. The ``xr_spatial_math`` function group accepts
+    explicit coordinate frames and
     performs deterministic spatial calculations without OpenXR, model, or MCP
     dependencies. ``xr_text_memory`` owns persistent per-source JSONL text
     history, and ``xr_conversation_memory`` composes it into a participant-
@@ -324,7 +334,7 @@ vec-mcp-server  (agent-mcp-servers/vec-mcp/)
 xr-ai-tests  (tests/)
     └── xr-ai-hub-client             [editable: ../agent-sdk/xr-ai-hub-client]
     └── xr-ai-models            [editable: ../agent-sdk/xr-ai-models]
-    └── xr-ai-nat[agents,services,vision] [editable: ../agent-sdk/xr-ai-nat]
+    └── xr-ai-nat[agents,relay,services,vision] [editable: ../agent-sdk/xr-ai-nat]
     └── xr-rag-service [editable: ../services/rag-service]
     └── xr-ai-pipecat           [editable: ../agent-sdk/xr-ai-pipecat]
     └── xr-ai-voice             [editable: ../agent-sdk/xr-ai-voice]

README.md

@@ -117,7 +117,7 @@ frames are dropped if it is closed.
 | Hub service | `services/xr-media-hub/` | XR-Media-Hub + LiveKit internal transport |
 | Launcher | `utils/xr-ai-launcher/` | stdlib-only process manager used by samples |
 | Logging | `utils/xr-ai-logging/` | shared loguru sink + stdlib bridge for every process |
-| Agent functions | `agent-sdk/xr-ai-nat/` | Typed, in-process NAT functions for XR capabilities |
+| Agent tools | `agent-sdk/xr-ai-nat/` | Relay-managed native tools and legacy NAT compatibility during migration |
 | Reusable services | `services/` | Model-serving and typed capability processes |
 | Agent interfaces | `agent-mcp-servers/` | MCP compatibility processes for XR data & rendering |
 | Agent demos | `agent-samples/` | End-to-end agent pipelines |

THIRD_PARTY_NOTICES.md

@@ -30,6 +30,7 @@ For the per-package dependency mapping, see [`DEPENDENCIES.md`](DEPENDENCIES.md)
 | `livekit-api`  | 0.7.0    | Apache-2.0    | https://github.com/livekit/python-sdks |
 | `numpy`        | 1.24.0   | BSD-3-Clause  | https://github.com/numpy/numpy |
 | `nvidia-nat-core` | 1.8.0 | Apache-2.0    | https://github.com/NVIDIA/NeMo-Agent-Toolkit |
+| `nemo-relay`   | >=0.7.2,<0.8 | Apache-2.0  | https://github.com/NVIDIA/NeMo-Relay |
 | `Pillow`       | 10.0.0   | HPND          | https://github.com/python-pillow/Pillow |
 | `pydantic`     | >=2.10   | MIT           | https://github.com/pydantic/pydantic |
 | `websockets`   | 12.0     | BSD-3-Clause  | https://github.com/python-websockets/websockets |

agent-sdk/xr-ai-models/README.md

@@ -140,7 +140,8 @@ class LLMService(Protocol):
     capabilities: Capabilities
     async def chat(self, messages, *, tools=None, max_tokens=None,
                    temperature=None, enable_thinking=False,
-                   thinking_budget=None, timeout=None) -> ChatResponse: ...
+                   thinking_budget=None, timeout=None,
+                   headers=None) -> ChatResponse: ...
     def stream(self, messages, *, ...) -> AsyncIterator[str]: ...
     async def health(self) -> bool: ...
     async def close(self) -> None: ...
@@ -174,6 +175,11 @@ class EmbeddingService(Protocol):
 `reasoning_field` knob normalizes `reasoning_content` (nemotron_v3 parser)
 into the same surface.
 
+`LLMService.chat` and `LLMService.stream` accept optional string-valued
+per-call headers for execution context such as Relay session lineage. The
+model profile remains the authority for credentials: callers cannot supply an
+`Authorization` header.
+
 ## Remote / hosted-NIM endpoints
 
 Cloud / remote endpoints (e.g. hosted [NVIDIA NIM](https://build.nvidia.com))

agent-sdk/xr-ai-models/xr_ai_models/_openai_compat.py

@@ -16,7 +16,7 @@
 import os
 import wave
 from pathlib import Path
-from typing import Any, AsyncIterator, Sequence
+from typing import Any, AsyncIterator, Mapping, Sequence
 from urllib.parse import urlparse
 
 import httpx
@@ -74,6 +74,22 @@ def _auth_headers(api_key: str | None) -> dict[str, str]:
     return {"Authorization": f"Bearer {api_key}"} if api_key else {}
 
 
+def _request_headers(
+    api_key: str | None,
+    headers: Mapping[str, str] | None,
+) -> dict[str, str]:
+    """Merge per-call context while keeping model credentials configuration-owned."""
+    result: dict[str, str] = {}
+    for name, value in (headers or {}).items():
+        if not isinstance(name, str) or not isinstance(value, str):
+            raise TypeError("LLM request headers must be strings")
+        if name.lower() == "authorization":
+            raise ValueError("LLM request headers cannot override Authorization")
+        result[name] = value
+    result.update(_auth_headers(api_key))
+    return result
+
+
 async def _http_health(client: httpx.AsyncClient, url: str, enabled: bool) -> bool:
     # Remote endpoints (hosted NIM) expose no local /health route; the spec
     # sets health_check=false, in which case readiness is assumed.
@@ -322,14 +338,15 @@ async def chat(
         enable_thinking: bool = False,
         thinking_budget: int | None = None,
         timeout: float | None = None,
+        headers: Mapping[str, str] | None = None,
     ) -> ChatResponse:
         payload = self._build_payload(
             messages,
             tools=tools, max_tokens=max_tokens, temperature=temperature,
             enable_thinking=enable_thinking, thinking_budget=thinking_budget,
             stream=False,
         )
-        kwargs: dict[str, Any] = {"json": payload, "headers": _auth_headers(self._api_key)}
+        kwargs: dict[str, Any] = {"json": payload, "headers": _request_headers(self._api_key, headers)}
         if timeout is not None:
             kwargs["timeout"] = timeout
         resp = await self._client.post(self._chat_url, **kwargs)
@@ -348,14 +365,15 @@ async def stream(
         enable_thinking: bool = False,
         thinking_budget: int | None = None,
         timeout: float | None = None,
+        headers: Mapping[str, str] | None = None,
     ) -> AsyncIterator[str]:
         payload = self._build_payload(
             messages,
             tools=tools, max_tokens=max_tokens, temperature=temperature,
             enable_thinking=enable_thinking, thinking_budget=thinking_budget,
             stream=True,
         )
-        kwargs: dict[str, Any] = {"json": payload, "headers": _auth_headers(self._api_key)}
+        kwargs: dict[str, Any] = {"json": payload, "headers": _request_headers(self._api_key, headers)}
         if timeout is not None:
             kwargs["timeout"] = timeout
         async with self._client.stream("POST", self._chat_url, **kwargs) as resp:

agent-sdk/xr-ai-models/xr_ai_models/_protocols.py

@@ -13,7 +13,7 @@
 
 from dataclasses import dataclass
 from pathlib import Path
-from typing import Any, AsyncIterator, Literal, Protocol, Sequence, runtime_checkable
+from typing import Any, AsyncIterator, Literal, Mapping, Protocol, Sequence, runtime_checkable
 
 
 ImageInput = bytes | Path | str
@@ -109,6 +109,7 @@ async def chat(
         enable_thinking: bool = False,
         thinking_budget: int | None = None,
         timeout: float | None = None,
+        headers: Mapping[str, str] | None = None,
     ) -> ChatResponse: pass
 
     def stream(
@@ -121,6 +122,7 @@ def stream(
         enable_thinking: bool = False,
         thinking_budget: int | None = None,
         timeout: float | None = None,
+        headers: Mapping[str, str] | None = None,
     ) -> AsyncIterator[str]: pass
 
     async def health(self) -> bool: pass

agent-sdk/xr-ai-nat/README.md

@@ -3,11 +3,64 @@
   SPDX-License-Identifier: Apache-2.0
 -->
 
-# XR AI functions for NeMo Agent Toolkit
+# XR AI native tools
 
-`xr-ai-nat` provides typed, in-process XR functions for NVIDIA NeMo Agent
-Toolkit (NAT). Applications compose these functions directly; process-backed
-or MCP compatibility adapters remain separate boundaries.
+`xr-ai-nat` is the toolkit-independent native tools layer for XR AI.
+`Tool` gives voice, background triggers, and model-driven agents one typed
+Pydantic invocation interface. NeMo Relay manages every new tool execution;
+model-backed tools use injected `xr-ai-models` services rather than exposing a
+model client to an application trigger.
+
+The existing NeMo Agent Toolkit function groups remain available through legacy
+extras while their concrete capabilities migrate. They are compatibility code,
+not the destination for new tools.
+
+## Native tools and tool-driven agents
+
+The base install supplies `Tool`, `AgentRunner`, and `as_agent_tool`. Install
+`xr-ai-nat[relay]` for the bundled bounded tool-driven `Agent`:
+
+```python
+from pydantic import BaseModel
+from xr_ai_nat import Tool
+from xr_ai_nat.agents import Agent
+
+
+class LookupRequest(BaseModel):
+    query: str
+
+
+class LookupResult(BaseModel):
+    answer: str
+
+
+async def lookup(request: LookupRequest) -> LookupResult:
+    return LookupResult(answer=request.query)
+
+
+lookup_tool = Tool(
+    "lookup",
+    "Look up one answer.",
+    LookupRequest,
+    LookupResult,
+    lookup,
+)
+agent = Agent(
+    name="assistant",
+    llm=llm,
+    system_prompt="Use the available tools.",
+    tools=(lookup_tool,),
+)
+```
+
+`AgentRunner` is the small async turn protocol behind `as_agent_tool(...)`.
+The bundled `Agent` is the basic stateless tool loop; applications can expose a
+custom, Fabric-backed, or framework-backed runner through the same registered
+`Tool`. That keeps voice, text, and autonomous background work on one
+invocation path. Relay observes model calls inside a tool-backed runner; the
+application never calls an LLM client as a separate control path.
+
+## Legacy NAT compatibility
 
 ## Shared value models and the service boundary
 
@@ -22,7 +75,7 @@ Capabilities that talk to an out-of-process service share one private transport,
 `RPCServer`). `_service` owns only the transport; the value models above live in
 `functions.types`, not in `_service`.
 
-## Model-backed agents
+## Legacy NAT model bridge
 
 Install `xr-ai-nat[agents]` to make an `xr-ai-models` `LLMService` available
 to NAT's built-in LangChain-backed agent types without bypassing the repository

agent-sdk/xr-ai-nat/pyproject.toml

@@ -8,19 +8,21 @@ build-backend = "hatchling.build"
 [project]
 name = "xr-ai-nat"
 version = "0.1.0"
-description = "NVIDIA NeMo Agent Toolkit functions for XR AI."
+description = "Native Relay-managed tools and legacy NeMo Agent Toolkit compatibility for XR AI."
 requires-python = ">=3.11,<3.13"
 dependencies = [
-    "nvidia-nat-core==1.8.0",
+    "nemo-relay>=0.7.2,<0.8",
     "pydantic>=2.10",
 ]
 
 [project.optional-dependencies]
-agents = ["nvidia-nat-langchain==1.8.0", "xr-ai-models"]
-mcp = ["fastmcp>=3.4,<4"]
-services = ["msgpack>=1.0", "pyzmq>=27.0"]
-vision = ["httpx>=0.27", "numpy>=1.24", "Pillow>=10.0", "xr-ai-hub-client", "xr-ai-models"]
-voice = ["xr-ai-voice"]
+relay = ["xr-ai-models"]
+live-vision = ["numpy>=1.24", "Pillow>=10.0", "xr-ai-hub-client", "xr-ai-models"]
+agents = ["nvidia-nat-core==1.8.0", "nvidia-nat-langchain==1.8.0", "xr-ai-models"]
+mcp = ["nvidia-nat-core==1.8.0", "fastmcp>=3.4,<4"]
+services = ["nvidia-nat-core==1.8.0", "msgpack>=1.0", "pyzmq>=27.0"]
+vision = ["nvidia-nat-core==1.8.0", "httpx>=0.27", "numpy>=1.24", "Pillow>=10.0", "xr-ai-hub-client", "xr-ai-models"]
+voice = ["nvidia-nat-core==1.8.0", "xr-ai-voice"]
 
 [project.entry-points."nat.plugins"]
 xr_ai_nat_llm = "xr_ai_nat.llm.config"

agent-sdk/xr-ai-nat/xr_ai_nat/__init__.py

@@ -1,6 +1,9 @@
 # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 
-"""NAT-native XR functions."""
+"""Toolkit-independent native XR tools with legacy NAT compatibility."""
 
-__all__: list[str] = []
+from .agent_runner import AgentRunner, as_agent_tool
+from .tools import Tool, ToolInvocationResult, ToolSet
+
+__all__ = ["AgentRunner", "Tool", "ToolInvocationResult", "ToolSet", "as_agent_tool"]

agent-sdk/xr-ai-nat/xr_ai_nat/agent_runner.py

@@ -0,0 +1,55 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Framework-neutral agent runners exposed through native tools."""
+
+from __future__ import annotations
+
+from collections.abc import Callable
+from typing import Protocol, TypeVar
+
+from pydantic import BaseModel
+
+from .tools import Tool
+
+RunnerRequestT = TypeVar("RunnerRequestT", contravariant=True)
+RunnerResultT = TypeVar("RunnerResultT", covariant=True)
+ToolRequestT = TypeVar("ToolRequestT", bound=BaseModel)
+ToolResultT = TypeVar("ToolResultT", bound=BaseModel)
+
+
+class AgentRunner(Protocol[RunnerRequestT, RunnerResultT]):
+    """An application-owned agent implementation that completes one asynchronous turn."""
+
+    async def run(self, request: RunnerRequestT) -> RunnerResultT:
+        """Run one turn and return the implementation-specific result."""
+        raise NotImplementedError
+
+
+def as_agent_tool(
+    *,
+    name: str,
+    description: str,
+    agent: AgentRunner[RunnerRequestT, RunnerResultT],
+    request_model: type[ToolRequestT],
+    result_model: type[ToolResultT],
+    request: Callable[[ToolRequestT], RunnerRequestT],
+    response: Callable[[RunnerResultT], ToolResultT],
+    return_direct: bool = False,
+) -> Tool[ToolRequestT, ToolResultT]:
+    """Expose any ``AgentRunner`` through the same ``Tool`` interface as capabilities."""
+
+    async def invoke(value: ToolRequestT) -> ToolResultT:
+        return response(await agent.run(request(value)))
+
+    return Tool(
+        name,
+        description,
+        request_model,
+        result_model,
+        invoke,
+        return_direct=return_direct,
+    )
+
+
+__all__ = ["AgentRunner", "as_agent_tool"]