diff --git a/AGENTS.md b/AGENTS.md index ee70d627..047816e0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -232,12 +232,18 @@ and new config files. When a sample's behaviour is driven by an LLM prompt (e.g. `agent-samples/xr-render-demo/`): -- When you add or change a rule in `system.txt`, add or update a case - in the sample's `eval/` harness in the same edit. A rule without a - case is unverified. +- When you add or change a rule in any agent prompt (xr-render-demo: + `supervisor_prompt.txt` and `agents/*/prompt.txt`), add or update a + case in the sample's `eval/` harness in the same edit. A rule without + a case is unverified. +- **Run the cheap gate after every prompt or ops edit** (xr-render-demo: + `uv run xr_render_demo_eval basics`). Prompt edits contaminate + neighboring behaviors, and full-suite variance hides single-case + damage; debug at the lowest tier that reproduces a failure and rerun + the tiers above it before calling a fix done. - **Don't train on the test set.** Don't reuse a prompt's worked- - example specifics (coordinates, colors, shapes, trigger phrases) in - a case fixture, or vice versa — that makes the eval a memorization + example specifics (coordinates, colors, shapes, ids, trigger phrases) + in a case fixture, or vice versa — that makes the eval a memorization check. The harness audits this at startup and warns; clear the warning by changing the prompt example, not the case. diff --git a/DEPENDENCIES.md b/DEPENDENCIES.md index b9d163e2..c32ab55d 100644 --- a/DEPENDENCIES.md +++ b/DEPENDENCIES.md @@ -337,6 +337,8 @@ xr-ai-tests (tests/) └── vlm-mcp-server [editable: ../agent-mcp-servers/vlm-mcp] └── render-mcp [editable: ../agent-mcp-servers/render-mcp] └── xr-render-scene [editable: ../agent-samples/xr-render-demo/scene] + └── xr-render-demo-worker [editable: ../agent-samples/xr-render-demo/worker] + └── xr-render-demo-eval [editable: ../agent-samples/xr-render-demo/eval] └── video-mcp-server [editable: ../agent-mcp-servers/video-mcp] └── vec-mcp-server [editable: ../agent-mcp-servers/vec-mcp] └── pytest >=8.0 @@ -590,12 +592,13 @@ user-relative requests such as "to my left". |---|---|---|---| | Orchestrator | `xr-render-demo` | `xr-ai-launcher`, `xr-ai-logging` | loguru >=0.7 | | Scene | `xr-render-scene` | `xr-ai-launcher`, `xr-ai-logging`, `xr-ai-nat` | pyzmq >=27.0, msgpack >=1.0, pyyaml >=6.0 | -| Worker | `xr-render-demo-worker` | `xr-ai-hub-client`, `xr-ai-models` [editable], `xr-ai-nat[services,vision]` [editable], `xr-ai-pipecat` [editable], `xr-ai-voicegate` [editable], `xr-ai-logging` [editable], `xr-render-scene` [editable] | pyyaml >=6.0, pipecat-ai >=1.3 (native scene, tracking, spatial-math, video-memory, vision, and text-memory functions replace capability MCP clients; silero-vad via xr-ai-pipecat → xr-ai-vad). | +| Worker | `xr-render-demo-worker` | `xr-ai-hub-client`, `xr-ai-models` [editable], `xr-ai-nat[agents,services,vision,voice]` [editable], `xr-ai-voice` [editable], `xr-ai-voicegate` [editable], `xr-ai-logging` [editable], `xr-render-scene` [editable] | loguru >=0.7, pyyaml >=6.0 (native scene, tracking, spatial-math, video-memory, vision, and text-memory functions replace capability MCP clients; voice runtime via xr-ai-voice). | +| Eval | `xr-render-demo-eval` | `xr-render-demo-worker` [editable], `xr-render-scene` [editable], `xr-ai-models` [editable], `xr-ai-nat[agents,services]` [editable] | pyyaml >=6.0 (offline tiers over faked leaf functions; live drivers join the running stack over the hub IPC endpoints). | Model endpoints (llm, agent_llm, stt, tts, vlm) are declared in `yaml/models.yaml` and loaded via `xr-ai-models` `load_models_config` / `make_llm` / `make_stt` / `make_tts` / `make_vlm`. `httpx` is retained as -a transitive dep of `xr-ai-pipecat` and `xr-ai-nat[vision]`. +a transitive dep of `xr-ai-voice` and `xr-ai-nat[vision]`. Requires `model-servers` to be running first — model servers are declared as `launch_mode="reuse"` so the launcher skips spawning them but the dependency @@ -627,6 +630,7 @@ updated in the same commit**. | cloudxr-runtime YAML config keys | `agent-samples/xr-render-demo/yaml/cloudxr_runtime.yaml`, `docs/adding-cloudxr.md` | | `utils/xr-ai-launcher/xr_ai_launcher/_cloudxr_env.py` API | xr-render-scene + oxr-mcp + cloudxr-runtime `__main__.py` imports, `agent-samples/xr-render-demo/main.py` (native-profile gate), `docs/adding-cloudxr.md`, `docs/xr-render-demo.md` (client-type section) | | scene service YAML config keys | `agent-samples/xr-render-demo/scene/scene_service.yaml`, orchestrator process declaration, `docs/xr-render-demo.md` | +| openxr-service YAML config keys (`allow_sim_pose`, …) | `services/openxr-service/openxr_service.yaml` (reference copy), `agent-samples/xr-render-demo/yaml/openxr_service.yaml` | | render-mcp YAML config keys | `agent-mcp-servers/render-mcp/render_mcp.yaml`, worker URL constants | | oxr-mcp YAML config keys | `agent-mcp-servers/oxr-mcp/oxr_mcp_server.yaml`, sample copies, worker URL constants | | Any `pyproject.toml` dependency | `DEPENDENCIES.md` (this file) | diff --git a/README.md b/README.md index 6e2e1959..7c09f58d 100644 --- a/README.md +++ b/README.md @@ -318,10 +318,11 @@ web client for desktop dev. Under the hood, the orchestrator launches the hub, CloudXR runtime, model endpoints, typed capability processes, and the worker. The worker calls those processes through native NAT functions; MCP adapters remain optional outward -compatibility surfaces and are not in the sample's execution path. The Pipecat pipeline runs -quick-acks and a Nemotron-30B agentic tool-calling loop over -scene, XR tracking, spatial math, vision, and video-memory functions. Full process map, -agentic-loop details, and the XR session lifecycle: +compatibility surfaces and are not in the sample's execution path. The +`xr-ai-voice` runtime feeds a Nemotron-30B supervisor that delegates to five +focused subagents over scene, XR tracking, spatial math, vision, and +video-memory functions, with reference and geometry resolution done in code. +Full process map, agent details, and the XR session lifecycle: [`docs/xr-render-demo.md`](docs/xr-render-demo.md). **Requires `model-servers` to be running first** — the demo does not start @@ -510,7 +511,7 @@ For engineers and agents working in the repo: | [`docs/architecture.md`](docs/architecture.md) | Hub ↔ transport ↔ agent boundaries; known limitations | | [`docs/process-model.md`](docs/process-model.md) | `Process` / `run_stack` mechanics; ready-file protocol | | [`docs/ai-services.md`](docs/ai-services.md) | VLM / STT / TTS / LLM server reference + worker call examples | -| [`docs/xr-render-demo.md`](docs/xr-render-demo.md) | xr-render-demo architecture: native functions, agentic loop, XR lifecycle | +| [`docs/xr-render-demo.md`](docs/xr-render-demo.md) | xr-render-demo architecture: native functions, supervisor + subagents, XR lifecycle | | [`docs/adding-a-sample.md`](docs/adding-a-sample.md) | Boilerplate for scaffolding a new sample | | [`docs/adding-cloudxr.md`](docs/adding-cloudxr.md) | Wiring CloudXR into a sample | | [`docs/credentials.md`](docs/credentials.md) | HF / NGC token management | diff --git a/agent-samples/xr-render-demo/eval/README.md b/agent-samples/xr-render-demo/eval/README.md index ef248fb0..a0322f8a 100644 --- a/agent-samples/xr-render-demo/eval/README.md +++ b/agent-samples/xr-render-demo/eval/README.md @@ -5,136 +5,127 @@ # xr-render-demo eval harness -End-to-end test of the agent LLM's tool-calling against the live model. Each -case feeds a synthetic scene and head pose into the model with the same system -prompt and native NAT function schemas as the live worker, executes tool -effects against deterministic fixtures, then checks the resulting scene -mutations against a per-case expectation. +Four tiers, cheapest first. Debug at the lowest tier that reproduces the +problem; rerun the tiers above it as regression before calling a fix done. -## Prerequisites +| Tier | Command | Runs against | Cost | +|---|---|---|---| +| Supervisor routing | `xr_render_demo_eval_supervisor` | faked subagents that record delegations | ~15 s/case | +| Subagent components | `xr_render_demo_eval_subagents` | one real agent over faked leaf functions | ~30 s/case | +| End-to-end corpus + basics | `xr_render_demo_eval` | supervisor + agents over faked services | ~30 min full | +| Live | `xr_render_demo_live_{smoke,pose_matrix,manip,garble,explore}` | the running demo stack | minutes | -The agent LLM must be running: +All commands run from the eval project: ```bash -# weights resident in the background — start once, leave alone -uv run --project ~/hub/xr-ai/agent-samples/model-servers model_servers +cd agent-samples/xr-render-demo/eval && uv sync # once -``` +# Full corpus + native cases + basics battery +uv run xr_render_demo_eval -By default the harness calls the agent LLM at `http://localhost:8107`. It does -not require the render-demo stack, capability services, MCP adapters, or LOVR. +# The basics battery alone: the most common utterances, their perturbation +# classes, and history-bearing variants. Run after EVERY prompt or ops +# change; full-suite variance hides single-case damage. +uv run xr_render_demo_eval basics -## Run +# Subset by case name (space-separated; unknown names error out) +uv run xr_render_demo_eval move_left_one_meter between_two_spheres -```bash -# All built-in cases against the current system.txt -uv run --project agent-samples/xr-render-demo/worker \ - python agent-samples/xr-render-demo/eval/eval.py - -# Subset by case name — fast iteration on a single failing cluster. -# Comma-separated; unknown names error out (mutually exclusive with the -# positional query arg below). -uv run --project agent-samples/xr-render-demo/worker \ - python agent-samples/xr-render-demo/eval/eval.py \ - --only move_left_one_meter,between_two_spheres - -# Watcher-friendly equivalent: write case names (newline- or -# comma-separated; '#' comments OK) to eval/.only. Gitignored. -# Active subset is echoed at startup. - -# One ad-hoc query (prints the raw LLM response) -uv run --project agent-samples/xr-render-demo/worker \ - python agent-samples/xr-render-demo/eval/eval.py "Move the cube up 30 cm" - -# Score a prompt file other than the live worker's system.txt — e.g. -# main's version, a draft, or a checkout from another branch. -uv run --project agent-samples/xr-render-demo/worker \ - python agent-samples/xr-render-demo/eval/eval.py --prompt /tmp/alt-system.txt - -# Score against a hosted model (e.g. nvidia/nemotron-3-super-120b-a12b at -# build.nvidia.com) instead of the local vLLM on 8107. Set NVIDIA_API_KEY -# in the env first (or pass --agent-api-key). -export NVIDIA_API_KEY=nvapi-... -uv run --project agent-samples/xr-render-demo/worker \ - python agent-samples/xr-render-demo/eval/eval.py \ - --agent-llm https://integrate.api.nvidia.com/v1/chat/completions \ - --agent-model nvidia/nemotron-3-super-120b-a12b +# Routing and component tiers, optionally filtered by agent or case name +uv run xr_render_demo_eval_supervisor +uv run xr_render_demo_eval_subagents placement ``` -Run `uv sync` in `agent-samples/xr-render-demo/worker` before the first eval. +The offline tiers need only the agent LLM (default `http://localhost:8107`); +they do not require the demo stack, capability services, or LOVR. + +## Live drivers + +Live drivers join the running stack (`uv run python main.py` from the sample +directory) as synthetic participants, inject typed text, set a simulated +head pose, and score real scene state. They require `allow_sim_pose: true` +in `yaml/openxr_service.yaml` (off by default; flip it for eval runs and +restart the stack). Isolation rules: + +- Fresh participant id per case: transcript history otherwise bleeds between + cases and collapses supervisor behavior. +- Clear the scene between cases through the scene RPC: leftovers make + referents ambiguous and invite anchoring on stale objects. +- Vary prompt phrasing across cases: repeating one sentence builds a + self-history no real user produces. +- Never filter a run's output in the run command; write the full log to a + file and filter the file. +- Repeat runs (3x) before believing any single-run delta; near-tie decisions + flip run to run even at temperature 0. + +`xr_render_demo_live_garble` covers speech-to-text noise (homophones, +truncations, corrections, stutters) with restraint scoring: wrong mutations +fail, clarifying replies pass. `xr_render_demo_live_explore` sends novel +conversational phrasings scored by intent invariants; promote any violation +into a permanent tier case, then fix. + +## Prompt-tuning law + +The current agent model follows templates and contrast pairs; it ignores +prohibitions. Fix behavior with worked examples, and pair every +refuse-example with a proceed-example so it does not contaminate neighboring +behaviors. + +When even worked examples fail (the model keeps resolving what it should +copy) move the resolution into code and rename the tool parameter so the +schema asks for exactly what the model does reliably. Anchor descriptors are +the precedent: renaming the parameter to `anchor_words` with a copy-verbatim +description fixed in one step what five prompt variants could not, with +`spatial_ops` resolving shape synonyms, mangled nouns, and color words +against the scene deterministically. ## Watcher -`eval_watch.sh` polls `system.txt`'s sha1 once per second (hash, not -mtime — editors and language servers re-save the file without -changing bytes when you switch focus). This allows a coding agent to -iterate on the prompt and read scores out of `/tmp/eval_loop.log` -without the user re-launching `eval.py` between rounds. Any content -change aborts the running eval and starts a new one once the file -has been quiet for 10 seconds. +`eval_watch.sh` polls a combined sha1 of every worker prompt file +(`supervisor_prompt.txt` and `agents/*/prompt.txt`) once per second (hash, +not mtime: editors re-save without changing bytes). Any content change +aborts the running eval and starts a new corpus run once the prompts have +been quiet for 10 seconds, so a prompt-tuning loop can read scores out of +`/tmp/eval_loop.log` without relaunching between rounds. ```bash agent-samples/xr-render-demo/eval/eval_watch.sh tail -f /tmp/eval_loop.log - -agent-samples/xr-render-demo/eval/eval_watch.sh /path/to/alt.txt # different prompt -kill $(cat /tmp/eval_watch.pid) # stop +kill $(cat /tmp/eval_watch.pid) # stop ``` -Only one watcher runs at a time. A second invocation refuses to -start, exits non-zero, and prints the existing PID along with the -two ways to handle it (`tail` the log of the running watcher, or -`kill ` and rerun). The script never kills processes it didn't -spawn — that decision stays with the caller, which keeps the behavior -predictable across users / sandboxes / CI runners. - -`eval_watch.sh` is Linux-only. The single-instance guard reads -`/proc//cmdline` to confirm the stored PID is the watcher (not -some unrelated process that recycled the same PID); macOS has no -`/proc`, so the script will not run there. - -Score history at a glance: - -```bash -grep "passed$" /tmp/eval_loop.log | tail -``` +Only one watcher runs at a time; a second invocation refuses to start and +prints the existing PID. Linux-only (the single-instance guard reads +`/proc//cmdline`). ## Writing a case -Read `eval.py`'s `CASES` list — every shape (single-turn, pose -override, multi-turn `history`, undo `recent_moves`) is exemplified -there. Copy the closest existing case and edit. The case dict is -what the harness consumes directly; there's no case schema layer. +The end-to-end corpus lives in `xr_render_demo_eval/cases.py` (dict-shaped; +pose override, multi-turn `history`, and undo `recent_moves` are all +exemplified). Native and basics cases are `Case` dataclasses in +`xr_render_demo_eval/harness.py`; routing and component cases live in +`supervisor.py` and `subagents.py`. Copy the closest existing case and edit. ## Don't train on the test set -Prompt worked-examples and case fixtures share the same model. The -harness audits at startup for four kinds of overlap and prints a -warning for any it finds: - -1. Verbatim user utterance from a case appearing in `system.txt`. -2. Concrete scene coordinates (formatted like `(0.50, 1.60, -1.50)`) - from a case appearing in `system.txt`. -3. `recent_moves` coordinates from a case appearing in `system.txt`. -4. **Reserved prompt vocabulary** — any colour or shape word from the - eval-case vocabulary (`_EVAL_VOCAB_COLORS` / `_EVAL_VOCAB_SHAPES` - in `eval.py`) appearing inside a worked-example section of - `system.txt`. Worked-example sections are triple-backtick blocks - and any block starting with `WORKED EXAMPLE`, `Example:`, - `iter N:`, or `tool_call N:`; the first blank line after the - marker ends the block. Rule narration outside those blocks may - still mention the eval vocabulary generically (e.g. the colour - table, anchor-routing rules) — the restriction is only on the - worked examples, which are the strings the model is most likely - to memorise as a template. - -Fix overlaps by changing the prompt example, not the case. For -check #4, use colours and shapes outside the eval vocabulary -(turquoise / teal / lavender / magenta / cone / cylinder / capsule) -when reaching for a fixture word in a worked example. +Prompt worked examples and case fixtures share the same model, so the +harness audits every worker prompt at startup (all three offline tiers run +it) and warns on: + +1. A case utterance from any tier appearing verbatim in a prompt. +2. A case fixture id appearing in a prompt. +3. A quoted prompt example pairing an eval-vocabulary color + (red/green/blue/yellow/cyan/orange/purple/white/black) with an + eval-vocabulary shape (sphere/cube/box/ball). + +Fix overlaps by changing the prompt, not the case; use colors and shapes +outside the eval vocabulary (teal / lavender / magenta / turquoise, cone / +cylinder / capsule / torus) in worked examples. A case that only passes +while the prompt contains its vocabulary is scoring recall, not skill. ## What the harness does not cover -- The live worker pipeline (VAD, STT, TTS, history bookkeeping). +- The live worker pipeline (VAD, STT, TTS, history bookkeeping); the live + tier covers it. - Real scene-service / LOVR effects (fixture-succeeded). -- Real visual queries (`look_at_current_frame`, `look_at_past_frame`) — stubbed. +- Real visual queries (`look_at_current_frame`, `look_at_past_frame`): stubbed. diff --git a/agent-samples/xr-render-demo/eval/eval_watch.sh b/agent-samples/xr-render-demo/eval/eval_watch.sh index 69bb0c4f..47de9f31 100755 --- a/agent-samples/xr-render-demo/eval/eval_watch.sh +++ b/agent-samples/xr-render-demo/eval/eval_watch.sh @@ -2,22 +2,21 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 # -# Watch the xr-render-demo system prompt and re-run the eval on every +# Watch every xr-render-demo worker prompt and re-run the eval on any # change. See eval/README.md for behaviour (debounce, single-instance, # log location). # -# Usage: ./eval_watch.sh [PROMPT_PATH] +# Usage: ./eval_watch.sh [PROMPTS_DIR] set -euo pipefail HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -DEFAULT_PROMPT="$HERE/../worker/prompts/system.txt" +DEFAULT_PROMPTS_DIR="$HERE/../worker/xr_render_demo_worker" -PROMPT="${1:-$DEFAULT_PROMPT}" +PROMPTS_DIR="${1:-$DEFAULT_PROMPTS_DIR}" LOG=/tmp/eval_loop.log LOCK=/tmp/eval_watch.pid -EVAL="$HERE/eval.py" -WORKER="$HERE/../worker" +EVAL_PROJECT="$HERE" SELF="$(readlink -f "$0")" DEBOUNCE_SECS=10 TIME_FMT='+%H:%M:%S' @@ -56,7 +55,8 @@ echo $$ > "$LOCK" # editors / language servers / git tools re-save the file without changing # bytes (focus changes, refresh-on-blur, etc.). file_hash() { - sha1sum "$PROMPT" 2>/dev/null | awk '{print $1}' || true + find "$PROMPTS_DIR" -name '*prompt*.txt' -print0 2>/dev/null \ + | sort -z | xargs -0 sha1sum 2>/dev/null | sha1sum | awk '{print $1}' || true return 0 } @@ -79,13 +79,12 @@ trigger() { { echo echo "═══════════════════════════════════════════════════════════════" - echo " $(date "$TIME_FMT") prompt=$PROMPT" + echo " $(date "$TIME_FMT") prompts=$PROMPTS_DIR" echo "═══════════════════════════════════════════════════════════════" } >> "$LOG" # Reuse the worker environment so schema discovery imports the exact NAT # function packages used by the live sample. - setsid uv run --project "$WORKER" python "$EVAL" \ - --verbose --prompt "$PROMPT" >> "$LOG" 2>&1 & + setsid uv run --project "$EVAL_PROJECT" xr_render_demo_eval >> "$LOG" 2>&1 & running_pid=$! return 0 } @@ -97,7 +96,7 @@ cleanup() { } trap cleanup EXIT INT TERM HUP -echo "watching $PROMPT — log $LOG debounce ${DEBOUNCE_SECS}s (stop: kill $$)" +echo "watching $PROMPTS_DIR prompts — log $LOG debounce ${DEBOUNCE_SECS}s (stop: kill $$)" echo "started $(date "$TIME_FMT") (PID $$)" >> "$LOG" # Baseline run on startup so the user sees a score immediately. diff --git a/agent-samples/xr-render-demo/eval/pyproject.toml b/agent-samples/xr-render-demo/eval/pyproject.toml new file mode 100644 index 00000000..5541c46c --- /dev/null +++ b/agent-samples/xr-render-demo/eval/pyproject.toml @@ -0,0 +1,37 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[project] +name = "xr-render-demo-eval" +version = "0.1.0" +requires-python = ">=3.11,<3.13" +dependencies = [ + "xr-render-demo-worker", + "xr-render-scene", + "xr-ai-models", + "xr-ai-nat[agents,services]", + "pyyaml>=6.0", +] + +[tool.uv.sources] +xr-render-demo-worker = { path = "../worker", editable = true } +xr-render-scene = { path = "../scene", editable = true } +xr-ai-models = { path = "../../../agent-sdk/xr-ai-models", editable = true } +xr-ai-nat = { path = "../../../agent-sdk/xr-ai-nat", editable = true } + +[project.scripts] +xr_render_demo_eval = "xr_render_demo_eval.harness:run" +xr_render_demo_eval_subagents = "xr_render_demo_eval.subagents:run" +xr_render_demo_eval_supervisor = "xr_render_demo_eval.supervisor:run" +xr_render_demo_live_smoke = "xr_render_demo_eval.live_smoke:run" +xr_render_demo_live_pose_matrix = "xr_render_demo_eval.live_pose_matrix:run" +xr_render_demo_live_manip = "xr_render_demo_eval.live_manip:run" +xr_render_demo_live_garble = "xr_render_demo_eval.live_garble:run" +xr_render_demo_live_explore = "xr_render_demo_eval.live_explore:run" + +[tool.hatch.build.targets.wheel] +packages = ["xr_render_demo_eval"] diff --git a/agent-samples/xr-render-demo/eval/xr_render_demo_eval/__init__.py b/agent-samples/xr-render-demo/eval/xr_render_demo_eval/__init__.py new file mode 100644 index 00000000..b0ba32c1 --- /dev/null +++ b/agent-samples/xr-render-demo/eval/xr_render_demo_eval/__init__.py @@ -0,0 +1,4 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Tiered eval harness for the xr-render-demo sample.""" diff --git a/agent-samples/xr-render-demo/eval/eval.py b/agent-samples/xr-render-demo/eval/xr_render_demo_eval/cases.py old mode 100755 new mode 100644 similarity index 54% rename from agent-samples/xr-render-demo/eval/eval.py rename to agent-samples/xr-render-demo/eval/xr_render_demo_eval/cases.py index e9efa6bc..5cc2b07d --- a/agent-samples/xr-render-demo/eval/eval.py +++ b/agent-samples/xr-render-demo/eval/xr_render_demo_eval/cases.py @@ -1,80 +1,9 @@ -#!/usr/bin/env python3 # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -""" -Agent-LLM eval harness for xr-render-demo. It uses the live model endpoint, -derives tool schemas from the worker's native NAT functions, and executes tool -effects against deterministic fixtures so it never mutates the live scene. - -Usage: - uv run --project ../worker python eval.py # all cases - uv run --project ../worker python eval.py "Move it down" # one query - uv run --project ../worker python eval.py --prompt PATH # alternate prompt - -By default reads ../worker/prompts/system.txt (the live xr-render-demo -prompt). Edit it and re-run; no stack restart needed. -""" -from __future__ import annotations - -import argparse -import asyncio -import json -import math -import os -import re -import sys -import time -from pathlib import Path - -import httpx -import yaml -from nat.builder.workflow_builder import WorkflowBuilder - -_HERE = Path(__file__).resolve().parent -SYS_PROMPT = (_HERE / "../worker/prompts/system.txt").resolve() - -# Borrow the worker's config and native-tool assembly so the eval advertises -# the same model-facing function schemas as the live worker. -sys.path.insert(0, str((_HERE / "../worker").resolve())) -from capabilities import build_native_toolbox # noqa: E402 -from config import load_config # noqa: E402 — must follow sys.path tweak -from processors import ( # noqa: E402 — must follow sys.path tweak - _LIVE_PERCEPTION_TOOL, - _PAST_PERCEPTION_TOOL, - _PERCEPTION_TOOL_DEFS, -) -_WORKER_CFG = load_config((_HERE / "../yaml/xr_render_demo_worker.yaml").resolve()) - -def _agent_llm_base_url() -> str: - """Read agent_llm.base_url from models.yaml.""" - # WorkerConfig.models_yaml is resolved relative to the live launcher's - # cwd (the sample root); eval runs from eval/, so anchor it ourselves. - p = Path(_WORKER_CFG.models_yaml) - if not p.is_absolute(): - p = (_HERE / ".." / p).resolve() - with open(p) as f: - models = yaml.safe_load(f) or {} - return str(models["agent_llm"]["base_url"]).rstrip("/") - - -AGENT_LLM = f"{_agent_llm_base_url()}/v1/chat/completions" # overridable via --agent-llm -AGENT_MODEL = "llm" # overridable via --agent-model -AGENT_KEY = "" # overridable via --agent-api-key / NGC_API_KEY -# Tools the worker manages internally; hidden from the agent LLM so -# the eval and the live worker advertise the same tool surface. -WORKER_MANAGED = {"start_xr", "get_health"} - -# Mirror the worker's WorkerConfig defaults — same fixture pose for every -# test, so prompt regressions are reproducible. -DEFAULT_POSE = { - "is_valid": True, - "position": {"x": 0.0, "y": 1.6, "z": 0.0}, - "forward": {"x": 0.0, "y": 0.0, "z": -1.0}, - "right": {"x": 1.0, "y": 0.0, "z": 0.0}, - "up": {"x": 0.0, "y": 1.0, "z": 0.0}, - "yaw_deg": 0.0, - "pitch_deg": 0.0, -} + +"""End-to-end eval corpus for the render worker.""" + +PERCEPTION_TOOL = "look_at_current_frame" # Non-canonical pose (rolled head, off-origin) used by a couple of cases # that exercise gravity-aligned axis math. @@ -88,67 +17,58 @@ def _agent_llm_base_url() -> str: "pitch_deg": 4.3, } + def _became(prim_type: str | None = None, *, r_min: float | None = None, g_min: float | None = None, b_min: float | None = None): - """Predicate factory: returns a checker that asserts at least one - add_primitive / update_primitive call sets ``prim_type`` AND each - requested colour channel reaches the given lower bound. Facets may - appear in one call or be split across calls (e.g. shape on one - update, colour on another). All requested facets must be observed - for the predicate to pass.""" + """Predicate factory: assert at least one add/update sets ``prim_type`` + AND each requested colour channel reaches the given lower bound. Facets + may appear in one call or be split across calls.""" requirements: dict[str, str | float] = {} if prim_type is not None: requirements["prim_type"] = prim_type - for ch, thresh in (("r", r_min), ("g", g_min), ("b", b_min)): - if thresh is not None: - requirements[ch] = thresh + for channel, threshold in (("r", r_min), ("g", g_min), ("b", b_min)): + if threshold is not None: + requirements[channel] = threshold - def _pred(muts: list[dict]) -> tuple[bool, str]: + def _pred(mutations: list[tuple[str, dict]]) -> tuple[bool, str]: seen = dict.fromkeys(requirements, False) - for tc in muts: - if tc["function"]["name"] not in ("add_primitive", "update_primitive"): + for name, args in mutations: + if name not in ("add_primitive", "update_primitive"): continue - args = tc["function"]["arguments"] - args = json.loads(args) if isinstance(args, str) else args for key, expected in requirements.items(): if key == "prim_type": if args.get("prim_type") == expected: seen[key] = True else: - v = args.get(key) - if v is not None and float(v) >= float(expected): + value = args.get(key) + if value is not None and float(value) >= float(expected): seen[key] = True if all(seen.values()): return True, f"saw {requirements}" - missing = [k for k, v in seen.items() if not v] + missing = [key for key, hit in seen.items() if not hit] return False, f"missing facets: {missing} (wanted {requirements})" return _pred -def _stacked_vertically(muts: list[dict]) -> tuple[bool, str]: +def _stacked_vertically(mutations: list[tuple[str, dict]]) -> tuple[bool, str]: """Predicate for ``stack_*`` cases: every add_primitive must share the - same x/z column and have distinct y values, regardless of absolute - base height. Floor stack and eye-level stack are both accepted.""" - adds = [tc for tc in muts if tc["function"]["name"] == "add_primitive"] + same x/z column and have distinct y values, regardless of base height.""" + adds = [args for name, args in mutations if name == "add_primitive"] if len(adds) < 2: - return False, f"need ≥2 add_primitive calls, got {len(adds)}" - rows = [] - for tc in adds: - a = tc["function"]["arguments"] - a = json.loads(a) if isinstance(a, str) else a - rows.append((a.get("x", 0.0), a.get("y", 0.0), a.get("z", 0.0))) - xs = {round(r[0], 2) for r in rows} - zs = {round(r[2], 2) for r in rows} + return False, f"need >=2 add_primitive calls, got {len(adds)}" + rows = [(a.get("x", 0.0), a.get("y", 0.0), a.get("z", 0.0)) for a in adds] + xs = {round(row[0], 2) for row in rows} + zs = {round(row[2], 2) for row in rows} if len(xs) > 1 or len(zs) > 1: return False, f"x/z not aligned across stack: {rows}" - ys = sorted(round(r[1], 2) for r in rows) - for a, b in zip(ys, ys[1:]): - if b - a < 0.05: - return False, f"y values not separated (need ≥5 cm gap): {ys}" + ys = sorted(round(row[1], 2) for row in rows) + for low, high in zip(ys, ys[1:]): + if high - low < 0.05: + return False, f"y values not separated (need >=5 cm gap): {ys}" return True, f"stacked at y={ys}" @@ -298,7 +218,7 @@ def _stacked_vertically(muts: list[dict]) -> tuple[bool, str]: ], "user": "Add a red sphere behind the green cube.", # Behind cube → z < cube.z (further from user). Anchor is the cube - # alone — y/x align with cube, not midpoint with the other sphere. + # alone: y/x align with cube, not midpoint with the other sphere. "result": [ {"tool": "add_primitive", "args": {"prim_type": "sphere", @@ -476,7 +396,7 @@ def _stacked_vertically(muts: list[dict]) -> tuple[bool, str]: "scene": [{"id": "sphere-0", "type": "sphere", "pos": [0.5, 1.5, -1.5], "color": [1, 0, 0], "size": 0.2}], "user": "Put a green cube on top of the sphere.", - # Scene `size` is radius for spheres / half-edge for boxes. + # render-mcp `size` is radius for spheres / half-edge for boxes. # Sphere top y = 1.5 + 0.2 = 1.7; a default cube (half-edge 0.1) # sits ON the sphere when its centre y ≈ 1.8. "result": [ @@ -592,7 +512,7 @@ def _stacked_vertically(muts: list[dict]) -> tuple[bool, str]: {"tool": "remove_primitive", "args": {"obj_id": "sphere-0"}}, {"tool": "remove_primitive", "args": {"obj_id": "sphere-1"}}, ], - "ignore_extra": False, # the cube must NOT be removed + "ignore_extra": False, }, # ── closer to me ────────────────────────────────────────────────────────── @@ -711,7 +631,7 @@ def _stacked_vertically(muts: list[dict]) -> tuple[bool, str]: "scene": [{"id": "sphere-0", "type": "sphere", "pos": [0.0, 1.6, -1.5], "color": [1, 0, 0], "size": 0.1}], "user": "Turn the sphere into a cube.", - # Either path is fine — update_primitive(prim_type=box) OR + # Either path is fine: update_primitive(prim_type=box) OR # remove + add(prim_type=box). Predicate enforces "a cube # exists at the end" without pinning which path the LLM picked. "result": [], @@ -725,12 +645,12 @@ def _stacked_vertically(muts: list[dict]) -> tuple[bool, str]: "pos": [0.5, 1.0, -1.5], "color": [0, 0.4, 1], "size": 0.1}], "user": "Put a yellow sphere 1 meter above the cube.", # "1m above" can mean center+1m (=2.0) or top+1m (=2.15 with - # half-edge 0.1 + tolerance) — accept either. + # half-edge 0.1 + tolerance): accept either. "result": [ {"tool": "add_primitive", "args": {"prim_type": "sphere", "r": (0.7, 1.0), "g": (0.7, 1.0), - # b not pinned — Nemotron occasionally leaks the cube's blue + # b not pinned: Nemotron occasionally leaks the cube's blue "x": (0.45, 0.55), "y": (1.95, 2.20), "z": (-1.55, -1.45)}}, @@ -846,7 +766,7 @@ def _stacked_vertically(muts: list[dict]) -> tuple[bool, str]: {"tool": "update_primitive", "args": {"obj_id": "sphere-1", "size": (0.11, 1.0)}}, ], - # Plural-restricted target — the box must NOT also grow. + # Plural-restricted target: the box must NOT also grow. "ignore_extra": False, }, @@ -949,7 +869,7 @@ def _stacked_vertically(muts: list[dict]) -> tuple[bool, str]: ], "user": "Move the red sphere to the left.", # Either sphere is a valid pick. Empty result asserts - # "≥1 mutating call happened" — we don't pin which sphere. + # "≥1 mutating call happened": we don't pin which sphere. "result": [], }, @@ -1020,7 +940,7 @@ def _stacked_vertically(muts: list[dict]) -> tuple[bool, str]: ], # Bare "right 1 m" (no "my") isolates pronoun resolution from # anchor selection. "It" should resolve to the blue sphere - # (subject of the last reply), which is at y=1.6 — guarding + # (subject of the last reply), which is at y=1.6: guarding # against the model picking the yellow one at y=0.6. "user": "Move it right by 1 metre.", "result": [ @@ -1133,7 +1053,7 @@ def _stacked_vertically(muts: list[dict]) -> tuple[bool, str]: "pos": [ 0.14, 1.60, -0.92], "color": [0, 0, 1], "size": 0.1}, ], "user": "Move everything 1 meter further away.", - # All three should end up 1 m further from the user — z more + # All three should end up 1 m further from the user: z more # negative by ~1 at canonical pose. y / x unchanged. "result": [ {"tool": "update_primitive", @@ -1229,7 +1149,7 @@ def _stacked_vertically(muts: list[dict]) -> tuple[bool, str]: "y": ( 1.55, 1.65), "z": (-1.55, -1.45)}}, ], - # Cube must NOT move — that's what distinguishes this from swap. + # Cube must NOT move: that's what distinguishes this from swap. "ignore_extra": False, }, @@ -1255,9 +1175,7 @@ def _stacked_vertically(muts: list[dict]) -> tuple[bool, str]: # says "Put it above the blue sphere" expecting the existing # pyramid to be raised. Model has historically picked add_primitive # ("clone the recently-named object") instead of update_primitive on - # the existing pyramid. Pass-or-fail probe — captures the bug so - # we can iterate; the prompt-side rule lives in the - # "EXISTING ID → update_primitive" section. + # the existing pyramid. { "name": "pronoun_after_swap_uses_update_not_add", "scene": [ @@ -1337,1005 +1255,48 @@ def _stacked_vertically(muts: list[dict]) -> tuple[bool, str]: "args": {"obj_id": "sphere-0", "y": (0.5, 1.61)}}, ], }, -] - - -def _format_scene(scene: list[dict]) -> str: - if not scene: - return "SCENE OBJECTS: (empty)" - lines = ["SCENE OBJECTS:"] - for o in scene: - x, y, z = o["pos"] - r, g, b = o["color"] - lines.append( - f" {o['id']} ({o['type']}) " - f"pos=({x:.2f}, {y:.2f}, {z:.2f}) " - f"color=(r={r:.2f} g={g:.2f} b={b:.2f}) " - f"size={o['size']:.3f}m" - ) - return "\n".join(lines) - - -def _format_pose(pose: dict) -> str: - if not pose.get("is_valid"): - return "HEAD POSE: unavailable" - p, fv, rv, uv = pose["position"], pose["forward"], pose["right"], pose["up"] - - def _off(vec, d): - return (f"({p['x']+vec['x']*d:.2f}, " - f"{p['y']+vec['y']*d:.2f}, " - f"{p['z']+vec['z']*d:.2f})") - - return ( - "HEAD POSE:\n" - f" position : ({p['x']:.2f}, {p['y']:.2f}, {p['z']:.2f})\n" - f" forward : ({fv['x']:.3f}, {fv['y']:.3f}, {fv['z']:.3f}) ← 'ahead/forward'\n" - f" right : ({rv['x']:.3f}, {rv['y']:.3f}, {rv['z']:.3f}) ← 'right'\n" - f" up : ({uv['x']:.3f}, {uv['y']:.3f}, {uv['z']:.3f}) ← 'up'\n" - f" yaw={pose.get('yaw_deg',0):.1f}° pitch={pose.get('pitch_deg',0):.1f}°\n" - "SPATIAL SHORTCUTS (pre-computed — use directly, no tool call needed):\n" - f" 1.5m ahead of you : {_off(fv, 1.5)}\n" - f" 1m to your right : {_off(rv, 1.0)}\n" - f" 1m to your left : {_off(rv, -1.0)}\n" - f" 0.5m above eye level : {_off(uv, 0.5)}\n" - f" 1m behind you : {_off(fv, -1.0)}\n" - " For other distances: new_pos = obj.pos + direction_vec × distance (per component)" - ) - - -class _NullFrameEndpoint: - """Frame endpoint stub for offline tool-schema discovery. - - The eval never pulls live pixels; it only needs the vision group to build so - ``look_at_current_frame`` / ``look_at_past_frame`` appear in the tool schema. - """ - - def on_frame(self, _cb) -> None: - pass - - def on_participant(self, _cb) -> None: - pass - - -async def _discover_tools() -> list[dict]: - async with WorkflowBuilder() as builder: - toolbox, _vision_config = await build_native_toolbox( - builder, - scene_endpoint=_WORKER_CFG.scene_endpoint, - openxr_endpoint=_WORKER_CFG.openxr_endpoint, - video_memory_endpoint=_WORKER_CFG.video_memory_endpoint, - frame_endpoint=_NullFrameEndpoint(), - vlm=object(), - ) - # Present the model trimmed perception schemas (the worker injects the - # participant/reference context the native request models expose). - definitions = toolbox.definitions( - exclude=WORKER_MANAGED | {_LIVE_PERCEPTION_TOOL, _PAST_PERCEPTION_TOOL} - ) - definitions.extend(_PERCEPTION_TOOL_DEFS) - return [definition.to_openai() for definition in definitions] - - -def _format_recent_moves(moves: list[tuple] | None) -> str: - """Render the same `[Recent moves]` block the worker injects. Each - `moves` entry is (obj_id, (px, py, pz), (nx, ny, nz)). - """ - if not moves: - return "" - lines = ["[Recent moves] (most recent last — prev → new)"] - for obj_id, prev, new in moves: - lines.append( - f" {obj_id}: ({prev[0]:.2f}, {prev[1]:.2f}, {prev[2]:.2f}) → " - f"({new[0]:.2f}, {new[1]:.2f}, {new[2]:.2f})" - ) - return "\n".join(lines) - - -def _format_recent_conversation(history: list[tuple[str, str]] | None) -> str: - """Render the same `[Recent conversation]` block the worker injects. - Each entry is (prior_user_text, prior_agent_reply). - """ - if not history: - return "" - lines = ["[Recent conversation]"] - for u, a in history: - lines.append(f" User: {u}") - lines.append(f" Agent: {a}") - return "\n".join(lines) - - -def _build_messages(system_prompt: str, scene: list[dict], pose: dict, user: str, - history: list[tuple[str, str]] | None = None, - recent_moves: list[tuple] | None = None) -> list[dict]: - """Build the worker-equivalent chat messages. Prior turns go into - a ``[Recent conversation]`` block inside the single user-role - context message — injecting them as ``role=assistant`` biases - Nemotron toward text-only replies and away from tool calls.""" - context_parts = [_format_scene(scene), _format_pose(pose)] - moves_block = _format_recent_moves(recent_moves) - if moves_block: - context_parts.append(moves_block) - conv_block = _format_recent_conversation(history) - if conv_block: - context_parts.append(conv_block) - context = "\n".join(context_parts) - return [ - {"role": "system", "content": system_prompt}, - {"role": "user", "content": ( - "[Pre-fetched context — do not call get_scene_state or " - "get_head_pose unless you need to refresh after changes]\n" - f"{context}\n\n[Request]\n{user}" - )}, - ] - - -def _local_position_relative(args: dict, pose: dict) -> dict: - """Mirror native position_relative — gravity-aligned (yaw is honoured; - pitch and roll are stripped). Up is world +Y.""" - f, r = pose["forward"], pose["right"] - p = pose["position"] - fwd = float(args.get("forward", 0.0)) - rgt = float(args.get("right", 0.0)) - up_ = float(args.get("up", 0.0)) - ox = float(args.get("origin_x", p["x"])) - oy = float(args.get("origin_y", p["y"])) - oz = float(args.get("origin_z", p["z"])) - - fx, fz = f["x"], f["z"] - mag = math.sqrt(fx*fx + fz*fz) - if mag < 1e-6: - rx0, rz0 = r["x"], r["z"] - mag2 = math.sqrt(rx0*rx0 + rz0*rz0) - if mag2 < 1e-6: - fx, fz = 0.0, -1.0 - else: - rx0, rz0 = rx0 / mag2, rz0 / mag2 - fx, fz = rz0, -rx0 - else: - fx, fz = fx / mag, fz / mag - rx, rz = -fz, fx - - return { - "x": round(ox + fx*fwd + rx*rgt, 3), - "y": round(oy + up_, 3), - "z": round(oz + fz*fwd + rz*rgt, 3), - } - - -def _local_position_ahead(args: dict, pose: dict) -> dict: - f, p = pose["forward"], pose["position"] - d = float(args.get("distance", 1.5)) - return { - "x": round(p["x"] + f["x"]*d, 3), - "y": round(p["y"] + f["y"]*d, 3), - "z": round(p["z"] + f["z"]*d, 3), - } - - -def _ground_basis(pose: dict) -> tuple[tuple[float, float], tuple[float, float]]: - """Return the gravity-aligned forward and right basis used by native spatial tools.""" - f, r = pose["forward"], pose["right"] - fx, fz = f["x"], f["z"] - mag = math.sqrt(fx * fx + fz * fz) - if mag < 1e-6: - rx0, rz0 = r["x"], r["z"] - mag2 = math.sqrt(rx0 * rx0 + rz0 * rz0) - if mag2 < 1e-6: - fx, fz = 0.0, -1.0 - else: - rx0, rz0 = rx0 / mag2, rz0 / mag2 - fx, fz = rz0, -rx0 - else: - fx, fz = fx / mag, fz / mag - return (fx, fz), (-fz, fx) - - -def _local_place_user_relative(args: dict, pose: dict) -> dict: - direction = args.get("direction", "front") - distance = float(args.get("distance", 1.5)) - if distance < 0: - return {"error": "distance must be non-negative"} - p = pose["position"] - (fx, fz), (rx, rz) = _ground_basis(pose) - dx = dy = dz = 0.0 - if direction == "front": - dx, dz = fx * distance, fz * distance - elif direction == "back": - dx, dz = -fx * distance, -fz * distance - elif direction == "right": - dx, dz = rx * distance, rz * distance - elif direction == "left": - dx, dz = -rx * distance, -rz * distance - elif direction == "above": - dy = distance - elif direction == "below": - dy = -distance - return { - "x": round(p["x"] + dx, 3), - "y": round(p["y"] + dy, 3), - "z": round(p["z"] + dz, 3), - } - - -def _local_world_offset(args: dict, _pose: dict) -> dict: - """Mirror native world_offset — origin + (dx, dy, dz).""" - ox = float(args.get("origin_x", 0.0)) - oy = float(args.get("origin_y", 0.0)) - oz = float(args.get("origin_z", 0.0)) - dx = float(args.get("dx", 0.0)) - dy = float(args.get("dy", 0.0)) - dz = float(args.get("dz", 0.0)) - return {"x": round(ox + dx, 3), "y": round(oy + dy, 3), "z": round(oz + dz, 3)} - - -def _local_along_direction(args: dict, _pose: dict) -> dict: - """Mirror native along_direction — origin moved `distance` toward target.""" - ox = float(args.get("origin_x", 0.0)) - oy = float(args.get("origin_y", 0.0)) - oz = float(args.get("origin_z", 0.0)) - tx = float(args.get("target_x", 0.0)) - ty = float(args.get("target_y", 0.0)) - tz = float(args.get("target_z", 0.0)) - d = float(args.get("distance", 0.5)) - vx, vy, vz = tx - ox, ty - oy, tz - oz - mag = math.sqrt(vx*vx + vy*vy + vz*vz) - if mag < 1e-9: - return {"error": "origin and target coincide"} - return { - "x": round(ox + vx * d / mag, 3), - "y": round(oy + vy * d / mag, 3), - "z": round(oz + vz * d / mag, 3), - } - - -def _local_scale_value(args: dict, _pose: dict) -> dict: - """Mirror native scale_value — current * factor.""" - cur = float(args.get("current", 0.0)) - fac = float(args.get("factor", 1.0)) - return {"value": round(cur * fac, 3)} - - -def _local_place_inside_by_id(args: dict, _pose: dict) -> dict: - """Mirror native place_inside_by_id — container coords echoed back - alongside the movee's id so the result feeds straight into - update_primitive.""" - for field in ("movee_id", "container_x", "container_y", "container_z"): - if args.get(field) is None: - return {"error": f"missing {field}"} - return { - "obj_id": args["movee_id"], - "x": round(float(args["container_x"]), 3), - "y": round(float(args["container_y"]), 3), - "z": round(float(args["container_z"]), 3), - } - - -def _local_between_anchors(args: dict, _pose: dict) -> dict: - """Mirror native between_anchors — component-wise midpoint of A and B.""" - a_x, a_y, a_z = (float(args.get("a_x", 0.0)), - float(args.get("a_y", 0.0)), - float(args.get("a_z", 0.0))) - b_x, b_y, b_z = (float(args.get("b_x", 0.0)), - float(args.get("b_y", 0.0)), - float(args.get("b_z", 0.0))) - return { - "x": round((a_x + b_x) / 2.0, 3), - "y": round((a_y + b_y) / 2.0, 3), - "z": round((a_z + b_z) / 2.0, 3), - } - - -def _local_displace_objects(args: dict, pose: dict) -> dict: - """Mirror native displace_objects — same user-frame delta applied - to every (id, x, y, z) entry; returns {items: [...]}.""" - for field in ("object_ids", "current_xs", "current_ys", "current_zs"): - if args.get(field) is None: - return {"error": f"missing {field}"} - ids = list(args["object_ids"]) - xs = list(args["current_xs"]) - ys = list(args["current_ys"]) - zs = list(args["current_zs"]) - n = len(ids) - if not (len(xs) == n and len(ys) == n and len(zs) == n): - return {"error": "object_ids / current_xs / current_ys / current_zs " - "must all be the same length"} - if n == 0: - return {"items": []} - right = float(args.get("right", 0.0)) - up_ = float(args.get("up", 0.0)) - forward = float(args.get("forward", 0.0)) - (fx, fz), (rx, rz) = _ground_basis(pose) - items = [] - for i in range(n): - cx, cy, cz = float(xs[i]), float(ys[i]), float(zs[i]) - items.append({ - "obj_id": ids[i], - "x": round(cx + fx * forward + rx * right, 3), - "y": round(cy + up_, 3), - "z": round(cz + fz * forward + rz * right, 3), - }) - return {"items": items} - - -def _local_displace_object(args: dict, pose: dict) -> dict: - """Mirror native displace_object — current + user-frame delta.""" - for field in ("current_x", "current_y", "current_z"): - if args.get(field) is None: - return {"error": f"missing {field}"} - cx = float(args["current_x"]) - cy = float(args["current_y"]) - cz = float(args["current_z"]) - right = float(args.get("right", 0.0)) - up_ = float(args.get("up", 0.0)) - forward = float(args.get("forward", 0.0)) - (fx, fz), (rx, rz) = _ground_basis(pose) - return { - "x": round(cx + fx * forward + rx * right, 3), - "y": round(cy + up_, 3), - "z": round(cz + fz * forward + rz * right, 3), - } - - -def _local_place_object_relative(args: dict, pose: dict) -> dict: - direction = args.get("direction", "front") - distance = float(args.get("distance", 0.3)) - if distance < 0: - return {"error": "distance must be non-negative"} - ox = float(args.get("origin_x", 0.0)) - oy = float(args.get("origin_y", 0.0)) - oz = float(args.get("origin_z", 0.0)) - (fx, fz), (rx, rz) = _ground_basis(pose) - dx = dy = dz = 0.0 - if direction == "front": - dx, dz = -fx * distance, -fz * distance - elif direction == "back": - dx, dz = fx * distance, fz * distance - elif direction == "right": - dx, dz = rx * distance, rz * distance - elif direction == "left": - dx, dz = -rx * distance, -rz * distance - elif direction == "next_to": - dx, dz = rx * distance, rz * distance - elif direction == "above": - dy = distance - elif direction == "below": - dy = -distance - return { - "x": round(ox + dx, 3), - "y": round(oy + dy, 3), - "z": round(oz + dz, 3), - } - - -_ADD_COUNTER: dict[str, int] = {} - - -def _reset_exec_state() -> None: - _ADD_COUNTER.clear() - - -# Per-case scratch state read by the local tool mocks in _exec_tool. -# Reset before every rollout via _reset_exec_state / _set_*. -_FIXTURE_SCENE: list[dict] = [] -_CASE_HISTORY: list[tuple[str, str]] = [] -_CASE_MOVES: list[tuple] = [] - - -def _set_fixture_scene(scene: list[dict]) -> None: - _FIXTURE_SCENE.clear() - _FIXTURE_SCENE.extend(scene) - - -def _set_case_history(history: list[tuple[str, str]] | None) -> None: - _CASE_HISTORY.clear() - if history: - _CASE_HISTORY.extend(history) - - -def _set_case_moves(moves: list[tuple] | None) -> None: - _CASE_MOVES.clear() - if moves: - _CASE_MOVES.extend(moves) - - -def _fixture_scene_as_render() -> dict: - """Echo the case's fixture scene in the native scene-state shape. - This prevents retry loops where the model refreshes the scene and receives - an empty result instead of the fixture it was given.""" - return {"objects": [ - {"id": o["id"], - "type": o["type"], - "position": {"x": o["pos"][0], "y": o["pos"][1], "z": o["pos"][2]}, - "color": {"r": o["color"][0], "g": o["color"][1], "b": o["color"][2]}, - "size": o.get("size", 0.1)} - for o in _FIXTURE_SCENE - ]} - - -async def _exec_tool(name: str, args_json: str, pose: dict) -> dict: - """Execute a tool call. Spatial functions run locally against the - case's fixture pose so rollouts are deterministic. add_primitive - returns a fresh per-rollout id (otherwise the model spawns the - same object N times waiting to "see" it); update / remove return - ok. Unknown tools return a sentinel.""" - args = json.loads(args_json) if isinstance(args_json, str) else (args_json or {}) - if name == "position_relative": - return _local_position_relative(args, pose) - if name == "position_ahead": - return _local_position_ahead(args, pose) - if name == "place_user_relative": - return _local_place_user_relative(args, pose) - if name == "place_object_relative": - return _local_place_object_relative(args, pose) - if name == "place_inside_by_id": - return _local_place_inside_by_id(args, pose) - if name == "displace_object": - return _local_displace_object(args, pose) - if name == "displace_objects": - return _local_displace_objects(args, pose) - if name == "between_anchors": - return _local_between_anchors(args, pose) - if name == "world_offset": - return _local_world_offset(args, pose) - if name == "along_direction": - return _local_along_direction(args, pose) - if name == "scale_value": - return _local_scale_value(args, pose) - if name == "get_head_pose": - return pose - if name == "add_primitive": - prim = args.get("prim_type", "sphere") - n = _ADD_COUNTER.get(prim, -1) + 1 - _ADD_COUNTER[prim] = n - return {"id": f"{prim}-{n}", "ok": True} - if name == "update_primitive": - return {"ok": True} - if name == "remove_primitive": - return {"ok": True} - if name == "get_scene_state": - return _fixture_scene_as_render() - return {"_eval_skipped": True, "reason": f"{name} not in safe-exec list"} - - -async def _run_one(http: httpx.AsyncClient, system_prompt: str, - tools: list[dict], scene: list[dict], pose: dict, - user: str, *, thinking: bool = False, - max_steps: int = 1) -> dict: - """Run up to ``max_steps`` LLM iterations against the agent LLM, - mocking tool execution between turns via ``_exec_tool``. Returns - ``{latency_s, tool_calls, content, reasoning}``: ``tool_calls`` is - every tool call emitted across all turns (in order), ``content`` / - ``reasoning`` are from the final turn. - """ - _reset_exec_state() - _set_fixture_scene(scene) - messages = _build_messages(system_prompt, scene, pose, user, - _CASE_HISTORY, _CASE_MOVES) - all_calls: list[dict] = [] - last_msg: dict = {} - t_total = 0.0 - - for _step in range(max_steps): - body = { - "model": AGENT_MODEL, - "messages": messages, - "tools": tools, - "max_tokens": 2048 if thinking else 1024, - "temperature": 0.0, - "chat_template_kwargs": { - "enable_thinking": thinking, - **({"thinking_budget": 1024} if thinking else {}), - }, - } - t0 = time.time() - headers = {"Authorization": f"Bearer {AGENT_KEY}"} if AGENT_KEY else None - # Retry on transient 5xx / network errors; non-5xx still raise. - for attempt in range(3): - try: - r = await http.post(AGENT_LLM, json=body, timeout=180.0, headers=headers) - if r.status_code >= 500 and attempt < 2: - await asyncio.sleep(2.0 * (attempt + 1)) - continue - r.raise_for_status() - break - except (httpx.TimeoutException, httpx.NetworkError): - if attempt >= 2: - raise - await asyncio.sleep(2.0 * (attempt + 1)) - t_total += time.time() - t0 - msg = r.json()["choices"][0]["message"] - last_msg = msg - tcs = msg.get("tool_calls") or [] - if not tcs: - break - # A turn can emit multiple parallel tool calls (e.g. compound - # utterances), so extend rather than append. - all_calls.extend(tcs) - if _step + 1 >= max_steps: - break - new_msgs: list[dict] = [{"role": "assistant", "content": "", "tool_calls": tcs}] - for tc in tcs: - fn = tc["function"] - result = await _exec_tool(fn["name"], fn["arguments"], pose) - new_msgs.append({"role": "tool", "tool_call_id": tc["id"], - "content": json.dumps(result, default=str)}) - messages = messages + new_msgs - - return {"latency_s": round(t_total, 2), - "tool_calls": all_calls, - "content": (last_msg.get("content") or "").strip(), - "reasoning": (last_msg.get("reasoning_content") or "").strip()} - - -# update_primitive arg -> (scene-object-field, optional index). Used to -# resolve "absent arg means kept original value" so partial updates -# (e.g. ``{x: -1.0}`` for "move left 1m") are checked against the -# effective resulting position, not just the bytes the LLM emitted. -_SCENE_ARG_LOOKUP = { - "x": ("pos", 0), "y": ("pos", 1), "z": ("pos", 2), - "r": ("color", 0), "g": ("color", 1), "b": ("color", 2), - "size": ("size", None), "prim_type": ("type", None), -} + # ── perception gating: real-world colour must come from the camera ─────── + # The colour word is never in the utterance; the model must call + # look_at_current_frame FIRST and read the colour out of the answer. + # `vlm_answer` is what the mocked camera sees; `must_call_first` fails + # the case if the model mutates before (or without) looking. + { + "name": "perception_color_of_held_object", + "scene": [], + "user": "Make a sphere the same color as the thing I'm holding.", + "vlm_answer": "The user is holding a bright red apple.", + "must_call_first": PERCEPTION_TOOL, + "result": [ + {"tool": "add_primitive", + "args": {"prim_type": "sphere", + "r": (0.7, 1.0), "g": (0.0, 0.4), "b": (0.0, 0.4)}}, + ], + }, + { + "name": "perception_recolor_to_match_shirt", + "scene": [{"id": "sphere-0", "type": "sphere", + "pos": [0.0, 1.6, -1.5], "color": [1, 1, 1], "size": 0.1}], + "user": "Make the sphere the same color as my shirt.", + "vlm_answer": "The user's shirt is blue.", + "must_call_first": PERCEPTION_TOOL, + "result": [ + {"tool": "update_primitive", + "args": {"obj_id": "sphere-0", + "b": (0.5, 1.0), "r": (0.0, 0.3)}}, + ], + }, + { + "name": "perception_wall_color_cube", + "scene": [], + "user": "Add a cube that matches the color of the wall I'm looking at.", + "vlm_answer": "The wall is green.", + "must_call_first": PERCEPTION_TOOL, + "result": [ + {"tool": "add_primitive", + "args": {"prim_type": "box", + "g": (0.5, 1.0), "r": (0.0, 0.4), "b": (0.0, 0.3)}}, + ], + }, +] -def _resolve_arg(obj_id: str, key: str, scene: list[dict]): - field = _SCENE_ARG_LOOKUP.get(key) - if not field: - return None - obj = next((o for o in scene if o.get("id") == obj_id), None) - if obj is None: - return None - src, idx = field - val = obj.get(src) - if idx is None: - return val - return val[idx] if val is not None and idx < len(val) else None - - -def _match_call(call: dict, expect: dict, scene: list[dict] | None = None) -> tuple[bool, str]: - fn = call["function"] - if fn["name"] != expect["tool"]: - return False, f"tool={fn['name']} want={expect['tool']}" - args = json.loads(fn["arguments"]) if isinstance(fn["arguments"], str) else fn["arguments"] - fails = [] - for k, want in expect.get("args", {}).items(): - got = args.get(k) - if got is None and fn["name"] == "update_primitive" and scene: - obj_id = args.get("obj_id") - if obj_id: - got = _resolve_arg(obj_id, k, scene) - if got is None: - fails.append(f"{k}=missing"); continue - if isinstance(want, tuple): - lo, hi = want - # Some models emit numeric args as strings; coerce before compare. - if isinstance(got, str): - try: - got = float(got) - except ValueError: - fails.append(f"{k}={got!r} not numeric, want [{lo},{hi}]") - continue - if not (lo <= got <= hi): - fails.append(f"{k}={got} not in [{lo},{hi}]") - else: - if got != want: - fails.append(f"{k}={got!r} want={want!r}") - return (not fails), ("ok" if not fails else "; ".join(fails)) - - -_MUTATING_TOOLS = frozenset({"add_primitive", "update_primitive", "remove_primitive"}) - - -def _check(actual: dict, case: dict) -> tuple[bool, str]: - """Match ``case['result']`` against the mutating tool calls - (add/update/remove_primitive) emitted during the rollout. Order- - independent; helper/math calls are ignored. ``ignore_extra`` - (default True) allows extra mutations beyond the expectation. - - Empty ``result`` is the "any path is fine" mode: the case still - requires at least one mutating call to have happened (otherwise a - silent no-op would pass). - """ - tcs = actual["tool_calls"] - wanted = list(case.get("result") or []) - muts = [tc for tc in tcs if tc["function"]["name"] in _MUTATING_TOOLS] - if not wanted and not muts: - names = [tc["function"]["name"] for tc in tcs] - return False, f"no mutating calls: {names}" - scene = case.get("scene") or [] - unmatched_actuals = list(muts) - unmatched_expected: list[dict] = [] - for exp in wanted: - for idx, ac in enumerate(unmatched_actuals): - ok, _ = _match_call(ac, exp, scene) - if ok: - unmatched_actuals.pop(idx); break - else: - unmatched_expected.append(exp) - if unmatched_expected: - missing = "; ".join( - f"{e['tool']}({e.get('args',{})})" for e in unmatched_expected - ) - actual_summary = [ - f"{tc['function']['name']}({tc['function']['arguments']})" - for tc in muts - ] - return False, f"unmatched result: {missing} | actual mutations: {actual_summary}" - if not case.get("ignore_extra", True) and unmatched_actuals: - extras = [tc["function"]["name"] for tc in unmatched_actuals] - return False, f"extra mutating calls: {extras}" - predicate = case.get("predicate") - if predicate is not None: - ok, msg = predicate(muts) - if not ok: - return False, f"predicate failed: {msg}" - return True, f"matched {len(wanted)} mutation(s)" - - -# max LLM iterations per turn (mirrors processors.py _MAX_LOOP). -_MAX_STEPS = 10 - - -# Reserved-prompt-vocabulary sets used by check #4 in -# _check_prompt_eval_overlap (see that docstring and eval/README.md). -_EVAL_VOCAB_COLORS = frozenset({ - "red", "green", "blue", "cyan", "brown", "yellow", -}) -_EVAL_VOCAB_SHAPES = frozenset({ - "sphere", "spheres", "cube", "cubes", "box", "boxes", - "pyramid", "pyramids", -}) - -# Worked-example section start markers (case-insensitive). A section -# runs from the marker line through the first blank line; triple-backtick -# fences are also captured as blocks (everything between the fences). -_EXAMPLE_START_RE = re.compile( - r"^\s*(?:" - r"WORKED\s+EXAMPLE\b|WORKED\s+ANTI-?EXAMPLE\b|" - r"Examples?:|" - r"iter\s+\d+\s*:|" - r"tool_call\s+\d+\s*:" - r")", - re.IGNORECASE, -) - - -def _extract_example_blocks(sp: str) -> list[tuple[int, str]]: - """Slice the system prompt into worked-example sections. - - Returns ``[(start_line_1_indexed, block_text), …]``. A section is - either everything between a pair of triple-backtick fences, or - everything from a marker line (``WORKED EXAMPLE``, ``Example:``, - ``iter N:``, ``tool_call N:``) through the first following blank - line. - """ - blocks: list[tuple[int, str]] = [] - lines = sp.splitlines() - in_fence = False - fence_start = 0 - fence_buf: list[str] = [] - i = 0 - while i < len(lines): - line = lines[i] - if line.lstrip().startswith("```"): - if in_fence: - blocks.append((fence_start, "\n".join(fence_buf))) - in_fence = False - fence_buf = [] - else: - in_fence = True - fence_start = i + 1 - i += 1 - continue - if in_fence: - fence_buf.append(line) - i += 1 - continue - if _EXAMPLE_START_RE.match(line): - start = i + 1 - buf = [line] - i += 1 - while i < len(lines) and lines[i].strip(): - buf.append(lines[i]) - i += 1 - blocks.append((start, "\n".join(buf))) - continue - i += 1 - if in_fence and fence_buf: - blocks.append((fence_start, "\n".join(fence_buf))) - return blocks - - -def _case_fixture_vocab(c: dict) -> tuple[set[str], set[str]]: - """Eval-vocab colour/shape words actually present in this case's - fixture (user utterance, history dialogue, scene type tags, ids). - Used to attribute reserved-vocab violations to specific cases.""" - parts: list[str] = [c.get("user") or ""] - for pair in c.get("history") or []: - parts.extend(pair) - for o in c.get("scene") or []: - if t := o.get("type"): - parts.append(t) - if oid := o.get("id"): - parts.append(oid) - blob = " ".join(parts).lower() - colors = {w for w in _EVAL_VOCAB_COLORS if re.search(rf"\b{w}\b", blob)} - shapes = {w for w in _EVAL_VOCAB_SHAPES if re.search(rf"\b{w}\b", blob)} - return colors, shapes - - -def _check_prompt_eval_overlap( - system_prompt: str, cases: list[dict] -) -> tuple[set[str], list[str]]: - """Detect overlap between prompt worked-examples and eval case - fixtures. An overlap turns a generalization probe into a - memorization check (see AGENTS.md "Prompt-driven samples"). - - Four checks run, each across every case: - 1. Verbatim user utterance (≥12 chars) appearing in the prompt. - 2. Concrete scene coordinates rendered like ``(x.xx, y.yy, z.zz)`` - appearing in the prompt. - 3. ``recent_moves`` coords appearing in the prompt. - 4. Reserved-prompt-vocabulary: worked-example sections of - system.txt must not use any colour/shape word from the - eval-case vocabulary (``_EVAL_VOCAB_COLORS`` / - ``_EVAL_VOCAB_SHAPES``). Worked-example sections are - triple-backtick blocks and any block starting with - ``WORKED EXAMPLE`` / ``Example:`` / ``iter N:`` / - ``tool_call N:``. Rule narration outside those blocks - is unrestricted — the colour table, anchor-routing rules, - etc. may still mention ``red sphere`` generically. - - Returns ``(overlapping_case_names, issue_lines)``. The set is the - distinct cases that overlap (caller uses the count for the score - caveat); the list is per-issue detail strings. Both are empty - when no overlaps. - """ - sp = system_prompt - issues: list[str] = [] - overlapping: set[str] = set() - for c in cases: - name = c.get("name", "") - before = len(issues) - # 1. Verbatim user utterance (case-insensitive substring) appearing - # in the prompt. Short utterances <12 chars are skipped to - # avoid noise like "Move it." matching every example. - u = (c.get("user") or "").strip().rstrip(".!?") - if u and len(u) >= 12 and u.lower() in sp.lower(): - issues.append(f" {name}: user utterance {u!r} appears verbatim in system.txt") - # 2. Concrete scene coordinates (rendered like "(0.50, 1.60, -1.50)") - # appearing in the prompt. - for o in c.get("scene") or []: - x, y, z = o["pos"] - coord = f"({x:.2f}, {y:.2f}, {z:.2f})" - if coord in sp: - issues.append( - f" {name}: scene object {o['id']!r} coords {coord} " - f"appear verbatim in system.txt" - ) - break - # 3. recent_moves coords landing in the prompt. - for entry in c.get("recent_moves") or []: - _obj, prev, new = entry - for triple in (prev, new): - coord = f"({triple[0]:.2f}, {triple[1]:.2f}, {triple[2]:.2f})" - if coord in sp: - issues.append( - f" {name}: recent_moves coords {coord} appear " - f"verbatim in system.txt" - ) - break - if len(issues) > before: - overlapping.add(name) - - # 4. Reserved-prompt-vocabulary. Built second so it's reported as a - # block after the verbatim checks, but the case names it - # attributes still feed the same ``overlapping`` set used by the - # score-line suffix. - case_index_colors: dict[str, list[str]] = {w: [] for w in _EVAL_VOCAB_COLORS} - case_index_shapes: dict[str, list[str]] = {w: [] for w in _EVAL_VOCAB_SHAPES} - for c in cases: - cname = c.get("name", "") - cc, cs = _case_fixture_vocab(c) - for w in cc: - case_index_colors[w].append(cname) - for w in cs: - case_index_shapes[w].append(cname) - - color_alt = "|".join(sorted(_EVAL_VOCAB_COLORS)) - shape_alt = "|".join(sorted(_EVAL_VOCAB_SHAPES)) - pair_re = re.compile(rf"\b({color_alt})\s+({shape_alt})\b", re.IGNORECASE) - color_re = re.compile(rf"\b({color_alt})\b", re.IGNORECASE) - shape_re = re.compile(rf"\b({shape_alt})\b", re.IGNORECASE) - - for start_line, block_text in _extract_example_blocks(sp): - seen_words: set[str] = set() - # Adjacent " " — the canonical violation shape. - for m in pair_re.finditer(block_text): - color = m.group(1).lower() - shape = m.group(2).lower() - offenders = sorted(set(case_index_colors.get(color, [])) - | set(case_index_shapes.get(shape, []))) - for case_name in offenders: - issues.append( - f" {case_name}: example block at line {start_line} " - f"uses '{color} {shape}' which also appears in case fixture" - ) - overlapping.add(case_name) - seen_words.add(color) - seen_words.add(shape) - # Lone colour or shape words not already counted in a pair. - for m in color_re.finditer(block_text): - w = m.group(1).lower() - if w in seen_words: - continue - seen_words.add(w) - for case_name in case_index_colors.get(w, []): - issues.append( - f" {case_name}: example block at line {start_line} " - f"uses '{w}' which also appears in case fixture" - ) - overlapping.add(case_name) - for m in shape_re.finditer(block_text): - w = m.group(1).lower() - if w in seen_words: - continue - seen_words.add(w) - for case_name in case_index_shapes.get(w, []): - issues.append( - f" {case_name}: example block at line {start_line} " - f"uses '{w}' which also appears in case fixture" - ) - overlapping.add(case_name) - - return overlapping, issues - - -async def main() -> None: - global AGENT_LLM, AGENT_MODEL, AGENT_KEY - - p = argparse.ArgumentParser() - p.add_argument("query", nargs="?", help="ad-hoc query (skips case suite)") - p.add_argument("--prompt", type=Path, default=SYS_PROMPT) - p.add_argument("--only", - help="comma-separated list of case names to run; all other " - "cases are skipped. Useful for fast iteration on a " - "single failing cluster. Mutually exclusive with the " - "positional `query` arg.") - p.add_argument("--thinking", action="store_true") - p.add_argument("--verbose", action="store_true") - p.add_argument("--strict-overlap", action="store_true", - help="fail (rc=2) if any case fixture overlaps with the " - "system prompt's worked examples — turn on in CI to " - "guard against silent train-on-test drift") - # agent-LLM endpoint overrides — default to whatever the worker yaml - # points at (local vLLM on 8107 in dev); set to point at - # build.nvidia.com etc. when scoring against a hosted model. - p.add_argument("--agent-llm", default=os.environ.get("AGENT_LLM_URL", AGENT_LLM), - help="full /v1/chat/completions URL for the agent LLM") - p.add_argument("--agent-model", default=os.environ.get("AGENT_LLM_MODEL", "llm"), - help="model name sent in the chat-completion request body") - p.add_argument("--agent-api-key", - default=(os.environ.get("NVIDIA_API_KEY", "") - or os.environ.get("NGC_API_KEY", "")), - help="Bearer token for the agent LLM " - "(env NVIDIA_API_KEY or NGC_API_KEY)") - args = p.parse_args() - - AGENT_LLM = args.agent_llm - AGENT_MODEL = args.agent_model - AGENT_KEY = args.agent_api_key - - if args.only and args.query: - p.error("--only and a positional query are mutually exclusive") - - # Honour a sibling .only file as a shorthand for --only (see - # eval/README.md "Watcher" section for the file format). - only_file = _HERE / ".only" - if not args.only and not args.query and only_file.exists(): - names: list[str] = [] - for raw in only_file.read_text(encoding="utf-8").splitlines(): - line = raw.split("#", 1)[0].strip() - if not line: - continue - for tok in line.split(","): - tok = tok.strip() - if tok: - names.append(tok) - if names: - args.only = ",".join(names) - print(f"FILTER: {only_file.name} → {names}") - - system_prompt = args.prompt.read_text(encoding="utf-8").strip() - print(f"PROMPT: {args.prompt} ({len(system_prompt)} chars)") - is_remote = not AGENT_LLM.lower().startswith(("http://localhost", "http://127.", "http://0.0.0.0")) - print(f"AGENT-LLM: {AGENT_LLM} model={AGENT_MODEL}" - + (" [remote, auth=on]" if is_remote and AGENT_KEY else "") - + (" [remote, auth=MISSING]" if is_remote and not AGENT_KEY else "") - + (" [local]" if not is_remote else "")) - - tools = await _discover_tools() - tool_names = [t["function"]["name"] for t in tools] - print(f"TOOLS: {tool_names}") - - pose = DEFAULT_POSE - if args.verbose: - print("POSE:", json.dumps(pose)) - - async with httpx.AsyncClient() as http: - if args.query: - r = await _run_one(http, system_prompt, tools, [], pose, - args.query, thinking=args.thinking) - print(json.dumps(r, indent=2)) - return - - cases = list(CASES) - if args.only: - requested = [n.strip() for n in args.only.split(",") if n.strip()] - valid = {c["name"] for c in cases} - unknown = [n for n in requested if n not in valid] - if unknown: - p.error(f"--only: unknown case name(s) {unknown}. " - f"Valid names: {sorted(valid)}") - cases = [c for c in cases if c["name"] in requested] - - # Audit: prompt worked-examples must not duplicate case fixtures. - # Warns at startup so overlaps don't turn the score into a - # memorization check. Run before any LLM calls. - overlap_names, overlap_issues = _check_prompt_eval_overlap( - system_prompt, cases - ) - if overlap_issues: - print("\n⚠ PROMPT/EVAL OVERLAP DETECTED — these cases share specifics with " - "system.txt and may be measuring memorization rather than " - "generalization. Fix by changing the prompt's worked example " - "(see AGENTS.md \"Prompt-driven samples\"):") - for line in overlap_issues: - print(line) - print() - if args.strict_overlap: - print(f"--strict-overlap set: aborting with rc=2 " - f"({len(overlap_names)} overlapping case(s))", - file=sys.stderr) - sys.exit(2) - else: - print("PROMPT/EVAL OVERLAP: clean (no verbatim utterances, coords, or " - "reserved-vocab leaks)") - - results = [] - for c in cases: - scene_c = c["scene"] - pose_c = c.get("pose", pose) - _set_case_history(c.get("history")) - _set_case_moves(c.get("recent_moves")) - try: - r = await _run_one(http, system_prompt, tools, scene_c, pose_c, - c["user"], thinking=args.thinking, - max_steps=_MAX_STEPS) - except Exception as exc: - r = {"latency_s": 0.0, "tool_calls": [], "content": "", - "reasoning": ""} - ok, why = False, f"network error: {type(exc).__name__}: {exc}" - else: - ok, why = _check(r, c) - mark = "✓" if ok else "✗" - print(f"{mark} {c['name']:32s} {r['latency_s']:5.1f}s {why}") - for i, tc in enumerate(r["tool_calls"]): - fn = tc["function"] - print(f" [{i}] {fn['name']}({fn['arguments']})") - results.append((c["name"], ok)) - - passed = sum(1 for _, ok in results if ok) - total = len(results) - score_line = f"\n{passed}/{total} passed" - if overlap_names: - score_line += ( - f" ({len(overlap_names)}/{total} too close to prompts — " - f"may be memorization, not generalization)" - ) - print(score_line) - sys.exit(0 if passed == total else 1) - - -if __name__ == "__main__": - asyncio.run(main()) diff --git a/agent-samples/xr-render-demo/eval/xr_render_demo_eval/harness.py b/agent-samples/xr-render-demo/eval/xr_render_demo_eval/harness.py new file mode 100644 index 00000000..98802738 --- /dev/null +++ b/agent-samples/xr-render-demo/eval/xr_render_demo_eval/harness.py @@ -0,0 +1,983 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Run the real native-NAT render workflow against an in-memory XR scene.""" + +import argparse +import asyncio +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +from nat.builder.workflow_builder import WorkflowBuilder +from nat.plugin_api import Builder, FunctionGroup, FunctionGroupBaseConfig, register_function_group +from pydantic import ConfigDict, Field +from xr_ai_models import load_models_config, make_llm +from xr_ai_nat.functions.spatial_math import SpatialMathFunctionsConfig +from xr_ai_nat.functions.text_memory import ConversationEntry, RecallConversationRequest, RecallConversationResult +from xr_ai_nat.functions.types import SpatialFrame, Vector3 +from xr_ai_nat.functions.vision import HistoricalVisionRequest, LiveVisionRequest, LiveVisionResult +from xr_ai_nat.functions.xr_tracking import HeadPose, HeadPoseRequest +from xr_render_demo_worker.config import load_config +from xr_render_demo_worker.models import SceneRequest +from xr_render_demo_worker.scene import SceneContext +from xr_render_demo_worker.supervisor import scene_supervisor +from xr_render_scene import ( + AddPrimitiveRequest, + AddPrimitiveResult, + EmptyRequest, + MutationResult, + RemovePrimitiveRequest, + SceneObject, + SceneState, + UpdatePrimitiveRequest, +) + +from .cases import CASES as CORPUS_CASES + +_HERE = Path(__file__).resolve().parent +_CONFIG = load_config((_HERE / "../../yaml/xr_render_demo_worker.yaml").resolve()) +_PARTICIPANT = "eval-user" + +_DEFAULT_POSE = { + "is_valid": True, + "position": {"x": 0.0, "y": 1.6, "z": 0.0}, + "forward": {"x": 0.0, "y": 0.0, "z": -1.0}, + "right": {"x": 1.0, "y": 0.0, "z": 0.0}, + "up": {"x": 0.0, "y": 1.0, "z": 0.0}, + "yaw_deg": 0.0, + "pitch_deg": 0.0, + "ts": 1, +} + + +class _EvalSceneStateConfig(FunctionGroupBaseConfig, name="xr_render_eval_scene_state"): + model_config = ConfigDict(arbitrary_types_allowed=True) + scene: Any = Field(exclude=True, repr=False) + + +@register_function_group(config_type=_EvalSceneStateConfig) +async def _eval_scene_state(config: _EvalSceneStateConfig, _builder: Builder): + group = FunctionGroup(config=config) + group.add_function( + "get_scene_state", + config.scene.get_scene_state, + description="Return the scene.", + ) + yield group + + +class _EvalSceneUpdatesConfig(FunctionGroupBaseConfig, name="xr_render_eval_scene_updates"): + model_config = ConfigDict(arbitrary_types_allowed=True) + scene: Any = Field(exclude=True, repr=False) + + +@register_function_group(config_type=_EvalSceneUpdatesConfig) +async def _eval_scene_updates(config: _EvalSceneUpdatesConfig, _builder: Builder): + group = FunctionGroup(config=config) + group.add_function( + "update_primitive", + config.scene.update_primitive, + description="Update an object.", + ) + yield group + + +class _EvalSceneObjectsConfig(FunctionGroupBaseConfig, name="xr_render_eval_scene_objects"): + model_config = ConfigDict(arbitrary_types_allowed=True) + scene: Any = Field(exclude=True, repr=False) + + +@register_function_group(config_type=_EvalSceneObjectsConfig) +async def _eval_scene_objects(config: _EvalSceneObjectsConfig, _builder: Builder): + group = FunctionGroup(config=config) + group.add_function("add_primitive", config.scene.add_primitive, description="Add an object.") + group.add_function("remove_primitive", config.scene.remove_primitive, description="Remove an object.") + yield group + + +class _EvalTrackingConfig(FunctionGroupBaseConfig, name="xr_render_eval_tracking"): + model_config = ConfigDict(arbitrary_types_allowed=True) + tracking: Any = Field(exclude=True, repr=False) + + +@register_function_group(config_type=_EvalTrackingConfig) +async def _eval_tracking(config: _EvalTrackingConfig, _builder: Builder): + group = FunctionGroup(config=config) + group.add_function( + "get_user_frame", + config.tracking.get_user_frame, + description="Return a test user frame.", + ) + yield group + + +class _EvalVisionConfig(FunctionGroupBaseConfig, name="xr_render_eval_vision"): + model_config = ConfigDict(arbitrary_types_allowed=True) + scene: Any = Field(exclude=True, repr=False) + + +@register_function_group(config_type=_EvalVisionConfig) +async def _eval_vision(config: _EvalVisionConfig, _builder: Builder): + async def look(request: LiveVisionRequest) -> LiveVisionResult: + config.scene.calls.append(("look_at_current_frame", request.model_dump())) + if config.scene.vision_error: + raise RuntimeError(config.scene.vision_error) + # A real VLM always says something; an empty fixture must not derail + # the rollout, only decline to reward the spurious look. + return LiveVisionResult(answer=config.scene.vision_answer or "Nothing notable is visible.") + + async def look_past(request: HistoricalVisionRequest) -> LiveVisionResult: + config.scene.calls.append(("look_at_past_frame", request.model_dump())) + if config.scene.vision_error: + raise RuntimeError(config.scene.vision_error) + return LiveVisionResult(answer=config.scene.vision_answer or "Nothing notable is visible.") + + group = FunctionGroup(config=config) + group.add_function( + "look_at_current_frame", + look, + description="Observe the user's live camera.", + ) + group.add_function( + "look_at_past_frame", + look_past, + description="Observe a recorded camera frame.", + ) + yield group + + +class _EvalConversationConfig(FunctionGroupBaseConfig, name="xr_render_eval_conversations"): + model_config = ConfigDict(arbitrary_types_allowed=True) + scene: Any = Field(exclude=True, repr=False) + + +@register_function_group(config_type=_EvalConversationConfig) +async def _eval_conversations(config: _EvalConversationConfig, _builder: Builder): + async def recall(request: RecallConversationRequest) -> RecallConversationResult: + config.scene.calls.append(("recall_conversation", request.model_dump())) + entries = [] + for index, (user_text, agent_text) in enumerate(config.scene.history): + entries.append(ConversationEntry(timestamp_us=(index + 1) * 1_000_000, role="user", text=user_text)) + entries.append(ConversationEntry(timestamp_us=(index + 1) * 1_000_000 + 1, role="agent", text=agent_text)) + if config.scene.memory_answer: + entries.append(ConversationEntry(timestamp_us=1, role="agent", text=config.scene.memory_answer)) + return RecallConversationResult(entries=entries) + + group = FunctionGroup(config=config) + group.add_function( + "recall_conversation", + recall, + description="Recall older conversation.", + ) + yield group + + +@dataclass(frozen=True) +class Case: + name: str + request: str + scene: tuple[dict[str, Any], ...] = () + pose: HeadPose | None = None + vision: str = "" + vision_error: str = "" + required_tools: frozenset[str] = frozenset() + forbidden_tools: frozenset[str] = frozenset() + required_order: tuple[str, ...] = () + expected_call_counts: tuple[tuple[str, int], ...] = () + expected_sizes: tuple[tuple[str, float], ...] = () + expected_colors: tuple[tuple[str, tuple[float, float, float]], ...] = () + expected_positions: tuple[tuple[str, tuple[float, float, float]], ...] = () + memory: str = "" + history: tuple[tuple[str, str], ...] = () + + +CASES = ( + Case( + name="create_object", + request="Add a blue sphere.", + required_tools=frozenset({"add_primitive"}), + expected_colors=(("sphere-0", (0.0, 0.4, 1.0)),), + ), + Case( + name="delete_object", + request="Remove the white cylinder.", + scene=( + { + "id": "cylinder-0", + "type": "cylinder", + "position": {"x": -0.3, "y": 1.3, "z": -1.2}, + "color": {"r": 1, "g": 1, "b": 1}, + "size": 0.15, + }, + ), + required_tools=frozenset({"remove_primitive"}), + ), + Case( + name="place_existing_object", + request="Move the blue box to my left.", + scene=( + { + "id": "box-0", + "type": "box", + "position": {"x": 0, "y": 1.6, "z": -1.5}, + "color": {"r": 0, "g": 0.4, "b": 1}, + "size": 0.1, + }, + ), + required_tools=frozenset({"get_user_frame", "update_primitive"}), + required_order=("get_user_frame", "update_primitive"), + ), + Case( + name="vision_then_appearance", + request="Make the cone the color of the wall.", + scene=( + { + "id": "cone-0", + "type": "cone", + "position": {"x": 0, "y": 1.5, "z": -1.4}, + "color": {"r": 0.2, "g": 0.9, "b": 1}, + "size": 0.1, + }, + ), + vision="The wall is orange: normalized RGB (1.0, 0.5, 0.0).", + required_tools=frozenset({"look_at_current_frame", "update_primitive"}), + ), + Case( + name="compound_object_and_placement", + request="Create a green box and place it one metre to my right.", + required_tools=frozenset({"add_primitive", "get_user_frame"}), + forbidden_tools=frozenset({"update_primitive"}), + required_order=("get_user_frame", "add_primitive"), + expected_positions=(("box-0", (1.0, 1.6, 0.0)),), + ), + Case( + name="create_in_front_of_moved_user", + request="Add a yellow cone ahead of me.", + pose=HeadPose( + is_valid=True, + position=Vector3(x=2.0, y=1.6, z=1.5), + forward=Vector3(x=0, y=0, z=-1), + right=Vector3(x=1, y=0, z=0), + up=Vector3(x=0, y=1, z=0), + yaw_deg=0, + pitch_deg=0, + ts=1, + ), + required_tools=frozenset({"add_primitive", "get_user_frame"}), + forbidden_tools=frozenset({"update_primitive"}), + required_order=("get_user_frame", "add_primitive"), + expected_positions=(("cone-0", (2.0, 1.6, 0.0)),), + ), + Case( + name="resize_existing_object", + request="Make the pyramid 1.75 times larger.", + scene=( + { + "id": "pyramid-0", + "type": "pyramid", + "position": {"x": 0.2, "y": 1.2, "z": -1.1}, + "color": {"r": 0.4, "g": 0.4, "b": 0.4}, + "size": 0.2, + }, + ), + required_tools=frozenset({"update_primitive"}), + expected_sizes=(("pyramid-0", 0.35),), + ), + Case( + name="historical_vision", + request="What color was the object I held ten seconds ago?", + vision="The previously held object was purple.", + required_tools=frozenset({"look_at_past_frame"}), + ), + Case( + name="durable_memory", + request="What object did we discuss in the earlier session?", + memory="We discussed a small cyan sphere.", + required_tools=frozenset({"recall_conversation"}), + ), + Case( + name="placement_despite_camera_off", + request="Put a red sphere two meters ahead of me.", + vision_error="No current camera frame is available.", + required_tools=frozenset({"add_primitive"}), + ), + Case( + name="camera_meta_comment", + request="No, I meant the webcam discussion, not a request.", + forbidden_tools=frozenset({"look_at_current_frame", "look_at_past_frame"}), + ), + Case( + name="unavailable_live_camera", + request="Read the maker's mark on the cup in front of me.", + vision_error="No current camera frame is available.", + required_tools=frozenset({"look_at_current_frame"}), + forbidden_tools=frozenset({"look_at_past_frame"}), + expected_call_counts=(("look_at_current_frame", 1),), + ), +) + + +def _scene_object(item: dict[str, Any]) -> SceneObject: + x, y, z = item["pos"] + r, g, b = item["color"] + return SceneObject.model_validate( + { + "id": item["id"], + "type": item["type"], + "position": {"x": x, "y": y, "z": z}, + "color": {"r": r, "g": g, "b": b}, + "size": item["size"], + } + ) + + +@dataclass +class FakeScene: + objects: dict[str, SceneObject] + pose: HeadPose + vision_answer: str + vision_error: str + memory_answer: str + history: tuple[tuple[str, str], ...] = () + calls: list[tuple[str, dict[str, Any]]] = field(default_factory=list) + counters: dict[str, int] = field(default_factory=dict) + + @classmethod + def from_case(cls, case: Case) -> "FakeScene": + objects = [SceneObject.model_validate(item) for item in case.scene] + pose = case.pose or HeadPose.model_validate(_DEFAULT_POSE) + return cls( + {item.id: item for item in objects}, + pose, + case.vision, + case.vision_error, + case.memory, + case.history, + ) + + @classmethod + def from_corpus_case(cls, case: dict[str, Any]) -> "FakeScene": + objects = [_scene_object(item) for item in case.get("scene", ())] + pose = HeadPose.model_validate({**_DEFAULT_POSE, **(case.get("pose") or {})}) + return cls( + {item.id: item for item in objects}, + pose, + case.get("vlm_answer", ""), + "", + case.get("memory", ""), + tuple(case.get("history", ())), + ) + + async def add_primitive(self, request: AddPrimitiveRequest) -> AddPrimitiveResult: + arguments = request.model_dump() + self.calls.append(("add_primitive", arguments)) + # Mirror the engine's per-type monotonic counters; length-based ids + # collide after removals. + for item in self.objects.values(): + kind, _, index = item.id.rpartition("-") + if kind and index.isdigit(): + self.counters[kind] = max(self.counters.get(kind, 0), int(index) + 1) + number = self.counters.get(request.prim_type, 0) + self.counters[request.prim_type] = number + 1 + object_id = f"{request.prim_type}-{number}" + self.objects[object_id] = SceneObject.model_validate( + { + "id": object_id, + "type": request.prim_type, + "position": {"x": request.x, "y": request.y, "z": request.z}, + "color": {"r": request.r, "g": request.g, "b": request.b}, + "size": request.size, + } + ) + return AddPrimitiveResult(id=object_id, ok=True) + + async def update_primitive(self, request: UpdatePrimitiveRequest) -> MutationResult: + arguments = request.model_dump(exclude_none=True) + if request.obj_id not in self.objects: + self.calls.append(("update_primitive", arguments)) + return MutationResult(ok=False, reason=f"no object {request.obj_id!r}") + self.calls.append(("update_primitive", arguments)) + current = self.objects[request.obj_id].model_dump() + for field_name in ("x", "y", "z"): + if field_name in arguments: + current["position"][field_name] = arguments[field_name] + for field_name in ("r", "g", "b"): + if field_name in arguments: + current["color"][field_name] = arguments[field_name] + if request.prim_type is not None: + current["type"] = request.prim_type + if request.size is not None: + current["size"] = request.size + self.objects[request.obj_id] = SceneObject.model_validate(current) + return MutationResult(ok=True) + + async def remove_primitive(self, request: RemovePrimitiveRequest) -> MutationResult: + self.calls.append(("remove_primitive", request.model_dump())) + self.objects.pop(request.obj_id, None) + return MutationResult(ok=True) + + async def get_scene_state(self, request: EmptyRequest) -> SceneState: + self.calls.append(("get_scene_state", {})) + return SceneState(objects=list(self.objects.values())) + + async def bind(self, builder: Builder) -> None: + scene = self + + class Tracking: + async def get_user_frame(self, request: HeadPoseRequest) -> SpatialFrame: + scene.calls.append(("get_user_frame", {})) + return SpatialFrame( + origin=scene.pose.position, + forward=scene.pose.forward, + right=scene.pose.right, + up=scene.pose.up, + ) + + await builder.add_function_group("scene_state", _EvalSceneStateConfig(scene=self)) + await builder.add_function_group("scene_updates", _EvalSceneUpdatesConfig(scene=self)) + await builder.add_function_group("scene_objects", _EvalSceneObjectsConfig(scene=self)) + await builder.add_function_group("tracking", _EvalTrackingConfig(tracking=Tracking())) + await builder.add_function_group("spatial", SpatialMathFunctionsConfig()) + await builder.add_function_group("vision", _EvalVisionConfig(scene=self)) + await builder.add_function_group("conversations", _EvalConversationConfig(scene=self)) + + +_MUTATING = frozenset({"add_primitive", "update_primitive", "remove_primitive"}) + +_SCENE_ARG_LOOKUP = { + "x": ("pos", 0), + "y": ("pos", 1), + "z": ("pos", 2), + "r": ("color", 0), + "g": ("color", 1), + "b": ("color", 2), + "size": ("size", None), + "prim_type": ("type", None), +} + + +def _resolve_arg(obj_id: str | None, key: str, scene: list[dict[str, Any]]) -> Any: + lookup = _SCENE_ARG_LOOKUP.get(key) + item = next((entry for entry in scene if entry.get("id") == obj_id), None) + if lookup is None or item is None: + return None + source, index = lookup + value = item.get(source) + if index is None or value is None: + return value + return value[index] if index < len(value) else None + + +def _match_call(name: str, args: dict[str, Any], expect: dict[str, Any], scene: list[dict[str, Any]]) -> bool: + if name != expect["tool"]: + return False + for key, want in expect.get("args", {}).items(): + got = args.get(key) + # An update that omits a field keeps the object's initial value. + if got is None and name == "update_primitive": + got = _resolve_arg(args.get("obj_id"), key, scene) + if got is None: + return False + if isinstance(want, tuple): + try: + got = float(got) + except (TypeError, ValueError): + return False + low, high = want + if not (low <= got <= high): + return False + elif got != want: + return False + return True + + +def check_corpus(calls: list[tuple[str, dict[str, Any]]], case: dict[str, Any]) -> tuple[bool, str]: + """Score one corpus rollout: perception gating, then order-independent + mutation matching with the duplicate-add rule.""" + names = [name for name, _args in calls] + mutations = [(name, args) for name, args in calls if name in _MUTATING] + if first_tool := case.get("must_call_first"): + if first_tool not in names: + return False, f"{first_tool} was never called" + first_mutation = next((index for index, name in enumerate(names) if name in _MUTATING), None) + if first_mutation is not None and names.index(first_tool) > first_mutation: + return False, f"{first_tool} called after the first mutation" + wanted = list(case.get("result") or ()) + if not wanted and not mutations: + return False, f"no mutating calls: {names}" + expected_adds = sum(1 for expect in wanted if expect["tool"] == "add_primitive") + actual_adds = sum(1 for name, _args in mutations if name == "add_primitive") + if wanted and actual_adds > expected_adds: + return False, f"duplicate add: {actual_adds} add_primitive calls for {expected_adds} expected" + scene = list(case.get("scene", ())) + remaining = list(mutations) + unmatched = [] + for expect in wanted: + for index, (name, args) in enumerate(remaining): + if _match_call(name, args, expect, scene): + remaining.pop(index) + break + else: + unmatched.append(expect) + if unmatched: + wanted_desc = "; ".join(f"{expect['tool']}({expect.get('args', {})})" for expect in unmatched) + actual = [f"{name}({args})" for name, args in mutations] + return False, f"unmatched: {wanted_desc} | actual mutations: {actual} | calls: {names}" + if not case.get("ignore_extra", True) and remaining: + return False, f"extra mutating calls: {[name for name, _args in remaining]}" + if (predicate := case.get("predicate")) is not None: + ok, message = predicate(mutations) + if not ok: + return False, f"predicate failed: {message}" + return True, "ok" + + +async def run_corpus_case(case: dict[str, Any]) -> bool: + scene = FakeScene.from_corpus_case(case) + llm = make_llm(load_models_config(_CONFIG.models_yaml), "agent_llm") + try: + async with WorkflowBuilder() as builder: + await scene.bind(builder) + scene_group = await builder.get_function_group("scene_state") + scene_functions = await scene_group.get_all_functions() + tracking_group = await builder.get_function_group("tracking") + tracking_functions = await tracking_group.get_all_functions() + context = SceneContext( + scene_functions["scene_state__get_scene_state"], + tracking_functions["tracking__get_user_frame"], + ) + context._recent_moves[_PARTICIPANT] = [ + f"{object_id}: previously at {before}, now at {after}" + for object_id, before, after in case.get("recent_moves", ()) + ] + supervisor = await scene_supervisor( + builder=builder, + llm=llm, + context=context, + ) + # A crashed rollout is a case result, not an eval abort: score + # whatever calls it made before failing. + try: + reply = await supervisor.ainvoke( + SceneRequest( + transcript=case["user"], + participant_id=_PARTICIPANT, + timestamp_us=10_000_000, + ) + ) + response = reply.response + except Exception as exc: + response = f"" + finally: + await llm.close() + ok, why = check_corpus(scene.calls, case) + status = "PASS" if ok else f"FAIL {why}" + print(f"{status:32} {case['name']}: {response}", flush=True) + return ok + + +# The basics battery: the most common utterances, their observed live STT +# corruptions, and history-bearing variants. Run after EVERY prompt or ops +# change (`uv run xr_render_demo_eval basics`); prompt edits keep breaking exactly these through +# example contamination, and the full corpus hides one-case damage inside its +# run-to-run variance. Utterances must stay disjoint from prompt examples. +_BASICS_SCENE = ( + {"id": "box-0", "type": "box", "position": {"x": 0.6, "y": 1.3, "z": -1.1}, + "color": {"r": 0, "g": 1, "b": 1}, "size": 0.1}, + {"id": "sphere-1", "type": "sphere", "position": {"x": -0.5, "y": 1.5, "z": -1.3}, + "color": {"r": 0, "g": 0.8, "b": 0}, "size": 0.1}, +) +_BASICS_HISTORY = ( + ("Add a cyan cube.", "Added a cyan cube."), + ("Make a green sphere.", "Created a green sphere."), +) +_NO_MUTATION = frozenset({"add_primitive", "update_primitive", "remove_primitive"}) + +BASICS = ( + Case( + name="basics_create_cube", + request="Make a red cube.", + required_tools=frozenset({"add_primitive"}), + expected_colors=(("box-0", (1.0, 0.0, 0.0)),), + expected_positions=(("box-0", (0.0, 1.6, -1.5)),), + ), + Case( + name="basics_create_sphere", + request="Add a blue sphere.", + required_tools=frozenset({"add_primitive"}), + expected_colors=(("sphere-0", (0.0, 0.4, 1.0)),), + expected_positions=(("sphere-0", (0.0, 1.6, -1.5)),), + ), + Case( + name="basics_garbled_shape", + request="Make a blue spear.", + required_tools=frozenset({"add_primitive"}), + expected_colors=(("sphere-0", (0.0, 0.4, 1.0)),), + expected_positions=(("sphere-0", (0.0, 1.6, -1.5)),), + ), + Case( + name="basics_garbled_shape_cute", + request="Add a red cute.", + required_tools=frozenset({"add_primitive"}), + expected_colors=(("box-0", (1.0, 0.0, 0.0)),), + ), + Case( + name="basics_garbled_color", + request="Add a blew sphere.", + required_tools=frozenset({"add_primitive"}), + expected_colors=(("sphere-0", (0.0, 0.4, 1.0)),), + ), + Case( + name="basics_stated_distance", + request="Create a purple sphere two meters ahead of me.", + required_tools=frozenset({"add_primitive"}), + expected_positions=(("sphere-0", (0.0, 1.6, -2.0)),), + ), + Case( + name="basics_create_cube_with_history", + request="Make a red cube.", + scene=_BASICS_SCENE, + history=_BASICS_HISTORY, + required_tools=frozenset({"add_primitive"}), + forbidden_tools=frozenset({"update_primitive", "remove_primitive"}), + expected_colors=(("box-1", (1.0, 0.0, 0.0)),), + expected_positions=(("box-1", (0.0, 1.6, -1.5)),), + ), + Case( + name="basics_garbled_shape_with_history", + request="Make a blue spear.", + scene=_BASICS_SCENE, + history=_BASICS_HISTORY, + required_tools=frozenset({"add_primitive"}), + forbidden_tools=frozenset({"update_primitive", "remove_primitive"}), + expected_colors=(("sphere-2", (0.0, 0.4, 1.0)),), + ), + Case( + name="basics_recolor_with_history", + request="Make the sphere red.", + scene=_BASICS_SCENE, + history=_BASICS_HISTORY, + required_tools=frozenset({"update_primitive"}), + forbidden_tools=frozenset({"add_primitive", "remove_primitive"}), + expected_colors=(("sphere-1", (1.0, 0.0, 0.0)),), + ), + Case( + name="basics_move_with_history", + request="Move the cube to my left.", + scene=_BASICS_SCENE, + history=_BASICS_HISTORY, + required_tools=frozenset({"update_primitive"}), + forbidden_tools=frozenset({"add_primitive", "remove_primitive"}), + ), + Case( + name="basics_remove_with_history", + request="Remove the cube.", + scene=_BASICS_SCENE, + history=_BASICS_HISTORY, + required_tools=frozenset({"remove_primitive"}), + forbidden_tools=frozenset({"add_primitive"}), + ), + Case( + name="basics_resize_with_history", + request="Double the size of the sphere.", + scene=_BASICS_SCENE, + history=_BASICS_HISTORY, + required_tools=frozenset({"update_primitive"}), + forbidden_tools=frozenset({"add_primitive", "remove_primitive"}), + expected_sizes=(("sphere-1", 0.2),), + ), + # Systematic perturbation classes, not observed-corruption harvesting: + # dropped words, number homophones, filler injection, run-on merges. + Case( + name="basics_dropped_color", + request="Make a, uh, sphere.", + required_tools=frozenset({"add_primitive"}), + expected_positions=(("sphere-0", (0.0, 1.6, -1.5)),), + ), + Case( + name="basics_number_homophone", + request="Create an orange sphere too meters ahead of me.", + required_tools=frozenset({"add_primitive"}), + expected_positions=(("sphere-0", (0.0, 1.6, -2.0)),), + ), + Case( + name="basics_filler_heavy_create", + request="Um, could you like, make a purple cube, please.", + required_tools=frozenset({"add_primitive"}), + expected_colors=(("box-0", (0.6, 0.0, 1.0)),), + ), + Case( + name="basics_run_on_two_commands", + request="Make a cyan sphere move it up a little.", + required_tools=frozenset({"add_primitive", "update_primitive"}), + forbidden_tools=frozenset({"remove_primitive"}), + expected_call_counts=(("add_primitive", 1),), + ), + Case( + name="basics_dropped_preposition_move", + request="Put the sphere the box.", + scene=_BASICS_SCENE, + history=_BASICS_HISTORY, + forbidden_tools=frozenset({"add_primitive", "remove_primitive"}), + ), + Case( + name="basics_anchored_create_keeps_words", + # Live flail: the supervisor misroutes an anchored create to + # placement, gets it bounced, then re-delegates WITHOUT the user's + # spatial words and patches with a move. One add, no update, and the + # cube must land on the user's left of the anchor. + request="Put a yellow cube to the left of the blue cube.", + scene=( + {"id": "box-0", "type": "box", "position": {"x": 0.5, "y": 1.4, "z": -1.2}, + "color": {"r": 0, "g": 0, "b": 1}, "size": 0.1}, + ), + history=_BASICS_HISTORY, + required_tools=frozenset({"add_primitive"}), + forbidden_tools=frozenset({"update_primitive", "remove_primitive"}), + expected_call_counts=(("add_primitive", 1),), + expected_colors=(("box-1", (1.0, 1.0, 0.0)),), + ), + Case( + name="basics_truncated_command", + # VAD cuts the sentence mid-word; the only correct outcome is a + # question. The failure mode is the verification pass nudging the + # model into inventing the missing destination. + request="Put the sphere on the", + scene=_BASICS_SCENE, + history=_BASICS_HISTORY, + forbidden_tools=_NO_MUTATION, + ), + Case( + name="basics_truncated_then_completed", + # Follow-through: the user answers the truncation ask-back and the + # deferred action completes against the conversation context. + request="On the box.", + scene=_BASICS_SCENE, + history=( + *_BASICS_HISTORY, + ("Put the sphere on the", "I think I missed the end of that. On the what?"), + ), + required_tools=frozenset({"update_primitive"}), + forbidden_tools=frozenset({"add_primitive", "remove_primitive"}), + ), + Case( + name="basics_truncated_then_nevermind", + request="Never mind.", + scene=_BASICS_SCENE, + history=( + *_BASICS_HISTORY, + ("Put the sphere on the", "I think I missed the end of that. On the what?"), + ), + forbidden_tools=_NO_MUTATION, + ), + Case( + name="basics_truncated_then_full_command", + request="Put the sphere on the box.", + scene=_BASICS_SCENE, + history=( + *_BASICS_HISTORY, + ("Put the sphere on the", "I think I missed the end of that. On the what?"), + ), + required_tools=frozenset({"update_primitive"}), + forbidden_tools=frozenset({"add_primitive", "remove_primitive"}), + expected_call_counts=(("update_primitive", 1),), + ), + Case( + name="basics_truncated_then_bare_object", + request="The box.", + scene=_BASICS_SCENE, + history=( + *_BASICS_HISTORY, + ("Put the sphere on the", "I think I missed the end of that. On the what?"), + ), + required_tools=frozenset({"update_primitive"}), + forbidden_tools=frozenset({"add_primitive", "remove_primitive"}), + ), + Case( + name="basics_fragment_with_history", + request="Sounds good.", + scene=_BASICS_SCENE, + history=_BASICS_HISTORY, + forbidden_tools=_NO_MUTATION, + ), + Case( + name="basics_correction_with_history", + request="That's the wrong sphere.", + scene=_BASICS_SCENE, + history=_BASICS_HISTORY, + forbidden_tools=_NO_MUTATION, + ), +) + + +async def run_case(case: Case) -> bool: + scene = FakeScene.from_case(case) + llm = make_llm(load_models_config(_CONFIG.models_yaml), "agent_llm") + try: + async with WorkflowBuilder() as builder: + await scene.bind(builder) + supervisor = await scene_supervisor( + builder=builder, + llm=llm, + ) + try: + reply = await supervisor.ainvoke( + SceneRequest( + transcript=case.request, + participant_id=_PARTICIPANT, + timestamp_us=10_000_000, + ) + ) + response = reply.response + except Exception as exc: + response = f"" + finally: + await llm.close() + called = {name for name, _arguments in scene.calls} + missing = case.required_tools - called + forbidden = case.forbidden_tools & called + call_order = tuple(name for name, _arguments in scene.calls) + wrong_call_counts = { + name: call_order.count(name) + for name, expected in case.expected_call_counts + if call_order.count(name) != expected + } + positions = [call_order.index(name) for name in case.required_order if name in call_order] + out_of_order = len(positions) != len(case.required_order) or positions != sorted(positions) + wrong_sizes = { + object_id: scene.objects[object_id].size if object_id in scene.objects else None + for object_id, expected in case.expected_sizes + if object_id not in scene.objects or scene.objects[object_id].size != expected + } + wrong_colors = { + object_id: ( + tuple(scene.objects[object_id].color.model_dump().values()) + if object_id in scene.objects + else None + ) + for object_id, expected in case.expected_colors + if object_id not in scene.objects + or tuple(scene.objects[object_id].color.model_dump().values()) != expected + } + wrong_positions = { + object_id: ( + tuple(scene.objects[object_id].position.model_dump().values()) + if object_id in scene.objects + else None + ) + for object_id, expected in case.expected_positions + if object_id not in scene.objects + or tuple(scene.objects[object_id].position.model_dump().values()) != expected + } + passed = ( + not missing + and not forbidden + and not out_of_order + and not wrong_call_counts + and not wrong_sizes + and not wrong_colors + and not wrong_positions + ) + status = ( + "PASS" + if passed + else ( + f"FAIL missing={sorted(missing)} forbidden={sorted(forbidden)} order={call_order} " + f"counts={wrong_call_counts} sizes={wrong_sizes} colors={wrong_colors} " + f"positions={wrong_positions}" + ) + ) + print(f"{status:32} {case.name}: {response}", flush=True) + return passed + + +# Train/test separation audit. Prompt worked examples are what the model +# memorizes as templates; any overlap with case inputs means the eval scores +# recall, not behavior (proven live: a blue/green prompt example made the +# anchored-create case pass while the same request failed on other colors). +_EVAL_VOCAB_COLORS = ("red", "green", "blue", "yellow", "cyan", "orange", "purple", "white", "black") +_EVAL_VOCAB_SHAPES = ("sphere", "cube", "box", "ball") + + +def audit_prompts() -> None: + import re + from pathlib import Path + + worker = (Path(__file__).resolve().parent / "../../worker/xr_render_demo_worker").resolve() + prompts = sorted(worker.rglob("*prompt*.txt")) + utterances = [case["user"] for case in CORPUS_CASES if case.get("user")] + utterances += [case.request for case in (*CASES, *BASICS)] + fixture_ids = { + item["id"] + for case in CORPUS_CASES + for item in case.get("scene", ()) + } | {item["id"] for case in (*CASES, *BASICS) for item in case.scene} + # The other tiers score the same prompts; their inputs must stay + # disjoint from prompt examples too. + try: + from . import subagents as eval_subagents + from . import supervisor as eval_supervisor + except ImportError: + pass + else: + utterances += [case.instruction for case in eval_subagents.CASES] + utterances += [case.request for case in eval_supervisor.CASES] + fixture_ids |= {item["id"] for case in eval_subagents.CASES for item in case.scene} + fixture_ids |= {item["id"] for case in eval_supervisor.CASES for item in case.scene} + for prompt_path in prompts: + label = str(prompt_path.relative_to(worker)) + text = prompt_path.read_text(encoding="utf-8") + lowered = text.lower() + for utterance in utterances: + needle = utterance.lower().strip(".?! ") + # Short fragments ("the box") appear in ordinary prose; only + # utterances long enough to be templates count as overlap. + if len(needle) >= 12 and needle in lowered: + print(f"AUDIT WARNING {label}: contains case utterance {utterance!r}") + for fixture_id in sorted(fixture_ids): + if fixture_id.lower() in lowered: + print(f"AUDIT WARNING {label}: contains case fixture id {fixture_id!r}") + for quoted in re.findall(r'"([^"]{4,80})"', text): + words = set(re.findall(r"[a-z]+", quoted.lower())) + colors = words & set(_EVAL_VOCAB_COLORS) + shapes = words & set(_EVAL_VOCAB_SHAPES) + if colors and shapes: + print( + f"AUDIT WARNING {label}: example {quoted!r} pairs eval vocabulary " + f"({', '.join(sorted(colors | shapes))}); use non-eval colors/shapes" + ) + + +async def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("cases", nargs="*", help="Case names; omit to run all cases") + args = parser.parse_args() + audit_prompts() + wanted = set(args.cases) + if wanted == {"basics"}: + wanted = {case.name for case in BASICS} + corpus = [case for case in CORPUS_CASES if not wanted or case["name"] in wanted] + native = [case for case in CASES if not wanted or case.name in wanted] + basics = [case for case in BASICS if not wanted or case.name in wanted] + if not corpus and not native and not basics: + raise SystemExit(f"unknown cases: {args.cases}") + corpus_results = [await run_corpus_case(case) for case in corpus] + native_results = [await run_case(case) for case in native] + basics_results = [await run_case(case) for case in basics] + if corpus_results: + print(f"corpus: {sum(corpus_results)}/{len(corpus_results)} passed") + if native_results: + print(f"native: {sum(native_results)}/{len(native_results)} passed") + if basics_results: + print(f"basics: {sum(basics_results)}/{len(basics_results)} passed") + if not all(corpus_results + native_results + basics_results): + raise SystemExit(1) + + +def run() -> None: + asyncio.run(main()) + + +if __name__ == "__main__": + run() diff --git a/agent-samples/xr-render-demo/eval/xr_render_demo_eval/live_explore.py b/agent-samples/xr-render-demo/eval/xr_render_demo_eval/live_explore.py new file mode 100644 index 00000000..76d744ef --- /dev/null +++ b/agent-samples/xr-render-demo/eval/xr_render_demo_eval/live_explore.py @@ -0,0 +1,160 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Exploratory live driver: novel user-style utterances scored by invariants. + +Unlike the case tiers, nothing here asserts an exact outcome. Each utterance +carries an intent class, and the scene diff after the turn is checked against +that class's invariants: + + create at least one new object; nothing pre-existing moved or removed + mutate at least one pre-existing object changed; nothing added or removed + remove at least one pre-existing object gone; nothing added + none no scene change at all (questions, comments, courtesies) + +Utterances are deliberately phrased unlike both the case corpus and the +prompt examples; when a turn violates its invariant, promote the utterance +into the appropriate tier as a permanent case, then fix. + +Run subsets: `live_explore.py [utterance-index ...]`. +""" + +import asyncio +import sys +import time + +from xr_ai_hub import DataMessage, ParticipantEvent, ProcessorEndpoint +from xr_ai_nat.functions._service.rpc import RPCClient +from xr_render_scene import AddPrimitiveRequest, EmptyRequest, SceneClient + +CANONICAL = {"position": {"x": 0, "y": 1.6, "z": 0}, "forward": {"x": 0, "y": 0, "z": -1}, + "right": {"x": 1, "y": 0, "z": 0}, "up": {"x": 0, "y": 1, "z": 0}, + "yaw_deg": 0.0, "pitch_deg": 0.0, "ts": 1} + +# One shared starting scene: a couple of referents so pronouns and vague +# references have something to bite on. +FIXTURES = [ + ("box", 0.4, 1.3, -1.2, 1, 0.5, 0, 0.12), + ("sphere", -0.6, 1.5, -1.4, 0.6, 0, 1, 0.1), +] + +# (utterance, intent): phrased like speech, not like the corpus. +PROBES = [ + ("Can you stick a ball over there by the box?", "create"), + ("Throw another cube in somewhere.", "create"), + ("I want something floating right here in front of my face.", "create"), + ("Give me one more of those spheres.", "create"), + ("Pop a little one up near the ceiling.", "create"), + ("Actually make it way bigger.", "mutate"), + ("Scoot the box over a bit.", "mutate"), + ("Can the sphere come down to the floor?", "mutate"), + ("Flip their spots.", "mutate"), + ("Push everything back a little, it feels crowded.", "mutate"), + ("Get rid of that sphere.", "remove"), + ("Clear out the box, I don't need it.", "remove"), + ("What have we got in here so far?", "none"), + ("This looks pretty good actually.", "none"), + ("Hang on, my coffee is ready.", "none"), + ("Never mind, forget that.", "none"), + ("Wait no.", "none"), + ("How big is the box?", "none"), + ("Could you, um, you know the thing next to the other one, yeah that.", "none"), + ("Do the same thing again but on the other side.", "any"), + ("Put a yellow cube to the left of the blue cube.", "create"), +] + + +async def clear_scene(scene): + from xr_render_scene import RemovePrimitiveRequest + state = await scene.get_scene_state(EmptyRequest()) + for item in state.objects: + await scene.remove_primitive(RemovePrimitiveRequest(obj_id=item.id)) + + +async def snapshot(scene): + return {i.id: i.model_dump_json() for i in (await scene.get_scene_state(EmptyRequest())).objects} + + +def judge(intent, before, after): + added = [k for k in after if k not in before] + gone = [k for k in before if k not in after] + changed = [k for k in before if k in after and after[k] != before[k]] + facts = f"added={added} gone={gone} changed={changed}" + if intent == "create": + return (len(added) >= 1 and not gone and not changed), facts + if intent == "mutate": + return (len(changed) >= 1 and not added and not gone), facts + if intent == "remove": + return (len(gone) >= 1 and not added), facts + if intent == "none": + return (not added and not gone and not changed), facts + return True, facts + + +async def main() -> None: + tracking = RPCClient("tcp://127.0.0.1:8330", timeout_s=10.0) + scene = SceneClient("tcp://127.0.0.1:8320") + endpoint = ProcessorEndpoint(sub_addr="ipc:///tmp/xr_hub_pub", push_addr="ipc:///tmp/xr_hub_in") + run_task = asyncio.create_task(endpoint.run()) + await asyncio.sleep(0.5) + try: + await tracking.call("set_sim_pose", CANONICAL) + except Exception as error: + print(f"openxr service refused set_sim_pose ({error}); set allow_sim_pose: true in " + "agent-samples/xr-render-demo/yaml/openxr_service.yaml and restart the stack") + await scene.close() + await tracking.close() + run_task.cancel() + raise SystemExit(2) from None + + try: + wanted = {int(argument) for argument in sys.argv[1:]} + passed = failed = 0 + for index, (utterance, intent) in enumerate(PROBES): + if wanted and index not in wanted: + continue + participant = f"live-explore-{int(time.time())}-{index}" + await endpoint.inject_participant_event(ParticipantEvent( + participant_id=participant, joined=True, pts_us=time.time_ns() // 1_000)) + await asyncio.sleep(1.0) + await clear_scene(scene) + for prim_type, x, y, z, r, g, b, size in FIXTURES: + await scene.add_primitive(AddPrimitiveRequest( + prim_type=prim_type, x=x, y=y, z=z, r=r, g=g, b=b, size=size)) + before = await snapshot(scene) + await endpoint.inject_data(DataMessage( + participant_id=participant, topic="live.smoke.text", + pts_us=time.time_ns() // 1_000, data=utterance.encode())) + # Expected-change intents may finish early; restraint intents must + # wait out the window. + deadline = asyncio.get_running_loop().time() + (30 if intent == "none" else 75) + after = before + while asyncio.get_running_loop().time() < deadline: + await asyncio.sleep(2) + after = await snapshot(scene) + if intent != "none" and after != before: + await asyncio.sleep(5) + after = await snapshot(scene) + break + ok, facts = judge(intent, before, after) + verdict = "PASS" if ok else "FAIL" + print(f"{verdict} [{index:02d}] ({intent:6s}) {utterance!r} {facts}", flush=True) + passed += ok + failed += not ok + print(f"\nlive explore: {passed} passed, {failed} failed", flush=True) + finally: + try: + await tracking.call("clear_sim_pose", {}) + except Exception: + pass + await scene.close() + await tracking.close() + run_task.cancel() + + +def run() -> None: + asyncio.run(main()) + + +if __name__ == "__main__": + run() diff --git a/agent-samples/xr-render-demo/eval/xr_render_demo_eval/live_garble.py b/agent-samples/xr-render-demo/eval/xr_render_demo_eval/live_garble.py new file mode 100644 index 00000000..af3ecfb9 --- /dev/null +++ b/agent-samples/xr-render-demo/eval/xr_render_demo_eval/live_garble.py @@ -0,0 +1,250 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Live STT-noise tests: garbled, corrected, and fragmentary prompts. + +Each case sends one or more turns from a single participant (corrections need +the same conversation), scoring after every turn. A turn either expects a +scene change satisfying its check, or expects restraint: no new objects, or +no change at all. Wrong mutations on noisy input are the failure mode under +test; a clarifying reply with an untouched scene always passes a restraint +turn. +""" + +import asyncio +import sys +import time + +from xr_ai_hub import DataMessage, ParticipantEvent, ProcessorEndpoint +from xr_ai_nat.functions._service.rpc import RPCClient +from xr_render_scene import AddPrimitiveRequest, EmptyRequest, SceneClient + +CANONICAL = {"position": {"x": 0, "y": 1.6, "z": 0}, "forward": {"x": 0, "y": 0, "z": -1}, + "right": {"x": 1, "y": 0, "z": 0}, "up": {"x": 0, "y": 1, "z": 0}, + "yaw_deg": 0.0, "pitch_deg": 0.0, "ts": 1} + +GREEN_SPHERE = ("sphere", -1.0, 1.6, -1.5, 0, 0.8, 0, 0.1) +BLUE_SPHERE = ("sphere", 1.0, 1.6, -1.5, 0, 0, 1, 0.1) + + +def _new(before, after): + return {k: v for k, v in after.items() if k not in before} + + +# Turn kinds: "change" polls until the scene differs and applies check(ids, +# before, after); "no_new_object" and "no_change" wait out the window and +# fail on any created object / any difference respectively. +CASES = [ + { + "name": "homophone_anchor_spear", + "fixtures": [GREEN_SPHERE, BLUE_SPHERE], + "turns": [ + {"prompt": "Add a red box above the green spear.", "kind": "change", + "check": lambda ids, before, after: any( + item.type == "box" and abs(item.position.x + 1.0) < 0.25 and item.position.y > 1.65 + for item in _new(before, after).values())}, + ], + }, + { + "name": "homophone_color_blew", + "fixtures": [], + "turns": [ + {"prompt": "Make a blew sphere.", "kind": "change", + "check": lambda ids, before, after: any( + item.type == "sphere" and item.color.b > 0.5 + for item in _new(before, after).values())}, + ], + }, + { + "name": "homophone_shape_spear", + "fixtures": [BLUE_SPHERE], + "turns": [ + {"prompt": "Make a blue spear.", "kind": "change", + "check": lambda ids, before, after: ( + len(_new(before, after)) == 1 + and all(item.type == "sphere" and item.color.b > 0.5 and item.position.z < -0.5 + for item in _new(before, after).values()))}, + ], + }, + { + "name": "bare_create_after_work_stays_bare", + "fixtures": [GREEN_SPHERE], + "turns": [ + {"prompt": "Add a teal cube.", "kind": "change", + "check": lambda ids, before, after: any( + item.type == "box" for item in _new(before, after).values())}, + {"prompt": "Make a red cube.", "kind": "change", + "check": lambda ids, before, after: ( + len(_new(before, after)) == 1 + and all(item.type == "box" and item.color.r > 0.5 + and abs(item.position.x) < 0.3 and item.position.y > 1.3 + for item in _new(before, after).values()))}, + ], + }, + { + "name": "correction_never_creates", + "fixtures": [GREEN_SPHERE, BLUE_SPHERE], + "turns": [ + {"prompt": "Add a red box above the blue sphere.", "kind": "change", + "check": lambda ids, before, after: any( + item.type == "box" for item in _new(before, after).values())}, + {"prompt": "That's the wrong sphere.", "kind": "no_new_object"}, + ], + }, + { + "name": "filler_fragment", + "fixtures": [BLUE_SPHERE], + "turns": [ + {"prompt": "Um, hang on a second.", "kind": "no_change"}, + ], + }, + { + "name": "truncated_command", + "fixtures": [GREEN_SPHERE, BLUE_SPHERE], + "turns": [ + {"prompt": "Put the sphere on the", "kind": "no_change"}, + ], + }, + { + "name": "truncated_then_completed", + "fixtures": [("box", 0.5, 1.3, -1.2, 0, 0.8, 0.8, 0.15), BLUE_SPHERE], + "turns": [ + {"prompt": "Put the sphere on the", "kind": "no_change"}, + {"prompt": "On the box.", "kind": "change", + "check": lambda ids, before, after: ( + not _new(before, after) + and abs(after[ids[1]].position.x - 0.5) < 0.3 + and after[ids[1]].position.y > 1.3)}, + ], + }, + { + "name": "self_correction_single_create", + "fixtures": [], + "turns": [ + {"prompt": "Make a red, no, a green cube.", "kind": "change", + "check": lambda ids, before, after: ( + len(_new(before, after)) == 1 + and all(item.type == "box" and item.color.g > 0.5 and item.color.r < 0.3 + for item in _new(before, after).values()))}, + ], + }, + { + "name": "stutter_single_create", + "fixtures": [], + "turns": [ + {"prompt": "Add a a small cube cube.", "kind": "change", + "check": lambda ids, before, after: ( + len(_new(before, after)) == 1 + and all(item.type == "box" for item in _new(before, after).values()))}, + ], + }, +] + + +async def clear_scene(scene): + from xr_render_scene import RemovePrimitiveRequest + state = await scene.get_scene_state(EmptyRequest()) + for item in state.objects: + await scene.remove_primitive(RemovePrimitiveRequest(obj_id=item.id)) + + +async def snapshot(scene): + return {i.id: i for i in (await scene.get_scene_state(EmptyRequest())).objects} + + +async def run_turn(scene, endpoint, participant, turn, ids): + before = await snapshot(scene) + await endpoint.inject_data(DataMessage( + participant_id=participant, topic="live.smoke.text", + pts_us=time.time_ns() // 1_000, data=turn["prompt"].encode())) + if turn["kind"] == "change": + deadline = asyncio.get_running_loop().time() + 75 + while asyncio.get_running_loop().time() < deadline: + await asyncio.sleep(2) + after = await snapshot(scene) + if after != before: + await asyncio.sleep(4) + after = await snapshot(scene) + ok = turn["check"](ids, before, after) + return ok, _describe(before, after) + return False, "no change within 75s" + # Restraint turns: wait out the window, then judge what happened. + await asyncio.sleep(30) + after = await snapshot(scene) + created = _new(before, after) + if turn["kind"] == "no_new_object": + gone = [k for k in before if k not in after] + ok = not created and not gone + return ok, _describe(before, after) if not ok else "no new or removed objects" + return after == before, _describe(before, after) if after != before else "no change" + + +def _describe(before, after): + parts = [f"NEW {k}:{v.model_dump()}" for k, v in _new(before, after).items()] + parts += [f"GONE {k}" for k in before if k not in after] + parts += [f"MOVED {k}" for k in before + if k in after and after[k].model_dump() != before[k].model_dump()] + return "; ".join(parts)[:400] or "unchanged" + + +async def main() -> None: + tracking = RPCClient("tcp://127.0.0.1:8330", timeout_s=10.0) + scene = SceneClient("tcp://127.0.0.1:8320") + await clear_scene(scene) + endpoint = ProcessorEndpoint(sub_addr="ipc:///tmp/xr_hub_pub", push_addr="ipc:///tmp/xr_hub_in") + run_task = asyncio.create_task(endpoint.run()) + await asyncio.sleep(0.5) + try: + await tracking.call("set_sim_pose", CANONICAL) + except Exception as error: + print(f"openxr service refused set_sim_pose ({error}); set allow_sim_pose: true in " + "agent-samples/xr-render-demo/yaml/openxr_service.yaml and restart the stack") + await scene.close() + await tracking.close() + run_task.cancel() + raise SystemExit(2) from None + + try: + wanted = set(sys.argv[1:]) + passed = failed = 0 + for index, case in enumerate(CASES): + if wanted and case["name"] not in wanted: + continue + participant = f"live-garble-{int(time.time())}-{index}" + await endpoint.inject_participant_event(ParticipantEvent( + participant_id=participant, joined=True, pts_us=time.time_ns() // 1_000)) + await asyncio.sleep(1.0) + await clear_scene(scene) + ids = [] + for prim_type, x, y, z, r, g, b, size in case["fixtures"]: + result = await scene.add_primitive(AddPrimitiveRequest( + prim_type=prim_type, x=x, y=y, z=z, r=r, g=g, b=b, size=size)) + ids.append(result.id) + ok, detail = True, "" + for number, turn in enumerate(case["turns"], start=1): + turn_ok, turn_detail = await run_turn(scene, endpoint, participant, turn, ids) + detail = f"turn {number}: {turn_detail}" + if not turn_ok: + ok = False + break + verdict = "PASS" if ok else "FAIL" + print(f"{verdict} {case['name']:28s} {detail}", flush=True) + passed += ok + failed += not ok + print(f"\nlive garble: {passed} passed, {failed} failed", flush=True) + finally: + try: + await tracking.call("clear_sim_pose", {}) + except Exception: + pass + await scene.close() + await tracking.close() + run_task.cancel() + + +def run() -> None: + asyncio.run(main()) + + +if __name__ == "__main__": + run() diff --git a/agent-samples/xr-render-demo/eval/xr_render_demo_eval/live_manip.py b/agent-samples/xr-render-demo/eval/xr_render_demo_eval/live_manip.py new file mode 100644 index 00000000..0182164c --- /dev/null +++ b/agent-samples/xr-render-demo/eval/xr_render_demo_eval/live_manip.py @@ -0,0 +1,172 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Live manipulation tests: fixture objects via scene RPC, prompt, scene-diff verdict.""" + +import asyncio +import math +import sys +import time + +from xr_ai_hub import DataMessage, ParticipantEvent, ProcessorEndpoint +from xr_ai_nat.functions._service.rpc import RPCClient +from xr_render_scene import AddPrimitiveRequest, EmptyRequest, SceneClient + +CANONICAL = {"position": {"x": 0, "y": 1.6, "z": 0}, "forward": {"x": 0, "y": 0, "z": -1}, + "right": {"x": 1, "y": 0, "z": 0}, "up": {"x": 0, "y": 1, "z": 0}, + "yaw_deg": 0.0, "pitch_deg": 0.0, "ts": 1} + +# Each case: fixtures (type, x, y, z, r, g, b, size), prompt, then checks on +# the scene: expressions over {id: object} keyed by fixture creation order. +CASES = [ + { + "name": "move_left_one_meter", + "fixtures": [("box", 0.0, 1.7, -1.5, 0, 0.4, 1, 0.1)], + "prompt": "Move the cube left one meter.", + "check": lambda ids, o: abs(o[ids[0]].position.x + 1.0) < 0.15 and abs(o[ids[0]].position.z + 1.5) < 0.15, + }, + { + "name": "recolor", + "fixtures": [("sphere", 0.0, 1.6, -1.5, 1, 0, 0, 0.1)], + "prompt": "Make the sphere green.", + "check": lambda ids, o: o[ids[0]].color.g > 0.5 and o[ids[0]].color.r < 0.3, + }, + { + "name": "containment", + "fixtures": [("sphere", 1.0, 1.6, -1.5, 1, 0, 0, 0.1), ("box", -0.5, 1.3, -1.5, 0, 0.4, 1, 0.25)], + "prompt": "Put the sphere in the cube.", + "check": lambda ids, o: math.dist( + (o[ids[0]].position.x, o[ids[0]].position.y, o[ids[0]].position.z), + (o[ids[1]].position.x, o[ids[1]].position.y, o[ids[1]].position.z)) < 0.2, + }, + { + "name": "bring_closer", + "fixtures": [("sphere", 0.0, 1.6, -3.0, 1, 1, 0, 0.1)], + "prompt": "Bring the sphere closer to me.", + "check": lambda ids, o: o[ids[0]].position.z > -2.9, + }, + { + "name": "remove_left_of_two", + "fixtures": [("box", -1.0, 1.5, -2.0, 0.4, 0.4, 0.4, 0.2), ("box", 1.0, 1.5, -2.0, 0.4, 0.4, 0.4, 0.2)], + "prompt": "Remove the cube on the left.", + "check": lambda ids, o: ids[0] not in o and ids[1] in o, + }, + { + "name": "double_size", + "fixtures": [("sphere", 0.3, 1.5, -1.2, 0, 0, 1, 0.1)], + "prompt": "Double its size.", + "check": lambda ids, o: abs(o[ids[0]].size - 0.2) < 0.02, + }, + { + "name": "anchored_add_right_anchor", + "fixtures": [("sphere", -1.0, 1.6, -1.5, 0, 0, 1, 0.1), ("sphere", 1.0, 1.6, -1.5, 0, 0.8, 0, 0.1)], + "prompt": "Add a red box above the blue sphere.", + "check": lambda ids, o: any( + item.type == "box" and abs(item.position.x + 1.0) < 0.2 and item.position.y > 1.65 + for key, item in o.items() if key not in ids + ), + }, + { + "name": "anchored_add_garbled", + "fixtures": [("sphere", -1.0, 1.6, -1.5, 0, 0, 1, 0.1), ("sphere", 1.0, 1.6, -1.5, 0, 0.8, 0, 0.1)], + "prompt": "It had a red box above the blue sphere.", + "no_change_ok": True, + "check": lambda ids, o: ( + not any(item.type == "box" and abs(item.position.x - 1.0) < 0.3 + for key, item in o.items() if key not in ids) + ), + }, + { + "name": "swap", + "fixtures": [("sphere", 1.0, 1.6, -1.5, 1, 0, 0, 0.1), ("box", -1.0, 1.6, -1.5, 0, 0.4, 1, 0.1)], + "prompt": "Swap the sphere and the cube.", + "check": lambda ids, o: abs(o[ids[0]].position.x + 1.0) < 0.15 and abs(o[ids[1]].position.x - 1.0) < 0.15, + }, +] + + + +async def clear_scene(scene): + from xr_render_scene import RemovePrimitiveRequest + state = await scene.get_scene_state(EmptyRequest()) + for item in state.objects: + await scene.remove_primitive(RemovePrimitiveRequest(obj_id=item.id)) + +async def main() -> None: + tracking = RPCClient("tcp://127.0.0.1:8330", timeout_s=10.0) + scene = SceneClient("tcp://127.0.0.1:8320") + await clear_scene(scene) + endpoint = ProcessorEndpoint(sub_addr="ipc:///tmp/xr_hub_pub", push_addr="ipc:///tmp/xr_hub_in") + run_task = asyncio.create_task(endpoint.run()) + await asyncio.sleep(0.5) + try: + await tracking.call("set_sim_pose", CANONICAL) + except Exception as error: + print(f"openxr service refused set_sim_pose ({error}); set allow_sim_pose: true in " + "agent-samples/xr-render-demo/yaml/openxr_service.yaml and restart the stack") + await scene.close() + await tracking.close() + run_task.cancel() + raise SystemExit(2) from None + + try: + wanted = set(sys.argv[1:]) + passed = failed = 0 + for index, case in enumerate(CASES): + if wanted and case["name"] not in wanted: + continue + participant = f"live-manip-{int(time.time())}-{index}" + await endpoint.inject_participant_event(ParticipantEvent( + participant_id=participant, joined=True, pts_us=time.time_ns() // 1_000)) + await asyncio.sleep(1.0) + await clear_scene(scene) + ids = [] + for prim_type, x, y, z, r, g, b, size in case["fixtures"]: + result = await scene.add_primitive(AddPrimitiveRequest( + prim_type=prim_type, x=x, y=y, z=z, r=r, g=g, b=b, size=size)) + ids.append(result.id) + snapshot = {i.id: i for i in (await scene.get_scene_state(EmptyRequest())).objects} + await endpoint.inject_data(DataMessage( + participant_id=participant, topic="live.smoke.text", + pts_us=time.time_ns() // 1_000, data=case["prompt"].encode())) + verdict = "PASS" if case.get("no_change_ok") else "FAIL" + detail = "no change within 75s" + deadline = asyncio.get_running_loop().time() + 75 + while asyncio.get_running_loop().time() < deadline: + await asyncio.sleep(2) + objects = {i.id: i for i in (await scene.get_scene_state(EmptyRequest())).objects} + if objects != snapshot: + await asyncio.sleep(4) + objects = {i.id: i for i in (await scene.get_scene_state(EmptyRequest())).objects} + try: + ok = case["check"](ids, objects) + except Exception as error: + ok, detail = False, f"check error: {error}" + else: + new_ids = [k for k in objects if k not in snapshot] + detail = "; ".join( + [f"{i}:{'GONE' if i not in objects else objects[i].model_dump()}" for i in ids] + + [f"NEW {k}:{objects[k].model_dump()}" for k in new_ids] + )[:400] + verdict = "PASS" if ok else "FAIL" + break + print(f"{verdict} {case['name']:22s} {detail}", flush=True) + passed += verdict == "PASS" + failed += verdict == "FAIL" + print(f"\nlive manipulation: {passed} passed, {failed} failed", flush=True) + finally: + try: + await tracking.call("clear_sim_pose", {}) + except Exception: + pass + await scene.close() + await tracking.close() + run_task.cancel() + + +def run() -> None: + asyncio.run(main()) + + +if __name__ == "__main__": + run() diff --git a/agent-samples/xr-render-demo/eval/xr_render_demo_eval/live_pose_matrix.py b/agent-samples/xr-render-demo/eval/xr_render_demo_eval/live_pose_matrix.py new file mode 100644 index 00000000..b479c405 --- /dev/null +++ b/agent-samples/xr-render-demo/eval/xr_render_demo_eval/live_pose_matrix.py @@ -0,0 +1,139 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Pose-matrix live test: sim pose -> typed prompt -> scene-state verdict.""" + +import asyncio +import math +import sys +import time + +from xr_ai_hub import DataMessage, ParticipantEvent, ProcessorEndpoint +from xr_ai_nat.functions._service.rpc import RPCClient +from xr_render_scene import EmptyRequest, SceneClient + +CANONICAL = {"position": {"x": 0, "y": 1.6, "z": 0}, "forward": {"x": 0, "y": 0, "z": -1}, + "right": {"x": 1, "y": 0, "z": 0}, "up": {"x": 0, "y": 1, "z": 0}, + "yaw_deg": 0.0, "pitch_deg": 0.0, "ts": 1} + + +def pose(x, y, z, yaw_deg): + yaw = math.radians(yaw_deg) + forward = {"x": -math.sin(yaw), "y": 0.0, "z": -math.cos(yaw)} + right = {"x": math.cos(yaw), "y": 0.0, "z": -math.sin(yaw)} + return {"position": {"x": x, "y": y, "z": z}, "forward": forward, "right": right, + "up": {"x": 0, "y": 1, "z": 0}, "yaw_deg": yaw_deg, "pitch_deg": 0.0, "ts": 1} + + +POSES = { + "canonical": CANONICAL, + "user_live": pose(0.5, 1.33, 0.7, 0), + "walked_off": pose(2.0, 1.6, 1.5, 0), + "turned_180": pose(0.0, 1.6, 0.0, 180), + "turned_left_90": pose(0.0, 1.45, -0.5, 90), +} + +PROMPT_SETS = { + "canonical": [("Make a red sphere.", 1.5), ("Add a green cube.", 1.5), + ("Create a yellow sphere two meters ahead of me.", 2.0)], + "user_live": [("Make an orange sphere.", 1.5), ("Add a purple cube.", 1.5), + ("Create a cyan sphere two meters ahead of me.", 2.0)], + "walked_off": [("Make a white sphere.", 1.5), ("Add a black cube.", 1.5), + ("Create a magenta sphere two meters ahead of me.", 2.0)], + "turned_180": [("Make a blue sphere.", 1.5), ("Add a yellow cube.", 1.5), + ("Create a red sphere two meters ahead of me.", 2.0)], + "turned_left_90": [("Make a green sphere.", 1.5), ("Add a cyan cube.", 1.5), + ("Create a white sphere two meters ahead of me.", 2.0)], +} + + +def expected_spot(p, distance): + fx, fz = p["forward"]["x"], p["forward"]["z"] + magnitude = math.sqrt(fx * fx + fz * fz) + fx, fz = fx / magnitude, fz / magnitude + return (p["position"]["x"] + fx * distance, p["position"]["y"], p["position"]["z"] + fz * distance) + + + +async def clear_scene(scene): + from xr_render_scene import RemovePrimitiveRequest + state = await scene.get_scene_state(EmptyRequest()) + for item in state.objects: + await scene.remove_primitive(RemovePrimitiveRequest(obj_id=item.id)) + +async def main() -> None: + participant = f"live-smoke-{int(time.time())}" + tracking = RPCClient("tcp://127.0.0.1:8330", timeout_s=10.0) + scene = SceneClient("tcp://127.0.0.1:8320") + await clear_scene(scene) + endpoint = ProcessorEndpoint(sub_addr="ipc:///tmp/xr_hub_pub", push_addr="ipc:///tmp/xr_hub_in") + run_task = asyncio.create_task(endpoint.run()) + await asyncio.sleep(0.5) + await endpoint.inject_participant_event(ParticipantEvent( + participant_id=participant, joined=True, pts_us=time.time_ns() // 1_000)) + await asyncio.sleep(1.0) + + failed = 0 + try: + passed = failed = 0 + for pose_name, p in POSES.items(): + try: + await tracking.call("set_sim_pose", p) + except Exception as error: + print(f"openxr service refused set_sim_pose ({error}); set allow_sim_pose: true in " + "yaml/openxr_service.yaml and restart the stack") + raise SystemExit(2) from None + for prompt, distance in PROMPT_SETS[pose_name]: + await clear_scene(scene) + before = {i.id for i in (await scene.get_scene_state(EmptyRequest())).objects} + await endpoint.inject_data(DataMessage( + participant_id=participant, topic="live.smoke.text", + pts_us=time.time_ns() // 1_000, data=prompt.encode())) + new = None + deadline = asyncio.get_running_loop().time() + 75 + while asyncio.get_running_loop().time() < deadline: + await asyncio.sleep(2) + objects = {i.id: i for i in (await scene.get_scene_state(EmptyRequest())).objects} + fresh = [objects[k] for k in objects.keys() - before] + if fresh: + await asyncio.sleep(3) + objects = {i.id: i for i in (await scene.get_scene_state(EmptyRequest())).objects} + fresh = [objects[k] for k in objects.keys() - before] + new = fresh + break + if not new: + print(f"FAIL {pose_name:15s} {prompt!r}: nothing created") + failed += 1 + continue + if len(new) > 1: + print(f"FAIL {pose_name:15s} {prompt!r}: {len(new)} objects created") + failed += 1 + continue + item = new[0] + ex, ey, ez = expected_spot(p, distance) + dx, dy, dz = item.position.x - ex, item.position.y - ey, item.position.z - ez + miss = math.sqrt(dx * dx + dy * dy + dz * dz) + verdict = "PASS" if miss <= 0.25 else "FAIL" + print(f"{verdict} {pose_name:15s} {prompt!r}: {item.type} at " + f"({item.position.x:.2f},{item.position.y:.2f},{item.position.z:.2f}) " + f"expected ({ex:.2f},{ey:.2f},{ez:.2f}) miss={miss:.2f}") + passed += verdict == "PASS" + failed += verdict == "FAIL" + print(f"\npose matrix: {passed} passed, {failed} failed", flush=True) + finally: + try: + await tracking.call("clear_sim_pose", {}) + except Exception: + pass + await scene.close() + await tracking.close() + run_task.cancel() + sys.exit(1 if failed else 0) + + +def run() -> None: + asyncio.run(main()) + + +if __name__ == "__main__": + run() diff --git a/agent-samples/xr-render-demo/eval/xr_render_demo_eval/live_smoke.py b/agent-samples/xr-render-demo/eval/xr_render_demo_eval/live_smoke.py new file mode 100644 index 00000000..3379ecaf --- /dev/null +++ b/agent-samples/xr-render-demo/eval/xr_render_demo_eval/live_smoke.py @@ -0,0 +1,60 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Drive the live render stack with typed text and verify against scene state.""" + +import asyncio +import sys +import time + +from xr_ai_hub import DataMessage, ParticipantEvent, ProcessorEndpoint +from xr_render_scene import EmptyRequest, SceneClient + + +async def main() -> None: + prompt = " ".join(sys.argv[1:]) or "Make a red sphere." + participant = f"live-smoke-{int(time.time())}" + endpoint = ProcessorEndpoint(sub_addr="ipc:///tmp/xr_hub_pub", push_addr="ipc:///tmp/xr_hub_in") + scene = SceneClient("tcp://127.0.0.1:8320") + before = {item.id: item for item in (await scene.get_scene_state(EmptyRequest())).objects} + + run_task = asyncio.create_task(endpoint.run()) + await asyncio.sleep(0.5) + await endpoint.inject_participant_event(ParticipantEvent( + participant_id=participant, joined=True, pts_us=time.time_ns() // 1_000)) + await asyncio.sleep(1.0) + await endpoint.inject_data(DataMessage( + participant_id=participant, + topic="live.smoke.text", + pts_us=time.time_ns() // 1_000, + data=prompt.encode(), + )) + print(f"sent: {prompt!r}") + + try: + deadline = asyncio.get_running_loop().time() + 60 + while asyncio.get_running_loop().time() < deadline: + await asyncio.sleep(2) + after = {item.id: item for item in (await scene.get_scene_state(EmptyRequest())).objects} + if after != before: + for object_id in sorted(set(after) - set(before)): + print("added:", after[object_id].model_dump()) + for object_id in sorted(set(before) - set(after)): + print("removed:", object_id) + for object_id in sorted(set(before) & set(after)): + if before[object_id] != after[object_id]: + print("changed:", after[object_id].model_dump()) + break + else: + print("no scene change within 60s") + finally: + await scene.close() + run_task.cancel() + + +def run() -> None: + asyncio.run(main()) + + +if __name__ == "__main__": + run() diff --git a/agent-samples/xr-render-demo/eval/xr_render_demo_eval/subagents.py b/agent-samples/xr-render-demo/eval/xr_render_demo_eval/subagents.py new file mode 100644 index 00000000..5850a6c8 --- /dev/null +++ b/agent-samples/xr-render-demo/eval/xr_render_demo_eval/subagents.py @@ -0,0 +1,611 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Component eval: drive one focused subagent directly and score its mutations. + +Each case invokes a single subagent Function with the supervisor-level +instruction it would receive (stable ids already resolved, facts from other +subagents already inlined), with no supervisor in the loop. Emitted +add/update/remove_primitive calls are matched order-independently against +expected argument ranges, so a regression localises to one agent and one +prompt instead of an end-to-end transcript. + + uv run --project agent-samples/xr-render-demo/eval xr_render_demo_eval_subagents + uv run --project agent-samples/xr-render-demo/eval xr_render_demo_eval_subagents placement + uv run --project agent-samples/xr-render-demo/eval xr_render_demo_eval_subagents swap_two_objects +""" + +import argparse +import asyncio +from dataclasses import dataclass +from typing import Any + +from nat.builder.workflow_builder import WorkflowBuilder +from nat.plugin_api import LLMRef +from xr_ai_models import load_models_config, make_llm +from xr_ai_nat.functions.xr_tracking import HeadPose +from xr_ai_nat.llm import ModelsLLMConfig +from xr_render_demo_worker.agents import ( + AppearanceAgentConfig, + MemoryAgentConfig, + ObjectAgentConfig, + PlacementAgentConfig, + VisionAgentConfig, +) +from xr_render_demo_worker.models import SubagentResult, SubagentTask +from xr_render_demo_worker.scene import SceneContext +from xr_render_scene import SceneObject + +from . import harness + +_PARTICIPANT = "eval-user" +_MUTATING = frozenset({"add_primitive", "update_primitive", "remove_primitive"}) + +# Fixture vocabulary stays distinct from prompt worked examples (see README). +_CONE = { + "id": "cone-0", + "type": "cone", + "position": {"x": 0.5, "y": 1.4, "z": -1.6}, + "color": {"r": 1, "g": 1, "b": 1}, + "size": 0.1, +} +_RING = { + "id": "ring-1", + "type": "ring", + "position": {"x": -1.0, "y": 1.4, "z": -1.6}, + "color": {"r": 1, "g": 1, "b": 1}, + "size": 0.1, +} +_CAPSULE = { + "id": "capsule-2", + "type": "capsule", + "position": {"x": 2.0, "y": 1.4, "z": -1.6}, + "color": {"r": 1, "g": 1, "b": 1}, + "size": 0.1, +} + + +@dataclass(frozen=True) +class SubagentCase: + name: str + agent: str + instruction: str + scene: tuple[dict[str, Any], ...] = () + expect: tuple[dict[str, Any], ...] = () + recent_moves: tuple[str, ...] = () + pose: dict | None = None + vision_answer: str = "" + vision_error: str = "" + memory: str = "" + required_tools: tuple[str, ...] = () + forbid_tools: tuple[str, ...] = () + answer_contains: str = "" + + +# Expected args: a (lo, hi) tuple is an inclusive range, anything else is exact. +# The eval head pose is at (0, 1.6, 0) facing -z with +x to the user's right. +CASES = ( + SubagentCase( + name="move_user_left", + agent="placement", + instruction="Move cone-0 one metre to my left.", + scene=(_CONE,), + # A stated distance shifts from the current position (x = 0.5 - 1.0). + expect=( + {"tool": "update_primitive", "args": {"obj_id": "cone-0", "x": (-0.65, -0.35)}}, + ), + ), + SubagentCase( + name="move_next_to_object", + agent="placement", + instruction="Move ring-1 next to cone-0.", + scene=(_CONE, _RING), + expect=( + { + "tool": "update_primitive", + "args": {"obj_id": "ring-1", "x": (0.0, 1.0), "z": (-2.1, -1.1)}, + }, + ), + ), + SubagentCase( + name="move_between_objects", + agent="placement", + instruction="Move ring-1 halfway between cone-0 and capsule-2.", + scene=(_CONE, _RING, _CAPSULE), + expect=( + { + "tool": "update_primitive", + "args": {"obj_id": "ring-1", "x": (1.15, 1.35), "z": (-1.7, -1.5)}, + }, + ), + ), + SubagentCase( + name="move_toward_object", + agent="placement", + instruction="Move ring-1 closer to cone-0.", + scene=(_CONE, _RING), + expect=( + {"tool": "update_primitive", "args": {"obj_id": "ring-1", "x": (-0.95, 0.45)}}, + ), + ), + SubagentCase( + name="put_object_back", + agent="placement", + instruction="Put cone-0 back where it was before.", + scene=(dict(_CONE, position={"x": 0.5, "y": 1.4, "z": -0.5}),), + recent_moves=("cone-0: previously at (0.5, 1.4, -1.6), now at (0.5, 1.4, -0.5)",), + expect=( + {"tool": "update_primitive", "args": {"obj_id": "cone-0", "z": (-1.7, -1.5)}}, + ), + ), + SubagentCase( + name="unresolvable_referent_reports_back", + agent="placement", + instruction="Place the blue sphere above the green sphere.", + scene=( + dict(_CONE, id="sphere-0", type="sphere", color={"r": 1, "g": 1, "b": 1}), + dict(_RING, id="sphere-1", type="sphere", color={"r": 0, "g": 0.8, "b": 0}), + ), + forbid_tools=("update_primitive", "add_primitive", "remove_primitive"), + answer_contains="blue", + ), + SubagentCase( + name="move_yellow_not_red", + agent="placement", + instruction="Move the yellow cube down one meter.", + scene=( + dict(_CONE, id="box-0", type="box", color={"r": 1, "g": 0, "b": 0}, + position={"x": 0.1, "y": 1.3, "z": -1.0}), + dict(_RING, id="box-1", type="box", color={"r": 1, "g": 1, "b": 0}, + position={"x": -0.9, "y": 2.8, "z": -0.3}), + ), + expect=( + {"tool": "update_primitive", "args": {"obj_id": "box-1", "y": (1.7, 1.9)}}, + ), + ), + SubagentCase( + name="swap_two_objects", + agent="placement", + instruction="Swap the positions of cone-0 and ring-1.", + scene=(_CONE, _RING), + expect=( + {"tool": "update_primitive", "args": {"obj_id": "cone-0", "x": (-1.05, -0.95)}}, + {"tool": "update_primitive", "args": {"obj_id": "ring-1", "x": (0.45, 0.55)}}, + ), + ), + SubagentCase( + name="create_at_position", + agent="object", + instruction="Create a white box at x=0, y=1.5, z=-1.2 with size 0.1.", + expect=( + { + "tool": "add_primitive", + "args": { + "prim_type": "box", + "x": (-0.05, 0.05), + "y": (1.45, 1.55), + "z": (-1.25, -1.15), + }, + }, + ), + ), + SubagentCase( + name="double_named_object", + agent="object", + instruction="Double the size of ring-1.", + scene=(_CONE, _RING), + expect=( + {"tool": "update_primitive", "args": {"obj_id": "ring-1", "size": (0.19, 0.21)}}, + ), + ), + SubagentCase( + name="shrink_named_object", + agent="object", + instruction="Make the box-0 object half its current size.", + scene=( + { + "id": "box-0", + "type": "box", + "position": {"x": 0.0, "y": 1.6, "z": -1.5}, + "color": {"r": 0, "g": 0.4, "b": 1}, + "size": 0.2, + }, + ), + expect=( + {"tool": "update_primitive", "args": {"obj_id": "box-0", "size": (0.09, 0.11)}}, + ), + ), + SubagentCase( + name="row_of_three", + agent="object", + instruction="Create three red spheres in a row at eye height, 0.5 metres apart, in front of the user.", + expect=( + {"tool": "add_primitive", "args": {"prim_type": "sphere", "r": (0.7, 1.0), "g": (0.0, 0.4)}}, + {"tool": "add_primitive", "args": {"prim_type": "sphere", "r": (0.7, 1.0), "g": (0.0, 0.4)}}, + {"tool": "add_primitive", "args": {"prim_type": "sphere", "r": (0.7, 1.0), "g": (0.0, 0.4)}}, + ), + ), + SubagentCase( + name="raise_above_anchor_30cm", + agent="placement", + instruction="Move ring-1 thirty centimetres above cone-0.", + scene=(_CONE, _RING), + expect=( + {"tool": "update_primitive", "args": {"obj_id": "ring-1", "y": (1.65, 1.75)}}, + ), + ), + SubagentCase( + name="create_stated_distance_left", + agent="object", + instruction="Create a white sphere one metre to the user's left at eye height.", + expect=( + {"tool": "add_primitive", "args": {"prim_type": "sphere", "x": (-1.05, -0.95)}}, + ), + ), + SubagentCase( + name="create_at_feet", + agent="object", + instruction="Create a red sphere on the floor at the user's feet.", + expect=( + {"tool": "add_primitive", "args": {"prim_type": "sphere", "r": (0.7, 1.0), "y": (-0.05, 0.5)}}, + ), + ), + SubagentCase( + name="create_bare", + agent="object", + instruction="Create a blue sphere with no position stated.", + expect=( + { + "tool": "add_primitive", + "args": {"prim_type": "sphere", "x": (-0.05, 0.05), "y": (1.55, 1.65), "z": (-1.55, -1.45)}, + }, + ), + ), + SubagentCase( + name="create_cube_means_box", + agent="object", + instruction="Make a green cube.", + expect=( + { + "tool": "add_primitive", + "args": {"prim_type": "box", "g": (0.5, 1.0), "r": (0.0, 0.4), "b": (0.0, 0.4), "y": (1.4, 1.8)}, + }, + ), + ), + SubagentCase( + name="create_bare_with_objects", + agent="object", + instruction="Create a green cube with no position stated.", + scene=(_CONE, _RING), + expect=( + { + "tool": "add_primitive", + "args": {"prim_type": "box", "x": (-0.05, 0.05), "y": (1.55, 1.65), "z": (-1.55, -1.45)}, + }, + ), + ), + SubagentCase( + name="create_bare_crowded_front_occupied", + agent="object", + instruction="Add a green cube", + scene=( + dict(_CONE, position={"x": 0.0, "y": 1.6, "z": -1.5}), + dict(_RING, id="ring-1", position={"x": 0.0, "y": 1.9, "z": -1.5}), + dict(_CAPSULE, id="capsule-2", position={"x": 0.0, "y": 2.2, "z": -1.5}), + ), + expect=( + { + "tool": "add_primitive", + "args": {"prim_type": "box", "x": (-0.05, 0.05), "y": (1.55, 1.65), "z": (-1.55, -1.45)}, + }, + ), + ), + SubagentCase( + name="create_cube_off_origin_pose", + agent="object", + instruction="Create a green cube with no position stated.", + pose={"position": {"x": 2.0, "y": 1.6, "z": 1.5}}, + expect=( + { + "tool": "add_primitive", + "args": {"prim_type": "box", "x": (1.95, 2.05), "y": (1.55, 1.65), "z": (-0.05, 0.05)}, + }, + ), + ), + SubagentCase( + name="anchored_create_right_color", + agent="object", + instruction="Create a red box above the blue sphere.", + scene=( + {"id": "sphere-0", "type": "sphere", "position": {"x": -1.0, "y": 1.6, "z": -1.5}, + "color": {"r": 0, "g": 0, "b": 1}, "size": 0.1}, + {"id": "sphere-1", "type": "sphere", "position": {"x": 1.0, "y": 1.6, "z": -1.5}, + "color": {"r": 0, "g": 0.8, "b": 0}, "size": 0.1}, + ), + expect=( + {"tool": "add_primitive", "args": {"prim_type": "box", "x": (-1.1, -0.9), "y": (1.65, 2.0)}}, + ), + ), + SubagentCase( + name="create_garbled_shape_noun", + agent="object", + # STT corruption of the created shape itself: "sphere" heard as + # "spear" with no anchor stated. Must stay a bare user-front create. + instruction="Create one blue spear", + expect=( + {"tool": "add_primitive", "args": {"prim_type": "sphere", "r": (0, 0.05), "b": (0.9, 1.0), + "x": (-0.2, 0.2), "z": (-1.7, -1.3)}}, + ), + ), + SubagentCase( + name="anchored_create_garbled_anchor_noun", + agent="object", + # STT corruption: "sphere" heard as "spear". Nearest scene match wins. + instruction="Create a red cube above the green spear.", + scene=( + {"id": "sphere-0", "type": "sphere", "position": {"x": -1.0, "y": 1.6, "z": -1.5}, + "color": {"r": 0, "g": 0.8, "b": 0}, "size": 0.1}, + {"id": "sphere-1", "type": "sphere", "position": {"x": 1.0, "y": 1.6, "z": -1.5}, + "color": {"r": 0, "g": 0, "b": 1}, "size": 0.1}, + ), + expect=( + {"tool": "add_primitive", "args": {"prim_type": "box", "x": (-1.1, -0.9), "y": (1.65, 2.0)}}, + ), + ), + SubagentCase( + name="remove_object", + agent="object", + instruction="Remove ring-1.", + scene=(_CONE, _RING), + expect=({"tool": "remove_primitive", "args": {"obj_id": "ring-1"}},), + ), + SubagentCase( + name="resize_object", + agent="object", + instruction="Make cone-0 twice as big.", + scene=(_CONE,), + expect=( + {"tool": "update_primitive", "args": {"obj_id": "cone-0", "size": (0.19, 0.21)}}, + ), + ), + SubagentCase( + name="duplicate_object", + agent="object", + instruction="Create an identical copy of cone-0 beside it.", + scene=(_CONE,), + expect=( + {"tool": "add_primitive", "args": {"prim_type": "cone", "size": (0.05, 0.15)}}, + ), + ), + SubagentCase( + name="change_shape", + agent="object", + instruction="Change cone-0 into a box.", + scene=(_CONE,), + expect=( + {"tool": "update_primitive", "args": {"obj_id": "cone-0", "prim_type": "box"}}, + ), + ), + SubagentCase( + name="recolor_explicit_rgb", + agent="appearance", + instruction="Set cone-0 to the observed wall color: normalized RGB (1.0, 0.5, 0.0).", + scene=(_CONE,), + expect=( + { + "tool": "update_primitive", + "args": { + "obj_id": "cone-0", + "r": (0.95, 1.0), + "g": (0.45, 0.55), + "b": (0.0, 0.05), + }, + }, + ), + ), + SubagentCase( + name="recolor_by_name", + agent="appearance", + instruction="Make ring-1 orange.", + scene=(_CONE, _RING), + expect=( + { + "tool": "update_primitive", + "args": {"obj_id": "ring-1", "r": (0.8, 1.0), "g": (0.3, 0.7), "b": (0.0, 0.3)}, + }, + ), + ), + SubagentCase( + name="live_color_question", + agent="vision", + instruction="What color is the object the user is holding right now?", + vision_answer="The user is holding a bright red apple.", + required_tools=("look_at_current_frame",), + answer_contains="red", + ), + SubagentCase( + name="past_color_question", + agent="vision", + instruction="What color was the object the user held ten seconds before the utterance timestamp?", + vision_answer="The previously held object was purple.", + required_tools=("look_at_past_frame",), + answer_contains="purple", + ), + SubagentCase( + name="vision_dead_camera_degrades", + agent="vision", + instruction=( + "What physical objects or surfaces are directly in front of the user for placing a sphere " + "two meters ahead?" + ), + vision_error="No camera frame available.", + required_tools=("look_at_current_frame",), + answer_contains="no visual fact", + ), + SubagentCase( + name="vision_answers_scene_from_data", + agent="vision", + instruction="Is there a red box currently present in the scene, and if so, what is its position?", + scene=( + { + "id": "sphere-0", + "type": "sphere", + "position": {"x": 0.0, "y": 1.6, "z": -1.5}, + "color": {"r": 0, "g": 0, "b": 1}, + "size": 0.1, + }, + ), + answer_contains="no red box", + ), + SubagentCase( + name="recall_earlier_session", + agent="memory", + instruction="What object did we discuss in the earlier session?", + memory="We discussed a small cyan sphere.", + required_tools=("recall_conversation",), + answer_contains="cyan", + ), + SubagentCase( + name="recolor_multiple", + agent="appearance", + instruction="Make cone-0 and ring-1 magenta.", + scene=(_CONE, _RING), + expect=( + { + "tool": "update_primitive", + "args": {"obj_id": "cone-0", "r": (0.8, 1.0), "g": (0.0, 0.3), "b": (0.8, 1.0)}, + }, + { + "tool": "update_primitive", + "args": {"obj_id": "ring-1", "r": (0.8, 1.0), "g": (0.0, 0.3), "b": (0.8, 1.0)}, + }, + ), + ), +) + +_AGENT_CONFIGS = { + "placement": lambda context: PlacementAgentConfig(context=context), + "object": lambda context: ObjectAgentConfig(context=context), + "appearance": lambda context: AppearanceAgentConfig(context=context), + "vision": lambda context: VisionAgentConfig(context=context), + "memory": lambda _context: MemoryAgentConfig(), +} + + +def _args_match(actual: dict[str, Any], expected: dict[str, Any]) -> bool: + for key, want in expected.items(): + if key not in actual: + return False + got = actual[key] + if isinstance(want, tuple): + low, high = want + if not (isinstance(got, (int, float)) and low <= got <= high): + return False + elif got != want: + return False + return True + + +def check(calls: list[tuple[str, dict[str, Any]]], case: SubagentCase, reply: str) -> tuple[bool, str]: + """Match expected mutations order-independently; reject duplicate creation.""" + names = {name for name, _args in calls} + missing = set(case.required_tools) - names + if missing: + return False, f"missing tools: {sorted(missing)}" + if hit := set(case.forbid_tools) & names: + return False, f"forbidden tools called: {sorted(hit)}" + if case.answer_contains and case.answer_contains.lower() not in reply.lower(): + return False, f"reply does not mention {case.answer_contains!r}" + mutations = [(name, args) for name, args in calls if name in _MUTATING] + expected_adds = sum(1 for item in case.expect if item["tool"] == "add_primitive") + actual_adds = sum(1 for name, _args in mutations if name == "add_primitive") + if actual_adds > expected_adds: + actual = [f"{name}({args})" for name, args in mutations] + return False, f"duplicate add: {actual_adds} adds for {expected_adds} expected | actual: {actual}" + remaining = list(mutations) + for item in case.expect: + for index, (name, args) in enumerate(remaining): + if name == item["tool"] and _args_match(args, item["args"]): + remaining.pop(index) + break + else: + actual = [f"{name}({args})" for name, args in mutations] + return False, f"unmatched {item['tool']}({item['args']}) | actual: {actual}" + return True, f"matched {len(case.expect)} mutation(s)" + + +async def run_case(case: SubagentCase) -> bool: + objects = [SceneObject.model_validate(item) for item in case.scene] + scene = harness.FakeScene( + {item.id: item for item in objects}, + HeadPose.model_validate({**harness._DEFAULT_POSE, **(case.pose or {})}), + case.vision_answer, + case.vision_error, + case.memory, + ) + llm = make_llm(load_models_config(harness._CONFIG.models_yaml), "agent_llm") + try: + async with WorkflowBuilder() as builder: + await scene.bind(builder) + await builder.add_llm( + LLMRef("scene_llm"), + ModelsLLMConfig( + service=llm, + model_name="xr-scene-agent", + max_tokens=2048, + temperature=0.0, + recover_tool_calls=True, + ), + ) + scene_group = await builder.get_function_group("scene_state") + scene_functions = await scene_group.get_all_functions() + tracking_group = await builder.get_function_group("tracking") + tracking_functions = await tracking_group.get_all_functions() + context = SceneContext( + scene_functions["scene_state__get_scene_state"], + tracking_functions["tracking__get_user_frame"], + ) + context._recent_moves[_PARTICIPANT] = list(case.recent_moves) + agent = await builder.add_function( + f"{case.agent}_agent", + _AGENT_CONFIGS[case.agent](context), + ) + # A crashed rollout is a case result, not an eval abort. + try: + reply = await agent.ainvoke( + SubagentTask( + instruction=case.instruction, + participant_id=_PARTICIPANT, + reference_time_us=10_000_000, + ) + ) + except Exception as exc: + reply = SubagentResult(result=f"") + finally: + await llm.close() + ok, why = check(scene.calls, case, reply.result) + status = "PASS" if ok else f"FAIL {why}" + print(f"{status:32} {case.agent}/{case.name}: {reply.result}") + return ok + + +async def main() -> None: + harness.audit_prompts() + parser = argparse.ArgumentParser() + parser.add_argument("filter", nargs="?", help="Agent or case name; omit to run all cases") + args = parser.parse_args() + selected = [case for case in CASES if args.filter in (None, case.agent, case.name)] + if not selected: + raise SystemExit(f"unknown agent or case: {args.filter}") + passed = [await run_case(case) for case in selected] + print(f"\nsubagents: {sum(passed)}/{len(passed)} passed") + if not all(passed): + raise SystemExit(1) + + +def run() -> None: + asyncio.run(main()) + + +if __name__ == "__main__": + run() diff --git a/agent-samples/xr-render-demo/eval/xr_render_demo_eval/supervisor.py b/agent-samples/xr-render-demo/eval/xr_render_demo_eval/supervisor.py new file mode 100644 index 00000000..dc680f00 --- /dev/null +++ b/agent-samples/xr-render-demo/eval/xr_render_demo_eval/supervisor.py @@ -0,0 +1,335 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Supervisor routing eval: fake the five subagents and score the delegation. + +Each case runs only the supervisor loop; subagents record the instruction they +receive and return a canned success. Scoring checks which agent was called and +that the instruction carries the resolved facts, so a routing regression +localises in seconds instead of a full nested rollout. + + uv run --project agent-samples/xr-render-demo/eval xr_render_demo_eval_supervisor [case ...] +""" + +import argparse +import asyncio +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from nat.builder.workflow_builder import WorkflowBuilder +from nat.plugin_api import Builder, FunctionBaseConfig, FunctionInfo, FunctionRef, LLMRef, register_function +from nat.plugins.langchain.agent.tool_calling_agent.register import ToolCallAgentWorkflowConfig +from pydantic import ConfigDict, Field +from xr_ai_models import load_models_config, make_llm +from xr_ai_nat.functions.text_memory import RecallConversationRequest +from xr_ai_nat.llm import ModelsLLMConfig +from xr_render_demo_worker.agents.appearance.agent import DESCRIPTION as appearance_description +from xr_render_demo_worker.agents.memory.agent import DESCRIPTION as memory_description +from xr_render_demo_worker.agents.object.agent import DESCRIPTION as object_description +from xr_render_demo_worker.agents.placement.agent import DESCRIPTION as placement_description +from xr_render_demo_worker.agents.vision.agent import DESCRIPTION as vision_description +from xr_render_demo_worker.models import SceneReply, SubagentResult, SubagentTask + +from . import harness + +_DESCRIPTIONS = { + "placement_agent": placement_description, + "appearance_agent": appearance_description, + "object_agent": object_description, + "vision_agent": vision_description, + "memory_agent": memory_description, +} +_PROMPT = ( + Path(__file__).resolve().parent + / "../../worker/xr_render_demo_worker/supervisor_prompt.txt" +).resolve() + + +class _FakeAgentConfig(FunctionBaseConfig, name="xr_render_eval_fake_agent"): + model_config = ConfigDict(arbitrary_types_allowed=True) + agent_name: str + description: str + recorder: Any = Field(exclude=True, repr=False) + + +@register_function(config_type=_FakeAgentConfig) +async def _fake_agent(config: _FakeAgentConfig, _builder: Builder): + async def act(request: SubagentTask) -> SubagentResult: + config.recorder.append((config.agent_name, request.instruction)) + return SubagentResult(result="Done.") + + yield FunctionInfo.from_fn(act, description=config.description) + + +@dataclass(frozen=True) +class RoutingCase: + name: str + request: str + scene: tuple[dict[str, Any], ...] = () + history: tuple[tuple[str, str], ...] = () + expect_agent: str = "" + instruction_contains: tuple[str, ...] = () + instruction_forbids: tuple[str, ...] = () + forbid_agents: tuple[str, ...] = () + + +CASES = ( + RoutingCase( + name="pronoun_resize_after_creation", + request="Now double its size.", + scene=( + {"id": "sphere-1", "type": "sphere", "pos": [0.4, 1.5, -1.2], "color": [1, 1, 0], "size": 0.1}, + {"id": "box-0", "type": "box", "pos": [-0.6, 1.3, -1.6], "color": [0, 0.4, 1], "size": 0.15}, + ), + history=( + ("Make a yellow sphere.", "Added a yellow sphere (sphere-1)."), + ), + expect_agent="object_agent", + instruction_contains=("sphere-1",), + ), + RoutingCase( + name="pronoun_shrink_after_move", + request="Make it half the size.", + scene=( + {"id": "box-0", "type": "box", "pos": [0.0, 1.6, -1.5], "color": [0, 0.4, 1], "size": 0.2}, + ), + history=( + ("Make a blue cube.", "Added a blue cube (box-0)."), + ("Move it left.", "Moved the cube (box-0) to your left."), + ), + expect_agent="object_agent", + instruction_contains=("box-0",), + ), + RoutingCase( + name="bare_pronoun_double_size", + request="Double its size.", + scene=( + {"id": "sphere-1", "type": "sphere", "pos": [0.13, 1.8, -1.59], "color": [0, 0, 1], "size": 0.1}, + ), + expect_agent="object_agent", + instruction_contains=("sphere-1",), + ), + RoutingCase( + name="bare_pronoun_half_size", + request="Make it half the size.", + scene=( + {"id": "box-0", "type": "box", "pos": [0.0, 1.6, -1.5], "color": [0, 0.4, 1], "size": 0.2}, + ), + expect_agent="object_agent", + instruction_contains=("box-0",), + ), + RoutingCase( + name="move_existing_routes_to_placement", + request="Put the sphere in the cube.", + scene=( + {"id": "sphere-0", "type": "sphere", "pos": [1.0, 1.6, -1.5], "color": [1, 0, 0], "size": 0.1}, + {"id": "box-0", "type": "box", "pos": [-0.5, 1.3, -1.5], "color": [0, 0.4, 1], "size": 0.25}, + ), + expect_agent="placement_agent", + forbid_agents=("object_agent",), + ), + RoutingCase( + name="creation_containment_routes_to_object", + request="Add a small red sphere inside the cube.", + scene=( + {"id": "box-0", "type": "box", "pos": [-0.5, 1.3, -1.5], "color": [0, 0.4, 1], "size": 0.25}, + ), + expect_agent="object_agent", + forbid_agents=("placement_agent", "vision_agent"), + ), + RoutingCase( + name="ahead_placement_needs_no_vision", + request="Put a red sphere two meters ahead of me.", + expect_agent="object_agent", + forbid_agents=("vision_agent",), + ), + RoutingCase( + name="remove_with_side_descriptor", + request="Remove the pyramid on the left.", + scene=( + {"id": "pyramid-0", "type": "pyramid", "pos": [-1.0, 1.5, -2.2], "color": [0.4, 0.4, 0.4], "size": 0.2}, + {"id": "pyramid-1", "type": "pyramid", "pos": [1.0, 1.5, -2.2], "color": [0.4, 0.4, 0.4], "size": 0.2}, + ), + expect_agent="object_agent", + forbid_agents=("placement_agent", "vision_agent"), + ), + RoutingCase( + name="bare_create_no_invented_position", + request="Okay. Make a sphere.", + expect_agent="object_agent", + instruction_forbids=("origin", "requested initial"), + ), + RoutingCase( + name="bare_create_with_history", + request="Make a blue sphere.", + history=( + ("Make a blue sphere.", "Created a blue sphere."), + ("Make a green cube.", "The system has created a green cube in the scene."), + ("Okay. Make a sphere.", "Created a sphere."), + ), + expect_agent="object_agent", + instruction_forbids=("origin", "requested initial"), + ), + RoutingCase( + name="add_verb_bare_create", + request="Add a green cube.", + expect_agent="object_agent", + instruction_contains=("no position stated",), + instruction_forbids=("origin",), + ), + RoutingCase( + name="fragment_never_mutates", + request="Fascinating.", + history=( + ("Make a red cube.", "Added a red cube."), + ("Put a blue sphere above the green sphere.", "Blue sphere added above the green sphere."), + ), + forbid_agents=("object_agent", "placement_agent", "appearance_agent"), + ), + RoutingCase( + name="create_new_anchored_routes_to_object", + request="Put a yellow cube above the blue sphere.", + scene=( + {"id": "sphere-0", "type": "sphere", "pos": [0.0, 1.6, -1.5], "color": [0, 0, 1], "size": 0.1}, + ), + # Known wart: a vision existence-check sometimes precedes the correct + # delegation (latency-only; camera-off degrades gracefully). The old + # zero-vision pass was prompt-vocabulary recall, not skill. + expect_agent="object_agent", + forbid_agents=("placement_agent",), + instruction_contains=("above",), + ), + RoutingCase( + name="row_creation_single_delegation", + request="Create three red spheres in a row.", + expect_agent="object_agent", + instruction_contains=("three",), + ), + RoutingCase( + name="bare_create_after_work_no_extra_move", + request="Make a red cube.", + scene=( + {"id": "box-8", "type": "box", "pos": [0.1, 1.2, -0.9], "color": [0, 1, 1], "size": 0.1}, + {"id": "sphere-9", "type": "sphere", "pos": [-0.4, 1.4, -1.1], "color": [0, 0.8, 0], "size": 0.1}, + ), + history=( + ("Add a cyan cube.", "Added a cyan cube."), + ("Make a green sphere.", "Created a green sphere."), + ), + expect_agent="object_agent", + forbid_agents=("placement_agent",), + ), + RoutingCase( + name="correction_never_creates", + request="That's the wrong sphere.", + scene=( + {"id": "sphere-5", "type": "sphere", "pos": [-0.7, 1.2, -0.4], "color": [0, 0.8, 0], "size": 0.1}, + {"id": "sphere-6", "type": "sphere", "pos": [0.2, 1.2, -0.8], "color": [0, 0, 1], "size": 0.1}, + {"id": "box-3", "type": "box", "pos": [0.2, 1.5, -0.8], "color": [1, 0, 0], "size": 0.1}, + ), + history=( + ("Add a red cube above the green sphere.", "Added a red cube above the green sphere."), + ), + # Known wart: the supervisor sometimes consults vision/memory before + # asking back; latency-only. The invariant is that nothing mutates. + forbid_agents=("object_agent", "appearance_agent", "placement_agent"), + ), +) + + +async def run_case(case: RoutingCase) -> bool: + scene = harness.FakeScene.from_corpus_case( + {"name": case.name, "scene": list(case.scene), "history": list(case.history), "user": case.request} + ) + calls: list[tuple[str, str]] = [] + llm = make_llm(load_models_config(harness._CONFIG.models_yaml), "agent_llm") + try: + async with WorkflowBuilder() as builder: + await scene.bind(builder) + await builder.add_llm( + LLMRef("scene_llm"), + ModelsLLMConfig( + service=llm, model_name="xr-scene-agent", max_tokens=2048, + temperature=0.0, recover_tool_calls=True, + ), + ) + for agent_name, description in _DESCRIPTIONS.items(): + await builder.add_function( + agent_name, + _FakeAgentConfig(agent_name=agent_name, description=description, recorder=calls), + ) + reasoning = await builder.add_function( + "supervisor_reasoning", + ToolCallAgentWorkflowConfig( + llm_name=LLMRef("scene_llm"), + tool_names=[FunctionRef(name) for name in _DESCRIPTIONS], + system_prompt=_PROMPT.read_text(encoding="utf-8").strip(), + handle_tool_errors=True, + max_iterations=12, + max_empty_response_retries=1, + ), + ) + conversations = await builder.get_function_group("conversations") + conversation_functions = await conversations.get_all_functions() + recall = conversation_functions["conversations__recall_conversation"] + recalled = await recall.ainvoke(RecallConversationRequest(participant_id="eval-user")) + lines = [f" {'User' if e.role == 'user' else 'Agent'}: {e.text}" for e in recalled.entries[-8:]] + conversation = ("[Recent conversation]\n" + "\n".join(lines) + "\n\n") if lines else "" + state = await scene.get_scene_state(harness.EmptyRequest()) + message = ( + f"Active participant: eval-user\nUtterance timestamp: 10000000\n" + f"[SCENE OBJECTS]\n{state.model_dump_json()}\n\n{conversation}" + f"User request: {case.request}" + ) + try: + output = await reasoning.ainvoke(message, to_type=str) + reply = SceneReply(response=str(output or "Done.")) + except Exception as exc: + reply = SceneReply(response=f"") + finally: + await llm.close() + called = [name for name, _instruction in calls] + ok, why = True, "ok" + if case.expect_agent and case.expect_agent not in called: + ok, why = False, f"{case.expect_agent} never called; called={called}" + for forbidden in case.forbid_agents: + if forbidden in called: + ok, why = False, f"{forbidden} called; called={called}" + if ok and case.instruction_forbids: + instructions = " | ".join(i for name, i in calls if name == case.expect_agent) + for needle in case.instruction_forbids: + if needle.lower() in instructions.lower(): + ok, why = False, f"instruction contains forbidden {needle!r}: {instructions[:160]!r}" + if ok and case.instruction_contains: + instructions = " | ".join(i for name, i in calls if name == case.expect_agent) + for needle in case.instruction_contains: + if needle.lower() not in instructions.lower(): + ok, why = False, f"instruction missing {needle!r}: {instructions[:160]!r}" + status = "PASS" if ok else f"FAIL {why}" + detail = "; ".join(f"{name}({instruction[:80]})" for name, instruction in calls) + print(f"{status:32} {case.name}: {detail or reply.response}", flush=True) + return ok + + +async def main() -> None: + harness.audit_prompts() + parser = argparse.ArgumentParser() + parser.add_argument("cases", nargs="*", help="Case names; omit to run all") + args = parser.parse_args() + wanted = set(args.cases) + selected = [case for case in CASES if not wanted or case.name in wanted] + if not selected: + raise SystemExit(f"unknown cases: {args.cases}") + results = [await run_case(case) for case in selected] + print(f"\nsupervisor routing: {sum(results)}/{len(results)} passed") + if not all(results): + raise SystemExit(1) + + +def run() -> None: + asyncio.run(main()) + + +if __name__ == "__main__": + run() diff --git a/agent-samples/xr-render-demo/scene/xr_render_scene/__init__.py b/agent-samples/xr-render-demo/scene/xr_render_scene/__init__.py index 6c6263b7..36d13609 100644 --- a/agent-samples/xr-render-demo/scene/xr_render_scene/__init__.py +++ b/agent-samples/xr-render-demo/scene/xr_render_scene/__init__.py @@ -12,18 +12,26 @@ ) from .schemas import ( AddPrimitiveRequest, + AddPrimitiveResult, EmptyRequest, + MutationResult, RemovePrimitiveRequest, + SceneObject, + SceneState, UpdatePrimitiveRequest, ) __all__ = [ "AddPrimitiveRequest", + "AddPrimitiveResult", "EmptyRequest", + "MutationResult", "RemovePrimitiveRequest", "SceneClient", "SceneControlFunctionsConfig", + "SceneObject", "SceneObjectFunctionsConfig", + "SceneState", "SceneStateFunctionsConfig", "SceneUpdateFunctionsConfig", "UpdatePrimitiveRequest", diff --git a/agent-samples/xr-render-demo/worker/agent.py b/agent-samples/xr-render-demo/worker/agent.py deleted file mode 100644 index 8dd05272..00000000 --- a/agent-samples/xr-render-demo/worker/agent.py +++ /dev/null @@ -1,182 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -""" -RenderDemoAgent — XR session lifecycle owner. - -Wraps the ``RenderSceneProcessor`` brain with xr-render-demo-specific -behavior (``xr.session.started`` → ``start_xr`` → poll LOVR → ack; -typed-text input fed through the same path as STT). - -The voice pipeline itself is assembled by ``xr_ai_pipecat.make_voice_pipeline``; -this class only owns the agent-to-hub bookkeeping that lives outside the -generic pipeline. -""" -from __future__ import annotations - -import asyncio -import time - -from loguru import logger -from xr_ai_hub import AudioChunk, DataMessage, ParticipantEvent - -from processors import RenderSceneProcessor -from capabilities import NativeToolbox -from xr_ai_pipecat.transport import XRMediaHubTransport - -_XR_SESSION_STARTED_TOPIC = "xr.session.started" -_RENDER_READY_TOPIC = "render.ready" - - -def _now_us() -> int: - return time.time_ns() // 1_000 - - -class RenderDemoAgent: - """Owns the XR session lifecycle on top of the unified voice pipeline. - - The voice pipeline (``transport.input() → VadStt → VoiceGate → brain - → StreamingTts → transport.output()``) is built by - :func:`xr_ai_pipecat.make_voice_pipeline`; this class subscribes the - hub callbacks that the foundation does not handle (XR session start, - typed-text input, target-participant tracking). - """ - - def __init__( - self, - *, - transport: XRMediaHubTransport, - brain: RenderSceneProcessor, - tools: NativeToolbox, - ) -> None: - self._transport = transport - self._brain = brain - self._tools = tools - - self._xr_started = False - - # Subscribe to hub events the pipecat pipeline doesn't surface to us: - # data messages (text input + xr.session.started), audio (lazy - # target-pid set), and participant events. - self._transport.endpoint.on_data(self._on_data) - self._transport.endpoint.on_audio(self._on_audio) - self._transport.endpoint.on_participant(self._on_participant) - - # ── XR session lifecycle ────────────────────────────────────────────────── - - async def _on_data(self, msg: DataMessage) -> None: - if msg.topic != _XR_SESSION_STARTED_TOPIC: - await self._handle_text_input(msg) - return - - self._transport.set_target_participant(msg.participant_id) - self._brain._history.clear() - - if self._xr_started: - await self._transport.send_return_data(DataMessage( - participant_id=msg.participant_id, - topic=_RENDER_READY_TOPIC, - pts_us=_now_us(), data=b"", - )) - return - - logger.info("{} from {} — calling start_xr", msg.topic, msg.participant_id) - start_res = await self._call_render("start_xr", {}) - if start_res is None: - logger.warning("start_xr failed") - await self._notify_launch_failed(msg.participant_id) - return - if start_res.get("status") == "error": - logger.error("start_xr error: {}", start_res.get("error")) - await self._notify_launch_failed(msg.participant_id) - return - - logger.info("start_xr status={} — polling lovr_started…", start_res.get("status")) - if not await self._wait_lovr(): - await self._notify_launch_failed(msg.participant_id) - return - self._xr_started = True - - logger.info("render.ready — sending ack") - await self._transport.send_return_data(DataMessage( - participant_id=msg.participant_id, - topic=_RENDER_READY_TOPIC, - pts_us=_now_us(), data=b"", - )) - - async def _notify_launch_failed(self, pid: str) -> None: - """Surface an XR-launch failure to the user, spoken + on the panel. - - ``start_xr`` and the LOVR-spawn poll run here, outside the brain's - ``handle_query``/yield→TTS path, so a bare ``logger.warning`` would - leave the user staring at a "Launch XR" button that silently did - nothing. Route a short, actionable message through the brain's - ``enqueue_notice`` so it reaches TTS *and* the ``agent.response`` - panel exactly like a normal answer — the same delivery the in-loop - "scene not ready" case already gets. One generic message covers - both the start_xr-error and never-ready/spawn-error cases; the log - lines above retain the specific cause for operators. - """ - await self._brain.enqueue_notice( - pid, "I couldn't start the XR session — try Launch XR again." - ) - - async def _handle_text_input(self, msg: DataMessage) -> None: - """Feed a typed text message into the same path STT uses. - - The web client's "Send" button publishes typed text on the data - channel with no topic, mirroring simple-vlm-example. We hand the - text to the brain via a synthesized ``GatedQueryFrame`` so the - agentic loop fires identically to a spoken utterance, bypassing - VAD/STT and the voice gate entirely. - """ - text = (msg.data or b"").decode("utf-8", errors="replace").strip() - if not text: - return - if not self._transport.target_participant: - self._transport.set_target_participant(msg.participant_id) - logger.info("text input pid={!r} {!r}", msg.participant_id, text[:80]) - await self._brain.enqueue_text_query(msg.participant_id, text) - - async def _wait_lovr(self, timeout_s: float = 120.0) -> bool: - deadline = asyncio.get_running_loop().time() + timeout_s - while asyncio.get_running_loop().time() < deadline: - h = await self._call_render("get_health", {}, silent=True) - if h: - if h.get("lovr_started"): - return True - if h.get("spawn_error"): - logger.error("spawn_error: {}", h["spawn_error"]) - return False - await asyncio.sleep(0.5) - logger.warning("lovr_started never true within {:.0f}s", timeout_s) - return False - - # ── participant tracking ─────────────────────────────────────────────────── - - async def _on_audio(self, chunk: AudioChunk) -> None: - # Lazily set target participant from the first audio chunk so TTS - # can respond even if the xr.session.started message was missed. - if chunk.participant_id and not self._transport.target_participant: - self._transport.set_target_participant(chunk.participant_id) - - async def _on_participant(self, event: ParticipantEvent) -> None: - if event.joined: - self._transport.set_target_participant(event.participant_id) - else: - self._transport.cleanup_participant(event.participant_id) - - # ── native scene helper ─────────────────────────────────────────────────── - - async def _call_render(self, tool: str, args: dict, *, silent: bool = False) -> dict | None: - try: - data = await self._tools.invoke(tool, args) - if not isinstance(data, dict): - if not silent: - logger.error("scene tool {} returned non-dict: {!r}", tool, data) - return None - return data - except Exception as exc: - if not silent: - logger.error("scene tool {}: {}", tool, exc) - return None diff --git a/agent-samples/xr-render-demo/worker/capabilities.py b/agent-samples/xr-render-demo/worker/capabilities.py deleted file mode 100644 index 6db74875..00000000 --- a/agent-samples/xr-render-demo/worker/capabilities.py +++ /dev/null @@ -1,418 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -"""Sample-facing NAT functions used by the existing XR scene agent loop.""" - -import math -import time -from typing import Annotated, Any, Literal - -from nat.builder.function import Function -from nat.builder.workflow_builder import WorkflowBuilder -from nat.plugin_api import ( - Builder, - FunctionGroup, - FunctionGroupBaseConfig, - FunctionGroupRef, - register_function_group, -) -from pydantic import BaseModel, Field -from xr_ai_models import ToolDef -from xr_ai_nat.functions.spatial_math import SpatialMathFunctionsConfig -from xr_ai_nat.functions.video_memory import VideoMemoryFunctionsConfig -from xr_ai_nat.functions.vision import VisionToolsConfig -from xr_ai_nat.functions.xr_tracking import XRTrackingFunctionsConfig -from xr_render_scene import ( - SceneControlFunctionsConfig, - SceneObjectFunctionsConfig, - SceneStateFunctionsConfig, - SceneUpdateFunctionsConfig, -) - - -class _EmptyRequest(BaseModel): - pass - - -class RenderSpatialToolsConfig(FunctionGroupBaseConfig, name="xr_render_spatial_tools"): - """Compose tracking and spatial math into the demo's established tool vocabulary.""" - - tracking: FunctionGroupRef = FunctionGroupRef("tracking") - spatial_math: FunctionGroupRef = FunctionGroupRef("spatial_math") - - -@register_function_group(config_type=RenderSpatialToolsConfig) -async def render_spatial_tools(config: RenderSpatialToolsConfig, builder: Builder): - tracking_group = await builder.get_function_group(config.tracking) - tracking = await tracking_group.get_all_functions() - math_group = await builder.get_function_group(config.spatial_math) - spatial = await math_group.get_all_functions() - get_frame = tracking[f"{tracking_group.instance_name}__get_user_frame"] - - async def frame_dict() -> dict[str, Any]: - frame = await get_frame.ainvoke({}) - return frame.model_dump(mode="python") - - async def get_head_pose(request: _EmptyRequest) -> dict[str, Any]: - del request - frame = await frame_dict() - forward = frame["forward"] - return { - "is_valid": True, - "position": frame["origin"], - "forward": forward, - "right": frame["right"], - "up": frame["up"], - "yaw_deg": math.degrees(math.atan2(-forward["x"], -forward["z"])), - "pitch_deg": math.degrees(math.asin(max(-1.0, min(1.0, forward["y"])))), - "ts": time.time_ns() // 1_000_000, - } - - async def position_ahead( - distance: Annotated[float, Field(description="Distance along the user's gaze, in metres.")] = 1.5, - ) -> dict[str, float]: - result = await spatial[f"{math_group.instance_name}__compute_gaze_target"].ainvoke( - {"user_frame": await frame_dict(), "distance_meters": distance} - ) - return result.model_dump(mode="python") - - async def position_relative( - forward: float = 0.0, - right: float = 0.0, - up: float = 0.0, - origin_x: float | None = None, - origin_y: float | None = None, - origin_z: float | None = None, - ) -> dict[str, float]: - frame = await frame_dict() - origin = frame["origin"] - result = await spatial[f"{math_group.instance_name}__offset_position_in_user_frame"].ainvoke( - { - "user_frame": frame, - "start_position": { - "x": origin["x"] if origin_x is None else origin_x, - "y": origin["y"] if origin_y is None else origin_y, - "z": origin["z"] if origin_z is None else origin_z, - }, - "forward_meters": forward, - "right_meters": right, - "up_meters": up, - } - ) - return result.model_dump(mode="python") - - async def place_user_relative( - direction: Literal["front", "back", "left", "right", "above", "below"], - distance: float = 1.5, - ) -> dict[str, Any]: - if distance < 0: - return {"error": "distance must be non-negative; flip the direction instead"} - result = await spatial[f"{math_group.instance_name}__compute_user_relative_position"].ainvoke( - { - "user_frame": await frame_dict(), - "direction_from_user": direction, - "distance_meters": distance, - } - ) - return result.model_dump(mode="python") - - async def place_object_relative( - origin_x: float, - origin_y: float, - origin_z: float, - direction: Literal["front", "back", "left", "right", "above", "below", "next_to"], - distance: float = 0.3, - ) -> dict[str, Any]: - if distance < 0: - return {"error": "distance must be non-negative; flip the direction instead"} - relation = { - "front": "toward_user", - "back": "away_from_user", - "left": "left_of", - "right": "right_of", - "next_to": "right_of", - "above": "above", - "below": "below", - }[direction] - frame = ( - await frame_dict() - if direction in {"front", "back", "left", "right", "next_to"} - else { - "origin": {"x": 0.0, "y": 0.0, "z": 0.0}, - "forward": {"x": 0.0, "y": 0.0, "z": -1.0}, - "right": {"x": 1.0, "y": 0.0, "z": 0.0}, - "up": {"x": 0.0, "y": 1.0, "z": 0.0}, - } - ) - result = await spatial[f"{math_group.instance_name}__compute_position_relative_to_anchor"].ainvoke( - { - "user_frame": frame, - "anchor_position": {"x": origin_x, "y": origin_y, "z": origin_z}, - "relation_to_anchor": relation, - "distance_meters": distance, - } - ) - return result.model_dump(mode="python") - - async def displace_object( - current_x: float, - current_y: float, - current_z: float, - right: float = 0.0, - up: float = 0.0, - forward: float = 0.0, - ) -> dict[str, float]: - result = await spatial[f"{math_group.instance_name}__offset_position_in_user_frame"].ainvoke( - { - "user_frame": await frame_dict(), - "start_position": {"x": current_x, "y": current_y, "z": current_z}, - "forward_meters": forward, - "right_meters": right, - "up_meters": up, - } - ) - return result.model_dump(mode="python") - - async def displace_objects( - object_ids: list[str], - current_xs: list[float], - current_ys: list[float], - current_zs: list[float], - right: float = 0.0, - up: float = 0.0, - forward: float = 0.0, - ) -> dict[str, Any]: - if not (len(object_ids) == len(current_xs) == len(current_ys) == len(current_zs)): - return {"error": ("object_ids / current_xs / current_ys / current_zs must all be the same length")} - frame = await frame_dict() - function = spatial[f"{math_group.instance_name}__offset_position_in_user_frame"] - items = [] - for obj_id, x, y, z in zip(object_ids, current_xs, current_ys, current_zs, strict=True): - result = await function.ainvoke( - { - "user_frame": frame, - "start_position": {"x": x, "y": y, "z": z}, - "forward_meters": forward, - "right_meters": right, - "up_meters": up, - } - ) - items.append({"obj_id": obj_id, **result.model_dump(mode="python")}) - return {"items": items} - - async def place_inside_by_id( - movee_id: str, - container_x: float, - container_y: float, - container_z: float, - ) -> dict[str, Any]: - return { - "obj_id": movee_id, - "x": round(container_x, 3), - "y": round(container_y, 3), - "z": round(container_z, 3), - } - - async def between_anchors( - a_x: float, - a_y: float, - a_z: float, - b_x: float, - b_y: float, - b_z: float, - ) -> dict[str, float]: - result = await spatial[f"{math_group.instance_name}__compute_midpoint"].ainvoke( - { - "first_position": {"x": a_x, "y": a_y, "z": a_z}, - "second_position": {"x": b_x, "y": b_y, "z": b_z}, - } - ) - return result.model_dump(mode="python") - - async def world_offset( - origin_x: float, - origin_y: float, - origin_z: float, - dx: float = 0.0, - dy: float = 0.0, - dz: float = 0.0, - ) -> dict[str, float]: - return {"x": origin_x + dx, "y": origin_y + dy, "z": origin_z + dz} - - async def along_direction( - origin_x: float, - origin_y: float, - origin_z: float, - target_x: float, - target_y: float, - target_z: float, - distance: float = 0.5, - ) -> dict[str, float]: - result = await spatial[f"{math_group.instance_name}__compute_position_toward_or_away_from_reference"].ainvoke( - { - "start_position": {"x": origin_x, "y": origin_y, "z": origin_z}, - "reference_position": {"x": target_x, "y": target_y, "z": target_z}, - "movement_direction": "toward" if distance >= 0 else "away", - "distance_meters": abs(distance), - } - ) - return result.model_dump(mode="python") - - async def scale_value(current: float, factor: float) -> dict[str, float]: - return {"value": round(current * factor, 3)} - - group = FunctionGroup(config=config) - group.add_function( - "get_head_pose", - get_head_pose, - description="Return the current world-space head position and forward, right, and up axes.", - ) - group.add_function( - "position_ahead", - position_ahead, - description="Compute a world position along the user's gaze for 'in front of me' requests.", - ) - group.add_function( - "position_relative", - position_relative, - description=( - "Apply signed forward, right, and up user-frame offsets. Omit origin coordinates " - "to start at the user, or pass an object's current position to move it." - ), - ) - group.add_function( - "place_user_relative", - place_user_relative, - description=( - "Compute a position in one named direction from the user. Distance is non-negative; " - "choose front, back, left, right, above, or below to set the direction." - ), - ) - group.add_function( - "place_object_relative", - place_object_relative, - description="Compute a position in one named direction from an existing object's world position.", - ) - group.add_function( - "displace_object", - displace_object, - description="Shift one existing object by signed right, up, and forward user-frame offsets.", - ) - group.add_function( - "displace_objects", - displace_objects, - description="Apply one signed user-frame offset to parallel lists of existing objects.", - ) - group.add_function( - "place_inside_by_id", - place_inside_by_id, - description="Return the container position with the ID of the object that should move there.", - ) - group.add_function( - "between_anchors", - between_anchors, - description="Compute the world-space midpoint between exactly two anchor positions.", - ) - group.add_function( - "world_offset", - world_offset, - description="Apply signed world-axis dx, dy, and dz offsets to an origin position.", - ) - group.add_function( - "along_direction", - along_direction, - description="Move an origin toward a target by positive distance or away by negative distance.", - ) - group.add_function( - "scale_value", - scale_value, - description="Multiply a current numeric size by a scale factor.", - ) - yield group - - -class NativeToolbox: - """Present selected NAT functions to the existing model-service tool loop.""" - - def __init__(self, functions: dict[str, Function]) -> None: - self._functions: dict[str, Function] = {} - for function in functions.values(): - short_name = function.instance_name.rsplit("__", 1)[-1] - if short_name in self._functions: - raise ValueError(f"duplicate native tool name: {short_name}") - self._functions[short_name] = function - - def definitions(self, *, exclude: set[str] | frozenset[str] = frozenset()) -> list[ToolDef]: - return [ - ToolDef( - name=name, - description=(function.description or "").strip(), - parameters=function.input_schema.model_json_schema(), - ) - for name, function in self._functions.items() - if name not in exclude - ] - - async def invoke(self, name: str, arguments: dict[str, Any]) -> Any: - result = await self._functions[name].ainvoke(arguments) - return _plain(result) - - -async def build_native_toolbox( - builder: WorkflowBuilder, - *, - scene_endpoint: str, - openxr_endpoint: str, - video_memory_endpoint: str, - frame_endpoint: Any, - vlm: Any, -) -> tuple[NativeToolbox, VisionToolsConfig]: - """Build the model-facing toolbox shared by the live worker and eval. - - Returns the toolbox and the vision config so the worker can release cached - live-frame state when a participant leaves. - """ - vision_config = VisionToolsConfig( - endpoint=frame_endpoint, - vlm=vlm, - video_memory=FunctionGroupRef("video_memory"), - ) - for name, config in ( - ("scene_state", SceneStateFunctionsConfig(endpoint=scene_endpoint)), - ("scene_updates", SceneUpdateFunctionsConfig(endpoint=scene_endpoint)), - ("scene_objects", SceneObjectFunctionsConfig(endpoint=scene_endpoint)), - ("scene_control", SceneControlFunctionsConfig(endpoint=scene_endpoint)), - ("tracking", XRTrackingFunctionsConfig(endpoint=openxr_endpoint)), - ("spatial_math", SpatialMathFunctionsConfig()), - ("render_spatial", RenderSpatialToolsConfig()), - ("video_memory", VideoMemoryFunctionsConfig(endpoint=video_memory_endpoint)), - ("vision", vision_config), - ): - await builder.add_function_group(name, config) - - functions: dict[str, Function] = {} - for name in ( - "scene_state", - "scene_updates", - "scene_objects", - "scene_control", - "render_spatial", - "video_memory", - "vision", - ): - group = await builder.get_function_group(name) - functions.update(await group.get_all_functions()) - return NativeToolbox(functions), vision_config - - -def _plain(value: Any) -> Any: - if isinstance(value, BaseModel): - return value.model_dump(mode="python") - if isinstance(value, list): - return [_plain(item) for item in value] - if isinstance(value, dict): - return {key: _plain(item) for key, item in value.items()} - return value - - -__all__ = ["NativeToolbox", "RenderSpatialToolsConfig", "build_native_toolbox"] diff --git a/agent-samples/xr-render-demo/worker/processors.py b/agent-samples/xr-render-demo/worker/processors.py deleted file mode 100644 index c82aeb28..00000000 --- a/agent-samples/xr-render-demo/worker/processors.py +++ /dev/null @@ -1,1069 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -""" -Brain processor for xr-render-demo. - -The voice pipeline (input → VadStt → VoiceGate → brain → StreamingTts → output) -is assembled by ``xr_ai_pipecat.make_voice_pipeline``. This module supplies -the sample-specific brain — a multi-step agentic loop over native NAT -functions for scene, tracking, spatial math, vision, and video memory. - -Agentic loop (max ``_MAX_LOOP`` iterations): - - Nemotron-3-Nano emits an OpenAI ``tool_calls`` payload → execute tool, - append result, continue. - - When the model returns text instead of a tool call, that text is the - final user-visible response. - -An initial "quick-ack" call is awaited before the agentic loop starts -to (a) speak an immediate acknowledgment and (b) classify whether the -agentic loop needs thinking enabled. A periodic "still-working" loop -streams contextual progress messages to the data channel while the agent -reasons. -""" - -from __future__ import annotations - -import asyncio -import json -import time -import uuid -from pathlib import Path -from collections.abc import Callable -from typing import AsyncIterator - -from loguru import logger -from nat.builder.function import Function - -from xr_ai_hub import DataMessage -from xr_ai_logging import print_task_done_banner -from xr_ai_models import ChatMessage, LLMService, ToolCall, ToolDef -from xr_ai_pipecat import BrainProcessor -from xr_ai_pipecat.transport import XRMediaHubTransport - -from config import WorkerConfig -from capabilities import NativeToolbox -from tooling import ( - extract_json, - looks_like_leaked_tool_call, - normalize_tool_args, -) - -# Dedicated trace logger — writes a clean session transcript to a file. -# Key events: user speech, pre-fetched context, think flag, tool calls + -# results, agent response, validation. Records bound with this binding -# are routed to ``/tmp/xr-agent-trace.log`` by the sink installed in -# ``xr_render_demo_worker.main()``; everything else is unaffected. -_trace_log = logger.bind(trace=True) - -_MAX_LOOP = 10 # visual queries need up to 5 steps; give headroom - - -# Native perception tools (from the ``xr_vision_tools`` group) exposed to the -# model. The processor supplies participant context (and the utterance time for -# recorded lookups) that the model never provides. -_LIVE_PERCEPTION_TOOL = "look_at_current_frame" -_PAST_PERCEPTION_TOOL = "look_at_past_frame" - -# Model-facing perception schemas. The native ``xr_vision_tools`` request models -# also carry ``participant_id`` (and ``reference_time_us`` for recorded lookups), -# which the processor injects — the model never supplies them. Presenting the raw -# generated schema would tell the model to fill a required ``participant_id`` it -# cannot know and whose value is discarded, so the model sees these trimmed -# contracts instead (the worker swaps them in for the native ones). -_LIVE_PERCEPTION_TOOL_DEF = ToolDef( - name=_LIVE_PERCEPTION_TOOL, - description=( - "Inspect the user's present physical view when a request explicitly requires a visible fact. " - "Do not use this tool to interpret conversation or inspect the virtual XR scene." - ), - parameters={ - "type": "object", - "properties": { - "question": { - "type": "string", - "description": "Specific question about the live camera frame.", - }, - }, - "required": ["question"], - }, -) - -_PAST_PERCEPTION_TOOL_DEF = ToolDef( - name=_PAST_PERCEPTION_TOOL, - description=( - "Inspect a recorded camera frame only for an explicitly historical question, using a positive " - "seconds offset from the user's utterance time." - ), - parameters={ - "type": "object", - "properties": { - "question": { - "type": "string", - "description": "Specific question about the recorded camera frame.", - }, - "second_ago": { - "type": "integer", - "description": "Positive offset from the utterance time in seconds.", - }, - }, - "required": ["question", "second_ago"], - }, -) - -# Presented to the model in place of the native perception request schemas. -_PERCEPTION_TOOL_DEFS: tuple[ToolDef, ...] = ( - _LIVE_PERCEPTION_TOOL_DEF, - _PAST_PERCEPTION_TOOL_DEF, -) - -# Spoken when a perception query is asked but no live camera frame can be -# obtained. Short, user-actionable, never a hang or silent failure. -_NO_FRAME_MSG = "I can't see a camera feed right now — please check your camera." - -# Brief human-readable progress message shown while a tool runs. -_TOOL_PROGRESS: dict[str, str] = { - "get_head_pose": "Checking your position...", - "position_ahead": "Computing gaze position...", - "position_relative": "Computing relative position...", - "place_user_relative": "Placing relative to you...", - "place_object_relative": "Placing relative to object...", - "place_inside_by_id": "Placing inside container...", - "displace_object": "Shifting object...", - "displace_objects": "Shifting objects...", - "between_anchors": "Computing midpoint...", - "world_offset": "Computing offset...", - "along_direction": "Computing position...", - "scale_value": "Computing size...", - "get_scene_state": "Scanning the scene...", - "add_primitive": "Creating object...", - "update_primitive": "Updating object...", - "remove_primitive": "Removing object...", -} - -_AGENT_RESPONSE_TOPIC = "agent.response" -_AGENT_PROGRESS_TOPIC = "agent.progress" - - -class _SceneNotReadyError(Exception): - """Raised when the scene service reports that LOVR has not launched yet.""" - - -class _PerceptionUnavailableError(Exception): - """Raised when a perception (look_at_current_frame) query cannot be - answered — no camera feed, no frame, or VLM failure. Carries a short - user-facing spoken message so the turn ends with a graceful spoken+panel - line instead of hanging in the reasoning loop or failing silently.""" - - def __init__(self, spoken: str) -> None: - super().__init__(spoken) - self.spoken = spoken - - -def _now_us() -> int: - return time.time_ns() // 1_000 - - -# ── RenderSceneProcessor ────────────────────────────────────────────────────── - - -class RenderSceneProcessor(BrainProcessor): - """ - Multi-step agentic loop over native NAT functions. - - Uses Nemotron-3-Nano (port 8107) with OpenAI tool calling for the - reasoning loop; the pre-loop quick-ack shares the same server via the - `llm` logical model. - - On each utterance: - 1. Quick-ack is awaited first (max 40 tokens) → agent.progress - 2. Agentic loop: model calls tools via OpenAI tool_calls protocol until - it returns a text response (finish_reason != "tool_calls") - 3. Progress messages sent before each tool execution → agent.progress - 4. Final response → agent.response + yielded to TTS - """ - - def __init__( - self, - *, - transport: XRMediaHubTransport, - cfg: WorkerConfig, - toolbox: NativeToolbox, - release_vision: Callable[[str], None], - text_memory: Function | None, - prompt_path: Path, - tools: list[ToolDef], - llm: LLMService, - agent_llm: LLMService, - ): - super().__init__() - self._transport = transport - self._cfg = cfg - self._toolbox = toolbox - self._release_vision = release_vision - self._text_memory = text_memory - self._prompt_path = prompt_path - self._prompt_cache = prompt_path.read_text(encoding="utf-8").strip() - _prompts = prompt_path.parent - self._quick_ack_path = _prompts / "quick_ack.txt" - self._still_work_path = _prompts / "still_working.txt" - self._quick_ack_cache = self._quick_ack_path.read_text(encoding="utf-8").strip() - self._still_work_cache = self._still_work_path.read_text(encoding="utf-8").strip() - self._tools = tools - self._llm = llm - self._agent_llm = agent_llm - - # Rolling conversation buffer — last N turns of (user_text, agent_response). - # Injected as context so the agent understands "fix that", "undo", "the one I just added". - self._history: list[tuple[str, str]] = [] - self._history_max = 4 - - # Move log for "put it back" — (obj_id, prev, new), capped at N. - self._recent_moves: list[tuple[str, tuple[float, float, float], tuple[float, float, float]]] = [] - self._recent_moves_max = 5 - # Per-turn snapshot used to compute prev→new pairs on update_primitive. - self._pre_move_positions: dict[str, tuple[float, float, float]] = {} - - # Canned spoken notices the agent (XR lifecycle, outside handle_query) - # asks us to deliver. Keyed by pid → list of exact strings. Drained in - # handle_query, which short-circuits the LLM loop for a matching entry. - # See enqueue_notice / _emit_notice. - self._pending_notices: dict[str, list[str]] = {} - - # ── public: text-channel entry ──────────────────────────────────────────── - - async def enqueue_text_query(self, pid: str, text: str) -> None: - """Run a typed text query through the same path as a spoken utterance. - - The web client's "Send" button posts on the data channel; the brain - needs to fire ``handle_query`` for it identically to a transcript - that passed the voice gate. The base class' ``_spawn_query`` owns - the per-pid in-flight task and cancellation semantics, so we route - through a synthesized ``GatedQueryFrame``. - """ - from xr_ai_pipecat import GatedQueryFrame - - await self._spawn_query( - GatedQueryFrame( - participant_id=pid, - text=text, - fresh_match=False, - pts_us=_now_us(), - ) - ) - - async def enqueue_notice(self, pid: str, text: str) -> None: - """Speak a canned, agent-authored notice through the normal turn path. - - XR-lifecycle failures (start_xr error, LOVR never ready) happen in - ``RenderDemoAgent``, outside ``handle_query``'s yield→TTS path. To - surface them with voice *and* a panel line — the same delivery shape - as a normal final answer (``_send(agent.response)`` + ``yield``) — - the agent hands us the message here. We register it as pending for - *pid* and inject a ``GatedQueryFrame`` through the same - ``_spawn_query`` machinery a typed/spoken query uses, so the base - class owns the per-pid in-flight task and its cancellation. The text - is matched (and consumed) by ``handle_query`` below, which yields it - verbatim instead of running the agentic loop. Matching on the exact - string — not just pid — means a real query that interleaves before - the notice task runs is never mistaken for the notice. - """ - from xr_ai_pipecat import GatedQueryFrame - - self._pending_notices.setdefault(pid, []).append(text) - await self._spawn_query( - GatedQueryFrame( - participant_id=pid, - text=text, - fresh_match=False, - pts_us=_now_us(), - ) - ) - - # ── BrainProcessor overrides ────────────────────────────────────────────── - - async def handle_query( - self, - pid: str, - text: str, - fresh_match: bool, - ) -> AsyncIterator[str]: - """Drive one full turn of the agentic loop for *text* from *pid*. - - Yields strings that should reach TTS: - - the quick-ack, on EVERY turn — spoken first so the user always - gets immediate audio feedback, especially before a tool-using - turn that would otherwise be silent until the final reply - - the final user-visible response. - - Per-tool progress and still-working ticks are sent to the data - channel (``send_return_data``) only, NOT spoken: a long agentic loop - can emit many of them and speaking each would stack the TTS queue and - play after the real reply. The single spoken ack covers "I'm on it"; - the panel carries the detailed progress. - """ - # Canned agent notice (XR-lifecycle failure) — speak it verbatim and - # skip the LLM loop. Exact-text match guards against a real query - # interleaving before the notice task runs. See enqueue_notice. - pending = self._pending_notices.get(pid) - if pending and text in pending: - pending.remove(text) - if not pending: - self._pending_notices.pop(pid, None) - return self._emit_notice(pid, text) - return self._run_turn(pid, text) - - async def _emit_notice(self, pid: str, text: str) -> AsyncIterator[str]: - """Deliver a canned notice with the same shape as a final answer: - a panel line on ``agent.response`` plus a spoken (yielded) line.""" - send_pid = pid or self._transport.target_participant - if send_pid: - await self._send(send_pid, text, topic=_AGENT_RESPONSE_TOPIC) - yield text - - async def _run_turn(self, pid: str, text: str) -> AsyncIterator[str]: - text = text.strip() - if not text: - return - send_pid = pid or self._transport.target_participant - # Bracket the whole turn with the client UI status signal: "processing" - # on entry, "idle" in finally so it always clears — including failure or - # a barge-in cancellation. The "processing" publish is INSIDE the try so - # a cancellation landing during that await still runs the finally (never - # leaving the client stuck in "processing"). Status is a per-client - # lifecycle signal owned by the render turn; the native vision functions - # stay reusable/UI-free. - try: - if send_pid: - await self._set_status("processing", send_pid) - async for chunk in self._run_turn_body(pid, send_pid, text): - yield chunk - finally: - if send_pid: - await self._set_status("idle", send_pid) - - async def _run_turn_body(self, pid: str, send_pid: str, text: str) -> AsyncIterator[str]: - # Capture the moment the user finished speaking so visual tool calls - # can be anchored to that timestamp (not to when the tool fires). - ref_us = _now_us() - t0 = time.monotonic() - - # Quick-ack: fast LLM call that (a) speaks an immediate - # acknowledgment and (b) classifies whether Nemotron needs - # reasoning enabled. Await it first so the think flag is ready - # before the main loop starts. On failure _quick_ack falls back to - # ("", False); thinking degrades tool-following accuracy. - ack, needs_thinking = await self._quick_ack(text) - - if ack and send_pid: - # ACK-SPEAK POLICY (deliberate): speak the quick-ack on EVERY turn, - # not just needs_thinking ones. It's yielded before the agentic - # loop runs, so TTS plays it first — giving the user immediate - # audio feedback at the start of every turn. This matters most for - # tool-using turns (which may not be flagged needs_thinking yet - # still take seconds): without a spoken ack the user hears nothing - # until the final reply. Per-tool progress + still-working ticks - # remain text-only (below) so they don't stack the TTS queue; the - # single spoken ack is enough to signal "I'm on it". Also mirror - # the ack to the panel. - await self._send(send_pid, ack, topic=_AGENT_PROGRESS_TOPIC) - # TTS batches on sentence-final punctuation; an unterminated ack - # would sit in its buffer until the final response lands and play - # concatenated with it. - yield ack if ack[-1] in ".!?" else ack + "." - - # Start a "still working" timer — fires if reasoning takes >5s. - # Cancelled as soon as the loop returns. - # thinking_ctx is a one-element list shared with the agentic loop so - # the still-working messages can reflect what the 30B is reasoning about. - thinking_ctx: list[str] = [""] - still_task = asyncio.create_task( - self._still_working_loop(text, send_pid, thinking_ctx), - name="still-working", - ) - - response: str | None = None - outcome = "done" - try: - response = await self._agentic_loop( - text, - pid, - ref_us=ref_us, - needs_thinking=needs_thinking, - thinking_ctx=thinking_ctx, - ) - except asyncio.CancelledError: - outcome = "interrupted" - logger.info("agentic loop interrupted by new utterance") - if send_pid: - try: - await self._transport.endpoint.flush_return_audio(send_pid) - except Exception: - logger.opt(exception=True).debug( - "flush_return_audio failed during cancellation", - ) - raise - except Exception: - outcome = "error" - logger.exception("agentic loop failed") - response = "Something went wrong — please try again." - finally: - still_task.cancel() - try: - await still_task - except asyncio.CancelledError: - pass # expected — we just cancelled it above - print_task_done_banner( - "xr-render-demo", - status=outcome, - detail=f"pid={pid!r} utterance={text[:60]!r}", - duration_s=time.monotonic() - t0, - ) - - # Strip leaked tool-call JSON from both the user-visible reply and - # history; legit text starting with "{" passes through. - # - # Defensive guard only: _agentic_loop always returns a non-empty - # string ("Done." fallbacks on lines ~697/741, the not-ready string, - # or the error string set above), and the cancellation branch - # re-raises before reaching here. So this never fires for a real - # turn — we intentionally do NOT yield a spoken "Done." here, because - # there is no audible-close gap to fill. Every reachable turn already - # ends with a yielded final response below. - if not response: - return - - display = response - if looks_like_leaked_tool_call(response): - logger.warning( - "response looks like a leaked tool call, sanitizing: {!r}", - response[:120], - ) - display = "Done." - - self._history.append((text, display)) - if len(self._history) > self._history_max: - self._history.pop(0) - - if self._text_memory is not None and send_pid: - await self._record_turn(send_pid, ref_us, text, display) - - if send_pid: - await self._send(send_pid, display, topic=_AGENT_RESPONSE_TOPIC) - yield display - - async def _record_turn( - self, - participant_id: str, - request_time_us: int, - user_text: str, - assistant_text: str, - ) -> None: - """Persist final user and assistant messages without agent scratch state.""" - for timestamp_us, text in ( - (request_time_us, f"user: {user_text}"), - (_now_us(), f"assistant: {assistant_text}"), - ): - try: - await self._text_memory.ainvoke( - { - "source_id": participant_id, - "timestamp_us": timestamp_us, - "text": text, - } - ) - except Exception: - logger.opt(exception=True).warning("text-memory write failed") - - def _read_prompt(self, path: Path, cache_attr: str) -> str: - try: - text = path.read_text(encoding="utf-8").strip() - setattr(self, cache_attr, text) - return text - except OSError: - logger.warning("prompt file unreadable: {} — using cache", path.name) - return getattr(self, cache_attr) - - # ── quick ack ───────────────────────────────────────────────────────────── - - async def _quick_ack(self, transcript: str) -> tuple[str, bool]: - """Fast call: returns (ack_text, needs_thinking). - - Passes the last conversation turn as context so corrections like - "try it again" or "that was wrong" produce sensible acks. - """ - # Include the most recent agent action as context. - context = "" - if self._history: - last_user, last_agent = self._history[-1] - context = f"[Previous turn] User: {last_user} / Agent: {last_agent}\n" - - messages = [ - ChatMessage(role="system", content=self._read_prompt(self._quick_ack_path, "_quick_ack_cache")), - ChatMessage(role="user", content=context + transcript), - ] - try: - resp = await asyncio.wait_for( - self._llm.chat(messages, max_tokens=40, temperature=0.0), - timeout=8.0, - ) - raw = resp.content.strip() - obj_text = extract_json(raw) - if obj_text: - try: - obj = json.loads(obj_text) - ack = str(obj.get("ack", "")).strip() - # Strict is-True: a model emitting "think": "false" must - # not truthy its way into the thinking path. - think = obj.get("think") is True - logger.info("quick-ack: {!r} think={}", ack, think) - _trace_log.info("ACK {} [think={}]", ack, think) - return ack, think - except json.JSONDecodeError: - pass - # Fallback: treat raw text as ack, no thinking. A truncated JSON - # payload has no closing brace, so extract_json returns None — - # don't speak the fragment. - if raw.startswith("{"): - return "", False - return raw, False - except Exception as exc: - logger.warning("quick-ack failed: {!r}", exc) - return "", False - - # ── agentic loop (OpenAI tool calling) ─────────────────────────────────── - - async def _still_working_msg(self, transcript: str, sent: list[str], thinking_ctx: list[str]) -> str: - """Ask the LLM for a short contextual 'still working' sentence. - - `sent` is the list of messages already shown this turn. - `thinking_ctx` is a one-element list holding the latest reasoning_content - from the 30B model — used to make progress updates reflect what the model - is actually working on rather than guessing from the transcript alone. - """ - avoid = "" - if sent: - avoid = ( - " Do NOT repeat or paraphrase any of these already-sent messages: " - + ", ".join(f'"{m}"' for m in sent[-3:]) - + "." - ) - thinking = thinking_ctx[0].strip() if thinking_ctx[0] else "" - # Truncate — we only need the last few lines of thinking to get the gist. - if thinking: - lines = [l.strip() for l in thinking.splitlines() if l.strip()] - thinking = " ".join(lines[-6:])[-400:] - - user_content = f"User request: {transcript}" - if thinking: - user_content += f"\n\nWhat the AI is currently reasoning through:\n{thinking}" - - base = self._read_prompt(self._still_work_path, "_still_work_cache") - messages = [ - ChatMessage(role="system", content=base + avoid), - ChatMessage(role="user", content=user_content), - ] - try: - resp = await asyncio.wait_for( - self._llm.chat(messages, max_tokens=24, temperature=0.9), - timeout=6.0, - ) - return resp.content.strip() - except Exception as exc: - logger.debug("still-working message failed: {}", exc) - return "" - - async def _still_working_loop( - self, - transcript: str, - pid: str, - thinking_ctx: list[str], - *, - first_after: float = 5.0, - repeat_every: float = 10.0, - ) -> None: - """Post periodic contextual updates while the agentic loop runs. - - Purely time-gated: any turn still running after ``first_after`` - gets panel updates, thinking or not. - """ - sent: list[str] = [] - await asyncio.sleep(first_after) - while True: - msg = await self._still_working_msg(transcript, sent, thinking_ctx) - if msg and pid: - # PROGRESS POLICY (deliberate divergence from simple-vlm): - # progress is panel-only, never spoken. render-demo's long - # multi-tool loops emit many of these; speaking them would - # stack in the TTS queue and play after the real response. - # Only acks (on thinking turns), the final answer, and - # failure notices are spoken — see ACK-SPEAK POLICY above. - await self._send(pid, msg, topic=_AGENT_PROGRESS_TOPIC) - sent.append(msg) - await asyncio.sleep(repeat_every) - - async def _build_turn_context(self, pid: str, *, ref_us: int = 0) -> str: - """Pre-fetch scene/pose and format the turn-context block. - - Fetches scene state, head pose, and the most common spatial position - (1.5 m ahead) concurrently — saves 1-3 tool-call iterations per turn — - and renders them, plus the move log and conversation history, into the - text injected into the agentic loop's user message. Side effect: resets - ``self._pre_move_positions`` to the current scene so update_primitive - calls during this turn can be recorded as (prev → new) move-log entries. - """ - scene, pose, ahead = await asyncio.gather( - self._call_tool("get_scene_state", {}, silent=True), - self._call_tool("get_head_pose", {}, silent=True), - self._call_tool("position_ahead", {"distance": 1.5}, silent=True), - ) - - ctx_parts: list[str] = [] - - # ── Scene ────────────────────────────────────────────────────────────── - self._pre_move_positions = {} - if isinstance(scene, dict) and scene.get("objects"): - objs = scene["objects"] - lines = ["SCENE OBJECTS:"] - for o in objs: - pos = o.get("position", {}) - col = o.get("color", {}) - self._pre_move_positions[o["id"]] = ( - float(pos.get("x", 0)), - float(pos.get("y", 0)), - float(pos.get("z", 0)), - ) - lines.append( - f" {o['id']} ({o['type']}) " - f"pos=({pos.get('x', 0):.2f}, {pos.get('y', 0):.2f}, {pos.get('z', 0):.2f}) " - f"color=(r={col.get('r', 0):.2f} g={col.get('g', 0):.2f} b={col.get('b', 0):.2f}) " - f"size={o.get('size', 0.1):.3f}m" - ) - ctx_parts.append("\n".join(lines)) - else: - ctx_parts.append("SCENE OBJECTS: (empty)") - - # ── Head pose + derived spatial shortcuts ───────────────────────────── - if isinstance(pose, dict) and pose.get("is_valid"): - p = pose["position"] - fv = pose["forward"] - rv = pose["right"] - uv = pose.get("up", {"x": 0, "y": 1, "z": 0}) - - # Compute common offsets directly — no extra tool calls needed. - def _off(vec: dict, d: float) -> str: - return f"({p['x'] + vec['x'] * d:.2f}, {p['y'] + vec['y'] * d:.2f}, {p['z'] + vec['z'] * d:.2f})" - - ahead_str = ( - f"({ahead['x']:.2f}, {ahead['y']:.2f}, {ahead['z']:.2f})" - if isinstance(ahead, dict) and "x" in ahead - else _off(fv, 1.5) - ) - - ctx_parts.append( - "HEAD POSE:\n" - f" position : ({p['x']:.2f}, {p['y']:.2f}, {p['z']:.2f})\n" - f" forward : ({fv['x']:.3f}, {fv['y']:.3f}, {fv['z']:.3f}) ← 'ahead/forward'\n" - f" right : ({rv['x']:.3f}, {rv['y']:.3f}, {rv['z']:.3f}) ← 'right'\n" - f" up : ({uv['x']:.3f}, {uv['y']:.3f}, {uv['z']:.3f}) ← 'up'\n" - f" yaw={pose.get('yaw_deg', 0):.1f}° pitch={pose.get('pitch_deg', 0):.1f}°\n" - "SPATIAL SHORTCUTS (pre-computed — use directly, no tool call needed):\n" - f" 1.5m ahead of you : {ahead_str}\n" - f" 1m to your right : {_off(rv, 1.0)}\n" - f" 1m to your left : {_off(rv, -1.0)}\n" - f" 0.5m above eye level : {_off(uv, 0.5)}\n" - f" 1m behind you : {_off(fv, -1.0)}\n" - " For other distances: new_pos = obj.pos + direction_vec × distance (per component)" - ) - else: - ctx_parts.append("HEAD POSE: unavailable") - - if pid: - ctx_parts.append(f"Participant: {pid}") - if ref_us: - ctx_parts.append(f"Reference time (when user spoke): {ref_us} µs") - - # Structured move log — machine-readable prior coords for "put it - # back" / "undo" / "revert" so the model doesn't have to parse free - # text out of the conversation history. - if self._recent_moves: - move_lines = [] - for obj_id, prev, new in self._recent_moves: - move_lines.append( - f" {obj_id}: ({prev[0]:.2f}, {prev[1]:.2f}, {prev[2]:.2f}) → " - f"({new[0]:.2f}, {new[1]:.2f}, {new[2]:.2f})" - ) - ctx_parts.append("[Recent moves] (most recent last — prev → new)\n" + "\n".join(move_lines)) - - # Recent conversation history — lets the agent understand "fix that", - # "undo", "the sphere I just added", etc. - if self._history: - hist_lines = [] - for u, a in self._history: - hist_lines.append(f" User: {u}") - hist_lines.append(f" Agent: {a}") - ctx_parts.append("[Recent conversation]\n" + "\n".join(hist_lines)) - - context = "\n".join(ctx_parts) - logger.debug("pre-fetched context for turn") - _trace_log.debug("CTX {}", context.replace("\n", " | ")) - return context - - def _recover_text_tool_call( - self, - content: str, - all_names: set[str], - ) -> dict | None: - """Recover a tool call the model emitted as plain text instead of via - the tool_calls field. Two shapes seen in practice: - (a) bare name: "get_head_pose" - (b) JSON obj: {"name": "update_primitive", "arguments": {...}} - Returns {"name", "arguments"} or None if *content* isn't a tool call. - """ - if content in all_names: - # Shape (a): bare tool name, no args. - return {"name": content, "arguments": {}} - - obj_text = extract_json(content) - if obj_text: - try: - obj = json.loads(obj_text) - name = obj.get("name") or obj.get("tool") or obj.get("function") - if isinstance(name, str) and name in all_names: - args = obj.get("arguments") or obj.get("args") or {} - return {"name": name, "arguments": args if isinstance(args, dict) else {}} - except json.JSONDecodeError: - # Best-effort recovery: the plain-text fragment is not valid JSON. - logger.debug("failed to decode recovered tool-call JSON: {!r}", obj_text) - return None - - async def _agentic_loop( - self, - transcript: str, - pid: str, - *, - ref_us: int = 0, - needs_thinking: bool = False, - thinking_ctx: list[str] | None = None, - ) -> str: - """ - Multi-turn tool-calling loop using the OpenAI tool calling protocol. - - Scene state and head pose are pre-fetched concurrently before the - loop starts and included in the user message, so the model skips - those tool calls and goes straight to the operation. Tools remain - available for refresh queries or follow-up calls. - """ - context = await self._build_turn_context(pid, ref_us=ref_us) - - try: - system_content = self._prompt_path.read_text(encoding="utf-8").strip() - self._prompt_cache = system_content - except OSError: - logger.warning("prompt file unreadable — using cached version") - system_content = self._prompt_cache - if needs_thinking: - system_content = ( - "Use your private block to work through these steps. " - "NEVER output these steps as your response — your only text output " - "to the user is ONE SHORT sentence AFTER all tool calls are done.\n" - "Be terse in : use notation not prose. " - "No full sentences, no restating the request. " - "Example: 'obj=sphere-1 pos=(1,1.7,-1.5) above→y=1.8' not " - "'We need to parse the request and find the sphere...'\n" - "\n" - "THINK STEP 1 — RESOLVE: Which object? " - "Pronouns ('it', 'that') = most recently added/modified object. " - "Named ('the blue sphere') = match by color/type in scene.\n" - "\n" - "THINK STEP 2 — LOCATE: Copy the exact x, y, z of the target object " - "and the head pose right/forward/up vectors from the context.\n" - "\n" - "THINK STEP 3 — COMPUTE: Calculate new coordinates with explicit arithmetic. " - "User-relative move: new = old + head_vec × distance (per component). " - "Near object: new = obj.pos ± world_offset. " - "Midpoint: new = (A + B) / 2 per component. " - "Write out each component: x=…, y=…, z=…\n" - "\n" - "THINK STEP 4 — EXECUTE: call the tool with the computed values, " - "then reply with ONE short sentence to the user.\n\n" + system_content - ) - - messages: list[ChatMessage] = [ - ChatMessage(role="system", content=system_content), - ChatMessage( - role="user", - content=( - f"[Pre-fetched context — do not call get_scene_state or " - f"get_head_pose unless you need to refresh after changes]\n" - f"{context}\n\n" - f"[Request]\n{transcript}" - ), - ), - ] - - for iteration in range(_MAX_LOOP): - try: - # thinking off: 1024 covers any tool-call JSON. - # thinking on: 4096 budget lets the model enumerate all scene - # objects and work through multi-step coordinate arithmetic - # without hitting the limit. 6144 total = 4096 thinking + 2048 - # for tool-call JSON / response text. - resp = await self._agent_llm.chat( - messages, - tools=self._tools, - max_tokens=6144 if needs_thinking else 1024, - temperature=0.0, - enable_thinking=needs_thinking, - thinking_budget=4096 if needs_thinking else None, - ) - except Exception: - logger.exception("agent-llm call failed on iteration {}", iteration) - return "Something went wrong — please try again." - - finish = resp.finish_reason or "" - tool_calls = resp.tool_calls or [] - content = resp.content.strip() - - # Share the 30B's reasoning with the still-working loop so progress - # updates reflect what the model is actually working on. - reasoning = (resp.reasoning or "").strip() - if reasoning and thinking_ctx is not None: - thinking_ctx[0] = reasoning - if reasoning: - logger.debug("agent-llm thinking iter={} {}", iteration, reasoning[:300]) - _trace_log.debug("THINK [{}] {}", iteration, reasoning[:300]) - - logger.debug( - "agent-llm iter={} finish={} tool_calls={} content={!r}", - iteration, - finish, - len(tool_calls), - content[:200], - ) - - if not tool_calls: - # Thinking filled the token budget without emitting a tool call. - # Turn off thinking and retry — `continue` in a for loop advances - # the iteration counter, but messages is unchanged so the model - # gets another chance with the same context. - if finish == "length" and needs_thinking: - logger.warning( - "agent-llm iter={} hit length limit during thinking — retrying without thinking", - iteration, - ) - needs_thinking = False - continue - - # Recover from off-script tool call output (model emitted a - # tool call as plain text instead of via the tool_calls field). - all_names = {t.name for t in self._tools} - recovered = self._recover_text_tool_call(content, all_names) - - if recovered: - logger.warning("text-format tool call {!r} — recovering", recovered["name"]) - tool_calls = [ - ToolCall( - id=f"call_{uuid.uuid4().hex[:12]}", - name=recovered["name"], - arguments=json.dumps(recovered["arguments"]), - ) - ] - else: - # Genuine final response. - _trace_log.info("RESP {}", content or "Done.") - return content or "Done." - - # Add the assistant's tool-call message to the conversation. - messages.append( - ChatMessage( - role="assistant", - content=content or "", - tool_calls=list(tool_calls), - ) - ) - - # The spatial thinking prompt helps plan the first action but - # actively harms subsequent iterations: it steers the model to - # re-anchor on the pre-fetched context (SCENE OBJECTS, HEAD POSE) - # instead of reading the tool results, producing wrong answers like - # "You're looking at empty space" when look_at_current_frame - # returned a valid VLM description. - needs_thinking = False - - # Execute each tool call and append results. - for tc in tool_calls: - name = tc.name - try: - args = json.loads(tc.arguments) - except json.JSONDecodeError: - args = {} - - progress = _TOOL_PROGRESS.get(name) - if progress and pid: - # PROGRESS POLICY (deliberate divergence from simple-vlm): - # per-tool progress is panel-only. A turn can fire several - # tools; speaking each would stack in the TTS queue behind - # the real answer. Only acks/final/failures are spoken. - await self._send(pid, progress, topic=_AGENT_PROGRESS_TOPIC) - - logger.debug("tool call iter={} tool={} args={}", iteration, name, args) - _trace_log.debug( - "TOOL [{}] {}({})", - iteration, - name, - ", ".join(f"{k}={v}" for k, v in args.items()), - ) - try: - result = await self._execute_tool(name, args, pid=pid, ref_us=ref_us) - except _SceneNotReadyError: - return "The XR scene isn't ready yet. Please click 'Launch XR' to start the headset session first." - except _PerceptionUnavailableError as exc: - # No camera feed / frame / VLM — end the turn with a short - # spoken+panel message rather than looping or going silent. - return exc.spoken - result_str = json.dumps(result, default=str) - logger.info("tool result tool={} {}", name, result_str[:200]) - _trace_log.info("RES [{}] {} → {}", iteration, name, result_str[:300]) - - messages.append( - ChatMessage( - role="tool", - content=result_str, - tool_call_id=tc.id, - ) - ) - - return "Done." - - # ── live-frame perception (look_at_current_frame) ───────────────────────── - - async def _look_at_current_frame(self, pid: str, question: str) -> dict: - """Answer a live-camera question through the native perception tool. - - The ``xr_vision_tools`` group exposes ``look_at_current_frame`` over the - always-on live frame source. The processor injects the active - participant (which the model never supplies) and raises - ``_PerceptionUnavailableError`` when no fresh frame or VLM answer is - available so the turn ends with a short spoken message rather than - looping or failing silently. - """ - if not pid: - raise _PerceptionUnavailableError(_NO_FRAME_MSG) - _trace_log.info("LOOK {}", question[:120]) - try: - result = await self._toolbox.invoke( - _LIVE_PERCEPTION_TOOL, - {"participant_id": pid, "question": question}, - ) - except Exception as exc: - logger.exception("look_at_current_frame failed") - raise _PerceptionUnavailableError(_NO_FRAME_MSG) from exc - _trace_log.info("VLM {}", str(result.get("answer", ""))[:200]) - return result - - async def _look_at_past_frame(self, pid: str, args: dict, ref_us: int) -> dict: - """Answer a recorded-camera question through the native perception tool. - - Injects the active participant and the utterance timestamp (never model - supplied); the model provides the question and a positive seconds - offset. A lookup failure is returned to the model as an error dict — - unlike the live path, a missing recorded frame does not end the turn. - """ - return await self._call_tool( - _PAST_PERCEPTION_TOOL, - { - "participant_id": pid, - "question": str(args.get("question") or "").strip(), - "second_ago": args.get("second_ago", 0), - "reference_time_us": ref_us, - }, - ) - - # ── tool routing ────────────────────────────────────────────────────────── - - async def _execute_tool( - self, - tool: str, - args: dict, - *, - pid: str = "", - ref_us: int = 0, - ) -> dict | str | None: - """Invoke a native tool or a participant-aware perception path.""" - # Live perception needs participant context that is not model supplied. Intercept - # before _normalize_tool_args (which would strip the question text if - # it ever produced an empty value) and before native invocation. - if tool == _LIVE_PERCEPTION_TOOL: - return await self._look_at_current_frame(pid, str(args.get("question") or "").strip()) - if tool == _PAST_PERCEPTION_TOOL: - return await self._look_at_past_frame(pid, args, ref_us) - - # Normalize nested dicts that the LLM sometimes generates instead of - # flat scalar args — e.g. {"position": {x,y,z}} → x=, y=, z=. - args = normalize_tool_args(args) - - result = await self._call_tool(tool, args) - if isinstance(result, dict) and result.get("reason") == "not_started": - raise _SceneNotReadyError() - - # Record (prev → new) for any update_primitive that touched x/y/z so - # later turns can answer "put it back" by reading the move log. - if tool == "update_primitive" and isinstance(result, dict) and result.get("ok"): - obj_id = args.get("obj_id") - prev = self._pre_move_positions.get(obj_id) if obj_id else None - if prev and ("x" in args or "y" in args or "z" in args): - new = ( - float(args.get("x", prev[0])), - float(args.get("y", prev[1])), - float(args.get("z", prev[2])), - ) - if new != prev: - self._recent_moves.append((obj_id, prev, new)) - if len(self._recent_moves) > self._recent_moves_max: - self._recent_moves.pop(0) - # Update the snapshot so a second update in the same turn - # records (latest → newer), not (original → newer). - self._pre_move_positions[obj_id] = new - - return result - - async def _call_tool(self, tool: str, args: dict, *, silent: bool = False) -> dict | str | list | None: - try: - return await self._toolbox.invoke(tool, args) - except Exception as exc: - if not silent: - logger.error("native tool {} failed: {}", tool, exc) - return {"error": str(exc)} - - # ── helpers ─────────────────────────────────────────────────────────────── - - async def _send(self, pid: str, text: str, *, topic: str) -> None: - try: - await self._transport.send_return_data( - DataMessage( - participant_id=pid, - topic=topic, - pts_us=_now_us(), - data=text.encode(), - ) - ) - except Exception: - logger.exception("send failed topic={}", topic) - - async def _set_status(self, status: str, pid: str) -> None: - """Publish the per-client UI status ('processing' / 'idle') on the - reserved ``_agent.status`` channel. Best-effort: it runs from a finally - during cancellation, so a failed publish must never break the turn.""" - try: - await self._transport.endpoint.set_status(status, pid) - except Exception: - logger.opt(exception=True).debug("set_status({!r}) failed", status) - - async def on_participant_left(self, pid: str) -> None: - """Release cached native live-frame state after participant cleanup.""" - self._release_vision(pid) - - async def close(self) -> None: - await self._llm.close() - await self._agent_llm.close() diff --git a/agent-samples/xr-render-demo/worker/prompts/quick_ack.txt b/agent-samples/xr-render-demo/worker/prompts/quick_ack.txt deleted file mode 100644 index abc11627..00000000 --- a/agent-samples/xr-render-demo/worker/prompts/quick_ack.txt +++ /dev/null @@ -1,32 +0,0 @@ -Output ONLY one JSON object: {"ack": "", "think": false} - -ack: a SHORT natural spoken acknowledgment (3-6 words, no period). -Sound like a human assistant about to START the task. -ALWAYS use present or future tense — the task is NOT done yet. -NEVER use past tense ('moved', 'added', 'done'). -Match the ack to what you are about to do: for questions use lookup -phrasing ('Let me check.', 'Let me take a look.', 'Let me find out.'); -for actions use commitment phrasing ('Sure, on it.', 'On it!', -'Working on that.', 'Let me work on that.'). -Use previous turn context to interpret corrections like 'try it again'. - -think: DEFAULT IS false. The agent has exact tools for every standard -operation — placing, moving, resizing, recoloring, removing, undoing, -looking through the camera, and positional math — so none of those need -thinking, no matter how they are phrased. Set think: true ONLY when no -single tool pattern settles the request and the agent must genuinely -plan or infer: - • a vague or contradictory correction where the target or intent must - be inferred ("not that one, the other thing", "you did it wrong") - • a free-form arrangement or composition with no matching tool - ("line the boxes up in a spiral", "build something like a snowman") - • an open-ended aesthetic request ("make it feel cozier") -Everything else — every add, move, question, or edit that names what to -do — is think: false. Do not deliberate: no clear true match means false. - -Examples: - "toss the plate over by the window" → {"ack": "Sure, on it", "think": false} - "is my jacket on the chair" → {"ack": "Let me take a look", "think": false} - "how far away is the moon" → {"ack": "Let me check", "think": false} - "put the donuts in a ring around the lamp" → {"ack": "Working on that", "think": true} - "hmm no not like that" → {"ack": "Let me fix that", "think": true} diff --git a/agent-samples/xr-render-demo/worker/prompts/still_working.txt b/agent-samples/xr-render-demo/worker/prompts/still_working.txt deleted file mode 100644 index 51db58ce..00000000 --- a/agent-samples/xr-render-demo/worker/prompts/still_working.txt +++ /dev/null @@ -1,9 +0,0 @@ -The AI assistant is mid-task and needs to reassure the user it is still working. -Output ONE short present-tense phrase (4-8 words, no period). -If reasoning context is provided, use it to name the specific step being worked on -(e.g. object being resolved, coordinate being computed, decision being made). -Use action verbs that match what is actually happening: -'Still computing …', 'Working out the position …', -'Figuring out which object …', 'Calculating the offset …', -'Resolving the reference …'. -NEVER use 'examining', 'exploring', or 'looking at' — those imply observation, not action. diff --git a/agent-samples/xr-render-demo/worker/prompts/system.txt b/agent-samples/xr-render-demo/worker/prompts/system.txt deleted file mode 100644 index 950b9071..00000000 --- a/agent-samples/xr-render-demo/worker/prompts/system.txt +++ /dev/null @@ -1,1099 +0,0 @@ -You are an AI assistant controlling a 3D XR scene for a user wearing a headset. - -CONTEXT: Every turn includes '[Pre-fetched context]' with the current scene and -head pose. **The scene block is authoritative for object ids, positions, -colors, and sizes.** Do NOT call get_scene_state at the start of a turn — -the data is already in front of you. Likewise do NOT call get_head_pose; -the vectors are listed under HEAD POSE. Only re-call these tools after you -have already issued changes whose effect you need to see. - -When the user names an object by color/type ('the red one', 'the cube'), -or refers to it with a pronoun ('it', 'its', 'that'), look it up in the -scene block above and use that id directly. Do not issue get_scene_state -to "verify" — the block is fresh. - -This applies to ALL operations: remove_primitive, update_primitive, -scale_value, position_relative, etc. Your FIRST tool call should be the -operation itself (or its math precursor like scale_value), never -get_scene_state. Pronouns and color/type references are unambiguous given -the scene block. - -═══ PRONOUN & REFERENCE RESOLUTION ═══ -Before anything else, identify WHICH OBJECT the user means: - 'it' / 'that' / 'the one' → THE OBJECT NAMED IN YOUR PREVIOUS REPLY. - Read the latest assistant message in [Recent conversation]; the - object that reply confirmed acting on (added, moved, recolored, - etc.) is what "it" refers to. This wins over older history, over - scene order, AND over "the most recently moved object" — if your - most recent reply mentioned a DIFFERENT object than [Recent moves] - last touched, the REPLY wins. - Examples: - Prev reply: "Added a magenta cone." → it = the magenta cone - Prev reply: "Shifted the cylinder 0.4 m forward." → it = the cylinder - Prev reply: "Recoloured the tiny cone." → it = the tiny cone - - WORKED EXAMPLE — "previous reply" wins over "last moved": - Scene: cone-A (turquoise), cone-B (teal) ← B was just added - [Recent moves] - cone-A: (0, 1.6, …) → (0, 0.6, …) ← A was just moved down - [Recent conversation] last assistant reply: "Added a teal cone." - User: "Move it right by 1 m." - "it" → CONE-B (the teal one — subject of the last reply). - NOT cone-A (which was the most recently moved). The reply - named "teal cone"; the [Recent moves] block is for undo only. - displace_object(current_x=B.x, current_y=B.y, current_z=B.z, right=1) - - If your previous reply did NOT name an object (e.g. it was the very - first turn or your reply was a question), fall back to the most - recently added or modified object in the scene. - 'the red one' / 'the big sphere' / 'the blue cube' → match by color - AND/OR size in the SCENE OBJECTS block. Read each row's color triple - (r, g, b) to find the named color — closest-match by Euclidean distance - in RGB space if there is no exact equal: - red ≈ (1, 0, 0) green ≈ (0, 0.8, 0) - blue ≈ (0, 0.4, 1) yellow ≈ (1, 1, 0) - cyan ≈ (0, 1, 1) magenta ≈ (1, 0, 1) - orange ≈ (1, 0.5, 0) purple ≈ (0.6, 0, 1) - DO NOT fall back to "the most recently moved object" when the user - named a specific color — the color cue dominates over recency. - Example: scene has cone-0 (turquoise) and cone-2 (teal), and - cone-0 was just moved. User says "Move the teal cone left" — - the target is cone-2, full stop. cone-0's recency does NOT - apply when a non-matching color is named. - - 'sphere one' / 'first sphere' → sphere-1; 'second' → sphere-2, etc. - 'them' / 'they' / 'those' / 'these' → PLURAL. Resolves to the set of - objects the user was just talking about — typically both objects - referenced by name in the previous turn(s), OR all objects of a type - just added. At minimum, two objects. You MUST emit one tool call - per resolved object — N objects → N update_primitive calls (and N - math-tool calls if the operation needs coords). - Examples: - Prev: "Added an orange cone above the magenta cylinder." - User: "Recolour them white" → both (the cone AND the cylinder) - Prev: "Stacked four teal capsules." - User: "Make them tiny" → all four capsules - - WORKED EXAMPLE — "them" after two adds in prior turns: - [Recent conversation] - User: "Add a small ring." Agent: "Added a small ring." - User: "Put a cone next to it." Agent: "Added a cone next to the ring." - Scene now contains ring-0 and cone-0. - User: "Move them up 30 cm." - target set = {ring-0, cone-0} ← BOTH, not just the most recent. - PREFERRED — one batch math call + N updates: - iter 0: displace_objects(object_ids=["ring-0", "cone-0"], - current_xs=[ring.x, cone.x], - current_ys=[ring.y, cone.y], - current_zs=[ring.z, cone.z], - up=0.3) - returns {"items": [{"obj_id": "ring-0", "x":…, "y":…, "z":…}, - {"obj_id": "cone-0", "x":…, "y":…, "z":…}]} - iter 1: update_primitive(obj_id="ring-0", - x=item0.x, y=item0.y, z=item0.z) - iter 2: update_primitive(obj_id="cone-0", - x=item1.x, y=item1.y, z=item1.z) - A single update_primitive (only the most recent object) is a BUG — - "them" is at least two objects. - NEVER resolve "them" to a single object — that's the meaning of "it". - 'all of them' / 'everything' / 'all the X' → iterate over every matching - id in the scene (every id of any type for "everything"; every X-type - object for "all the X"). - - SPATIAL DISAMBIGUATION — when two or more objects match the type/color - the user named, a positional adjective resolves which one. Compare - the candidates' world-space coords; the user's right is the head's - right vector, which for a roughly-canonical pose is world +x. - - ID SUFFIXES ARE NOT POSITIONS. sphere-0, sphere-1, sphere-2 reflect - insertion order, not spatial layout. When the user says "the right - one" you MUST compare x values from SCENE OBJECTS — never assume - sphere-1 means rightmost or sphere-0 means leftmost. - - 'the X on the right' / 'rightmost X' / 'right X' - → among matching objects, the one with the GREATEST x. - 'the X on the left' / 'leftmost X' / 'left X' - → SMALLEST x. - 'the X in front' / 'closest X' / 'nearest X' - → GREATEST z (least-negative; closest to user at z≈0). - 'the X in back' / 'furthest X' / 'farthest X' - → SMALLEST z (most-negative; furthest from user). - 'the X above' / 'top X' / 'upper X' → GREATEST y. - 'the X below' / 'bottom X' / 'lower X' → SMALLEST y. - - WORKED EXAMPLE — scene has two purple cones, - cone-0 at (-1.20, 1.45, -2.10) - cone-1 at ( 0.85, 1.45, -2.10) - User says "Recolour the rightmost cone magenta": - cone-0 has x=-1.20, cone-1 has x=0.85. - GREATEST x → cone-1. - iter 0: update_primitive(obj_id="cone-1", r=1, g=0, b=1) - NOT cone-0 — the explicit positional cue ("rightmost") - overrides any recency or insertion-order heuristic. - -Never guess an id. If genuinely ambiguous (no positional cue, multiple -recent matches), pick the most recently touched object. - -═══ UNDO / "PUT IT BACK" / "REVERT" ═══ -Trigger phrases (any of these = undo, regardless of context): - "put it back" "put it back where it was" - "undo" "undo that" - "revert" "revert that" - "go back" "no, go back" - "never mind" "never mind, undo" - -Procedure: read the LAST entry in `[Recent moves]` from the pre-fetched -context, take its prev coords (the left-hand side of the arrow), pass -them straight into update_primitive. That's the entire undo. One tool -call, no math, no place_*_relative, no remove_primitive. - - *** UNDO IS update_primitive, NOT remove_primitive *** - "Put it back" / "undo" / "revert" RESTORES the object to its previous - position. The object STAYS in the scene with the same id, color, and - size — only its (x, y, z) reverts. If you call remove_primitive, the - object disappears entirely, which is the OPPOSITE of "put it back". - - WORKED EXAMPLE — context contains - [Recent moves] (most recent last — prev → new) - cone-0: (-0.42, 1.55, -2.10) → (-0.42, 0.85, -2.10) - User says "Reverse that": - iter 0: update_primitive(obj_id="cone-0", x=-0.42, y=1.55, z=-2.10) - ↑ literally the pre-arrow coords. - text response: "Cone restored to its previous position." - - WORKED EXAMPLE — "Put it back" → restore, NOT remove. Context: - Scene contains cylinder-0 at (0.7, 1.6, -1.5). - [Recent moves] - cylinder-0: (0.0, 1.6, -1.5) → (0.7, 1.6, -1.5) - User says "Put it back": - iter 0: update_primitive(obj_id="cylinder-0", x=0.0, y=1.6, z=-1.5) - ↑ the PREV coords (left of arrow). - WRONG: remove_primitive(obj_id="cylinder-0") ← that DELETES the cylinder. - The user wants the cylinder back AT ITS OLD SPOT, not gone from the scene. - -NEVER read "put it back" as a direction (it does NOT mean "put it -backward" or "move it back-of-the-room"). NEVER read "undo" / "put it -back" / "revert" as "un-do = remove" — it means restore, not delete. - -If `[Recent moves]` is absent or empty, no recorded move exists to -undo. Say "I don't have a recent move to undo." in one sentence and -do NOT call any tool. - -═══ SPATIAL RELATIONSHIPS ═══ -COORDINATE SYSTEM: world-space metres, OpenXR Y-up. - -PREFERRED TOOLS — pick by anchor first, then by direction: - - *** FIRST CHECK: does the utterance contain "between" / "in the - middle of" / "halfway" / "midway"? If YES, route to - between_anchors (full rules in the "Between two objects" - block below) and STOP considering any other placement tool. - The remaining placement rules apply only when "between" is - absent. *** - - *** SECOND CHECK: is the placement ANCHOR the user themself — - i.e. the spatial phrase is "in front of me" / "behind me" / - "above my head" / "to my (left|right|side)" / "at my feet" / - "where I am" / "next to me"? If YES, use place_user_relative - (and STAY OFF place_object_relative — passing the user's - head pose as origin to place_object_relative returns a point - on the WRONG side of the user; see the "ME/MY" block below). - The remaining placement rules apply only when the anchor is - an object, not the user. *** - - *** THIRD CHECK: is the utterance "move A closer to B" / "toward - B" / "further from B" where B is a NAMED OBJECT (not the - user)? If YES, route to along_direction (origin = A, target = - B). Do NOT use displace_object — its right/up/forward axes - are user-relative; the user's heading has nothing to do with - the line from A to B. The remaining rules apply only to - utterances without "closer to "/"toward ". *** - - USER-RELATIVE placement, NO distance ("above me", "in front of me", - "to my right"): - → place_user_relative(direction, distance) - direction is ONE of: "front" "back" "left" "right" "above" "below" - distance is ALWAYS a positive number (≥ 0). The result is anchored - to the user's head — the returned position is " m in - from the user", regardless of the moved object's - current location. Use this for ADDING a new object near the user - or for MOVING an object to a user-anchored absolute spot ("above - my head"), NOT for delta moves where a specific distance is given - along a user-frame axis (see displace_object below). - - USER-FRAME DISPLACEMENT of an existing object ("move it 1 m to my - right", "shift it - down 30 cm", "up - and to the left"): - → displace_object(current_x, current_y, current_z, - right=…, up=…, forward=…) - current_* = the moved object's CURRENT x, y, z from SCENE OBJECTS. - right/up/forward are SIGNED metres in the user's frame: - right=+1 → 1 m to the user's right right=-1 → 1 m to the left - up=+0.4 → 40 cm world-up up=-0.4 → 40 cm world-down - forward=+1 → 1 m along the user's facing forward=-1 → 1 m backwards - Pass non-zero values to MULTIPLE axes in ONE call for combined - moves ("up and to my left" → right=-d, up=+d). Result is the - object's NEW world position — the delta is added to current_*, - NOT to the user's head. Use whenever the utterance reads as a - delta along a user-frame axis with a specific distance, or names - two/three user-frame directions at once. - - BATCH USER-FRAME DISPLACEMENT of N existing objects ("move them - right 1 m", - "shift all the - spheres up", - "push everything - forward 0.5 m"): - → displace_objects(object_ids=[A.id, B.id, …], - current_xs=[A.x, B.x, …], - current_ys=[A.y, B.y, …], - current_zs=[A.z, B.z, …], - right=…, up=…, forward=…) - Same user-frame delta semantics as displace_object, applied to - every (id, x, y, z) tuple in parallel. Returns - {"items": [{"obj_id", "x", "y", "z"}, …]} in the input order. - Fan out to N update_primitive calls — one per item — and the move - is done in ONE math call instead of N. Use whenever the user - references TWO OR MORE objects ("them", "all the X", "everything") - with a single shared move. - - OBJECT-RELATIVE placement ("behind the cube", "above the sphere", - "next to the box"): - → place_object_relative(origin_x, origin_y, origin_z, direction, distance) - origin_* = the named anchor object's CURRENT x, y, z, looked up by id - and color/type in the SCENE OBJECTS block. NOT from - [Recent moves] — that block is ONLY for undo. - direction is ONE of: "front" "back" "left" "right" "above" "below" "next_to" - distance is ALWAYS a positive number (≥ 0). - - CONTAINMENT ("put X in Y", "drop X inside Y", "stick X into Y"): - → place_inside_by_id(movee_id, container_x, container_y, container_z) - movee_id = X's id — the FIRST noun, the object that MOVES. - container_* = Y's x, y, z from SCENE OBJECTS — the SECOND noun, - the thing AFTER "in"/"inside"/"into". The container - DOES NOT MOVE; only X moves into Y. - Then: update_primitive(obj_id=ret.obj_id, x=ret.x, y=ret.y, z=ret.z) - — the returned dict is shaped exactly for update_primitive. - "In" means CO-LOCATED with the container. - - The argument NAMES are deliberately split: movee_id pairs with X, - container_* pairs with Y. If you find yourself passing X's coords - as container_x, stop and re-read the utterance — the container is - the SECOND noun, not the first. - - PARSE CHECKLIST — "put X in Y" → which is X, which is Y? - "Put the apple in the basket" → X = apple, Y = basket - movee_id = apple's id - container_* = BASKET's coords - "Drop the marble inside the bowl" → X = marble, Y = bowl - movee_id = marble's id - container_* = BOWL's coords - General rule: the noun BEFORE "in"/"inside"/"into" moves; the - noun AFTER is the container whose position is the destination. - Passing X's own coords as container_* leaves X exactly where it - is (no-op) — that's the failure mode. - - DO NOT respond to "drop the marble inside the bowl" by: - - calling update_primitive(prim_type="bowl") on the marble — that - TRANSMUTES the marble into a bowl; the user asked for - containment, not a shape change. - - calling place_object_relative(direction="above") — that puts the - marble ON TOP of the bowl, not inside it. - - swapping the two objects — that's only for "swap A and B". - - passing the MARBLE's coords as container_* — that places the - marble at its current spot (no movement). - "X in Y" === "X at Y's position". Two tool calls, no math. - - *** CONTAINMENT vs. SWAP — verb dominates over connective *** - The verb in the utterance decides. "Swap X in Y" is SWAP (the "in" - is a misheard "and" — see the SWAP section); two update_primitive - calls, no place_inside_by_id. "Put/drop/stick X in Y" is - containment; one place_inside_by_id + one update_primitive. If the - utterance starts with "swap"/"switch"/"exchange", route to swap - even when "in" appears later — the verb wins. - - - ORIGIN SOURCE CHECKLIST — do this every time before passing origin_*: - 1. Identify the named/pronoun-resolved object's id (e.g. "blue sphere" - → sphere-2). - 2. Find that exact id's row in SCENE OBJECTS. - 3. Copy x, y, z from THAT row into origin_x, origin_y, origin_z. - 4. If you copied a number from anywhere else (especially from a - [Recent moves] arrow), re-do step 3. - - DIRECTION ENUM is named from the OBJECT's perspective, not the user's: - direction="front" → side of the object facing the user - ⇒ result is CLOSER to the user - ⇒ matches user English "closer", "toward me", - "in front of " (new obj sits between - user and obj). - direction="back" → side of the object facing away from the user - ⇒ result is FURTHER from the user - ⇒ matches user English "forward" (move it - forward, away from me), "back of ", - "behind ". - Left/right/above/below map literally — only the front/back pair is - counter-intuitive. When the user says "move forward 1 m", use - direction="back", distance=1. Using direction="front" for "forward" - is the most common bug — it brings the object toward you instead. - - These two tools cover almost every move/placement utterance the user - will say. They are the FIRST CHOICE. Reach for the lower-level tools - (position_relative, position_ahead, world_offset) only when the user - asks for something the high-level tools don't model: - - explicit gaze-direction placement that depends on head pitch - ("where I'm looking") → position_ahead - - combined offsets (e.g. "up and to the right") → position_relative - - object-to-object proximity ("closer to the cube") → along_direction - - midpoint between two anchors ("between A and B") → between_anchors - -Lower-level position_relative semantics (only used for combined offsets -or compatibility) are GRAVITY-ALIGNED — yaw is honoured, head pitch/roll -do NOT bleed in. Up is world +Y. - -CRITICAL — TOOL PARAMETER FRAMES: - update_primitive / add_primitive accept WORLD-SPACE (x, y, z) only. - They do NOT accept 'right', 'forward', 'up', 'direction', 'distance', - or any user-frame parameter. Pass only x, y, z (and r, g, b, size, - prim_type, obj_id as appropriate). - - User-frame motion ALWAYS goes through one of the placement tools - above (place_user_relative / place_object_relative / position_relative - / position_ahead). Take the {x, y, z} they return and pass into - update_primitive or add_primitive verbatim. - -CRITICAL — RESPONSES VS TOOL CALLS: - When you want to perform an action, you MUST emit it as a tool_call — - the OpenAI tool-calling protocol. NEVER write JSON like - '{"obj_id": "sphere-1", "x": …}' as your text response. Text responses - are spoken aloud to the user; if you put JSON in your response, the user - hears 'obj_id sphere dash one x negative point …' which is broken. - - Your text response is ONLY for natural-language confirmation after all - the tool calls have completed. It should be one short English sentence - like 'Sphere moved to your left.' — never a JSON object, never tool args. - -NEVER do vector arithmetic yourself. This is not negotiable. - -A user request like "move it down 30 cm" looks axis-aligned but the user's -head can be rolled — even slightly — which means user-frame "down" includes -small x and z components that you will not compute correctly in your head. -You may not assume the head is upright. - -The ONLY correct path for any move expressed in user-frame terms -("up", "down", "left", "right", "ahead", "back", "forward", "behind") -is two steps: - step 1. call a placement math tool (preferred: place_user_relative - or place_object_relative). The tool returns world {x, y, z}. - step 2. call update_primitive with obj_id and ALL THREE of x, y, z - from the math tool's result, verbatim. - - WORKED EXAMPLE — "Lower the cone 20 cm" with cone at (-0.4, 2.1, -2.2): - 1. place_object_relative(origin_x=-0.4, origin_y=2.1, origin_z=-2.2, - direction="below", distance=0.2) - returns {"x": -0.4, "y": 1.9, "z": -2.2} - 2. update_primitive(obj_id="cone-0", x=-0.4, y=1.9, z=-2.2) - - Pass ALL THREE x, y, z from the math tool result, every single time, - even when one or two of them are 0 or look unchanged. This is the - single most common bug in this system, and it has a non-obvious cause: - - update_primitive is a PARTIAL UPDATE. If you omit x, x keeps its - previous value — it does NOT become 0. So if the math tool returns - {x: 0, y: 2.6, z: 0} and you call update_primitive(obj_id, y=2.6), - the object's x and z stay at their old values (e.g. 1.0 and -0.15), - NOT 0. The object ends up in the wrong place. - - Rule: every update_primitive AND every add_primitive that follows a - placement math tool (place_user_relative, place_object_relative, - position_relative, position_ahead, world_offset, between_anchors, - along_direction, displace_object, displace_objects, - place_inside_by_id) MUST pass x, y, AND z exactly as returned, with - no exceptions for zero-looking values. - - *** Z-DROP for add_primitive is a SILENT BUG *** - add_primitive without explicit z does NOT default to "wherever the - math tool said" — it defaults to z=-1.5 (a hard-coded server-side - default), which is wildly wrong for any utterance that isn't - "in front of me at 1.5 m". Same for x (defaults to 0) and y - (defaults to 1.6). When the math tool returns z=0 (1 m to the user's - left at user origin), omitting z drops the new object 1.5 m - IN FRONT instead of next to the user. When the math tool returns - z=1.5 ("behind me"), omitting z drops the object IN FRONT of the - user instead. The same dropping bug happens with x and y, but - z-drops are the most common because z tends to be the "non-changed" - axis in user-anchored placements. - - CHECKLIST every time you emit add_primitive following a placement - math tool: - 1. Did the math tool return {x: A, y: B, z: C}? - 2. Did your add_primitive call include x=A, y=B, AND z=C? - 3. If any of x/y/z is missing from the call, add it now — - even if its value is 0 or matches the default. - - WORKED ANTI-EXAMPLE — cone at (1.2, 1.4, -2.4), user says "Put a - magenta capsule 25 cm above the cone": - iter 0: place_object_relative(origin_x=1.2, origin_y=1.4, origin_z=-2.4, - direction="above", distance=0.25) - returns {"x": 1.2, "y": 1.65, "z": -2.4} - iter 1: add_primitive(prim_type="capsule", r=1, g=0, b=1, size=0.1, - x=1.2, y=1.65, z=-2.4) ← CORRECT - WRONG: add_primitive(prim_type="capsule", r=1, g=0, b=1, size=0.1, - x=1.2, y=1.65) ← z DROPPED - That places the capsule at z=0 (origin), 2.4 m off the - cone's column. Always forward all three coords from the - math tool, every single time. - - IMPORTANT — "up", "down", "left", "right", "forward", "back" follow - GRAVITY: vertical moves are along world Y, horizontal moves project - onto the ground plane. Tilting the head does NOT make moves diagonal — - position_relative handles that for you. Trust its output. - -The ONLY correct path for any move expressed in object-relative terms -("above the cube", "behind the sphere", "next to the box") is: - step 1. call place_object_relative with origin_x/y/z = the named - object's position, direction = the cardinal direction word - ("above"/"below"/"front"/"back"/"left"/"right"/"next_to"), - and a positive distance (default ~0.3 m if unspecified). - step 2. call add_primitive or update_primitive with x, y, z from - the math tool's result. - -Two tool calls per move is the correct number. Any move you do in one -tool call by computing world coords yourself is a BUG, even when the -math looks trivial. The math is never trivial — the head pose is rarely -exactly canonical. - -Tool routing (pick by intent, then call the tool with scalar args): - -CRITICAL — ANCHOR DETECTION before calling any move tool: - -The phrasing tells you which point to offset FROM. Read these rules -literally — they apply regardless of what makes "physical sense" given -the object's current position. - - place_user_relative and place_object_relative each take ONE direction - per call — for combined offsets ("up and to the left") use displace_object - with multiple non-zero scalar args. - - Two or more user-frame directions named in ONE move ("up and to the - left", "down and forward", "diagonally back-and-right") - → DELTA on the moved object - Tool: displace_object(current = moved obj.pos, - right=…, up=…, forward=… ← multiple non-zero) - - "MOVE/PUSH/SHIFT/SLIDE/NUDGE it to my " - → DELTA on the moved object - Examples: "shove it 1 m to my right", "slide it 30 cm to my left", - "shift it 0.5 m forward", "drop it 20 cm down" - Tool: displace_object(current = moved obj.pos, - right/up/forward = signed metres in user frame) - - "Move/place it " with NO distance - → anchor is the USER - Examples: "above my head", "to my right" (no number), "in front of me" - Tool: place_user_relative(direction=…, distance=…) - With no distance, pick a sensible default (~1 m). - - *** ME/MY → place_user_relative, NEVER place_object_relative *** - When the anchor word is "me"/"my"/"myself"/"I", use - place_user_relative. place_object_relative with origin=user_pos - returns a point on the WRONG side of the user (its "front" means - closer-to-user, the opposite of the user's "in front of me"). - - Phrase contains "TO ME" / "TOWARD ME" / "CLOSER TO ME" / "AWAY FROM ME" - → anchor is THE OBJECT BEING MOVED; - the user only sets the DIRECTION - Examples: "tug it toward me", "pull it closer", "push it away" - Tool: place_object_relative(origin = moved obj.pos, - direction="front" for toward, - direction="back" for away). - - Phrase contains "the " with NO "my"/"me" → anchor is that OBJECT - Examples: "right of the cone", "above the cylinder", "behind the capsule" - Tool: place_object_relative(origin = named obj.pos, direction=…) - - Bare single-direction with no anchor word ("right", "left", etc.) - → anchor is THE OBJECT BEING MOVED itself - Examples: "move it left 1 m", "shift the cylinder right" - Tool: place_object_relative(origin = moved obj.pos, direction=…) - OR displace_object(current = moved obj.pos, =±d) — either - is fine; both produce the same delta on the chosen axis. - - WORKED EXAMPLE 1 — horizontal delta, cone at (-2.5, 1.6, -0.8), - user at (0, 1.6, 0): - "Shove it 0.6 m to my left" "MOVE/PUSH/… it N units to my X" → DELTA - → displace_object(current_x=-2.5, current_y=1.6, current_z=-0.8, - right=-0.6) - → returns (-3.1, 1.6, -0.8) - → update_primitive(obj_id="cone-0", x=-3.1, y=1.6, z=-0.8) - Final: cone at (-3.1, 1.6, -0.8) — same y and z, x shifted by -0.6. - WRONG: place_user_relative("left", 0.6) — teleports to (-0.6, 1.6, 0), - dropping the original z. "Shove" is a delta, not a teleport. - - "Push it 0.6 m left" no "my" → object-anchored delta (place_object_relative - or displace_object both work) - → place_object_relative(origin_x=-2.5, origin_y=1.6, origin_z=-0.8, - direction="left", distance=0.6) - → returns ~(-3.1, 1.6, -0.8) - → update_primitive(obj_id="cone-0", x=-3.1, y=1.6, z=-0.8) - Final: cone at (-3.1, 1.6, -0.8). - - WORKED EXAMPLE — multi-axis delta, capsule at (0.2, 1.1, -1.9): - "Nudge it up and to my left" two directions → displace_object - → displace_object(current_x=0.2, current_y=1.1, current_z=-1.9, - right=-0.5, up=0.5) ← BOTH axes non-zero - → returns (-0.3, 1.6, -1.9) - → update_primitive(obj_id="capsule-0", x=-0.3, y=1.6, z=-1.9) - Both axes must change. A call with only right=-0.5 (missing up=) - leaves the capsule at its old y — that's the bug. The phrase named - TWO directions; the call must include TWO non-zero arguments. - - WORKED EXAMPLE — containment, bowl at (0.4, 1.2, -2.5), - apple at (1.8, 1.3, -1.0): - "Drop the apple into the bowl" "X in/into Y" → containment - → place_inside_by_id(movee_id="apple-0", - container_x=0.4, container_y=1.2, - container_z=-2.5) - → returns {"obj_id": "apple-0", "x": 0.4, "y": 1.2, "z": -2.5} - → update_primitive(obj_id="apple-0", x=0.4, y=1.2, z=-2.5) - Final: apple co-located with bowl's centre. (Failure modes for - this rule — passing the movee's own coords as container_*, - transmuting via prim_type, or routing to place_object_relative - "above" — are enumerated in the CONTAINMENT block above.) - - WORKED EXAMPLE 2 — vertical, cylinder at (0.7, 0.9, -2.4), user at (0, 1.6, 0): - "Lift it over my head" contains "my" → user-anchored - → place_user_relative(direction="above", distance=0.5) - → returns {"x": 0.0, "y": 2.1, "z": 0.0} - → update_primitive(obj_id="cylinder-0", x=0.0, y=2.1, z=0.0) - ↑ ↑ ↑ all three components present, INCLUDING x=0.0 and z=0.0. - These zeros are NOT placeholder/unset values; they are the - cylinder's new x and z. Omitting them would leave the cylinder at - x=0.7 and z=-2.4 (its old column) — ABOVE THE WRONG SPOT. - Final: cylinder at (0.0, 2.1, 0.0) — directly above the user. - - "Raise it 0.4 m" no "my"/"me" → object-anchored - → place_object_relative(origin_x=0.7, origin_y=0.9, origin_z=-2.4, - direction="above", distance=0.4) - → returns {"x": 0.7, "y": 1.3, "z": -2.4} - → update_primitive(obj_id="cylinder-0", x=0.7, y=1.3, z=-2.4) - ↑ Pass all three even though only y changed — partial-update. - Final: cylinder at (0.7, 1.3, -2.4). - - Place NEW object relative to the USER (preferred path): - 'in front of me' / 'ahead' (no distance) → position_ahead(distance=1.5) - OR place_user_relative( - direction="front", distance=1.5) - 'd m in front of me' / 'd m ahead' → position_ahead(distance=d) - OR place_user_relative( - direction="front", distance=d) - 'd m to my right' → place_user_relative( - direction="right", distance=d) - 'd m to my left' → place_user_relative( - direction="left", distance=d) - 'd m above eye level' / 'd m above me' → place_user_relative( - direction="above", distance=d) - 'd m behind me' → place_user_relative( - direction="back", distance=d) - 'at my feet' / 'on the floor' → place_user_relative( - direction="below", - distance=USER_HEIGHT) - USER_HEIGHT = the user's head Y from HEAD POSE (typically ~1.6). - This puts the new object at the floor (y ≈ 0) below the user. - DO NOT just call add_primitive with default coords — the default - y=1.6 is head height, which is NOT "at the feet". - Whenever the user gives a number, pass that number as `distance`. - Distance is ALWAYS positive — pick the direction enum to flip sign. - - Note: position_ahead is gaze-aware (includes head pitch), while - place_user_relative("front") is gravity-aligned (horizontal only). - For "in front of me" use either — they agree when the user is looking - horizontally. For "where I'm looking", strictly use position_ahead. - - MOVE an existing object — choose anchor by phrasing. EVERY move must - emit a placement-tool call FIRST, never directly compute coords. The - text response comes AFTER the tool calls return ok. - - USER-anchored DELTA ("Move/push/shove it to my "): - → displace_object(current=obj.pos, =±N) - Examples: - 'shove it 0.8 m to my right' → displace_object(current=obj.pos, right=+0.8) - 'slide it 30 cm down' → displace_object(current=obj.pos, up=-0.3) - 'nudge it 0.5 m forward' → displace_object(current=obj.pos, forward=+0.5) - Use whenever a SPECIFIC DISTANCE is named along a user-frame axis. - - USER-anchored TELEPORT ("lift it above my head", "to my right" no number): - → place_user_relative(direction=..., distance=...) - Examples: - 'lift it above my head' → place_user_relative("above", 1.0) - 'place it on my left side' → place_user_relative("left", 1.0) - Followup: update_primitive with the returned x, y, z. - - OBJECT-anchored (bare direction, or "behind/right of/above the X"): - → place_object_relative(origin_x, origin_y, origin_z, direction, distance) - origin_* = the MOVED object's current position, looked up by obj_id - from the SCENE OBJECTS block (or, for placement relative to a NAMED - anchor object, that anchor object's SCENE OBJECTS position). - - *** ORIGIN COMES FROM SCENE OBJECTS, NOT FROM [Recent moves] *** - [Recent moves] is for undo only. When the user asks to move a - different object than the most recent move, do NOT use the recent- - move's coordinates as origin — re-read the SCENE block and look up - the named object's row by id and color/type. - - WORKED ANTI-EXAMPLE — scene contains - cylinder-0 (orange) pos=(2.05, 0.95, -1.85) ← just moved here - cone-0 (teal) pos=(-0.55, 1.30, -2.40) - [Recent moves] - cylinder-0: (0.00, 0.95, -3.00) → (2.05, 0.95, -1.85) - - User: "Lift the teal cone up 0.5 m." - WRONG: place_object_relative(origin_x=2.05, origin_y=0.95, origin_z=-1.85, - direction="above", distance=0.5) - (those are cylinder-0/orange's coords from [Recent moves]). - RIGHT: place_object_relative(origin_x=-0.55, origin_y=1.30, origin_z=-2.40, - direction="above", distance=0.5) - (cone-0/teal's row from SCENE OBJECTS). - Examples: - 'shift the cone right 0.4 m' → place_object_relative(cone.x, cone.y, - cone.z, "right", 0.4) - 'raise it 0.5 m' → place_object_relative(obj.x, obj.y, - obj.z, "above", 0.5) - 'spawn a capsule in front of the cone' - → place_object_relative(cone.x, cone.y, - cone.z, "front", 0.3) - 'beside the cone' → place_object_relative(cone.x, cone.y, - cone.z, "next_to") - - *** COMBINED OFFSETS ARE NOT place_object_relative *** - If the user names TWO directions in one move (e.g. "up and to the - right", "down and back", "diagonally up-left"), place_object_relative - is the WRONG tool — it only handles a single direction. Use - displace_object with multiple non-zero scalar args: - - Words → displace_object axis - "right" / "to the right" → right=+d - "left" / "to the left" → right=-d - "up" / "raise" / "lift" → up=+d - "down" / "lower" / "drop" → up=-d - "forward" / "ahead" → forward=+d - "back" / "backwards" → forward=-d - - WORKED EXAMPLE — "Shift the cone down and to the right", cone - at (-0.5, 2.0, -2.3), canonical pose. BOTH directions must move: - iter 0: displace_object(current_x=-0.5, current_y=2.0, current_z=-2.3, - right=0.4, up=-0.4) - returns {"x": -0.1, "y": 1.6, "z": -2.3} - iter 1: update_primitive(obj_id="cone-0", x=-0.1, y=1.6, z=-2.3) - - WRONG: place_object_relative(direction="right", distance=0.4) — - ignores "down"; only x changes. Both named axes must move. - - Default unspecified distance for a combined offset: 0.5 m per axis. - - "Next to" / "beside" / "adjacent to" ALWAYS goes through - place_object_relative with direction="next_to" (or a small distance - in left/right). NEVER place the new object at a fresh user-relative - position when the user said "next to obj"; that puts it nowhere near - the obj. - WORKED EXAMPLE — cone at (-1.3, 1.4, -2.7), user says "Place a - purple cylinder beside the cone": - iter 0: place_object_relative(origin_x=-1.3, origin_y=1.4, origin_z=-2.7, - direction="next_to") - iter 0 result: {"x": -1.0, "y": 1.4, "z": -2.7} - iter 1: add_primitive(prim_type="cylinder", r=0.6, g=0, b=1, size=0.1, - x=-1.0, y=1.4, z=-2.7) - - Stacking ON TOP OF an existing object ("put X on top of Y", "stack - X on Y") — clearance for BOTH objects. render-mcp's `size` is the - primitive's half-extent on every axis (radius for spheres, half-edge - for boxes/pyramids), so the new object's CENTRE must sit at - base.y + base.size + new.size for the two to touch flush instead of - intersecting: - - → place_object_relative(origin_x=BASE.x, origin_y=BASE.y, - origin_z=BASE.z, direction="above", - distance = BASE.size + NEW.size) - Reuse BASE.x / BASE.z as the new object's x / z so the stack stays - in one column. - - WORKED EXAMPLE — base capsule at y=1.2 with size=0.15 (radius); - adding a default-size cylinder (size=0.1): - iter 0: place_object_relative(origin_x=base.x, origin_y=1.2, - origin_z=base.z, direction="above", - distance = 0.15 + 0.1 = 0.25) - → returns y = 1.45 - iter 1: add_primitive(prim_type="cylinder", r=1, g=0, b=1, - size=0.1, x=base.x, y=1.45, z=base.z) - Notice the second size term is the NEW object's size, NOT the - base's. If you're MOVING an existing object onto the base, the - second call is update_primitive(obj_id=mover.id, x=base.x, - y=1.45, z=base.z). Using only base.size as the distance puts the - new object's centre at the base's top, intersecting by exactly - one half-edge. - - STACKING FROM AN EMPTY SCENE ("stack three cubes", "pile up four - spheres") — there is no existing object to stack on. Pick one - arbitrary column (x=0, z=0 is fine) and assign each new object its - OWN y on the way in. Pass the y value DIRECTLY in each - add_primitive call so the stack is in place when the object spawns: - - WORKED EXAMPLE — "Stack four small magenta cylinders" (empty scene, default - size 0.1 → full-edge 0.2): - iter 0: add_primitive(prim_type="cylinder", r=1, g=0, b=1, size=0.1, - x=0, y=0.1, z=0) ← y SET HERE - iter 1: add_primitive(prim_type="cylinder", r=1, g=0, b=1, size=0.1, - x=0, y=0.3, z=0) ← y SET HERE - iter 2: add_primitive(prim_type="cylinder", r=1, g=0, b=1, size=0.1, - x=0, y=0.5, z=0) ← y SET HERE - iter 3: add_primitive(prim_type="cylinder", r=1, g=0, b=1, size=0.1, - x=0, y=0.7, z=0) ← y SET HERE - Each cylinder's centre is 2 × half-edge above the previous. Do NOT add - four cylinders with no y and then patch them up with update_primitive - afterwards — pass the right y on the way in. - - Move A toward / closer to / further from another OBJECT B (proximity - to a NAMED OBJECT, NOT the user): - 'move A closer to B' / 'move A toward B' - → along_direction(origin_x=A.x, origin_y=A.y, origin_z=A.z, - target_x=B.x, target_y=B.y, target_z=B.z, - distance=+d) [d default 0.5] - 'move A further from B' → same with distance=-d - DO NOT use position_relative or displace_object for these — both - offset along the USER'S facing direction, which is unrelated to - where B is. along_direction is the only tool that interpolates - between two points. - This rule applies ONLY when the proximity target is a named OBJECT. - "Closer to me" is a different rule (user is the direction; see - "TO ME / TOWARD ME" above). - - *** "CLOSER TO " / "TOWARD " / "DRAG TO - " → along_direction, NEVER displace_object *** - "Drag the pyramid toward the cone" reads as a user-frame delta - (move + direction), which TEMPTS displace_object(forward=…). That - is WRONG — the user's facing direction has no relationship to where - the cone is. The direction must be derived from the line FROM the - moved object TO the named target. Only along_direction does that - math; pass the moved object's coords as origin_*, the named - target's coords as target_*, and a positive distance for "closer" - (negative for "further"). - - WORKED EXAMPLE — cone-0 at (1.5, 1.4, -3.2), cylinder-0 at (-1.5, 1.4, -3.2), - user says "Drag the cone toward the cylinder": - iter 0: along_direction(origin_x=1.5, origin_y=1.4, origin_z=-3.2, - target_x=-1.5, target_y=1.4, target_z=-3.2, - distance=0.7) - iter 0 result: {"x": 0.8, "y": 1.4, "z": -3.2} - iter 1: update_primitive(obj_id="cone-0", x=0.8, y=1.4, z=-3.2) - Notice we did NOT call position_relative(forward=…) — the user's - gaze direction has nothing to do with where the cylinder is. - - WORKED EXAMPLE — "Drop a magenta cylinder in front of the orange capsule" - with capsule at (1.85, 1.42, -2.35), user at canonical pose. There may - be other objects in the scene; ignore them. The anchor is the CAPSULE alone. - iter 0: place_object_relative(origin_x=1.85, origin_y=1.42, - origin_z=-2.35, direction="front", - distance=0.25) - iter 0 result: {"x": 1.85, "y": 1.42, "z": -2.10} - ↑ z is LESS NEGATIVE than the capsule's z=-2.35, meaning the new - object is CLOSER to the user along the gaze — "in front of" the - capsule. The tool handles signs internally; you just pick the - direction enum. - iter 1: add_primitive(prim_type="cylinder", r=1, g=0, b=1, size=0.1, - x=1.85, y=1.42, z=-2.10) - iter 2: text response → "Added a magenta cylinder in front of the capsule." - - WORKED EXAMPLE — "Push it away from me", cylinder at (0.6, 1.5, -1.2), - canonical pose. "Away from me" → direction = "back" (away from user's - face). The MOVED OBJECT is the cylinder → origin = cylinder's position. - iter 0: place_object_relative(origin_x=0.6, origin_y=1.5, origin_z=-1.2, - direction="back", distance=0.8) - iter 0 result: {"x": 0.6, "y": 1.5, "z": -2.0} - iter 1: update_primitive(obj_id="cylinder-0", x=0.6, y=1.5, z=-2.0) - The cylinder's z shrank from -1.2 to -2.0 — further from the user at z=0. - Common mistake: passing origin = USER pos. That puts the cylinder on - the WRONG side — origin is always the moved object. - - Between two objects: - 'between A and B' / 'in the middle of A and B' / 'halfway between' - → between_anchors(a_x=A.x, a_y=A.y, a_z=A.z, - b_x=B.x, b_y=B.y, b_z=B.z) - 'between me and obj' - → between_anchors(a_x=user.x, a_y=user.y, a_z=user.z, - b_x=obj.x, b_y=obj.y, b_z=obj.z) - The USER'S head-pose position is the FIRST anchor (a_x, a_y, a_z); - the object's position is the SECOND anchor (b_x, b_y, b_z). - Read user_pos from the HEAD POSE block's "position" line — - do NOT use the object's coords for both anchors, that just - gives the object's own position back as the midpoint. - - *** "BETWEEN" / "MIDDLE" / "HALFWAY" → between_anchors, ALWAYS *** - The single trigger word "between" / "middle" / "halfway" / - "midway" / "in between" in the utterance is sufficient — route - to between_anchors before considering any other placement tool. - Read the TWO named anchors (objects, or "me" + one object) once, - pass them as (a_x, a_y, a_z) and (b_x, b_y, b_z), and feed the - returned {x, y, z} verbatim into add_primitive or - update_primitive. - - WRONG for "between": - • place_object_relative(direction=…) — offsets from ONE - anchor along a named direction. "0.5 m to A's right" is NOT - "between A and B". Setting direction="front" with B as the - origin gives "in front of B", not "midway between A and B". - Doing TWO of these (one from each anchor) is also wrong; - between_anchors does the midpoint in one call. - • place_user_relative — ignores the second anchor. - • place_inside_by_id — "between" is NOT - containment. The new object is not placed at A's position - nor at B's; it is placed at the MIDPOINT. Routing "between" - to a containment tool gives the wrong spot AND, when the - movee object doesn't exist yet (you're adding it), produces - zero mutations because place_inside_by_id needs an existing - obj_id. - • computing (Ax+Bx)/2 yourself — the model is not a - calculator; call the tool. - - WORKED EXAMPLE — cone at (1.4, 1.55, -3.2), user at (0, 1.6, 0), - user says "Drop a magenta cylinder halfway between me and the cone": - iter 0: between_anchors(a_x=0, a_y=1.6, a_z=0, - b_x=1.4, b_y=1.55, b_z=-3.2) - returns {"x": 0.7, "y": 1.575, "z": -1.6} - iter 1: add_primitive(prim_type="cylinder", r=1, g=0, b=1, - size=0.1, x=0.7, y=1.575, z=-1.6) - WRONG: place_user_relative("front", 1.6) — ignores the cone's - x; lands at x=0 instead of x=0.7. ALWAYS between_anchors - when the user says "between". - - Even when both positions are visible in the scene block and the - average looks trivial, you MUST call between_anchors — do not - compute (Ax+Bx)/2 yourself. Same rule as for moves. - - Use between_anchors ONLY when the user explicitly says "between A - and B" / "midpoint of A and B" / "between me and X" / "halfway - between" / "in the middle of". Phrases like "behind the cube", - "right of the cube", "next to the cube" name a SINGLE anchor - object, even when other objects are visible in the scene — route - those to the object-relative rules above. Do not pull a second - object into the computation. - - WORKED EXAMPLE — orange cylinder at (-1.4, 1.55, -2.8), purple cylinder at - (1.4, 1.55, -2.8), user says "Put a white cone in the middle": - 1. between_anchors(a_x=-1.4, a_y=1.55, a_z=-2.8, - b_x= 1.4, b_y=1.55, b_z=-2.8) - returns {"x": 0.0, "y": 1.55, "z": -2.8} - 2. add_primitive(prim_type="cone", r=1, g=1, b=1, size=0.1, - x=0.0, y=1.55, z=-2.8) - - WORKED EXAMPLE — scene also has distractor objects you should ignore: - cylinder-0 (ORANGE) at (-1.4, 1.55, -2.8) - cylinder-1 (PURPLE) at ( 1.4, 1.55, -2.8) - capsule-0 (TEAL) at ( 0.0, 1.55, -4.0) ← distractor, ignore - cone-0 (PINK) at (-3.0, 1.55, 0.5) ← distractor, ignore - User says "Put a white capsule midway between the orange cylinder and the purple cylinder": - 1. between_anchors(a_x=-1.4, a_y=1.55, a_z=-2.8, - b_x= 1.4, b_y=1.55, b_z=-2.8) - returns {"x": 0.0, "y": 1.55, "z": -2.8} - 2. add_primitive(prim_type="capsule", r=1, g=1, b=1, size=0.1, - x=0.0, y=1.55, z=-2.8) - Notice we used ONLY the two named objects' positions (orange, purple). The - teal capsule and pink cone are visible in the scene block but the - user did not name them, so they do NOT enter the between_anchors call. - NEVER substitute "between A and B" with "to the right of A" or any - other place_object_relative call — that is not what the user asked - for. - - Resize: - 'd× bigger' / 'half its size' → scale_value(current=obj.size, factor=d) - - Swap two objects' positions: - Trigger phrases — "swap" naming two objects in any of these forms means - swap, regardless of the small connective word in between: - 'swap A and B' 'swap A with B' 'swap A for B' - 'swap A in B' (STT often mishears "and" as "in") - 'switch A and B' 'switch their positions' - 'exchange A and B' - - "Swap" is NEVER an add-at-midpoint and NEVER a remove. It is exactly - TWO update_primitive calls. The objects must keep their existing ids, - types, colours, and sizes — only their (x, y, z) swap. - - Read both A.pos and B.pos from the scene block FIRST (the scene block - is your snapshot of the BEFORE state). Then issue: - update_primitive(obj_id=A.id, x=B.x, y=B.y, z=B.z) - update_primitive(obj_id=B.id, x=A.x, y=A.y, z=A.z) - - Pass all three of x, y, z on each call. The order of the two calls - does not matter because both use the BEFORE positions captured from - the scene block — overwriting in either order leaves A and B at - each other's original locations. - - WORKED EXAMPLE — "Switch the orange cone and the magenta cylinder", with - cone-0 at (-2.1, 1.4, -3.0) and cylinder-0 at (0.4, 2.2, -1.1): - tool_call 1: update_primitive(obj_id="cone-0", x=0.4, y=2.2, z=-1.1) - tool_call 2: update_primitive(obj_id="cylinder-0", x=-2.1, y=1.4, z=-3.0) - text response: "Switched the cone and cylinder." - - NEVER call remove_primitive followed by add_primitive to implement a - swap — that destroys the original ids and any history attached to - them, and leaks a new id on every swap. - -After the math tool returns, immediately call add_primitive or -update_primitive with the returned coordinates. Two tool calls per move -is normal and fast — do not try to skip the math tool to save a call. - -═══ BATCH MOVES — "all", "everything", "them" ═══ -Every targeted object needs its own update_primitive — math without -update_primitive is a no-op. - -PREFERRED PATH for a USER-FRAME delta (right/up/forward on every -target): ONE displace_objects call + N update_primitive calls. The -batch math tool returns {"items": [{"obj_id", "x", "y", "z"}, …]} — -fan out each item into its own update_primitive. Total tool calls: -1 + N. - - WORKED EXAMPLE — three objects in scene, user says "Move everything - forward 1 m": - iter 0: displace_objects(object_ids=[A.id, B.id, C.id], - current_xs=[A.x, B.x, C.x], - current_ys=[A.y, B.y, C.y], - current_zs=[A.z, B.z, C.z], - forward=1) - returns {"items": [{"obj_id": A.id, ...}, - {"obj_id": B.id, ...}, - {"obj_id": C.id, ...}]} - iter 1: update_primitive(obj_id=item0.obj_id, - x=item0.x, y=item0.y, z=item0.z) - iter 2: update_primitive(obj_id=item1.obj_id, - x=item1.x, y=item1.y, z=item1.z) - iter 3: update_primitive(obj_id=item2.obj_id, - x=item2.x, y=item2.y, z=item2.z) - iter 4: text response → "Moved all three objects forward one metre." - - COMPATIBILITY PATH — if the delta is object-anchored (e.g. "move all - spheres closer to the cube") or otherwise unavailable as a user-frame - delta, fall back to N (math + update_primitive) pairs, one per - object. EACH OBJECT'S MATH STARTS FROM ITS OWN ORIGIN — "Move them - closer to the cube" with two spheres at different positions means - TWO independent math calls with TWO different origin triples, NOT - one math call whose result is reused for every object. Reusing the - same result stacks every object at the same world position. - -═══ REAL-WORLD VISUAL QUERIES ═══ -If the user references anything about the real world that you cannot determine -from the scene state alone — what they are holding, pointing at, or looking at; -a real-world color, shape, text, or object — you MUST call -look_at_current_frame with a specific question about the live camera feed. - • look_at_current_frame inspects the user's current live camera frame — the - camera is always on, so you do not need to capture a frame or pass a file path. - • It returns {"answer": "..."} — base your reply (and any scene action) on - that answer. If it returns {"error": ...} with a "spoken" message, relay - that spoken message to the user verbatim and do not retry. - • Use the answer directly: if the user asked "what colour is this?", your - reply IS the colour from the answer; if they asked to add an object that - matches a real-world colour, read the colour from the answer first. -NEVER guess or assume a real-world color or object. -If you are not certain from the scene context alone, use look_at_current_frame. - -═══ RULES ═══ -1. Never invent or guess an object id — only use ids from the scene context. -2. add_primitive always creates a NEW object, even if similar ones exist. -3. COLOR — always set all three of r, g, b. Use these exact triples: - red=(1,0,0) green=(0,0.8,0) blue=(0,0.4,1) - yellow=(1,1,0) orange=(1,0.5,0) purple=(0.6,0,1) - cyan=(0,1,1) magenta=(1,0,1) pink=(1,0.4,0.7) - white=(1,1,1) black=(0,0,0) grey=(0.5,0.5,0.5) - The ONLY accepted parameters on add_primitive / update_primitive are: - prim_type (add only), obj_id (update only), x, y, z, r, g, b, size. - Do NOT invent new keys ('c', 'color', 'rgb', 'colour', etc.) — they - are dropped silently and the call fails. ALSO do not include a - 'color' string-value key alongside r/g/b. - - COMPOUND COLOUR ISOLATION — when a single user utterance creates two - or more objects with DIFFERENT colours (e.g. "a green sphere AND a - blue cube"), every add_primitive call must use only the channels - from its OWN colour row. Cross-contamination ("green sphere" coming - out as r=0, g=0.8, b=1, which is teal because b leaked from the - blue cube) is the most common failure mode. Treat each add_primitive - as if you'd never heard of the other object. - - NEW-OBJECT COLOUR LOOKUP — when ADDING a new object, the colour for - r/g/b comes from the user's spoken word and the table above — NOT - from any nearby scene object's colour. "Yellow sphere" is always - (1, 1, 0) regardless of whether a blue cube is also in the scene. - (Scene colours ARE used for REFERENCE — "the green cube" still - identifies the green-coloured box. They just don't supply r/g/b for - a brand-new object.) - - SCENE-CONTEXT CONTAMINATION (the failure mode) — if the scene - already contains a BLUE sphere and the user asks for a GREEN sphere - "between them", a tempting but WRONG output is (r=0, g=0.8, b=0.8) - because the model "remembered" that there's a blue object nearby - and lets b leak in. There is NO rule that mixes scene colours into - new objects. Treat the scene's colour values as INVISIBLE while - you fill r/g/b for a new object — they exist only to identify - referenced objects, never to contribute channels. - - GREEN is (0, 0.8, 0). Its b value MUST be 0. If you ever emit a - green object with b ≠ 0 you have contaminated; that output is wrong. - - COLOUR-LOOKUP CHECKLIST — every time you emit add_primitive, before - filling r, g, b: - 1. Find the colour word the user actually said for THIS object. - ("yellow sphere" → yellow; ignore other objects' colours.) - 2. Read the EXACT triple off the colour table (line 1 of rule 3). - yellow → (1, 1, 0). Not (1, 1, 1). Not (0, 1, 1). Not (1, 0.5, 0). - 3. If the result has any digit you didn't read off the table, - you contaminated — re-emit with the table's literal triple. - Same rule for update_primitive when changing colour. - - WORKED EXAMPLE — "Put an orange cone to my right and a magenta cylinder - to my left", canonical pose: - place_user_relative(direction="right", distance=1.2) - → {x: 1.2, y: 1.6, z: 0} - add_primitive(prim_type="cone", - r=1, g=0.5, b=0, size=0.1, - x=1.2, y=1.6, z=0) ← orange: b MUST be 0 - place_user_relative(direction="left", distance=1.2) - → {x: -1.2, y: 1.6, z: 0} - add_primitive(prim_type="cylinder", - r=1, g=0, b=1, size=0.1, - x=-1.2, y=1.6, z=0) ← magenta: g MUST be 0 - Notice the two calls share zero g/b digits between them. Anytime a - cone's b is anything other than 0 in this scenario, you have crossed - the wires. - - COMBINED SHAPE + COLOUR CHANGE — when a SINGLE utterance asks an - existing object to change BOTH shape and colour, emit ONE - update_primitive carrying BOTH prim_type AND r/g/b — don't drop - the colour. - - WORKED EXAMPLE — "Turn the cone into a turquoise cylinder": - → update_primitive(obj_id="cone-0", - prim_type="cylinder", r=0, g=0.8, b=0.8) - WRONG: update_primitive(obj_id="cone-0", prim_type="cylinder") - — the cone becomes a cylinder but keeps its old colour; - the user explicitly said "turquoise". Update everything - the user named in one call. - -4. SIZE in metres — radius for spheres, half-edge for boxes: - tiny=0.05 small=0.08 default=0.1 medium=0.2 large=0.5 huge=1.0 - Resize: call scale_value(current=obj.size, factor=k) and use the result. - '3× bigger' → factor=3 'half' → factor=0.5 'twice' → factor=2. -5. NEVER ask the user for clarification. If a detail is missing, pick a sensible default - and act immediately: unspecified type → white sphere; unspecified position → 1.5 m ahead; - unspecified color → white; unspecified size → default (0.1 m). - - ONE call per request. Once a tool call returns {"id": "...", "ok": true} - (or any success result), the operation is complete. Do NOT emit another - add_primitive / update_primitive / remove_primitive after a successful - result for the same intent — emit a SHORT TEXT RESPONSE instead. - - WORKED EXAMPLE — "Make a magenta cone", scene empty, user at canonical pose: - iter 0: tool_call → add_primitive(prim_type="cone", r=1, g=0, b=1, size=0.1) - iter 0 result: {"id": "cone-0", "ok": true} - iter 1: text response → "Added a magenta cone." [finish_reason="stop"] - STOP HERE. Do not emit add_primitive again. The cone already exists. -6. After completing the task respond with ONE short sentence confirming what was done. - diff --git a/agent-samples/xr-render-demo/worker/pyproject.toml b/agent-samples/xr-render-demo/worker/pyproject.toml index 2017a7f9..b3031b90 100644 --- a/agent-samples/xr-render-demo/worker/pyproject.toml +++ b/agent-samples/xr-render-demo/worker/pyproject.toml @@ -12,33 +12,26 @@ requires-python = ">=3.11,<3.13" dependencies = [ "xr-ai-hub-client", "xr-ai-models", - "xr-ai-nat[services,vision]", - "xr-ai-pipecat", + "xr-ai-nat[agents,services,vision,voice]", + "xr-ai-voice", "xr-ai-voicegate", "xr-ai-logging", "xr-render-scene", + "loguru>=0.7", "pyyaml>=6.0", - "pipecat-ai>=1.3", ] [tool.uv.sources] xr-ai-hub-client = { path = "../../../agent-sdk/xr-ai-hub-client", editable = true } xr-ai-models = { path = "../../../agent-sdk/xr-ai-models", editable = true } xr-ai-nat = { path = "../../../agent-sdk/xr-ai-nat", editable = true } -xr-ai-pipecat = { path = "../../../agent-sdk/xr-ai-pipecat", editable = true } +xr-ai-voice = { path = "../../../agent-sdk/xr-ai-voice", editable = true } xr-ai-voicegate = { path = "../../../utils/xr-ai-voicegate", editable = true } xr-ai-logging = { path = "../../../utils/xr-ai-logging", editable = true } xr-render-scene = { path = "../scene", editable = true } [project.scripts] -xr_render_demo_worker = "xr_render_demo_worker:run" +xr_render_demo_worker = "xr_render_demo_worker.__main__:run" [tool.hatch.build.targets.wheel] -only-include = [ - "xr_render_demo_worker.py", - "agent.py", - "capabilities.py", - "processors.py", - "tooling.py", - "config.py", -] +packages = ["xr_render_demo_worker"] diff --git a/agent-samples/xr-render-demo/worker/tooling.py b/agent-samples/xr-render-demo/worker/tooling.py deleted file mode 100644 index 80ae60a5..00000000 --- a/agent-samples/xr-render-demo/worker/tooling.py +++ /dev/null @@ -1,108 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -""" -Module-level tool-call helpers for the xr-render-demo brain. - -Pure functions shared between ``processors.py`` (the agentic loop) and -``agent.py`` (the XR-lifecycle render-mcp calls): unwrapping FastMCP tool -results, detecting leaked tool-call JSON in model text, extracting the first -balanced JSON object from a string, and flattening nested position/color args. -""" -from __future__ import annotations - -import json - - -def tool_payload(result) -> dict | list | None: - if hasattr(result, "data") and result.data is not None: - return result.data - return getattr(result, "structured_content", None) - - -_TOOL_CALL_KEY_SHAPES: tuple[frozenset[str], ...] = ( - frozenset({"name", "arguments"}), - frozenset({"tool", "args"}), - frozenset({"function", "arguments"}), -) - - -def looks_like_leaked_tool_call(text: str) -> bool: - """True if *text* should never reach TTS: a whole-message JSON object or - array (e.g. an echoed tool result), or JSON whose top level matches an - OpenAI-style tool-call envelope (name+arguments, tool+args, or - function+arguments). Prose that merely contains JSON passes through. - """ - stripped = text.strip() - if stripped.startswith("["): - try: - return isinstance(json.loads(stripped), list) - except json.JSONDecodeError: - return False - obj_text = extract_json(text) - if obj_text is None: - return False - try: - obj = json.loads(obj_text) - except json.JSONDecodeError: - return False - if obj_text.strip() == stripped and isinstance(obj, dict): - return True - candidates: list[dict] = [] - if isinstance(obj, dict): - candidates.append(obj) - elif isinstance(obj, list): - candidates.extend(c for c in obj if isinstance(c, dict)) - else: - return False - for c in candidates: - keys = set(c.keys()) - if any(shape <= keys for shape in _TOOL_CALL_KEY_SHAPES): - return True - return False - - -def extract_json(text: str) -> str | None: - depth, start, in_string, escape = 0, -1, False, False - for i, ch in enumerate(text): - if in_string: - if escape: escape = False - elif ch == "\\": escape = True - elif ch == '"': in_string = False - continue - if ch == '"': in_string = True; continue - if ch == "{": - if depth == 0: start = i - depth += 1 - elif ch == "}": - if depth == 0: continue - depth -= 1 - if depth == 0 and start >= 0: - return text[start:i + 1] - return None - - -def normalize_tool_args(args: dict) -> dict: - """Flatten nested position/color dicts that the LLM sometimes generates. - - The LLM may produce {"position": {"x":0,"y":1.6,"z":-1.5}} because it - pattern-matches the get_scene_state output format. Flatten to scalar kwargs - so FastMCP validation passes. - """ - args = dict(args) - - if "position" in args and isinstance(args["position"], dict): - pos = args.pop("position") - for k in ("x", "y", "z"): - if k in pos and k not in args: - args[k] = float(pos[k]) - - if "color" in args and isinstance(args["color"], dict): - col = args.pop("color") - for k in ("r", "g", "b"): - if k in col and k not in args: - args[k] = float(col[k]) - - # Strip None and empty-string values — the model sometimes emits r='' - # when thinking is enabled and the value wasn't filled in. - return {k: v for k, v in args.items() if v is not None and v != ""} diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker.py b/agent-samples/xr-render-demo/worker/xr_render_demo_worker.py deleted file mode 100644 index 3c980c92..00000000 --- a/agent-samples/xr-render-demo/worker/xr_render_demo_worker.py +++ /dev/null @@ -1,223 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -""" -xr-render-demo agent worker — voice-driven XR scene control via Pipecat. - -Voice pipeline (assembled by ``xr_ai_pipecat.make_voice_pipeline``): - transport.input → VadStt → VoiceGate → RenderSceneProcessor (brain) - → StreamingTts → transport.output - -Launched as a subprocess by ``uv run xr_render_demo``. -""" - -from __future__ import annotations - -import argparse -import asyncio -import pathlib -import signal -from pathlib import Path - -from loguru import logger -from nat.builder.function import Function -from nat.builder.workflow_builder import WorkflowBuilder -from xr_ai_logging import setup_logging -from xr_ai_models import ChatMessage, load_models_config, make_llm, make_stt, make_tts, make_vlm -from xr_ai_nat.functions.text_memory import TextMemoryFunctionsConfig -from xr_ai_pipecat import VadConfig, make_voice_pipeline, run_voice_pipeline -from xr_ai_pipecat.services import wait_for_services -from xr_ai_pipecat.transport import XRMediaHubTransport -from xr_ai_voicegate import load_voice_gate_config - -from agent import RenderDemoAgent -from capabilities import build_native_toolbox -from config import WorkerConfig, load_config -from processors import ( - _LIVE_PERCEPTION_TOOL, - _PAST_PERCEPTION_TOOL, - _PERCEPTION_TOOL_DEFS, - RenderSceneProcessor, -) - -_TRACE_FILE = "/tmp/xr-agent-trace.log" - -# Tools the worker calls directly (control-plane). Excluded from the LLM tool -# list so the model can't trigger them — the worker manages XR lifecycle. -# get_scene_state is intentionally absent: the model must call it to discover -# object ids before any manipulation. -_WORKER_MANAGED_TOOLS = frozenset({"start_xr", "get_health"}) - - -async def _group_functions(builder: WorkflowBuilder, *names: str) -> dict[str, Function]: - functions: dict[str, Function] = {} - for name in names: - group = await builder.get_function_group(name) - functions.update(await group.get_all_functions()) - return functions - - -_PROMPT_FILE = Path(__file__).resolve().parent / "prompts" / "system.txt" - - -async def main( - cfg: WorkerConfig, - config_path: pathlib.Path | None = None, - ready_file: pathlib.Path | None = None, -) -> None: - setup_logging("worker") - - # Curated session transcript — only records bound with extra={"trace": True} - # via ``logger.bind(trace=True)`` reach this sink. Tail this file (or - # paste it) to see USER/CTX/TOOL/RES/RESP events without the full chatter. - # DEBUG so verbose CTX / TOOL records (demoted out of the terminal) still - # land here. - logger.add( - _TRACE_FILE, - filter=lambda r: r["extra"].get("trace") is True, - format="{time:HH:mm:ss} {message}", - mode="w", - level="DEBUG", - ) - logger.bind(trace=True).info("=== trace started ===") - - models_cfg = load_models_config(cfg.models_yaml) - llm = make_llm(models_cfg, "llm") - agent_llm = make_llm(models_cfg, "agent_llm") - stt = make_stt(models_cfg, "stt") - tts = make_tts(models_cfg, "tts") - vlm_service = make_vlm(models_cfg, "vlm") - - # VLM /health only returns 200 after weights are fully loaded — this ensures - # GPU 0 memory has settled before LOVR starts its Vulkan device, preventing - # the transient OOM race condition. - probes = { - "LLM": llm.health, - "agent-LLM": agent_llm.health, - "STT": stt.health, - "TTS": tts.health, - "VLM": vlm_service.health, - } - await wait_for_services(probes) - - # A cold vLLM engine pays kernel-autotune costs on its first inference - # that can exceed the quick-ack timeout; absorb them before the first - # turn with a request shaped like a real quick-ack. Hosted backends - # (health_check: false, e.g. NIM) are always warm — skip the paid call. - if models_cfg.llm("llm").health_check: - try: - await llm.chat( - [ChatMessage(role="user", content="Add a small cube.")], - max_tokens=40, - timeout=120.0, - ) - except Exception: - logger.opt(exception=True).warning("LLM warmup failed") - - voice_gate_cfg = load_voice_gate_config(pathlib.Path(cfg.voice_gate_yaml)) - - transport = XRMediaHubTransport() - async with WorkflowBuilder() as builder: - toolbox, vision_config = await build_native_toolbox( - builder, - scene_endpoint=cfg.scene_endpoint, - openxr_endpoint=cfg.openxr_endpoint, - video_memory_endpoint=cfg.video_memory_endpoint, - frame_endpoint=transport.endpoint, - vlm=vlm_service, - ) - await builder.add_function_group( - "text_memory", TextMemoryFunctionsConfig(directory=cfg.text_memory_dir) - ) - - text_memory_functions = await _group_functions(builder, "text_memory") - text_memory = text_memory_functions["text_memory__add_transcript"] - # The native perception request models carry participant/reference - # context the processor injects; present the model trimmed schemas - # (question, and question+second_ago) in place of the raw native ones. - tools = toolbox.definitions( - exclude=_WORKER_MANAGED_TOOLS | {_LIVE_PERCEPTION_TOOL, _PAST_PERCEPTION_TOOL} - ) - tools.extend(_PERCEPTION_TOOL_DEFS) - logger.info("native tool-calling functions: {}", [tool.name for tool in tools]) - - brain = RenderSceneProcessor( - transport=transport, - cfg=cfg, - toolbox=toolbox, - release_vision=vision_config.release, - text_memory=text_memory, - prompt_path=_PROMPT_FILE, - tools=tools, - llm=llm, - agent_llm=agent_llm, - ) - # The endpoint retains the agent's bound callbacks for the worker lifetime. - RenderDemoAgent(transport=transport, brain=brain, tools=toolbox) - - _, task = make_voice_pipeline( - transport=transport, - stt=stt, - tts=tts, - brain=brain, - vad_cfg=VadConfig( - silence_duration=cfg.silence_duration, - min_speech=cfg.min_speech, - silero_threshold=cfg.silero_threshold, - ), - voice_gate_cfg=voice_gate_cfg, - # Brain pushes its own per-turn ``agent.response`` data - # message with the sanitized "display" string (see - # ``RenderDemoBrain._run_turn``); opting out of the - # pipeline-level echo here avoids a duplicate send. - text_topic="", - # Idle-timeout auto-cancel — disabled unless set in the worker YAML. - idle_timeout_secs=cfg.idle_timeout_secs, - ) - - loop = asyncio.get_running_loop() - cancel_requested = False - - def _request_cancel() -> None: - # PipelineTask.cancel is a coroutine; add_signal_handler needs a - # sync callable. Guard against a second signal (e.g. double - # ctrl-c) spawning a redundant cancel task while the first is - # still draining the pipeline. - nonlocal cancel_requested - if cancel_requested: - return - cancel_requested = True - asyncio.create_task(task.cancel()) - - for sig in (signal.SIGINT, signal.SIGTERM): - loop.add_signal_handler(sig, _request_cancel) - - logger.info("xr_render_demo starting") - try: - await run_voice_pipeline( - task, - transport, - on_ready=ready_file.touch if ready_file else None, - ) - finally: - transport.shutdown() - await brain.close() - for service in (stt, tts, vlm_service): - try: - await service.close() - except Exception: - logger.opt(exception=True).warning("service close failed") - logger.info("xr_render_demo stopped") - - -def run() -> None: - p = argparse.ArgumentParser(add_help=False) - p.add_argument("--config", type=pathlib.Path, default=None) - p.add_argument("--ready-file", type=pathlib.Path, default=None) - ns, _ = p.parse_known_args() - cfg = load_config(ns.config) - asyncio.run(main(cfg, config_path=ns.config, ready_file=ns.ready_file)) - - -if __name__ == "__main__": - run() diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/__init__.py b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/__init__.py new file mode 100644 index 00000000..149f872c --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/__init__.py @@ -0,0 +1,10 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""NAT-native xr-render-demo worker: a supervisor over focused subagents.""" + +from .app import run_app +from .models import SceneReply, SceneRequest +from .supervisor import scene_supervisor + +__all__ = ["SceneReply", "SceneRequest", "run_app", "scene_supervisor"] diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/__main__.py b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/__main__.py new file mode 100644 index 00000000..6fdf4382 --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/__main__.py @@ -0,0 +1,23 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Command-line entry point for the xr-render-demo worker.""" + +import argparse +import asyncio +import pathlib + +from .app import run_app +from .config import load_config + + +def run() -> None: + parser = argparse.ArgumentParser(add_help=False) + parser.add_argument("--config", type=pathlib.Path, default=None) + parser.add_argument("--ready-file", type=pathlib.Path, default=None) + arguments, _unknown = parser.parse_known_args() + asyncio.run(run_app(load_config(arguments.config), ready_file=arguments.ready_file)) + + +if __name__ == "__main__": + run() diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/__init__.py b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/__init__.py new file mode 100644 index 00000000..104d7495 --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/__init__.py @@ -0,0 +1,18 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Focused subagents composed by the scene supervisor.""" + +from .appearance import AppearanceAgentConfig +from .memory import MemoryAgentConfig +from .object import ObjectAgentConfig +from .placement import PlacementAgentConfig +from .vision import VisionAgentConfig + +__all__ = [ + "AppearanceAgentConfig", + "MemoryAgentConfig", + "ObjectAgentConfig", + "PlacementAgentConfig", + "VisionAgentConfig", +] diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/appearance/__init__.py b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/appearance/__init__.py new file mode 100644 index 00000000..60048a40 --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/appearance/__init__.py @@ -0,0 +1,8 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Appearance subagent.""" + +from .agent import AppearanceAgentConfig + +__all__ = ["AppearanceAgentConfig"] diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/appearance/agent.py b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/appearance/agent.py new file mode 100644 index 00000000..5b95e089 --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/appearance/agent.py @@ -0,0 +1,76 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Define the appearance subagent and its immediate NAT dependencies.""" + +from pathlib import Path + +from loguru import logger +from nat.plugin_api import ( + Builder, + FunctionBaseConfig, + FunctionGroupRef, + FunctionInfo, + LLMRef, + register_function, +) +from nat.plugins.langchain.agent.tool_calling_agent.register import ToolCallAgentWorkflowConfig +from pydantic import ConfigDict, Field + +from ...models import SubagentResult, SubagentTask +from ...scene import SceneContext +from ...spatial_ops import AppearanceOpsConfig, TurnGuard + +_PROMPT = Path(__file__).with_name("prompt.txt") +DESCRIPTION = "Change only the color of existing XR objects." + + +class AppearanceAgentConfig(FunctionBaseConfig, name="xr_render_appearance_agent"): + model_config = ConfigDict(arbitrary_types_allowed=True) + + llm_name: LLMRef = LLMRef("scene_llm") + scene_state: FunctionGroupRef = FunctionGroupRef("scene_state") + scene_updates: FunctionGroupRef = FunctionGroupRef("scene_updates") + context: SceneContext = Field(exclude=True, repr=False) + + +@register_function(config_type=AppearanceAgentConfig) +async def appearance_agent(config: AppearanceAgentConfig, builder: Builder): + guard = TurnGuard() + ops = FunctionGroupRef("appearance_ops") + await builder.add_function_group( + ops, + AppearanceOpsConfig(guard=guard, scene_state=config.scene_state, scene_updates=config.scene_updates), + ) + reasoning = await builder.add_function( + "appearance_reasoning", + ToolCallAgentWorkflowConfig( + llm_name=config.llm_name, + tool_names=[config.scene_state, ops], + system_prompt=_PROMPT.read_text(encoding="utf-8").strip(), + handle_tool_errors=True, + max_iterations=12, + max_empty_response_retries=1, + ), + ) + + async def change_appearance(request: SubagentTask) -> SubagentResult: + logger.debug("appearance agent instruction={!r}", request.instruction[:200]) + config.context.mark_mutating(request.participant_id) + guard.reset() + message = ( + f"Active participant: {request.participant_id}\n" + f"Utterance timestamp: {request.reference_time_us}\n" + f"{await config.context.describe(request.participant_id)}\n\n" + f"Focused instruction: {request.instruction}" + ) + output = await reasoning.ainvoke(message, to_type=str) + return SubagentResult(result=str(output or "Done.")) + + yield FunctionInfo.from_fn( + change_appearance, + description=DESCRIPTION, + ) + + +__all__ = ["AppearanceAgentConfig"] diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/appearance/prompt.txt b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/appearance/prompt.txt new file mode 100644 index 00000000..3fce6bf2 --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/appearance/prompt.txt @@ -0,0 +1,19 @@ +You are one focused subagent controlling appearance in a 3D XR scene. Follow +only the supervisor instruction and use only the tools provided to you. + +The SCENE OBJECTS context is authoritative for ids, types, positions, RGB +colors, and sizes. Pronouns refer to the object named in the latest +assistant reply; plural pronouns require acting on every resolved object. + +Handle color changes only, with one recolor call per object. Quote both +arguments from the instruction verbatim: object_words is the instruction's +words for the target ("the teal capsule", or an id the instruction states), +and color_words is the instruction's color word, mangled spellings +included; the tool resolves both against the scene itself and reports back +when nothing or several objects match — relay that answer instead of +guessing. An explicit color used to identify an object is a reference, not +a request to recolor it. + +Execute only the focused task, then report a concise result with stable ids +and resolved values the supervisor may need. Never expose JSON, tool +syntax, or private reasoning. diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/memory/__init__.py b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/memory/__init__.py new file mode 100644 index 00000000..f3ad4c62 --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/memory/__init__.py @@ -0,0 +1,8 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Memory subagent.""" + +from .agent import MemoryAgentConfig + +__all__ = ["MemoryAgentConfig"] diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/memory/agent.py b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/memory/agent.py new file mode 100644 index 00000000..7811d945 --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/memory/agent.py @@ -0,0 +1,53 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Define the memory subagent and its immediate NAT dependencies.""" + +from pathlib import Path + +from loguru import logger +from nat.plugin_api import Builder, FunctionBaseConfig, FunctionGroupRef, FunctionInfo, LLMRef, register_function +from nat.plugins.langchain.agent.tool_calling_agent.register import ToolCallAgentWorkflowConfig + +from ...models import SubagentResult, SubagentTask + +_PROMPT = Path(__file__).with_name("prompt.txt") +DESCRIPTION = "Recall earlier conversation turns; never mutates the XR scene." + + +class MemoryAgentConfig(FunctionBaseConfig, name="xr_render_memory_agent"): + llm_name: LLMRef = LLMRef("scene_llm") + conversations: FunctionGroupRef = FunctionGroupRef("conversations") + + +@register_function(config_type=MemoryAgentConfig) +async def memory_agent(config: MemoryAgentConfig, builder: Builder): + reasoning = await builder.add_function( + "memory_reasoning", + ToolCallAgentWorkflowConfig( + llm_name=config.llm_name, + tool_names=[config.conversations], + system_prompt=_PROMPT.read_text(encoding="utf-8").strip(), + handle_tool_errors=True, + max_iterations=12, + max_empty_response_retries=1, + ), + ) + + async def recall(request: SubagentTask) -> SubagentResult: + logger.debug("memory agent instruction={!r}", request.instruction[:200]) + message = ( + f"Active participant: {request.participant_id}\n" + f"Utterance timestamp: {request.reference_time_us}\n\n" + f"Focused instruction: {request.instruction}" + ) + output = await reasoning.ainvoke(message, to_type=str) + return SubagentResult(result=str(output or "Done.")) + + yield FunctionInfo.from_fn( + recall, + description=DESCRIPTION, + ) + + +__all__ = ["MemoryAgentConfig"] diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/memory/prompt.txt b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/memory/prompt.txt new file mode 100644 index 00000000..82378433 --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/memory/prompt.txt @@ -0,0 +1,9 @@ +You are the memory subagent. Follow only the supervisor instruction and use +only the transcript recall tool provided to you. Never invent a participant id +or tool argument. + +Recall earlier conversation when the supervisor needs a fact from a prior +turn. Call conversations__recall_conversation with the active participant id +and a suitable time window ending before the active utterance timestamp. +Return only relevant user and agent turns. Never mutate the XR scene, expose +tool syntax, or reveal private reasoning. diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/object/__init__.py b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/object/__init__.py new file mode 100644 index 00000000..37316f21 --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/object/__init__.py @@ -0,0 +1,8 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Object lifecycle subagent.""" + +from .agent import ObjectAgentConfig + +__all__ = ["ObjectAgentConfig"] diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/object/agent.py b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/object/agent.py new file mode 100644 index 00000000..b8d4355a --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/object/agent.py @@ -0,0 +1,105 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Define the object subagent and its immediate NAT dependencies.""" + +import asyncio +from pathlib import Path + +from loguru import logger +from nat.plugin_api import ( + Builder, + FunctionBaseConfig, + FunctionGroupRef, + FunctionInfo, + LLMRef, + register_function, +) +from nat.plugins.langchain.agent.tool_calling_agent.register import ToolCallAgentWorkflowConfig +from pydantic import ConfigDict, Field + +from ...models import SubagentResult, SubagentTask +from ...scene import SceneContext +from ...spatial_ops import CreationLedger, ObjectOpsConfig, TurnGuard + +_PROMPT = Path(__file__).with_name("prompt.txt") +DESCRIPTION = ( + "Create new XR objects at their requested initial positions, and remove, resize, duplicate, or " + "reshape existing ones; never moves an existing object, including putting one in, on, or next to another." +) + + +class ObjectAgentConfig(FunctionBaseConfig, name="xr_render_object_agent"): + model_config = ConfigDict(arbitrary_types_allowed=True) + + llm_name: LLMRef = LLMRef("scene_llm") + scene_state: FunctionGroupRef = FunctionGroupRef("scene_state") + scene_updates: FunctionGroupRef = FunctionGroupRef("scene_updates") + scene_objects: FunctionGroupRef = FunctionGroupRef("scene_objects") + tracking: FunctionGroupRef = FunctionGroupRef("tracking") + spatial: FunctionGroupRef = FunctionGroupRef("spatial") + context: SceneContext = Field(exclude=True, repr=False) + # A supervisor-owned ledger spans the whole user turn, including the + # verification pass; without one, dedup only covers a single delegation. + ledger: CreationLedger | None = Field(default=None, exclude=True, repr=False) + + +@register_function(config_type=ObjectAgentConfig) +async def object_agent(config: ObjectAgentConfig, builder: Builder): + ledger = config.ledger if config.ledger is not None else CreationLedger() + owns_ledger = config.ledger is None + guard = TurnGuard() + delegation_lock = asyncio.Lock() + ops = FunctionGroupRef("object_ops") + await builder.add_function_group( + ops, + ObjectOpsConfig( + ledger=ledger, + guard=guard, + scene_state=config.scene_state, + scene_updates=config.scene_updates, + scene_objects=config.scene_objects, + tracking=config.tracking, + spatial=config.spatial, + ), + ) + reasoning = await builder.add_function( + "object_reasoning", + ToolCallAgentWorkflowConfig( + llm_name=config.llm_name, + tool_names=[config.scene_state, ops], + system_prompt=_PROMPT.read_text(encoding="utf-8").strip(), + handle_tool_errors=True, + max_iterations=12, + max_empty_response_retries=1, + ), + ) + + async def change_object(request: SubagentTask) -> SubagentResult: + logger.debug("object agent instruction={!r}", request.instruction[:200]) + config.context.mark_mutating(request.participant_id) + # One delegation at a time: the guard and ledger are agent-scoped, + # and interleaved delegations would clobber each other's state. + async with delegation_lock: + if owns_ledger: + ledger.reset() + guard.reset() + return await _run(request) + + async def _run(request: SubagentTask) -> SubagentResult: + message = ( + f"Active participant: {request.participant_id}\n" + f"Utterance timestamp: {request.reference_time_us}\n" + f"{await config.context.describe(request.participant_id, bearings=True)}\n\n" + f"Focused instruction: {request.instruction}" + ) + output = await reasoning.ainvoke(message, to_type=str) + return SubagentResult(result=str(output or "Done.")) + + yield FunctionInfo.from_fn( + change_object, + description=DESCRIPTION, + ) + + +__all__ = ["ObjectAgentConfig"] diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/object/prompt.txt b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/object/prompt.txt new file mode 100644 index 00000000..3050e90e --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/object/prompt.txt @@ -0,0 +1,77 @@ +You are one focused subagent controlling object lifecycle, shape, and size in a +3D XR scene. Follow only the supervisor instruction and use only the tools +provided to you. + +The SCENE OBJECTS context is authoritative for ids, types, positions, RGB +colors, and sizes. Resolve explicit type, color, and positional descriptions +before recency. Pronouns refer to the object named in the latest assistant +reply; plural pronouns require acting on every resolved object. Never invent an +id or tool argument. + +Handle creation, including a new object's initial position, plus deletion, +shape changes, resizing, and duplication. Do not change color or move an +existing object; the supervisor will invoke appearance or placement separately. + +Quote the instruction's own shape word as prim_type, mangled spellings +included; the tools map it to a renderer shape themselves. A shape word you +do not recognize is still a creation, never a reason to anchor on or reuse +an existing object. +Creation always makes a new object with the stated shape. A request to change +or turn an existing object into another shape is not creation: make one +change_shape call with that object's id and the new shape. The creation tools +resolve the user frame and geometry themselves and return the new object's id +and final position; never compute coordinates yourself. When the request +states a distance, pass exactly that number; never substitute a tool's +default distance. + +- A user-relative position ("in front of me", "two metres ahead", "to my + left", "above my head", "at my feet") uses create_user_relative with the + named direction and the stated distance. "At my feet" or "on the floor" is + direction below with distance about 1.5. When the instruction names no + position and no anchor object (including "no position stated" or a bare + "Add/Make a "), always use create_user_relative with direction + front; never anchor on an existing object the instruction does not name, + and never stack on one because the space seems occupied. +- A position relative to an existing object ("above the box", "behind the + sphere", "on the cube") uses create_object_relative. The anchor_words + value is a quotation of the instruction, never a lookup: for "Create a + cone above the teal capsule", anchor_words is "teal capsule" even though + SCENE OBJECTS lists capsule-8 and capsule-9; for "Create a cone above + capsule-9", anchor_words is "capsule-9" because the instruction itself + said so. The tool resolves quoted descriptions, including + transcription-mangled nouns, against the scene itself, and reports a tie + when several objects fit; relay that question. + Horizontal relations use your view of the anchor: "in front of the cone" is + toward_user, "behind the cone" is away_from_user. For stacking, anchor + each new object on the previously created one with relation above and a + distance of about the two sizes combined; never reuse one anchor for + every layer. +- create_at is only for coordinates stated numerically in the instruction; + with no numbers present, never call it. With no + requested spatial relationship at all, use create_user_relative with + direction front and the default distance, so the object appears in front + of the user; never direction above unless the request says above. + +The creation tools take color_words: quote the instruction's color word +exactly as written, mangled spellings included, and the tool resolves the +RGB itself. "Create a teal capsule" passes color_words "teal"; an identical +copy of capsule-8 passes color_words "same as capsule-8"; when the +instruction states no color, leave color_words empty and the standard +default applies. Never translate a color to numbers yourself. + +Never create first and move the new object afterward, and never substitute +renderer defaults for requested placement. + +Deletion uses remove_object with the resolved id. Shape changes use one +change_shape call; change_shape changes only the primitive type, never size. +Any size change uses one resize_object call with the requested factor: +"half the size" is factor 0.5, "double" is factor 2, "three times larger" is +factor 3. + +If the instruction asks to move, contain, swap, or restore an existing +object, make no tool call; report that the task is movement of an existing +object so the supervisor can route it to placement_agent. + +Tool calls must use their declared schemas. Report the stable id of every +created or changed object so the supervisor can pass it to another subagent. +Never expose JSON, tool syntax, or private reasoning. diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/placement/__init__.py b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/placement/__init__.py new file mode 100644 index 00000000..5ac251f9 --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/placement/__init__.py @@ -0,0 +1,8 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Placement subagent.""" + +from .agent import PlacementAgentConfig + +__all__ = ["PlacementAgentConfig"] diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/placement/agent.py b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/placement/agent.py new file mode 100644 index 00000000..c6083ccf --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/placement/agent.py @@ -0,0 +1,93 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Define the placement subagent and its immediate NAT dependencies.""" + +import asyncio +from pathlib import Path + +from loguru import logger +from nat.plugin_api import ( + Builder, + FunctionBaseConfig, + FunctionGroupRef, + FunctionInfo, + LLMRef, + register_function, +) +from nat.plugins.langchain.agent.tool_calling_agent.register import ToolCallAgentWorkflowConfig +from pydantic import ConfigDict, Field + +from ...models import SubagentResult, SubagentTask +from ...scene import SceneContext +from ...spatial_ops import PlacementOpsConfig, TurnGuard + +_PROMPT = Path(__file__).with_name("prompt.txt") +DESCRIPTION = ( + "Move, swap, contain, stack, or restore existing XR objects; never creates, recolors, or removes them." +) + + +class PlacementAgentConfig(FunctionBaseConfig, name="xr_render_placement_agent"): + model_config = ConfigDict(arbitrary_types_allowed=True) + + llm_name: LLMRef = LLMRef("scene_llm") + scene_state: FunctionGroupRef = FunctionGroupRef("scene_state") + scene_updates: FunctionGroupRef = FunctionGroupRef("scene_updates") + tracking: FunctionGroupRef = FunctionGroupRef("tracking") + spatial: FunctionGroupRef = FunctionGroupRef("spatial") + context: SceneContext = Field(exclude=True, repr=False) + + +@register_function(config_type=PlacementAgentConfig) +async def placement_agent(config: PlacementAgentConfig, builder: Builder): + guard = TurnGuard() + delegation_lock = asyncio.Lock() + ops = FunctionGroupRef("placement_ops") + await builder.add_function_group( + ops, + PlacementOpsConfig( + guard=guard, + scene_state=config.scene_state, + scene_updates=config.scene_updates, + tracking=config.tracking, + spatial=config.spatial, + ), + ) + reasoning = await builder.add_function( + "placement_reasoning", + ToolCallAgentWorkflowConfig( + llm_name=config.llm_name, + tool_names=[ops], + system_prompt=_PROMPT.read_text(encoding="utf-8").strip(), + handle_tool_errors=True, + max_iterations=12, + max_empty_response_retries=1, + ), + ) + + async def place(request: SubagentTask) -> SubagentResult: + logger.debug("placement agent instruction={!r}", request.instruction[:200]) + config.context.mark_mutating(request.participant_id) + # The guard is agent-scoped; serialize delegations. + async with delegation_lock: + guard.reset() + return await _run(request) + + async def _run(request: SubagentTask) -> SubagentResult: + message = ( + f"Active participant: {request.participant_id}\n" + f"Utterance timestamp: {request.reference_time_us}\n" + f"{await config.context.describe(request.participant_id, bearings=True)}\n\n" + f"Focused instruction: {request.instruction}" + ) + output = await reasoning.ainvoke(message, to_type=str) + return SubagentResult(result=str(output or "Done.")) + + yield FunctionInfo.from_fn( + place, + description=DESCRIPTION, + ) + + +__all__ = ["PlacementAgentConfig"] diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/placement/prompt.txt b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/placement/prompt.txt new file mode 100644 index 00000000..cc8d2d61 --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/placement/prompt.txt @@ -0,0 +1,68 @@ +You are one focused subagent moving existing objects in a 3D XR scene. Follow +only the supervisor instruction and use only the tools provided to you. + +The SCENE OBJECTS context is authoritative for ids, types, positions, RGB +colors, and sizes. Every object argument is a quotation of the instruction, +never a lookup: for "Raise the teal capsule a little", pass "teal capsule" +even though SCENE OBJECTS lists capsule-3 and capsule-7; for "Raise +capsule-7", pass "capsule-7" because the instruction itself said so. The +tools resolve quoted descriptions, mangled nouns included, against the +scene themselves, and report back when nothing or several objects match. +When a tool answers that nothing matches, the turn is over: instruction +says "raise the magenta capsule" and the tool answers that no scene object +matches "magenta capsule" — make no other tool call, and reply that no +magenta capsule exists, naming what the scene does contain. Trying a +different object instead is the one forbidden move. Name the id +you moved in your report. Instruction says "put cone-7 back" and cone-7 is +in SCENE OBJECTS with [Recent moves]: proceed normally with move_to. Resolve explicit type, color, and positional descriptions +before recency. Pronouns refer to the object named in the latest assistant +reply; plural pronouns require acting on every resolved object. Never invent an +id or tool argument. + +Every tool takes object ids, direction names, and metres, performs the move +itself, and returns the final position. When the request states a distance, +pass exactly that number; never substitute a tool's default distance, and +never scale it down. Never compute or copy coordinates +yourself except when the request states them. Pick the operation whose meaning +matches the request: + +- A stated distance in a direction ("one metre to my right", "down 30 cm", + "left one meter") shifts the object from its current position: use nudge + with signed user-frame offsets. +- A user-anchored destination with no stated offset from the object ("to my + left", "in front of me", "above my head", "at my feet") uses + move_user_relative. +- A destination anchored on another object ("beside/left of/above/on top of + the cone") uses move_object_relative. Horizontal relations use your view of + the anchor: "in front of the cone" is toward_user, "behind the cone" is + away_from_user. For stacking on an anchor, use relation above with a + distance of about the two sizes combined. +- "Into/inside the cube" uses move_inside. +- "Between A and B" uses move_between. +- "Toward/closer to X" or "away from X" uses move_toward with the matching + direction. +- "Closer to me/bring it closer/away from me" uses move_toward_user. +- Swaps use swap_positions once; never create or delete objects to swap. +- Restoring an object to where it was earlier uses move_to with the + previous coordinates from [Recent moves]. +- Explicit coordinates, or a novel arrangement you must reason out, use + move_to per object. + +If the instruction asks to create something new, make no tool call; report +back that the task is creation, repeating the instruction word for word +("This is a creation task: ") so the supervisor +can re-route it with nothing lost. + +Contrast: "move the sphere above my head" is a user-anchored destination +(move_user_relative, direction above); "move the sphere up one metre" is a +shift (nudge, up=1). The [Object bearings from the user] block gives each +object's computed offsets for resolving "the one on the right/closest/behind +me" style references. + +Do not change appearance, shape, or size, and do not create, remove, or +delete objects. "Remove/delete/get rid of X" is never a movement: make no +tool call and report that removal belongs to another subagent so the +supervisor can re-route it. Execute only +the focused task, then report a concise result with stable ids and final +positions the supervisor may need. Never expose JSON, tool syntax, or private +reasoning. diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/vision/__init__.py b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/vision/__init__.py new file mode 100644 index 00000000..ded4af50 --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/vision/__init__.py @@ -0,0 +1,8 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Vision subagent.""" + +from .agent import VisionAgentConfig + +__all__ = ["VisionAgentConfig"] diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/vision/agent.py b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/vision/agent.py new file mode 100644 index 00000000..963056c7 --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/vision/agent.py @@ -0,0 +1,82 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Define the vision subagent and its immediate NAT dependencies.""" + +from pathlib import Path + +from loguru import logger +from nat.plugin_api import Builder, FunctionBaseConfig, FunctionGroupRef, FunctionInfo, LLMRef, register_function +from nat.plugins.langchain.agent.tool_calling_agent.register import ToolCallAgentWorkflowConfig +from pydantic import ConfigDict, Field + +from ...models import SubagentResult, SubagentTask +from ...scene import SceneContext + +_PROMPT = Path(__file__).with_name("prompt.txt") +DESCRIPTION = ( + "Answer a question about the physical world from the live or recorded camera; never for XR scene " + "state or placement." +) + + +class VisionAgentConfig(FunctionBaseConfig, name="xr_render_vision_agent"): + model_config = ConfigDict(arbitrary_types_allowed=True) + + llm_name: LLMRef = LLMRef("scene_llm") + vision: FunctionGroupRef = FunctionGroupRef("vision") + context: SceneContext | None = Field(default=None, exclude=True, repr=False) + + +@register_function(config_type=VisionAgentConfig) +async def vision_agent(config: VisionAgentConfig, builder: Builder): + reasoning = await builder.add_function( + "vision_reasoning", + ToolCallAgentWorkflowConfig( + llm_name=config.llm_name, + tool_names=[config.vision], + system_prompt=_PROMPT.read_text(encoding="utf-8").strip(), + # A missing live frame must return to the supervisor instead of + # letting this agent improvise a recorded-frame fallback. + handle_tool_errors=False, + max_iterations=12, + max_empty_response_retries=1, + ), + ) + + async def observe(request: SubagentTask) -> SubagentResult: + logger.debug("vision agent instruction={!r}", request.instruction[:200]) + if config.context is not None: + config.context.mark_delegated(request.participant_id) + scene_block = "" + if config.context is not None: + scene_block = f"{await config.context.describe(request.participant_id)}\n\n" + message = ( + f"Active participant: {request.participant_id}\n" + f"Utterance timestamp: {request.reference_time_us}\n" + f"{scene_block}" + f"Focused instruction: {request.instruction}" + ) + # A dead camera is an answer, not a workflow failure: retry once + # against scene data alone, then degrade explicitly. + try: + output = await reasoning.ainvoke(message, to_type=str) + except Exception as error: + retry = ( + f"{message}\n\n(The camera is unavailable: {error}. If SCENE OBJECTS" + " answers the instruction, answer from it with no tool call;" + " otherwise state that no visual fact is available.)" + ) + try: + output = await reasoning.ainvoke(retry, to_type=str) + except Exception: + return SubagentResult(result=f"No visual fact available ({error}); proceed without vision.") + return SubagentResult(result=str(output or "Done.")) + + yield FunctionInfo.from_fn( + observe, + description=DESCRIPTION, + ) + + +__all__ = ["VisionAgentConfig"] diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/vision/prompt.txt b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/vision/prompt.txt new file mode 100644 index 00000000..a70cc47c --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/agents/vision/prompt.txt @@ -0,0 +1,24 @@ +You are the vision subagent. Follow only the supervisor instruction and use +only the perception tools provided to you. Never invent a participant id or +tool argument. + +Observe the user's live camera or recorded video and return facts to the +supervisor. When SCENE OBJECTS is present, it lists every XR object; +contrast: "Is there a magenta cone currently in the scene?": answer directly from +SCENE OBJECTS with the ids and positions it shows, no tool call. "What color +is the mug the user is holding?": a physical-world fact, call +vision__look_at_current_frame with that question. Call vision__look_at_current_frame with a specific question only +for the present physical view. Never guess a real-world object, color, shape, +or text. If the current frame is unavailable, report that failure immediately; +never substitute a recorded frame or repeat the failed call. + +Call vision__look_at_past_frame only when the supervisor explicitly asks about +a past moment. Pass the active participant, the utterance timestamp as +reference_time_us, and a positive seconds offset. Never reinterpret a present +question as historical and never call the historical tool with a zero offset. + +Never create, update, remove, or place an XR object. When asked about color, +include explicit normalized r, g, and b values in the result. When asked about +shape, text, identity, or another property, describe only what the frame +supports. Your concise result is evidence for the supervisor, not a scene +mutation. Never expose JSON, tool syntax, or private reasoning. diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/app.py b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/app.py new file mode 100644 index 00000000..d915d62b --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/app.py @@ -0,0 +1,102 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Wire model services, native NAT function groups, subagents, and the voice session.""" + +from __future__ import annotations + +from pathlib import Path + +from loguru import logger +from nat.builder.workflow_builder import WorkflowBuilder +from xr_ai_logging import setup_logging +from xr_ai_models import load_models_config, make_llm, make_stt, make_tts, make_vlm +from xr_ai_nat.adapters import as_voice_handler, record_voice_transcripts +from xr_ai_nat.functions.spatial_math import SpatialMathFunctionsConfig +from xr_ai_nat.functions.text_memory import ConversationMemoryFunctionsConfig, TextMemoryFunctionsConfig +from xr_ai_nat.functions.video_memory import VideoMemoryFunctionsConfig +from xr_ai_nat.functions.vision import VisionToolsConfig +from xr_ai_nat.functions.xr_tracking import XRTrackingFunctionsConfig +from xr_ai_voice import VadConfig, VoiceSession +from xr_ai_voicegate import load_voice_gate_config +from xr_render_scene import ( + SceneControlFunctionsConfig, + SceneObjectFunctionsConfig, + SceneStateFunctionsConfig, + SceneUpdateFunctionsConfig, +) + +from .config import WorkerConfig +from .models import SceneRequest +from .supervisor import scene_supervisor +from .xr_session import XRSessionController + + +async def run_app(config: WorkerConfig, *, ready_file: Path | None = None) -> None: + """Run the render sample until the shared voice session exits.""" + setup_logging("worker") + models = load_models_config(config.models_yaml) + llm = make_llm(models, "agent_llm") + stt = make_stt(models, "stt") + tts = make_tts(models, "tts") + vlm = make_vlm(models, "vlm") + + session = VoiceSession( + stt=stt, + tts=tts, + vad=VadConfig( + silence_duration=config.silence_duration, + min_speech=config.min_speech, + silero_threshold=config.silero_threshold, + ), + voice_gate=load_voice_gate_config(Path(config.voice_gate_yaml)), + probes={"agent-llm": llm.health, "vlm": vlm.health}, + ready_file=ready_file, + closeables=(llm, vlm), + idle_timeout_secs=config.idle_timeout_secs, + ) + + async with session, WorkflowBuilder() as builder: + await builder.add_function_group("scene_state", SceneStateFunctionsConfig(endpoint=config.scene_endpoint)) + await builder.add_function_group("scene_updates", SceneUpdateFunctionsConfig(endpoint=config.scene_endpoint)) + await builder.add_function_group("scene_objects", SceneObjectFunctionsConfig(endpoint=config.scene_endpoint)) + await builder.add_function_group("scene_control", SceneControlFunctionsConfig(endpoint=config.scene_endpoint)) + await builder.add_function_group("tracking", XRTrackingFunctionsConfig(endpoint=config.openxr_endpoint)) + await builder.add_function_group("spatial", SpatialMathFunctionsConfig()) + await builder.add_function_group("text_memory", TextMemoryFunctionsConfig(directory=config.text_memory_dir)) + await builder.add_function_group("conversations", ConversationMemoryFunctionsConfig()) + await builder.add_function_group( + "video_memory", + VideoMemoryFunctionsConfig(endpoint=config.video_memory_endpoint), + ) + vision = VisionToolsConfig(endpoint=session.transport.endpoint, vlm=vlm) + await builder.add_function_group("vision", vision) + + supervisor = await scene_supervisor(builder=builder, llm=llm) + handler = as_voice_handler( + supervisor, + request=lambda turn: SceneRequest( + transcript=turn.text, + participant_id=turn.participant_id, + timestamp_us=turn.timestamp_us, + ), + response=lambda reply: reply.response, + ) + text_memory = await builder.get_function_group("text_memory") + text_memory_functions = await text_memory.get_all_functions() + scene_control = await builder.get_function_group("scene_control") + scene_control_functions = await scene_control.get_all_functions() + xr_session = XRSessionController( + session=session, + start_xr=scene_control_functions["scene_control__start_xr"], + get_render_health=scene_control_functions["scene_control__get_health"], + ) + xr_session.attach() + + logger.info("xr-render-demo worker starting") + await session.run( + handler, + observer=record_voice_transcripts(text_memory_functions["text_memory__add_transcript"]), + on_participant_left=vision.release, + ) + logger.info("xr-render-demo worker stopped") diff --git a/agent-samples/xr-render-demo/worker/config.py b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/config.py similarity index 78% rename from agent-samples/xr-render-demo/worker/config.py rename to agent-samples/xr-render-demo/worker/xr_render_demo_worker/config.py index 5af4c741..b3da09d2 100644 --- a/agent-samples/xr-render-demo/worker/config.py +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/config.py @@ -12,11 +12,9 @@ @dataclass(frozen=True) class WorkerConfig: - # Path to the models.yaml file (resolved relative to cwd). models_yaml: str - # Path to the voice gate YAML. Resolved the same way as models_yaml — - # bare basenames sit next to this worker's config YAML. + # Resolved relative to the config file, like models_yaml. voice_gate_yaml: str scene_endpoint: str @@ -40,13 +38,8 @@ def load_config(path: pathlib.Path | None) -> WorkerConfig: with open(path) as f: data = yaml.safe_load(f) or {} - # Resolve models_yaml relative to the config file's directory so the path - # works regardless of where the worker process is launched from. The - # default `"models.yaml"` is a bare basename — it sits next to this - # worker's config YAML in `agent-samples/xr-render-demo/yaml/`. When the - # launcher passes `--config`, `path.parent` is that yaml dir; when run - # bare without `--config`, the relative path falls back to CWD. - # + # Relative yaml paths resolve against the config file's directory, so + # bare basenames sit next to this worker's config YAML. # `model_backend: nim` selects the NIM overlay for hosted LLM/VLM services. backend = str(data.get("model_backend", "local")).lower() models_yaml_raw = ( @@ -68,7 +61,6 @@ def load_config(path: pathlib.Path | None) -> WorkerConfig: silence_duration = float(data.get("silence_duration", 0.8)), min_speech = float(data.get("min_speech", 0.15)), silero_threshold = float(data.get("silero_threshold", 0.5)), - # 0 / unset → disabled (None); a positive value opts into idle cancel. idle_timeout_secs = (float(data["idle_timeout_secs"]) if data.get("idle_timeout_secs") else None), ) diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/models.py b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/models.py new file mode 100644 index 00000000..1cd35cc8 --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/models.py @@ -0,0 +1,38 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Typed requests and results exchanged by the scene workflow.""" + +from pydantic import BaseModel, ConfigDict, Field + + +class SceneRequest(BaseModel): + model_config = ConfigDict(extra="forbid") + transcript: str + participant_id: str = "" + timestamp_us: int = 0 + + +class SceneReply(BaseModel): + response: str + + +class SubagentTask(BaseModel): + """Self-contained task passed from the supervisor to one focused agent. + + Model-visible: NAT lifts this docstring into every subagent's tool + schema, so it shapes how the supervisor phrases delegations. + """ + + instruction: str = Field(description="Focused task including facts returned by earlier subagents.") + participant_id: str = Field(description="Active participant ID copied from the user request.") + reference_time_us: int = Field(default=0, description="Timestamp of the active user utterance.") + + +class SubagentResult(BaseModel): + """Focused result returned to the supervisor for further planning.""" + + result: str + + +__all__ = ["SceneReply", "SceneRequest", "SubagentResult", "SubagentTask"] diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/scene.py b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/scene.py new file mode 100644 index 00000000..adb6f9c9 --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/scene.py @@ -0,0 +1,134 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Authoritative scene state and movement history for the workflow.""" + +from __future__ import annotations + +from nat.plugin_api import Function +from xr_ai_nat.functions.xr_tracking import HeadPoseRequest +from xr_render_scene import EmptyRequest, SceneState + + +class SceneContext: + """Read authoritative scene state and retain the last movement for undo.""" + + def __init__(self, get_scene_state: Function, get_user_frame: Function | None = None) -> None: + self._get_scene_state = get_scene_state + self._get_user_frame = get_user_frame + self._recent_moves: dict[str, list[str]] = {} + self._recent_moves_age: dict[str, int] = {} + self._mutating_delegations: set[str] = set() + self._any_delegations: set[str] = set() + + def mark_mutating(self, participant_id: str) -> None: + """Flag that this turn delegated mutating work.""" + self._mutating_delegations.add(participant_id) + self._any_delegations.add(participant_id) + + def mark_delegated(self, participant_id: str) -> None: + """Flag that this turn delegated any subagent work.""" + self._any_delegations.add(participant_id) + + def take_delegated(self, participant_id: str) -> bool: + """Consume and return whether the turn delegated at all.""" + if participant_id in self._any_delegations: + self._any_delegations.discard(participant_id) + return True + return False + + def take_mutating(self, participant_id: str) -> bool: + """Consume and return whether the turn delegated any mutating work.""" + if participant_id in self._mutating_delegations: + self._mutating_delegations.discard(participant_id) + return True + return False + + async def snapshot(self) -> SceneState: + """Return the renderer's current typed scene state.""" + return await self._get_scene_state.ainvoke(EmptyRequest()) + + async def describe(self, participant_id: str, *, bearings: bool = False) -> str: + """Format the current scene and participant's last movement for a subagent.""" + state = await self.snapshot() + parts = [f"[SCENE OBJECTS]\n{state.model_dump_json()}"] + if bearings and (computed := await self._bearings(state)): + parts.append( + "[Object bearings from the user] (computed; +right/-left, +ahead/-behind, +up/-down)\n" + computed + ) + if moves := self._recent_moves.get(participant_id): + parts.append("[Recent moves]\n" + "\n".join(moves)) + return "\n\n".join(parts) + + async def _bearings(self, state: SceneState) -> str: + if self._get_user_frame is None or not state.objects: + return "" + try: + frame = await self._get_user_frame.ainvoke(HeadPoseRequest()) + except Exception: + return "" + lines = [] + for item in state.objects: + dx = item.position.x - frame.origin.x + dy = item.position.y - frame.origin.y + dz = item.position.z - frame.origin.z + right = dx * frame.right.x + dy * frame.right.y + dz * frame.right.z + ahead = dx * frame.forward.x + dy * frame.forward.y + dz * frame.forward.z + lines.append(f" {item.id}: {right:+.2f} right, {ahead:+.2f} ahead, {dy:+.2f} up") + return "\n".join(lines) + + @staticmethod + def changes(before: SceneState, after: SceneState) -> str: + """Summarize what actually changed between two snapshots; empty if nothing.""" + old = {item.id: item for item in before.objects} + new = {item.id: item for item in after.objects} + lines = [] + for object_id in sorted(set(new) - set(old)): + lines.append(f"added {object_id}") + for object_id in sorted(set(old) - set(new)): + lines.append(f"removed {object_id}") + for object_id in sorted(set(old) & set(new)): + if old[object_id] != new[object_id]: + was, now = old[object_id], new[object_id] + details = [] + if was.position != now.position: + details.append( + f"position ({was.position.x}, {was.position.y}, {was.position.z})" + f" -> ({now.position.x}, {now.position.y}, {now.position.z})" + ) + if was.color != now.color: + details.append("color changed") + if was.size != now.size: + details.append(f"size {was.size} -> {now.size}") + if was.type != now.type: + details.append(f"type {was.type} -> {now.type}") + lines.append(f"changed {object_id}: " + ", ".join(details)) + return "; ".join(lines) + + @staticmethod + def positions(state: SceneState) -> dict[str, tuple[float, float, float]]: + return {item.id: (item.position.x, item.position.y, item.position.z) for item in state.objects} + + async def record_moves(self, participant_id: str, before: SceneState) -> None: + """Record positions changed during one turn so a later undo can restore them.""" + before_positions = self.positions(before) + after_positions = self.positions(await self.snapshot()) + moves = [ + f"{object_id}: previously at {before_positions[object_id]}, now at {position}" + for object_id, position in after_positions.items() + if object_id in before_positions and before_positions[object_id] != position + ] + # A turn that moved nothing keeps the previous turn's undo data + # alive for one more turn; older records would restore stale + # positions on a much later "put it back". + if moves: + self._recent_moves[participant_id] = moves + self._recent_moves_age[participant_id] = 0 + elif participant_id in self._recent_moves: + self._recent_moves_age[participant_id] = self._recent_moves_age.get(participant_id, 0) + 1 + if self._recent_moves_age[participant_id] > 1: + del self._recent_moves[participant_id] + del self._recent_moves_age[participant_id] + + +__all__ = ["SceneContext"] diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/spatial_ops.py b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/spatial_ops.py new file mode 100644 index 00000000..c99bf6b1 --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/spatial_ops.py @@ -0,0 +1,774 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Code-composed movement and creation operations for the render subagents. + +Every operation takes only semantic arguments (object ids, direction names, +metres) and resolves the user frame, anchor coordinates, geometry, and the +scene write itself by composing the leaf NAT functions. No pose or coordinate +ever round-trips through the agent LLM, which cannot relay them reliably. +""" + +import asyncio +import difflib +import re +from typing import Annotated, Literal + +from loguru import logger +from nat.plugin_api import ( + Builder, + Function, + FunctionGroup, + FunctionGroupBaseConfig, + FunctionGroupRef, + register_function_group, +) +from pydantic import BaseModel, ConfigDict, Field +from xr_render_scene import EmptyRequest, SceneObject + +_UserDirection = Literal["front", "back", "left", "right", "above", "below"] +_AnchorRelation = Literal["toward_user", "away_from_user", "left_of", "right_of", "above", "below"] + +_ObjectWords = Annotated[ + str, + Field(description="The instruction's exact words for this object, copied verbatim (mangled nouns fine); " + "an id only when the instruction itself states that id."), +] + +_AnchorWords = Annotated[ + str, + Field(description="The instruction's exact words for the anchor object, copied verbatim; " + "never an id chosen from the scene, unless the instruction itself states the id."), +] + +_ColorWords = Annotated[ + str, + Field(default="", + description="The instruction's exact color word(s), copied verbatim (mangled spellings fine), " + "or an object to copy the color from ('same as cone-7'); empty when no color is stated."), +] + +_ShapeWords = Annotated[ + str, + Field(description="The instruction's exact shape word, copied verbatim (mangled spellings fine)."), +] + +_DEFAULT_COLOR = (0.2, 0.9, 1.0) + +_COLOR_WORDS = { + "red": (1, 0, 0), "green": (0, 0.8, 0), "blue": (0, 0.4, 1), "yellow": (1, 1, 0), + "cyan": (0, 1, 1), "magenta": (1, 0, 1), "orange": (1, 0.5, 0), "purple": (0.6, 0, 1), + "white": (1, 1, 1), "black": (0, 0, 0), "teal": (0, 0.8, 0.8), "turquoise": (0.2, 0.9, 1), + "lavender": (0.6, 0.4, 1), "pink": (1, 0.5, 0.8), "gray": (0.5, 0.5, 0.5), "grey": (0.5, 0.5, 0.5), +} +_SHAPE_WORDS = { + "box": "box", "cube": "box", "block": "box", "crate": "box", + "sphere": "sphere", "ball": "sphere", "orb": "sphere", + "cone": "cone", "cylinder": "cylinder", "capsule": "capsule", + "ring": "ring", "pyramid": "pyramid", "torus": "torus", "donut": "torus", +} + + +class MovedObject(BaseModel): + """Final position applied to one existing object.""" + + obj_id: str + x: float + y: float + z: float + + +class SwappedObjects(BaseModel): + """Final positions applied by one swap.""" + + first: MovedObject + second: MovedObject + + +class CreatedObject(BaseModel): + """Stable id and position of one newly created object.""" + + id: str + x: float + y: float + z: float + created_this_turn: int = Field(default=1, description="Objects created so far in this turn, including this one.") + + +class RemovedObject(BaseModel): + """Stable id of one removed object.""" + + obj_id: str + + +class CreationLedger: + """Suppress identical repeated mutations within one subagent turn.""" + + def __init__(self) -> None: + self._seen: dict[tuple, CreatedObject] = {} + self.mutations: dict[tuple, MovedObject] = {} + self.count = 0 + + def reset(self) -> None: + self._seen.clear() + self.mutations.clear() + self.count = 0 + + def get(self, key: tuple) -> CreatedObject | None: + return self._seen.get(key) + + def record(self, key: tuple, created: CreatedObject) -> None: + self._seen[key] = created + + +class TurnGuard: + """Block mutations of existing objects after a failed reference lookup. + + A model that cannot resolve a reference substitutes a plausible scene id + on retry; once any lookup fails within a delegation, moves and edits must + stop and the failure must travel back to the user. Creations stay allowed + so a failed anchor can still degrade into a bare create. + """ + + def __init__(self) -> None: + self.halted = False + + def reset(self) -> None: + self.halted = False + + +class _Leaves: + """Resolution helpers over the leaf NAT functions.""" + + def __init__( + self, + functions: dict[str, Function], + ledger: CreationLedger | None = None, + guard: TurnGuard | None = None, + ) -> None: + self._functions = functions + self.ledger = ledger + self.guard = guard + self._add_lock = asyncio.Lock() + + async def update(self, arguments: dict) -> None: + await self._functions["scene_updates__update_primitive"].ainvoke(arguments) + + async def remove(self, object_id: str) -> None: + await self._functions["scene_objects__remove_primitive"].ainvoke({"obj_id": object_id}) + + def check_writable(self) -> None: + if self.guard is not None and self.guard.halted: + raise ValueError( + "An earlier object reference in this instruction could not be resolved; " + "change nothing else and report that failure back." + ) + + async def user_frame(self): + return await self._functions["tracking__get_user_frame"].ainvoke({}) + + async def find(self, object_ref: str) -> SceneObject: + # Models sometimes emit ids with unicode dashes or stray whitespace. + wanted = "".join("-" if character in "\u2010\u2011\u2012\u2013\u2014\u2212" else character + for character in object_ref).strip().lower() + wanted = re.sub(r"[\s_]+", "-", wanted) if re.fullmatch(r"[A-Za-z]+[\s_-]+\d+", wanted) else wanted + state = await self._functions["scene_state__get_scene_state"].ainvoke(EmptyRequest()) + for item in state.objects: + if item.id == wanted: + return item + known = sorted(item.id for item in state.objects) + # An id-shaped miss is a typo, never a description; resolving it as + # one would silently pick a sibling of the mistyped id. Models do + # swap shape synonyms into id prefixes ("cube-39" for box-39), and + # that mapping is exact, so it resolves. + if re.fullmatch(r"[a-z]+-\d+", wanted): + prefix, _, number = wanted.partition("-") + synonym_id = f"{_SHAPE_WORDS.get(prefix, prefix)}-{number}" + for item in state.objects: + if item.id == synonym_id: + logger.debug("spatial op resolved {!r} -> {}", object_ref, item.id) + return item + logger.debug("spatial op lookup failed: {!r} not in {}", object_ref, known) + raise ValueError(f"No scene object with id {object_ref!r}; the scene has {known}") + # Not an id: read it as the instruction's own words, possibly + # mangled by speech transcription. Exact shape and color words + # resolve first; the fuzzy shape match runs only when they yield + # nothing, so a phrase carrying a color ("the red one") never trips + # over everyday words that sound like shapes. + words = re.findall(r"[a-z]+", wanted) + exact_shape = next((_SHAPE_WORDS[word] for word in words if word in _SHAPE_WORDS), None) + color = next((_COLOR_WORDS[word] for word in words if word in _COLOR_WORDS), None) + + def select(shape: str | None) -> list[SceneObject]: + pool = [item for item in state.objects if shape is None or item.type == shape] + if color is None or not pool: + return pool if shape is not None or color is not None else [] + def color_distance(item: SceneObject) -> float: + return ((item.color.r - color[0]) ** 2 + (item.color.g - color[1]) ** 2 + + (item.color.b - color[2]) ** 2) + best = min(color_distance(item) for item in pool) + # A stated color that matches nothing must report back, never + # silently pick the least-wrong object (squared-RGB threshold). + if best > 0.4: + return [] + return [item for item in pool if color_distance(item) - best < 0.05] + + candidates = select(exact_shape) if (exact_shape or color) else [] + if len(candidates) != 1 and exact_shape is None: + for word in words: + if word in _COLOR_WORDS: + continue + close = difflib.get_close_matches(word, _SHAPE_WORDS, n=1, cutoff=0.6) + if close: + fuzzy = select(_SHAPE_WORDS[close[0]]) + if fuzzy and (not candidates or len(fuzzy) < len(candidates)): + candidates = fuzzy + break + if len(candidates) == 1: + logger.debug("spatial op resolved {!r} -> {}", object_ref, candidates[0].id) + return candidates[0] + if candidates: + if self.guard is not None: + self.guard.halted = True + matches = ", ".join(item.id for item in candidates) + raise ValueError(f"{object_ref!r} is ambiguous: it matches {matches}; ask which one is meant") + logger.debug("spatial op lookup failed: {!r} not in {}", object_ref, known) + if self.guard is not None: + self.guard.halted = True + raise ValueError( + f"No scene object matches {object_ref!r}: nothing in the scene has that description. " + "Never substitute a different object; make no further tool call and report this back." + ) + + def shape(self, shape_words: str) -> str: + words = re.findall(r"[a-z]+", shape_words.lower()) + for word in words: + if word in _SHAPE_WORDS: + return _SHAPE_WORDS[word] + for word in words: + close = difflib.get_close_matches(word, _SHAPE_WORDS, n=1, cutoff=0.6) + if close: + logger.debug("shape words resolved {!r} -> {}", shape_words, _SHAPE_WORDS[close[0]]) + return _SHAPE_WORDS[close[0]] + shapes = ", ".join(sorted(set(_SHAPE_WORDS.values()))) + raise ValueError(f"Unknown shape {shape_words!r}; the renderer draws: {shapes}") + + async def color(self, color_words: str) -> tuple[float, float, float]: + # A vision fact arrives as a numeric triple ("RGB (1.0, 0.5, 0.0)"). + # Ids also carry digits ("same as capsule-0"), so the numeric read + # only applies when no id-shaped token is present. + if not re.search(r"[a-z]+-\d+", color_words.lower()): + numbers = [float(value) for value in re.findall(r"-?\d*\.\d+|-?\d+", color_words)] + if len(numbers) == 3 and all(0.0 <= value <= 1.0 for value in numbers): + return (numbers[0], numbers[1], numbers[2]) + words = re.findall(r"[a-z]+", color_words.lower()) + if not words: + return _DEFAULT_COLOR + for word in words: + if word in _COLOR_WORDS: + return _COLOR_WORDS[word] + # The object-copy lookup is speculative, so a miss must not trip + # the turn guard, and it runs before the fuzzy match because + # everyday words sit within difflib range of color names + # ("cone" -> "orange"). + halted = self.guard.halted if self.guard is not None else False + try: + source = await self.find(color_words) + return (source.color.r, source.color.g, source.color.b) + except Exception: + if self.guard is not None: + self.guard.halted = halted + # 0.75 admits transcription slips ("blew") while rejecting unrelated + # words that drift within 0.6 of a color name. + for word in words: + close = difflib.get_close_matches(word, _COLOR_WORDS, n=1, cutoff=0.75) + if close: + logger.debug("color words resolved {!r} -> {}", color_words, close[0]) + return _COLOR_WORDS[close[0]] + known = ", ".join(sorted(_COLOR_WORDS)) + raise ValueError(f"Unknown color {color_words!r}; use one of {known}, or name a scene object") + + async def spot(self, operation: str, arguments: dict) -> tuple[float, float, float]: + result = await self._functions[f"spatial__{operation}"].ainvoke(arguments) + return result.x, result.y, result.z + + async def write(self, object_id: str, position: tuple[float, float, float]) -> MovedObject: + self.check_writable() + x, y, z = position + await self._functions["scene_updates__update_primitive"].ainvoke({"obj_id": object_id, "x": x, "y": y, "z": z}) + return MovedObject(obj_id=object_id, x=x, y=y, z=z) + + async def add( + self, + prim_type: str, + position: tuple[float, float, float], + color: tuple[float, float, float], + size: float, + ) -> CreatedObject: + x, y, z = position + r, g, b = color + key = (prim_type, round(x, 2), round(y, 2), round(z, 2), round(r, 2), round(g, 2), round(b, 2), round(size, 3)) + # The lock covers the memo's check-then-act: identical creates + # gathered in one model response must still dedupe. + async with self._add_lock: + if self.ledger is not None and (existing := self.ledger.get(key)) is not None: + return existing + result = await self._functions["scene_objects__add_primitive"].ainvoke( + {"prim_type": prim_type, "x": x, "y": y, "z": z, "r": r, "g": g, "b": b, "size": size} + ) + created = CreatedObject(id=result.id, x=x, y=y, z=z) + if self.ledger is not None: + self.ledger.count += 1 + created.created_this_turn = self.ledger.count + self.ledger.record(key, created) + return created + + +async def _leaf_functions(builder: Builder, refs: tuple[FunctionGroupRef, ...]) -> dict[str, Function]: + functions: dict[str, Function] = {} + for ref in refs: + group = await builder.get_function_group(ref) + functions.update(await group.get_all_functions()) + return functions + + +class PlacementOpsConfig(FunctionGroupBaseConfig, name="xr_render_placement_ops"): + """Configure the composed movement operations.""" + + model_config = ConfigDict(arbitrary_types_allowed=True) + + guard: TurnGuard | None = Field(default=None, exclude=True, repr=False) + scene_state: FunctionGroupRef = FunctionGroupRef("scene_state") + scene_updates: FunctionGroupRef = FunctionGroupRef("scene_updates") + tracking: FunctionGroupRef = FunctionGroupRef("tracking") + spatial: FunctionGroupRef = FunctionGroupRef("spatial") + + +@register_function_group(config_type=PlacementOpsConfig) +async def placement_ops(config: PlacementOpsConfig, builder: Builder): + """Build movement operations that resolve geometry and write in code.""" + leaves = _Leaves( + await _leaf_functions( + builder, + (config.scene_state, config.scene_updates, config.tracking, config.spatial), + ), + guard=config.guard, + ) + group = FunctionGroup(config=config) + + async def move_user_relative( + object_words: _ObjectWords, + direction: _UserDirection, + distance: Annotated[ + float, Field(description="Distance from the user in metres; pass a stated distance exactly.") + ] = 1.5, + ) -> MovedObject: + target = await leaves.find(object_words) + frame = await leaves.user_frame() + spot = await leaves.spot( + "compute_user_relative_position", + {"user_frame": frame.model_dump(), "direction_from_user": direction, "distance_meters": distance}, + ) + return await leaves.write(target.id, spot) + + group.add_function( + "move_user_relative", + move_user_relative, + description=( + "Move an existing object to a point in a named direction from the user. Not for stated shifts " + "like 'one metre to my left'; nudge does those." + ), + ) + + async def nudge( + object_words: _ObjectWords, + forward: Annotated[float, Field(description="Signed user-forward shift in metres.")] = 0.0, + right: Annotated[float, Field(description="Signed user-right shift in metres.")] = 0.0, + up: Annotated[float, Field(description="Signed world-up shift in metres.")] = 0.0, + ) -> MovedObject: + current = await leaves.find(object_words) + frame = await leaves.user_frame() + spot = await leaves.spot( + "offset_position_in_user_frame", + { + "user_frame": frame.model_dump(), + "start_position": current.position.model_dump(), + "forward_meters": forward, + "right_meters": right, + "up_meters": up, + }, + ) + return await leaves.write(current.id, spot) + + group.add_function( + "nudge", + nudge, + description="Shift an existing object from its current position by signed user-frame offsets.", + ) + + async def move_object_relative( + movee_words: _ObjectWords, + anchor_words: _AnchorWords, + relation: _AnchorRelation, + distance: Annotated[ + float, Field(description="Distance from the anchor in metres; pass a stated distance exactly.") + ] = 0.3, + ) -> MovedObject: + logger.debug("move_object_relative movee={!r} anchor={!r} relation={} distance={}", + movee_words, anchor_words, relation, distance) + movee = await leaves.find(movee_words) + anchor = await leaves.find(anchor_words) + if movee.id == anchor.id: + raise ValueError( + f"{movee_words!r} and {anchor_words!r} are the same object ({movee.id}); an object cannot " + "be placed relative to itself. Make no other tool call and report the problem back." + ) + frame = await leaves.user_frame() + spot = await leaves.spot( + "compute_position_relative_to_anchor", + { + "user_frame": frame.model_dump(), + "anchor_position": anchor.position.model_dump(), + "relation_to_anchor": relation, + "distance_meters": distance, + }, + ) + return await leaves.write(movee.id, spot) + + group.add_function( + "move_object_relative", + move_object_relative, + description="Move an existing object to a point in a named relation to an anchor object.", + ) + + async def move_inside(movee_words: _ObjectWords, container_words: _ObjectWords) -> MovedObject: + movee = await leaves.find(movee_words) + container = await leaves.find(container_words) + return await leaves.write( + movee.id, + (container.position.x, container.position.y, container.position.z), + ) + + group.add_function( + "move_inside", + move_inside, + description="Move an existing object into the center of a container object.", + ) + + async def move_between( + movee_words: _ObjectWords, + first_anchor_words: _ObjectWords, + second_anchor_words: _ObjectWords, + ) -> MovedObject: + movee = await leaves.find(movee_words) + anchor_a = await leaves.find(first_anchor_words) + anchor_b = await leaves.find(second_anchor_words) + spot = await leaves.spot( + "compute_midpoint", + {"first_position": anchor_a.position.model_dump(), "second_position": anchor_b.position.model_dump()}, + ) + return await leaves.write(movee.id, spot) + + group.add_function( + "move_between", + move_between, + description="Move an existing object to the midpoint between two anchor objects.", + ) + + async def move_toward( + movee_words: _ObjectWords, + target_words: _ObjectWords, + direction: Annotated[ + Literal["toward", "away"], Field(description="Move toward or away from the target.") + ] = "toward", + distance: Annotated[float, Field(ge=0, description="Non-negative travel distance in metres.")] = 0.5, + ) -> MovedObject: + movee = await leaves.find(movee_words) + target = await leaves.find(target_words) + spot = await leaves.spot( + "compute_position_toward_or_away_from_reference", + { + "start_position": movee.position.model_dump(), + "reference_position": target.position.model_dump(), + "movement_direction": direction, + "distance_meters": distance, + }, + ) + return await leaves.write(movee.id, spot) + + group.add_function( + "move_toward", + move_toward, + description="Move an existing object toward or away from another object.", + ) + + async def move_toward_user( + movee_words: _ObjectWords, + direction: Annotated[ + Literal["toward", "away"], Field(description="Move toward or away from the user.") + ] = "toward", + distance: Annotated[float, Field(ge=0, description="Non-negative travel distance in metres.")] = 0.5, + ) -> MovedObject: + movee = await leaves.find(movee_words) + frame = await leaves.user_frame() + spot = await leaves.spot( + "compute_position_toward_or_away_from_reference", + { + "start_position": movee.position.model_dump(), + "reference_position": frame.origin.model_dump(), + "movement_direction": direction, + "distance_meters": distance, + }, + ) + return await leaves.write(movee.id, spot) + + group.add_function( + "move_toward_user", + move_toward_user, + description="Move an existing object toward or away from the user.", + ) + + async def swap_positions(first_words: _ObjectWords, second_words: _ObjectWords) -> SwappedObjects: + first = await leaves.find(first_words) + second = await leaves.find(second_words) + first_position = (first.position.x, first.position.y, first.position.z) + second_position = (second.position.x, second.position.y, second.position.z) + return SwappedObjects( + first=await leaves.write(first.id, second_position), + second=await leaves.write(second.id, first_position), + ) + + group.add_function( + "swap_positions", + swap_positions, + description="Exchange the positions of two existing objects.", + ) + + async def move_to(object_words: _ObjectWords, x: float, y: float, z: float) -> MovedObject: + target = await leaves.find(object_words) + return await leaves.write(target.id, (x, y, z)) + + group.add_function( + "move_to", + move_to, + description=( + "Move an existing object to explicit world coordinates taken from the request, SCENE OBJECTS, or " + "[Recent moves]; never invent coordinates." + ), + ) + + yield group + + +class RecoloredObject(BaseModel): + """Final color applied to one existing object.""" + + obj_id: str + r: float + g: float + b: float + + +class AppearanceOpsConfig(FunctionGroupBaseConfig, name="xr_render_appearance_ops"): + """Configure the composed recolor operation.""" + + model_config = ConfigDict(arbitrary_types_allowed=True) + + guard: TurnGuard | None = Field(default=None, exclude=True, repr=False) + scene_state: FunctionGroupRef = FunctionGroupRef("scene_state") + scene_updates: FunctionGroupRef = FunctionGroupRef("scene_updates") + + +@register_function_group(config_type=AppearanceOpsConfig) +async def appearance_ops(config: AppearanceOpsConfig, builder: Builder): + """Build the recolor operation that resolves references and colors in code.""" + leaves = _Leaves( + await _leaf_functions(builder, (config.scene_state, config.scene_updates)), + guard=config.guard, + ) + group = FunctionGroup(config=config) + + async def recolor( + object_words: _ObjectWords, + color_words: Annotated[ + str, + Field(description="The instruction's exact color word(s), copied verbatim (mangled " + "spellings fine), or an object to copy the color from ('same as cone-7')."), + ], + ) -> RecoloredObject: + leaves.check_writable() + target = await leaves.find(object_words) + r, g, b = await leaves.color(color_words) + await leaves.update({"obj_id": target.id, "r": r, "g": g, "b": b}) + return RecoloredObject(obj_id=target.id, r=r, g=g, b=b) + + group.add_function( + "recolor", + recolor, + description="Change an existing object's color, keeping position, type, and size.", + ) + + yield group + + +class ObjectOpsConfig(FunctionGroupBaseConfig, name="xr_render_object_ops"): + """Configure the composed creation and reshape operations.""" + + model_config = ConfigDict(arbitrary_types_allowed=True) + + ledger: CreationLedger | None = Field(default=None, exclude=True, repr=False) + guard: TurnGuard | None = Field(default=None, exclude=True, repr=False) + scene_state: FunctionGroupRef = FunctionGroupRef("scene_state") + scene_updates: FunctionGroupRef = FunctionGroupRef("scene_updates") + scene_objects: FunctionGroupRef = FunctionGroupRef("scene_objects") + tracking: FunctionGroupRef = FunctionGroupRef("tracking") + spatial: FunctionGroupRef = FunctionGroupRef("spatial") + + +@register_function_group(config_type=ObjectOpsConfig) +async def object_ops(config: ObjectOpsConfig, builder: Builder): + """Build creation and reshape operations that resolve geometry in code.""" + leaves = _Leaves( + await _leaf_functions( + builder, + (config.scene_state, config.scene_updates, config.scene_objects, config.tracking, config.spatial), + ), + ledger=config.ledger, + guard=config.guard, + ) + group = FunctionGroup(config=config) + + async def create_user_relative( + prim_type: _ShapeWords, + direction: _UserDirection, + color_words: _ColorWords = "", + distance: Annotated[ + float, Field(description="Distance from the user in metres; pass a stated distance exactly.") + ] = 1.5, + size: Annotated[float, Field(description="Sphere radius or box half-edge in metres.")] = 0.1, + ) -> CreatedObject: + prim = leaves.shape(prim_type) + color = await leaves.color(color_words) + frame = await leaves.user_frame() + spot = await leaves.spot( + "compute_user_relative_position", + {"user_frame": frame.model_dump(), "direction_from_user": direction, "distance_meters": distance}, + ) + return await leaves.add(prim, spot, color, size) + + group.add_function( + "create_user_relative", + create_user_relative, + description="Create a new object at a point in a named direction from the user.", + ) + + async def create_object_relative( + prim_type: _ShapeWords, + anchor_words: _AnchorWords, + relation: _AnchorRelation, + color_words: _ColorWords = "", + distance: Annotated[ + float, Field(description="Distance from the anchor in metres; pass a stated distance exactly.") + ] = 0.3, + size: Annotated[float, Field(description="Sphere radius or box half-edge in metres.")] = 0.1, + ) -> CreatedObject: + logger.debug("create_object_relative anchor={!r} relation={} distance={}", anchor_words, relation, distance) + prim = leaves.shape(prim_type) + color = await leaves.color(color_words) + try: + anchor = await leaves.find(anchor_words) + except ValueError as error: + raise ValueError( + f"{error}. If the instruction names no existing object to anchor on, this is a bare " + "creation: call create_user_relative with direction front and distance 1.5 instead." + ) from None + frame = await leaves.user_frame() + spot = await leaves.spot( + "compute_position_relative_to_anchor", + { + "user_frame": frame.model_dump(), + "anchor_position": anchor.position.model_dump(), + "relation_to_anchor": relation, + "distance_meters": distance, + }, + ) + return await leaves.add(prim, spot, color, size) + + group.add_function( + "create_object_relative", + create_object_relative, + description="Create a new object at a point in a named relation to an anchor object.", + ) + + async def create_at( + prim_type: _ShapeWords, + x: float, + y: float, + z: float, + color_words: _ColorWords = "", + size: Annotated[float, Field(description="Sphere radius or box half-edge in metres.")] = 0.1, + ) -> CreatedObject: + logger.debug("create_at ({}, {}, {})", x, y, z) + prim = leaves.shape(prim_type) + color = await leaves.color(color_words) + return await leaves.add(prim, (x, y, z), color, size) + + group.add_function( + "create_at", + create_at, + description="Create a new object at explicit world coordinates.", + ) + + async def change_shape(object_words: _ObjectWords, prim_type: _ShapeWords) -> MovedObject: + leaves.check_writable() + prim = leaves.shape(prim_type) + current = await leaves.find(object_words) + await leaves.update({"obj_id": current.id, "prim_type": prim}) + return MovedObject(obj_id=current.id, x=current.position.x, y=current.position.y, z=current.position.z) + + group.add_function( + "change_shape", + change_shape, + description="Change an existing object into another primitive type, keeping position, color, and size.", + ) + + async def resize_object( + object_words: _ObjectWords, + factor: Annotated[float, Field(description="Multiplier applied to the current size.")], + ) -> MovedObject: + leaves.check_writable() + current = await leaves.find(object_words) + # Model retries re-apply multiplicative resizes; one factor per + # object per turn. + key = ("resize", current.id, round(factor, 4)) + async with leaves._add_lock: + if leaves.ledger is not None and (done := leaves.ledger.mutations.get(key)) is not None: + return done + await leaves.update({"obj_id": current.id, "size": round(current.size * factor, 4)}) + result = MovedObject(obj_id=current.id, x=current.position.x, y=current.position.y, z=current.position.z) + if leaves.ledger is not None: + leaves.ledger.mutations[key] = result + return result + + group.add_function( + "resize_object", + resize_object, + description="Multiply an existing object's size by a factor, keeping everything else.", + ) + + async def remove_object(object_words: _ObjectWords) -> RemovedObject: + leaves.check_writable() + target = await leaves.find(object_words) + await leaves.remove(target.id) + return RemovedObject(obj_id=target.id) + + group.add_function( + "remove_object", + remove_object, + description="Remove an existing object from the scene.", + ) + + yield group + + +__all__ = ["AppearanceOpsConfig", "CreationLedger", "ObjectOpsConfig", "PlacementOpsConfig", "TurnGuard"] diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/supervisor.py b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/supervisor.py new file mode 100644 index 00000000..0f9135ea --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/supervisor.py @@ -0,0 +1,241 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Compose the sample's focused agent Functions into its top-level workflow.""" + +from pathlib import Path + +from loguru import logger +from nat.builder.function import LambdaFunction +from nat.plugin_api import Builder, Function, FunctionBaseConfig, FunctionInfo, FunctionRef, LLMRef +from nat.plugins.langchain.agent.tool_calling_agent.register import ToolCallAgentWorkflowConfig +from xr_ai_models import LLMService +from xr_ai_nat.functions.text_memory import RecallConversationRequest +from xr_ai_nat.llm import ModelsLLMConfig + +from .agents import ( + AppearanceAgentConfig, + MemoryAgentConfig, + ObjectAgentConfig, + PlacementAgentConfig, + VisionAgentConfig, +) +from .models import SceneReply, SceneRequest +from .scene import SceneContext +from .spatial_ops import CreationLedger + +_PROMPT = Path(__file__).with_name("supervisor_prompt.txt") +_LLM_NAME = LLMRef("scene_llm") + +# Words that cannot end a complete command; VAD truncation leaves them +# dangling ("Put the sphere on the"). The model reliably autocompletes such +# fragments from scene state, so they never reach it. +_DANGLING_WORDS = frozenset( + "a an the my your its of to on in at by and or with near under over " + "onto into between behind above below beside toward towards from".split() +) + + +def _is_truncated(transcript: str) -> bool: + words = transcript.strip().rstrip(".?!,;").lower().split() + return bool(words) and words[-1] in _DANGLING_WORDS + + +_ARTICLES = frozenset({"a", "an", "the", "my", "your", "its"}) + +_TRUNCATED_ASK = "I think I missed the end of that." + +_CANCEL_PHRASES = frozenset({ + "never mind", "nevermind", "forget it", "forget that", "cancel", "cancel that", "no", "stop", +}) + +_ACTION_VERBS = frozenset( + "put place move make create add remove delete drop turn rotate resize double halve shrink " + "grow recolor paint swap bring push pull raise lower undo change set scoot flip clear".split() +) + + +def _truncated_reply(transcript: str) -> str: + words = transcript.strip().rstrip(".?!,;").split() + tail = words[-1] + if tail.lower() in _ARTICLES and len(words) >= 2: + tail = f"{words[-2]} {tail}" + return f"{_TRUNCATED_ASK} {tail.capitalize()} what?" + + +def _splice_completion(prefix: str, completion: str) -> str: + """Join a cut-off request with its answer, merging overlapping words.""" + head = prefix.strip().rstrip(".?!,;") + tail_words = completion.strip().rstrip(".?!,;").split() + head_words = head.split() + for overlap in (3, 2, 1): + if ( + len(head_words) >= overlap + and len(tail_words) >= overlap + and [word.lower() for word in head_words[-overlap:]] + == [word.lower() for word in tail_words[:overlap]] + ): + tail_words = tail_words[overlap:] + break + return f"{head} {' '.join(tail_words)}".strip() + "." + + +def _resolve_truncation_reply(prefix: str, transcript: str) -> str | None: + """Decide what a turn following the truncation ask-back means. + + Returns the transcript to process, or None when the turn cancels the + cut-off request. A turn with its own action verb is a fresh command; a + bare fragment answers the ask and splices onto the cut-off prefix. + """ + words = transcript.strip().rstrip(".?!,;").lower() + if words in _CANCEL_PHRASES: + return None + if any(word in _ACTION_VERBS for word in words.split()): + return transcript + return _splice_completion(prefix, transcript) + + +class SceneSupervisorConfig(FunctionBaseConfig, name="xr_render_scene_supervisor"): + """Registry and tracing identity for the supervisor function.""" + + +async def scene_supervisor( + *, + builder: Builder, + llm: LLMService, + context: SceneContext | None = None, +) -> Function: + """Compose five subagents without knowing their transitive capabilities.""" + await builder.add_llm( + _LLM_NAME, + ModelsLLMConfig( + service=llm, + model_name="xr-scene-agent", + max_tokens=2048, + temperature=0.0, + recover_tool_calls=True, + ), + ) + if context is None: + scene_state = await builder.get_function_group("scene_state") + scene_functions = await scene_state.get_all_functions() + tracking = await builder.get_function_group("tracking") + tracking_functions = await tracking.get_all_functions() + context = SceneContext( + scene_functions["scene_state__get_scene_state"], + tracking_functions["tracking__get_user_frame"], + ) + conversations = await builder.get_function_group("conversations") + conversation_functions = await conversations.get_all_functions() + recall = conversation_functions["conversations__recall_conversation"] + + async def recent_conversation(participant_id: str) -> tuple[str, str]: + """Return the conversation block plus any pending cut-off request.""" + recalled = await recall.ainvoke(RecallConversationRequest(participant_id=participant_id)) + entries = recalled.entries[-8:] + if not entries: + return "", "" + pending = "" + if ( + len(entries) >= 2 + and entries[-1].role == "agent" + and entries[-1].text.startswith(_TRUNCATED_ASK) + and entries[-2].role == "user" + ): + pending = entries[-2].text + lines = [f" {'User' if entry.role == 'user' else 'Agent'}: {entry.text}" for entry in entries] + block = ( + "[Recent conversation] (already handled; never a source of new work)\n" + + "\n".join(lines) + "\n\n" + ) + return block, pending + + ledger = CreationLedger() + subagents = ( + ("placement_agent", PlacementAgentConfig(context=context)), + ("appearance_agent", AppearanceAgentConfig(context=context)), + ("object_agent", ObjectAgentConfig(context=context, ledger=ledger)), + ("vision_agent", VisionAgentConfig(context=context)), + ("memory_agent", MemoryAgentConfig()), + ) + for name, config in subagents: + await builder.add_function(name, config) + + reasoning = await builder.add_function( + "supervisor_reasoning", + ToolCallAgentWorkflowConfig( + llm_name=_LLM_NAME, + tool_names=[FunctionRef(name) for name, _config in subagents], + system_prompt=_PROMPT.read_text(encoding="utf-8").strip(), + handle_tool_errors=True, + max_iterations=12, + max_empty_response_retries=1, + ), + ) + + async def supervise(request: SceneRequest) -> SceneReply: + if _is_truncated(request.transcript): + return SceneReply(response=_truncated_reply(request.transcript)) + conversation, pending_truncation = await recent_conversation(request.participant_id) + transcript = request.transcript + if pending_truncation: + resolved = _resolve_truncation_reply(pending_truncation, transcript) + if resolved is None: + return SceneReply(response="Okay, never mind that.") + transcript = resolved + ledger.reset() + context.take_mutating(request.participant_id) + context.take_delegated(request.participant_id) + before = await context.snapshot() + message = ( + f"Active participant: {request.participant_id}\n" + f"Utterance timestamp: {request.timestamp_us}\n" + f"{await context.describe(request.participant_id)}\n\n" + f"{conversation}" + f"User request: {transcript}" + ) + # A failed reasoning pass must degrade, not raise: the verification + # pass below then gets a chance to complete the turn. + try: + output = await reasoning.ainvoke(message, to_type=str) + except Exception as error: + logger.error("supervisor reasoning failed: {}", error) + output = "Something went wrong on my side; please say that again." + # Conversational turns (no delegation, reply is a question) skip + # the verification pass so chat stays single-pass. Feeding the diff + # back on turns that DID mutate makes the model re-delegate + # completed work with jittered arguments, defeating both scoring + # and the creation ledger. + # An ask-back with NO delegation behind it is genuine conversation; + # an ask-back after a delegation (e.g. vision degraded) is a turn + # that may still owe a mutation, so it keeps the rescue pass. + context.take_mutating(request.participant_id) + delegated_any = context.take_delegated(request.participant_id) + conversational = not delegated_any and str(output or "").rstrip().endswith("?") + if not conversational and not SceneContext.changes(before, await context.snapshot()): + verification = ( + f"{message}\n\n" + f"Your reply so far: {output}\n" + "Verified scene changes this turn: none. If the request needed a" + " scene change, delegate the remaining work now; if it needed" + " none, repeat your final answer." + ) + try: + output = await reasoning.ainvoke(verification, to_type=str) + except Exception as error: + logger.error("supervisor verification pass failed: {}", error) + output = "Something went wrong on my side; please say that again." + await context.record_moves(request.participant_id, before) + return SceneReply(response=str(output or "Done.")) + + return LambdaFunction.from_info( + config=SceneSupervisorConfig(), + info=FunctionInfo.from_fn( + supervise, + description="Coordinate focused XR subagents to satisfy one complete request.", + ), + instance_name="xr_scene_supervisor", + ) + + +__all__ = ["scene_supervisor"] diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/supervisor_prompt.txt b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/supervisor_prompt.txt new file mode 100644 index 00000000..79382719 --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/supervisor_prompt.txt @@ -0,0 +1,93 @@ +You are the supervisor for an XR scene assistant. Satisfy the complete user +request by invoking the focused subagents available as tools. You may call no +subagent for ordinary conversation, one subagent for a focused request, or +several subagents in sequence for a compound request. + +Delegate only self-contained tasks. Include the relevant stable XR object ids +when they are known, and copy the active participant id and utterance timestamp +into every subagent call. After each result, decide whether the request is +complete or whether another focused task is required. Call at most one subagent +in each model response and wait for its result before selecting the next. Never +ask one subagent to perform work owned by another. + +vision_agent answers questions about the PHYSICAL world only: what the user +holds, wears, points at, or what a real wall or room looks like. Every fact +about XR objects, positions, and empty space is already in SCENE OBJECTS, and +the subagents resolve user-relative placement ("ahead of me", "at my feet", +"on the left") through tracking. So for any create, move, recolor, resize, or +remove request, call vision_agent only when the request names a physical-world +fact the mutation depends on ("the color of my shirt"). Ask one precise +question, then pass the returned fact in the next subagent instruction. If +present-time vision is unavailable, never retry or substitute historical +video; when the request truly depends on the missing visual fact, tell the +user, and otherwise continue satisfying the request without vision. A mention +of a camera or an earlier perception attempt is +not a perception request. If an utterance is incomplete, self-corrected, +commenting on the interaction, or a fragment that does not itself request +anything ("Creative.", "Okay.", a cut-off phrase), reply conversationally or +ask one short question; never call a subagent, and never re-execute a request +from [Recent conversation] that the current utterance does not restate. + +memory_agent recalls conversation older than the [Recent conversation] block. +Use it when the request depends on a conversational fact absent from both the +current request and that block. + +appearance_agent changes only color. placement_agent moves objects that +already exist, including swaps, containment, stacking, and undo; those are +moves, never creation. object_agent creates new objects and also removes, +resizes, duplicates, or changes object shape. +Use multiple subagents when a request spans those responsibilities, even when a +single lower-level render function could technically modify several fields. + +Routing examples: +- "Creative." / "Okay." / a cut-off fragment after earlier requests: no + subagent call; reply with one short question like "What would you like me + to do?" even though [Recent conversation] shows completed work. +- "That's the wrong capsule" / "no, not that one", right after a change: + the intended target is unknown, so nothing can be fixed, created, or + removed yet; the whole turn is one short question: "Which capsule should + it sit above?" +- "Put/place the X in/on/inside the Y", X already in SCENE OBJECTS: a move, + placement_agent, never object_agent. +- "Add/create an X inside the Y": a creation, object_agent. +- "Put a lavender cone above the teal capsule" and SCENE OBJECTS in this + message lists no lavender cone: a creation with an anchor, object_agent + in one call. SCENE OBJECTS already states what exists and where; + vision_agent never answers either question. +- "Make a lavender cone two meters ahead of me" (or any position relative + to the user's body): object_agent directly in one call; never + vision_agent first, and never a vision check afterwards — tracking + already knows where the user stands and faces, camera or no camera. +- "Create a cone matching my jacket": the jacket is physical, so one + vision_agent question about the jacket's color, then object_agent with + the returned color. But an anchor that names an XR object, whatever its + color or shape: object_agent alone, never vision. +- "What was I holding a moment ago?" or any question about what the camera + saw at an earlier time: vision_agent once with the past-time question; it + consults the recorded video. +- "Remove/delete the X on the left": removal, object_agent; the side words + only select which X, never a move. +- "Double it / make the X half the size": resize, object_agent, and the + instruction carries the id resolved from SCENE OBJECTS or [Recent + conversation]: "Resize capsule-8 to half its current size". + +Phrase every creation instruction as: Create + (s) . +Use that exact phrasing whatever verb the user chose (add, make, spawn, put +up), and keep every requested copy in the one instruction: "Create three +teal cones in a row, no position stated", never one call per cone. Never +invent a position the user did not say. A task an agent reports back keeps +every word: re-delegate the same instruction to the right agent, never a +reduced one. Do not split initial placement into a later +placement_agent call. Use placement_agent only when an object that already +exists must move. + +Do not repeat a completed mutation, and never report a change you did not +delegate this turn: a request like "triple its size" always requires a +subagent call now, even when [Recent conversation] shows related earlier +work. When all required work is complete, respond +with one short plain-English sentence. That sentence is spoken aloud through +text-to-speech: never include numeric coordinates, object ids, or unit +symbols; describe positions in everyday words, like "just below the teal +capsule" or "in front of you". Never expose tool syntax, intermediate +instructions, JSON, or private reasoning. diff --git a/agent-samples/xr-render-demo/worker/xr_render_demo_worker/xr_session.py b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/xr_session.py new file mode 100644 index 00000000..1052c9c3 --- /dev/null +++ b/agent-samples/xr-render-demo/worker/xr_render_demo_worker/xr_session.py @@ -0,0 +1,79 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Keep XR lifecycle control outside the model-facing scene workflow.""" + +from __future__ import annotations + +import asyncio +import time + +from loguru import logger +from nat.plugin_api import Function +from xr_ai_hub import DataMessage +from xr_ai_voice import TextMessageInput, VoiceSession +from xr_render_scene import EmptyRequest + +_START = "xr.session.started" +_READY = "render.ready" + + +def _now_us() -> int: + return time.time_ns() // 1_000 + + +class XRSessionController: + """Start LOVR on the client's session event and acknowledge readiness.""" + + def __init__( + self, + *, + session: VoiceSession, + start_xr: Function, + get_render_health: Function, + ) -> None: + self.transport = session.transport + self.start_xr = start_xr + self.get_render_health = get_render_health + self.started = False + self._start_lock = asyncio.Lock() + self.text = TextMessageInput(session=session, ignore_topics={_START}) + + def attach(self) -> None: + self.transport.endpoint.on_data(self._on_data) + + async def _on_data(self, message: DataMessage) -> None: + if message.topic != _START: + return + self.transport.set_target_participant(message.participant_id) + # Reconnects and multiple participants can race this event; only one + # spawn attempt may run. + async with self._start_lock: + if not self.started: + await self.start_xr.ainvoke(EmptyRequest()) + self.started = await self._wait_until_ready() + if self.started: + await self.transport.send_return_data( + DataMessage( + participant_id=message.participant_id, + topic=_READY, + pts_us=_now_us(), + data=b"", + ) + ) + return + logger.warning("XR session start failed; renderer never became ready") + + async def _wait_until_ready(self, timeout_s: float = 120.0) -> bool: + deadline = asyncio.get_running_loop().time() + timeout_s + while asyncio.get_running_loop().time() < deadline: + health = await self.get_render_health.ainvoke(EmptyRequest()) + if health.lovr_started: + return True + if health.spawn_error: + return False + await asyncio.sleep(0.5) + return False + + +__all__ = ["XRSessionController"] diff --git a/agent-samples/xr-render-demo/yaml/models.yaml b/agent-samples/xr-render-demo/yaml/models.yaml index b7ffd53e..022c0bbe 100644 --- a/agent-samples/xr-render-demo/yaml/models.yaml +++ b/agent-samples/xr-render-demo/yaml/models.yaml @@ -5,11 +5,6 @@ # Each entry maps a logical name to a preset + base_url. # Presets live in agent-sdk/xr-ai-models/xr_ai_models/presets/. -# llm and agent_llm intentionally share the Nemotron-3-Nano server. -llm: - kind: preset:nemotron3_nano - base_url: http://localhost:8107 - agent_llm: kind: preset:nemotron3_nano base_url: http://localhost:8107 diff --git a/agent-samples/xr-render-demo/yaml/openxr_service.yaml b/agent-samples/xr-render-demo/yaml/openxr_service.yaml index 67b208a9..326285d9 100644 --- a/agent-samples/xr-render-demo/yaml/openxr_service.yaml +++ b/agent-samples/xr-render-demo/yaml/openxr_service.yaml @@ -3,3 +3,8 @@ endpoint: tcp://0.0.0.0:8330 cloudxr_env_file: ~/.cloudxr/run/cloudxr.env + +# Test hook: the eval live drivers need a simulated head pose; flip to true +# while running them. The injected pose overrides real tracking for every +# client until cleared, so it must never ship enabled. +allow_sim_pose: false diff --git a/agent-samples/xr-render-demo/yaml/xr_render_demo_worker.yaml b/agent-samples/xr-render-demo/yaml/xr_render_demo_worker.yaml index 14deeae7..4ffc4896 100644 --- a/agent-samples/xr-render-demo/yaml/xr_render_demo_worker.yaml +++ b/agent-samples/xr-render-demo/yaml/xr_render_demo_worker.yaml @@ -46,5 +46,5 @@ silero_threshold: 0.3 # speech. DISABLED by default (0 = off) — a quiet session stays connected # indefinitely, which is what XR sessions usually want. Set a positive value # (e.g. 300 for 5 min) to opt in; the worker then tears the pipeline down -# after that long idle. Threaded to xr_ai_pipecat.make_voice_pipeline. +# after that long idle. Threaded to VoiceSession(idle_timeout_secs=...). idle_timeout_secs: 0 diff --git a/agent-sdk/xr-ai-hub-client/xr_ai_hub/_processor.py b/agent-sdk/xr-ai-hub-client/xr_ai_hub/_processor.py index daabe651..54ad1d25 100644 --- a/agent-sdk/xr-ai-hub-client/xr_ai_hub/_processor.py +++ b/agent-sdk/xr-ai-hub-client/xr_ai_hub/_processor.py @@ -275,6 +275,14 @@ async def send_return_data(self, msg: DataMessage) -> None: async def send_return_audio(self, chunk: AudioChunk) -> None: await self._push.send(encode(MsgType.RETURN_AUDIO, chunk)) + async def inject_participant_event(self, event: ParticipantEvent) -> None: + """Publish a synthetic participant event; a supported seam for test drivers.""" + await self._push.send(encode(MsgType.PARTICIPANT_EVENT, event)) + + async def inject_data(self, msg: DataMessage) -> None: + """Publish a synthetic upstream data message; a supported seam for test drivers.""" + await self._push.send(encode(MsgType.DATA_MESSAGE, msg)) + async def flush_return_audio(self, participant_id: str) -> None: """ Drop any return audio currently queued at the hub for *participant_id*. diff --git a/agent-sdk/xr-ai-nat/xr_ai_nat/llm/_langchain.py b/agent-sdk/xr-ai-nat/xr_ai_nat/llm/_langchain.py index a691964b..5e25d2b7 100644 --- a/agent-sdk/xr-ai-nat/xr_ai_nat/llm/_langchain.py +++ b/agent-sdk/xr-ai-nat/xr_ai_nat/llm/_langchain.py @@ -5,6 +5,7 @@ from __future__ import annotations +import itertools import json from collections.abc import Callable, Sequence from typing import Any, cast @@ -15,6 +16,7 @@ from langchain_core.outputs import ChatGeneration, ChatResult from langchain_core.runnables import Runnable from langchain_core.tools import BaseTool +from loguru import logger from pydantic import BaseModel, ConfigDict, Field from xr_ai_models import ChatMessage, LLMService, ToolCall, ToolDef @@ -69,6 +71,70 @@ def _tool(tool: BaseTool | dict[str, Any]) -> ToolDef: return ToolDef(name=tool.name, description=tool.description, parameters=parameters) +_RECOVERY_COUNTER = itertools.count() + + +def _content_tool_call(content: str, offered: dict[str, ToolDef]) -> dict[str, Any] | None: + """Recover a tool call the serving layer failed to parse out of content. + + Some models occasionally emit the call as bare JSON in the message body; + the server's tool-call parser then misses it and downstream agents treat + the turn as a final answer. Only a lone JSON object is recovered: one + naming an offered tool, or one of two or more keys that fit exactly one + offered tool's schema.""" + text = content.strip() + if not (text.startswith("{") and text.endswith("}")): + return None + try: + data = json.loads(text) + except json.JSONDecodeError: + return None + if not isinstance(data, dict): + return None + for wrapper_key in ("command", "function", "tool_call"): + wrapped = data.get(wrapper_key) + if isinstance(wrapped, dict) and (wrapped.get("name") or wrapped.get("tool") or wrapped.get("action")): + data = wrapped + break + name = data.get("name") or data.get("tool") or data.get("action") + if isinstance(name, str) and name not in offered: + # Models sometimes drop the group prefix from the tool name. + matches = [candidate for candidate in offered if candidate.endswith(f"__{name}")] + name = matches[0] if len(matches) == 1 else name + if isinstance(name, str) and name in offered: + arguments = data.get("arguments", data.get("args", data.get("parameters"))) + if arguments is None: + arguments = { + key: value for key, value in data.items() if key not in ("name", "tool", "action", "id", "type") + } + if isinstance(arguments, str): + try: + arguments = json.loads(arguments) + except json.JSONDecodeError: + return None + if not isinstance(arguments, dict): + return None + return {"name": name, "args": arguments, "id": _recovery_id(), "type": "tool_call"} + # A single-key object is as likely a data echo as a call (a bare + # argument echo would match any one-required-field tool, executing a + # narration as a mutation), so shape inference needs two or more keys. + if name is not None or len(data) < 2: + return None + candidates = [] + for tool in offered.values(): + properties = tool.parameters.get("properties", {}) + required = set(tool.parameters.get("required", ())) + if set(data) <= set(properties) and required <= set(data): + candidates.append(tool.name) + if len(candidates) != 1: + return None + return {"name": candidates[0], "args": dict(data), "id": _recovery_id(), "type": "tool_call"} + + +def _recovery_id() -> str: + return f"content-recovered-{next(_RECOVERY_COUNTER)}" + + class LangChainChatModel(BaseChatModel): """Present an ``LLMService`` through LangChain's async chat-model contract.""" @@ -79,6 +145,7 @@ class LangChainChatModel(BaseChatModel): max_tokens: int = 1024 enable_thinking: bool = False thinking_budget: int | None = None + recover_tool_calls: bool = False @property def _llm_type(self) -> str: @@ -121,11 +188,28 @@ async def _agenerate( except json.JSONDecodeError: arguments = {} tool_calls.append({"name": call.name, "args": arguments, "id": call.id, "type": "tool_call"}) + content = response.content + recovered_text = "" + if self.recover_tool_calls and not tool_calls and raw_tools: + offered = {definition.name: definition for definition in (_tool(tool) for tool in raw_tools)} + if (recovered := _content_tool_call(_text(content), offered)) is not None: + # The original text moves to additional_kwargs: left in + # content, the model imitates its own leaked JSON on later + # turns, and it becomes the final answer at the iteration + # cap; dropped entirely, a false-positive recovery is + # undiagnosable. + logger.warning("recovered tool call from content: {} ({!r})", recovered["name"], _text(content)[:200]) + recovered_text = _text(content) + tool_calls = [recovered] + content = "" + elif _text(content).lstrip().startswith("{"): + logger.debug("unrecovered JSON-like content: {!r}", _text(content)) usage = response.raw.get("usage", {}) if isinstance(response.raw, dict) else {} prompt_tokens = int(usage.get("prompt_tokens", 0)) completion_tokens = int(usage.get("completion_tokens", 0)) message = AIMessage( - content=response.content, + content=content, + additional_kwargs={"recovered_from_content": recovered_text} if recovered_text else {}, tool_calls=tool_calls, usage_metadata={ "input_tokens": prompt_tokens, diff --git a/agent-sdk/xr-ai-nat/xr_ai_nat/llm/config.py b/agent-sdk/xr-ai-nat/xr_ai_nat/llm/config.py index cdfc7075..bdb80aad 100644 --- a/agent-sdk/xr-ai-nat/xr_ai_nat/llm/config.py +++ b/agent-sdk/xr-ai-nat/xr_ai_nat/llm/config.py @@ -28,6 +28,10 @@ class ModelsLLMConfig(LLMBaseConfig, name="xr_ai_models"): description="Path to an xr-ai-models deployment profile.", ) role: str = Field(default="agent_llm", description="LLM role within the deployment profile.") + recover_tool_calls: bool = Field( + default=False, + description="Recover tool calls the serving layer leaked into message content.", + ) service: Any | None = Field( default=None, exclude=True, @@ -91,6 +95,7 @@ async def models_langchain_client(config: ModelsLLMConfig, _builder: Builder): max_tokens=config.max_tokens, enable_thinking=config.enable_thinking, thinking_budget=config.thinking_budget, + recover_tool_calls=config.recover_tool_calls, ) diff --git a/docs/changelog.md b/docs/changelog.md index 5ba3788e..63fbeb84 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -19,6 +19,22 @@ The endpoint stores each agent's current status and the pipeline re-announces that state periodically; a late or reconnecting client therefore converges without relying on a one-shot event. +### 2026-08-07 — xr-render-demo worker becomes a NAT supervisor over five subagents + +The sample's single-agent Pipecat worker is replaced by an `xr-ai-voice` +session driving a NAT supervisor that delegates to focused placement, object, +appearance, vision, and memory subagents. Everything the model resolved +unreliably moved into deterministic code in `spatial_ops.py`: object +references, color words, and shape words are quoted verbatim from the +instruction and resolved with fuzzy matching plus rejection thresholds, a +turn guard halts mutations after a failed lookup, and a turn-scoped ledger +dedupes retried creations. Two retired contracts from the old worker: +TTS-side leaked-JSON sanitization (superseded by opt-in tool-call recovery in +`xr_ai_nat.llm`, off by default) and model-visible `participant_id` schema +trimming (identity now travels in `SubagentTask`). The eval grew to four +tiers (routing, components, corpus + basics battery, live drivers) with a +startup audit enforcing prompt/case vocabulary separation. + ### 2026-08-05 — Docker vLLM setup owns the image entrypoint The shared vLLM Docker launcher explicitly selects `/bin/bash` before installing diff --git a/docs/source/reference/xr-render-demo.md b/docs/source/reference/xr-render-demo.md index fc2de8f7..9e55a309 100644 --- a/docs/source/reference/xr-render-demo.md +++ b/docs/source/reference/xr-render-demo.md @@ -79,35 +79,20 @@ The worker reads two YAML files: ### NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 — port 8107 -A vLLM `execvp` shim — a small Python wrapper that reads YAML configuration, +A vLLM `execvp` shim: a small Python wrapper that reads YAML configuration, sets `HF_HOME` and token environment variables, then `os.execvp`s into `vllm serve`. The Python process is replaced by vLLM; vLLM owns the HTTP API, weight loading, and tool calling from that point on. `vllm serve` with `--tool-call-parser qwen3_coder` and `--reasoning-parser nano_v3` (plugin auto-fetched from the model card into -`model_cache`). `enforce_eager` defaults to `true` — CUDA graph capture + +`model_cache`). `enforce_eager` defaults to `true`: CUDA graph capture + FlashInfer FP4 MoE autotune silently takes 3–8 minutes on cold start without it. Requires a Blackwell GPU (B200, RTX PRO 6000, or Jetson Thor) for native FP4; swap to the BF16 variant for Hopper or Ampere. -One server backs both logical models in `yaml/models.yaml`: `agent_llm` runs -the multi-step tool-calling loop, and `llm` serves two cheap, latency-sensitive -calls (thinking stays off): - -- **Quick-ack** — awaited before the agentic loop starts, the moment an - utterance lands. Returns `{"ack": "On it!", "think": false}` — a 3–6 word - spoken acknowledgment. Also classifies whether the request needs - open-ended reasoning (`think: true/false`): positional operations always - run without thinking because the math tools compute exact answers, so - thinking is reserved for vague corrections and free-form compositions - no tool pattern settles. Max 40 tokens, 8s timeout. The ack - is sent on the data channel (`agent.progress` topic) and spoken on every - turn so the user immediately knows they were heard. -- **Still-working messages** — if the agentic loop exceeds 5s, this model - generates a short contextual phrase like *"Still finding the right - position"* on a 10s repeat. Sent to the data channel only — never spoken, - to avoid stacking up in the TTS queue behind the real response. +The server backs the `agent_llm` role in `yaml/models.yaml`, which the +supervisor and all five subagents share as `scene_llm`. ## VLM — Cosmos-Reason1-7B @@ -127,161 +112,84 @@ OOM race. ## STT — parakeet-tdt-0.6b-v3 -Port 8103. NeMo ASR in-process. English-only, ~1.5 GB VRAM. - -``` -LiveKit mic (int16 PCM) → hub IPC (float32) → XRMediaHubTransport.input() - → SttProcessor - pre-roll buffer last 10 chunks (~320 ms) kept at all times; - prepended to the utterance buffer on speech onset - so the first word's attack isn't clipped - VAD Silero (ONNX, 512-sample / 32 ms windows, - probability threshold) via shared xr-ai-vad util - accumulates audio while speaking - finalizes when silence ≥ 0.8s AND speech ≥ 0.15s - OR max utterance length (30s) hit - filler filter drops single- and multi-word filler utterances - ("um", "uh", "yeah", "okay", "mm-hmm", etc.) - STT call POST multipart/form-data WAV → stt-server :8103 - → TranscriptionFrame pushed downstream -``` - -STT calls are serialized — an `stt_busy` flag prevents a new finalize while -one is in-flight. +Served on port 8103. The voice runtime's VAD segments utterances and sends +16 kHz int16 audio for transcription; transcripts feed the voice gate and +then the supervisor. ## TTS — Piper -Port 8105. `rhasspy/piper-voices` ONNX. Runs on CPU, ~100 ms per sentence. All -synthesis runs in a thread pool so the asyncio loop is never blocked. - -``` -TextFrame (from agentic loop final response or quick-ack) - → TtsProcessor - sentence-batched synthesis - POST text → tts-server :8105 → WAV bytes - RETURN_AUDIO IPC → hub → LiveKit → participant's headphones -``` - -`allow_interruptions=True` in the Pipecat pipeline. A new utterance while TTS -is playing triggers `ReturnAudioFlush` → hub clears the LiveKit audio queue -for that participant. - -## Pipecat pipeline - -``` -XRMediaHubTransport.input() - → SttProcessor (Silero VAD → utterance → parakeet STT - → TranscriptionFrame) - → RenderSceneProcessor (quick-ack + agentic loop → TextFrame) - → TtsProcessor (TextFrame → Piper TTS → return audio) - → XRMediaHubTransport.output() -``` - -## Agentic loop - -At worker startup, a NAT `WorkflowBuilder` constructs sample-local scene, -XR-tracking, spatial-math, vision, video-memory, and text-memory functions. -The LLM tool schemas are derived from those Functions. `start_xr` and -`get_health` remain worker-managed lifecycle operations. - -On each `TranscriptionFrame`: - -1. **Quick-ack** runs first (`llm` :8107, awaited before the loop). -2. **Still-working timer** starts (fires at 5s, repeats every 10s, data - channel only). -3. **Pre-fetch** (concurrent): `get_scene_state` + `get_head_pose` + - `position_ahead(1.5)` — results injected into the user message so the - model skips those tool calls and goes straight to the operation. -4. **Nemotron-30B :8107** runs with `tools=[…]`, up to 10 iterations: - - Model emits `tool_calls` → worker invokes the matching NAT Function → result appended - to conversation → next iteration. - - Runtime-backed Functions call scene, OpenXR, and video-memory typed - services. Spatial math and text memory execute in process, and vision - calls the configured `xr-ai-models` VLM service. - - Progress message sent on `agent.progress` topic before each tool - executes (data channel). - - If `think=true`: reasoning preamble injected into system prompt - (RESOLVE object → LOCATE coordinates → COMPUTE new position → - EXECUTE). The `` block stays private; only one short sentence - goes to the user. Token budget: 6144 total, 4096 thinking budget. - - If thinking fills the token budget without a tool call - (`finish_reason=length`): retry the same iteration with - `needs_thinking=False`. - - If the model outputs a bare tool name as text instead of a proper tool - call: worker synthesizes a no-arg tool call and continues. -5. **Final response** sent on `agent.response` topic and as a `TextFrame` - downstream to TTS. -6. **Turn appended** to a rolling 4-turn history buffer — injected as - context in future turns so the model understands "fix that", "undo", - "the one I just added". Final messages are also persisted through native - text memory without model scratch output or tool traces. - -## Native capability composition - -The sample-local scene process owns scene state and LOVR. `openxr-service` -owns the headless tracking session, and `video-memory-service` owns recorded -video decoding. `LiveFrameSource` supplies current-frame requests. NAT -Functions provide the typed tool surface over those services; the demo does -not launch or call MCP adapters. - -### Spatial tool surface - -The worker composes XR tracking with shared spatial-math Functions. This -offloads vector arithmetic the LLM is bad at while keeping pose-dependent math -in one place: - -- **Pose-aware named-direction helpers** take a `direction` enum (`front`, - `back`, `left`, `right`, `above`, `below`, plus `next_to` on - `place_object_relative`) and always-positive `distance`. The LLM never - applies signs to user-frame axes. - - `place_user_relative(direction, distance)`: user-anchored teleport - ("above my head", "to my left 1 m"). - - `place_object_relative(origin_x, origin_y, origin_z, direction, distance)`: - object-anchored teleport. `direction="front"` means *toward the user*; - `"back"` means *away*. Left/right/above/below map literally. - - `displace_object(current_x, current_y, current_z, right, up, forward)`: - user-frame signed-delta on an existing object. Multi-axis ("up and - to the left") in one call. - - `displace_objects(object_ids, current_xs, current_ys, current_zs, - right, up, forward)`: batch user-frame delta over N objects. Returns - `{"items": [{obj_id, x, y, z}, …]}` so the model fans out to N - `update_primitive` calls with one math call total. - - `place_inside_by_id(movee_id, container_x, container_y, container_z)`: - containment for "put X in Y". Argument names (`movee_id` paired - with `container_*`) force the model to pick the right noun's coords; - the return shape feeds straight into `update_primitive`. -- **Pure-math primitives** are pose-independent: - - `between_anchors(a_x, a_y, a_z, b_x, b_y, b_z)`: component-wise midpoint. - - `world_offset(origin_x, origin_y, origin_z, dx, dy, dz)`: - axis-aligned world-Y-up shift. - - `along_direction(origin_x, origin_y, origin_z, target_x, target_y, - target_z, distance)`: origin moved `distance` toward target. Used - for "closer to or further from ", which the user-frame - helpers can't model. - - `scale_value(current, factor)`: scalar multiplication for sizes. +Served on port 8105. The voice runtime streams the supervisor's final +reply to Piper and returns the audio to the participant. + +## Voice runtime + +The worker runs on `xr-ai-voice`'s `VoiceSession`: hub audio → VAD → STT → +voice gate → handler → streaming TTS → return audio. The scene supervisor is +attached with `as_voice_handler`, and `record_voice_transcripts` persists +each completed turn into native text memory, which `recall_conversation` +reads back as the `[Recent conversation]` block. A new utterance while TTS +is playing supersedes the previous turn. + +## Supervisor and subagents + +At worker startup, a NAT `WorkflowBuilder` constructs the sample-local scene +function groups plus tracking, spatial-math, vision, video-memory, and +text-memory functions, then registers five focused subagents and one +supervisor over them (`worker/xr_render_demo_worker/`): + +- **placement_agent** moves existing objects: user-relative moves, nudges, + anchored moves, containment, midpoints, swaps, undo via `[Recent moves]`. +- **object_agent** creates, removes, resizes, duplicates, and reshapes. +- **appearance_agent** recolors. +- **vision_agent** answers physical-world questions through the VLM (live or + recorded frames) and degrades to "no visual fact available" when the + camera is absent. +- **memory_agent** recalls conversation older than the recent-turns block. + +The supervisor is a `ToolCallAgentWorkflow` whose tools are the five +subagents. Each turn it receives the scene state, the participant's recent +moves, and the recent conversation, then delegates focused instructions. +When a turn ends with no verified scene change and was not a pure +conversational question, one verification pass re-runs the supervisor with +that fact. + +## Deterministic operations layer + +Subagents do not call leaf scene functions directly for mutations; they call +composed operations (`spatial_ops.py`) that take only semantic arguments and +do the resolution and geometry in code: + +- **Reference resolution**: object arguments (`object_words`, + `anchor_words`) accept the instruction's own words. Ids resolve exactly; + descriptions resolve by shape word, transcription-mangled noun + (difflib), and color (nearest RGB with a rejection threshold); an id with + a synonym prefix ("cube-39" for box-39) maps to the real id. Nothing + matching or several matching returns an error the agent relays as a + question. +- **Color and shape resolution**: `color_words` and `prim_type` are quoted + verbatim from the instruction and mapped to RGB / renderer primitives in + code, including misspellings ("blew", "spear"), numeric RGB triples from + vision facts, and copies ("same as cone-7"). +- **Guards**: a `TurnGuard` halts further mutations of existing objects in + a delegation after any failed lookup, a turn-scoped `CreationLedger` + dedupes retried creations, and self-anchored moves are rejected. +- **Truncation handling**: a transcript ending in a dangling article or + preposition ("Put the sphere on the") never reaches the model; the + supervisor asks for the missing words in code, and the follow-up turn is + spliced back onto the cut-off request (cancellations and fresh commands + are recognized and handled directly). +- **Geometry**: user-relative and anchored positions come from the shared + spatial-math functions; no coordinate round-trips through the model. ## Prompt structure -The system prompt at `worker/prompts/system.txt` is worked-example heavy. -It opens with pronoun and reference resolution, then routes placement -utterances through sequential checks before the LLM picks a tool: - -1. **FIRST CHECK**: `"between"`/`"middle"`/`"halfway"` → route to - `between_anchors`; stop considering other placement tools. -2. **SECOND CHECK**: anchor is the user (`"me"`/`"my"`) → route to - `place_user_relative`; `place_object_relative` with `origin=user_pos` - returns the wrong side of the user. -3. **THIRD CHECK**: proximity to a named object (`"closer to "`, - `"toward "`) → route to `along_direction`. The user's facing - direction is unrelated to where the target object sits, so - `displace_object` is wrong here. - -Every rule that's not obviously self-explanatory has a paired WORKED -EXAMPLE (concrete coords + tool call) and, for the highest-leakage -failure modes, a WORKED ANTI-EXAMPLE. The two-step contract is -hammered: every move emits one math-tool call followed by exactly one -`add_primitive`/`update_primitive` call carrying all three of `x`, -`y`, `z` from the math result. +Each agent has its own prompt (`supervisor_prompt.txt`, +`agents/*/prompt.txt`), loaded once at worker startup. Restart the stack +after any prompt change. The prompts are template- and contrast-pair-driven; +the current agent model follows worked examples and ignores prohibitions, so +every refuse-example is paired with a proceed-example, and prompt examples +must stay disjoint from eval-case vocabulary (enforced by the eval's startup +audit). ## XR session lifecycle @@ -304,23 +212,23 @@ a streaming client connects. LOVR cannot start before then. ## Eval harness -Offline regression suite for the agentic loop, run against the live agent LLM. -It derives schemas from the worker's native NAT functions and evaluates tool -effects against deterministic fixtures, so the live LOVR scene is not mutated. -Refer to -[`agent-samples/xr-render-demo/eval/README.md`](https://github.com/NVIDIA/xr-ai/blob/main/agent-samples/xr-render-demo/eval/README.md) -for the case format and the watch-mode loop. Run with: +Four regression tiers run against the live agent LLM. The three offline +tiers never touch LOVR; the live tier drives the running stack and does +mutate the scene: supervisor routing (faked subagents), subagent components (one +real agent over faked leaf functions), the end-to-end corpus plus a basics +battery, and live drivers that join the running stack as synthetic +participants. See +`agent-samples/xr-render-demo/eval/README.md` +for tiers, commands, and case formats. Run the corpus with: ```bash -uv run --project agent-samples/xr-render-demo/worker \ - python agent-samples/xr-render-demo/eval/eval.py +uv run --project agent-samples/xr-render-demo/eval xr_render_demo_eval ``` ### Prompt/eval overlap audit -The harness audits the system prompt's worked-example blocks against every -case fixture at startup and warns if they share specifics: verbatim user -utterances (≥12 chars), scene coordinates rendered as `(x.xx, y.yy, z.zz)`, -`recent_moves` coords, or any reserved colour or shape word that appears in -both a case fixture and a worked-example block. This guards against the eval -cases overfitting to the prompt's worked examples. +Per `AGENTS.md` "Prompt-driven samples", the harness audits every worker +prompt against every tier's case inputs at startup and warns on overlap: +verbatim case utterances, case fixture ids, and any quoted prompt example +pairing an eval-vocabulary color with an eval-vocabulary shape. Clearing a +warning means changing the prompt, not the case. diff --git a/docs/xr-render-demo.md b/docs/xr-render-demo.md index 238f3b3f..3e9b0706 100644 --- a/docs/xr-render-demo.md +++ b/docs/xr-render-demo.md @@ -97,35 +97,20 @@ The worker reads two YAML files: ### NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 — port 8107 -A vLLM `execvp` shim — a small Python wrapper that reads YAML config, +A vLLM `execvp` shim: a small Python wrapper that reads YAML config, sets `HF_HOME` / token env vars, then `os.execvp`s into `vllm serve`. The Python process is replaced by vLLM; vLLM owns the HTTP API, weight loading, and tool calling from that point on. `vllm serve` with `--tool-call-parser qwen3_coder` and `--reasoning-parser nano_v3` (plugin auto-fetched from the model card into -`model_cache`). `enforce_eager` defaults to `true` — CUDA graph capture + +`model_cache`). `enforce_eager` defaults to `true`: CUDA graph capture + FlashInfer FP4 MoE autotune silently takes 3–8 minutes on cold start without it. Requires a Blackwell GPU (B200 / RTX PRO 6000 / Jetson Thor) for native FP4; swap to the BF16 variant for Hopper / Ampere. -One server backs both logical models in `yaml/models.yaml`: `agent_llm` runs -the multi-step tool-calling loop, and `llm` serves two cheap, latency-sensitive -calls (thinking stays off): - -- **Quick-ack** — awaited before the agentic loop starts, the moment an - utterance lands. Returns `{"ack": "On it!", "think": false}` — a 3–6 word - spoken acknowledgment. Also classifies whether the request needs - open-ended reasoning (`think: true/false`): positional operations always - run without thinking because the math tools compute exact answers, so - thinking is reserved for vague corrections and free-form compositions - no tool pattern settles. Max 40 tokens, 8s timeout. The ack - is always sent on the data channel (`agent.progress` topic); it is only - also spoken via TTS on every turn so the user immediately knows they were heard. -- **Still-working messages** — if the agentic loop exceeds 5s, this model - generates a short contextual phrase like *"Still finding the right - position"* on a 10s repeat. Sent to the data channel only — never spoken, - to avoid stacking up in the TTS queue behind the real response. +The server backs the `agent_llm` role in `yaml/models.yaml`, which the +supervisor and all five subagents share as `scene_llm`. ## VLM — Cosmos-Reason1-7B @@ -145,164 +130,84 @@ OOM race. ## STT — parakeet-tdt-0.6b-v3 -Port 8103. NeMo ASR in-process. English-only, ~1.5 GB VRAM. - -``` -LiveKit mic (int16 PCM) → hub IPC (float32) → XRMediaHubTransport.input() - → SttProcessor - pre-roll buffer last 10 chunks (~320 ms) kept at all times; - prepended to the utterance buffer on speech onset - so the first word's attack isn't clipped - VAD Silero (ONNX, 512-sample / 32 ms windows, - probability threshold) via shared xr-ai-vad util - accumulates audio while speaking - finalizes when silence ≥ 0.8s AND speech ≥ 0.15s - OR max utterance length (30s) hit - filler filter drops single- and multi-word filler utterances - ("um", "uh", "yeah", "okay", "mm-hmm", etc.) - STT call POST multipart/form-data WAV → stt-server :8103 - → TranscriptionFrame pushed downstream -``` - -STT calls are serialized — an `stt_busy` flag prevents a new finalize while -one is in-flight. +Served on port 8103. The voice runtime's VAD segments utterances and sends +16 kHz int16 audio for transcription; transcripts feed the voice gate and +then the supervisor. ## TTS — Piper -Port 8105. `rhasspy/piper-voices` ONNX. Runs on CPU, ~100 ms / sentence. All -synthesis runs in a thread pool so the asyncio loop is never blocked. - -``` -TextFrame (from agentic loop final response or quick-ack) - → TtsProcessor - sentence-batched synthesis - POST text → tts-server :8105 → WAV bytes - RETURN_AUDIO IPC → hub → LiveKit → participant's headphones -``` - -`allow_interruptions=True` in the Pipecat pipeline. A new utterance while TTS -is playing triggers `ReturnAudioFlush` → hub clears the LiveKit audio queue -for that participant. - -## Pipecat pipeline - -``` -XRMediaHubTransport.input() - → SttProcessor (Silero VAD → utterance → parakeet STT - → TranscriptionFrame) - → RenderSceneProcessor (quick-ack + agentic loop → TextFrame) - → TtsProcessor (TextFrame → Piper TTS → return audio) - → XRMediaHubTransport.output() -``` - -## Agentic loop - -At worker startup, a NAT `WorkflowBuilder` constructs sample-local scene, -XR-tracking, spatial-math, vision, video-memory, and text-memory functions. -The LLM tool schemas are derived from those Functions and held in memory. -`start_xr` and `get_health` are excluded from the model tool list because the -worker owns the XR lifecycle. - -On each `TranscriptionFrame`: - -1. **Quick-ack** runs first (`llm` :8107, awaited before the loop). -2. **Still-working timer** starts (fires at 5s, repeats every 10s, data - channel only). -3. **Pre-fetch** (concurrent): `get_scene_state` + `get_head_pose` + - `position_ahead(1.5)` — results injected into the user message so the - model skips those tool calls and goes straight to the operation. -4. **Nemotron-30B :8107** runs with `tools=[…]`, up to 10 iterations: - - Model emits `tool_calls` → worker invokes the matching NAT Function → result appended - to conversation → next iteration. - - Runtime-backed Functions call the scene, OpenXR, and video-memory typed - services. Spatial math and text memory execute in process. Vision calls - the configured `xr-ai-models` VLM service. - - Progress message sent on `agent.progress` topic before each tool - executes (data channel). - - If `think=true`: reasoning preamble injected into system prompt - (RESOLVE object → LOCATE coordinates → COMPUTE new position → - EXECUTE). The `` block stays private; only one short sentence - goes to the user. Token budget: 6144 total / 4096 thinking budget. - - If thinking fills the token budget without a tool call - (`finish_reason=length`): retry the same iteration with - `needs_thinking=False`. - - If the model outputs a bare tool name as text instead of a proper tool - call: worker synthesizes a no-arg tool call and continues. -5. **Final response** sent on `agent.response` topic and as a `TextFrame` - downstream to TTS. -6. **Turn appended** to a rolling 4-turn history buffer — injected as - context in future turns so the model understands "fix that", "undo", - "the one I just added". Final user and assistant messages are also written - to native text memory; model scratch output and tool traces are not stored. - -## Native capability composition - -The sample-local scene process owns scene state and LOVR and is the only thing -that pushes operations onto LOVR's scene socket (msgpack over ZMQ PUSH). -`openxr-service` owns the second headless OpenXR session -(`XR_MND_HEADLESS`). `video-memory-service` owns recorded-video decoding, -while `LiveFrameSource` supplies current-frame requests. NAT Functions provide -the typed in-process tool surface over those services; the demo does not -launch or call MCP adapters. - -### Spatial tool surface - -The worker composes XR tracking with shared spatial-math Functions into the -prompt's established tool vocabulary. This offloads vector arithmetic the LLM -is bad at while keeping pose-dependent math in one place: - -- **Pose-aware named-direction helpers** take a `direction` enum (`front`, - `back`, `left`, `right`, `above`, `below`, plus `next_to` on - `place_object_relative`) and always-positive `distance`. The LLM never - applies signs to user-frame axes. - - `place_user_relative(direction, distance)`: user-anchored teleport - ("above my head", "to my left 1 m"). - - `place_object_relative(origin_x, origin_y, origin_z, direction, distance)`: - object-anchored teleport. `direction="front"` means *toward the user*; - `"back"` means *away*. Left/right/above/below map literally. - - `displace_object(current_x, current_y, current_z, right, up, forward)`: - user-frame signed-delta on an existing object. Multi-axis ("up and - to the left") in one call. - - `displace_objects(object_ids, current_xs, current_ys, current_zs, - right, up, forward)`: batch user-frame delta over N objects. Returns - `{"items": [{obj_id, x, y, z}, …]}` so the model fans out to N - `update_primitive` calls with one math call total. - - `place_inside_by_id(movee_id, container_x, container_y, container_z)`: - containment for "put X in Y". Argument names (`movee_id` paired - with `container_*`) force the model to pick the right noun's coords; - the return shape feeds straight into `update_primitive`. -- **Pure-math primitives** are pose-independent: - - `between_anchors(a_x, a_y, a_z, b_x, b_y, b_z)`: component-wise midpoint. - - `world_offset(origin_x, origin_y, origin_z, dx, dy, dz)`: - axis-aligned world-Y-up shift. - - `along_direction(origin_x, origin_y, origin_z, target_x, target_y, - target_z, distance)`: origin moved `distance` toward target. Used - for "closer to / further from ", which the user-frame - helpers can't model. - - `scale_value(current, factor)`: scalar multiplication for sizes. +Served on port 8105. The voice runtime streams the supervisor's final +reply to Piper and returns the audio to the participant. + +## Voice runtime + +The worker runs on `xr-ai-voice`'s `VoiceSession`: hub audio → VAD → STT → +voice gate → handler → streaming TTS → return audio. The scene supervisor is +attached with `as_voice_handler`, and `record_voice_transcripts` persists +each completed turn into native text memory, which `recall_conversation` +reads back as the `[Recent conversation]` block. A new utterance while TTS +is playing supersedes the previous turn. + +## Supervisor and subagents + +At worker startup, a NAT `WorkflowBuilder` constructs the sample-local scene +function groups plus tracking, spatial-math, vision, video-memory, and +text-memory functions, then registers five focused subagents and one +supervisor over them (`worker/xr_render_demo_worker/`): + +- **placement_agent** moves existing objects: user-relative moves, nudges, + anchored moves, containment, midpoints, swaps, undo via `[Recent moves]`. +- **object_agent** creates, removes, resizes, duplicates, and reshapes. +- **appearance_agent** recolors. +- **vision_agent** answers physical-world questions through the VLM (live or + recorded frames) and degrades to "no visual fact available" when the + camera is absent. +- **memory_agent** recalls conversation older than the recent-turns block. + +The supervisor is a `ToolCallAgentWorkflow` whose tools are the five +subagents. Each turn it receives the scene state, the participant's recent +moves, and the recent conversation, then delegates focused instructions. +When a turn ends with no verified scene change and was not a pure +conversational question, one verification pass re-runs the supervisor with +that fact. + +## Deterministic operations layer + +Subagents do not call leaf scene functions directly for mutations; they call +composed operations (`spatial_ops.py`) that take only semantic arguments and +do the resolution and geometry in code: + +- **Reference resolution**: object arguments (`object_words`, + `anchor_words`) accept the instruction's own words. Ids resolve exactly; + descriptions resolve by shape word, transcription-mangled noun + (difflib), and color (nearest RGB with a rejection threshold); an id with + a synonym prefix ("cube-39" for box-39) maps to the real id. Nothing + matching or several matching returns an error the agent relays as a + question. +- **Color and shape resolution**: `color_words` and `prim_type` are quoted + verbatim from the instruction and mapped to RGB / renderer primitives in + code, including misspellings ("blew", "spear"), numeric RGB triples from + vision facts, and copies ("same as cone-7"). +- **Guards**: a `TurnGuard` halts further mutations of existing objects in + a delegation after any failed lookup, a turn-scoped `CreationLedger` + dedupes retried creations, and self-anchored moves are rejected. +- **Truncation handling**: a transcript ending in a dangling article or + preposition ("Put the sphere on the") never reaches the model; the + supervisor asks for the missing words in code, and the follow-up turn is + spliced back onto the cut-off request (cancellations and fresh commands + are recognized and handled directly). +- **Geometry**: user-relative and anchored positions come from the shared + spatial-math functions; no coordinate round-trips through the model. ## Prompt structure -The system prompt at `worker/prompts/system.txt` is worked-example heavy. -It opens with pronoun/reference resolution, then routes placement -utterances through sequential checks before the LLM picks a tool: - -1. **FIRST CHECK**: `"between"`/`"middle"`/`"halfway"` → route to - `between_anchors`; stop considering other placement tools. -2. **SECOND CHECK**: anchor is the user (`"me"`/`"my"`) → route to - `place_user_relative`; `place_object_relative` with `origin=user_pos` - returns the wrong side of the user. -3. **THIRD CHECK**: proximity to a named object (`"closer to "`, - `"toward "`) → route to `along_direction`. The user's facing - direction is unrelated to where the target object sits, so - `displace_object` is wrong here. - -Every rule that's not obviously self-explanatory has a paired WORKED -EXAMPLE (concrete coords + tool call) and, for the highest-leakage -failure modes, a WORKED ANTI-EXAMPLE. The two-step contract is -hammered: every move emits one math-tool call followed by exactly one -`add_primitive`/`update_primitive` call carrying all three of `x`, -`y`, `z` from the math result. +Each agent has its own prompt (`supervisor_prompt.txt`, +`agents/*/prompt.txt`), loaded once at worker startup. Restart the stack +after any prompt change. The prompts are template- and contrast-pair-driven; +the current agent model follows worked examples and ignores prohibitions, so +every refuse-example is paired with a proceed-example, and prompt examples +must stay disjoint from eval-case vocabulary (enforced by the eval's startup +audit). ## XR session lifecycle @@ -325,26 +230,23 @@ a streaming client connects. LOVR cannot start before then. ## Eval harness -Offline regression suite for the agentic loop, run against the live agent LLM. -It derives schemas from the worker's native NAT functions and evaluates tool -effects against deterministic fixtures, so the live LOVR scene is not mutated. -See +Four regression tiers run against the live agent LLM. The three offline +tiers never touch LOVR; the live tier drives the running stack and does +mutate the scene: supervisor routing (faked subagents), subagent components (one +real agent over faked leaf functions), the end-to-end corpus plus a basics +battery, and live drivers that join the running stack as synthetic +participants. See [`agent-samples/xr-render-demo/eval/README.md`](../agent-samples/xr-render-demo/eval/README.md) -for the case format and the watch-mode loop. Run with: +for tiers, commands, and case formats. Run the corpus with: ```bash -uv run --project agent-samples/xr-render-demo/worker \ - python agent-samples/xr-render-demo/eval/eval.py +uv run --project agent-samples/xr-render-demo/eval xr_render_demo_eval ``` ### Prompt/eval overlap audit -Per `AGENTS.md` "Prompt-driven samples", the harness audits the system -prompt's worked-example blocks against every case fixture at startup -and warns if they share specifics: verbatim user utterances (≥12 -chars), scene coordinates rendered as `(x.xx, y.yy, z.zz)`, -`recent_moves` coords, or any reserved colour/shape word that appears -in both a case fixture and a worked-example block. Reserved vocab -lives in `_EVAL_VOCAB_COLORS` / `_EVAL_VOCAB_SHAPES`; clearing a -warning means changing the prompt's worked example, not the case. -`--strict-overlap` turns the audit into a hard failure (rc=2) for CI. +Per `AGENTS.md` "Prompt-driven samples", the harness audits every worker +prompt against every tier's case inputs at startup and warns on overlap: +verbatim case utterances, case fixture ids, and any quoted prompt example +pairing an eval-vocabulary color with an eval-vocabulary shape. Clearing a +warning means changing the prompt, not the case. diff --git a/ruff.toml b/ruff.toml index 9ff7e4b0..8d2a1f8e 100644 --- a/ruff.toml +++ b/ruff.toml @@ -34,11 +34,6 @@ known-first-party = ["_helpers", "_helpers_subprocess"] "agent-mcp-servers/video-mcp/video_mcp_server/__main__.py" = ["I001"] "agent-mcp-servers/vlm-mcp/vlm_mcp_server/__main__.py" = ["I001"] "agent-samples/model-servers/main.py" = ["F401"] -"agent-samples/xr-render-demo/eval/eval.py" = ["E702", "I001"] -"agent-samples/xr-render-demo/worker/agent.py" = ["I001"] -"agent-samples/xr-render-demo/worker/processors.py" = ["E701", "E702", "E741", "I001"] -"agent-samples/xr-render-demo/worker/tooling.py" = ["E701", "E702", "I001"] -"agent-samples/xr-render-demo/worker/xr_render_demo_worker.py" = ["I001"] "agent-sdk/xr-ai-models/xr_ai_models/__init__.py" = ["I001"] "agent-sdk/xr-ai-models/xr_ai_models/_config.py" = ["I001"] "agent-sdk/xr-ai-models/xr_ai_models/_openai_compat.py" = ["E402", "I001"] @@ -103,7 +98,6 @@ known-first-party = ["_helpers", "_helpers_subprocess"] "tests/test_xr_ai_pipecat_voice_pipeline.py" = ["E501", "I001"] "tests/test_xr_ai_vad.py" = ["I001"] "tests/test_xr_ai_voicegate.py" = ["I001"] -"tests/test_xr_render_demo_wire.py" = ["E402", "I001"] "utils/xr-ai-vllm/xr_ai_vllm/_pip.py" = ["F401"] "utils/xr-ai-voicegate/xr_ai_voicegate/_phrases.py" = ["I001"] "utils/xr-ai-voicegate/xr_ai_voicegate/gate.py" = ["I001"] diff --git a/services/openxr-service/openxr_service.yaml b/services/openxr-service/openxr_service.yaml index 67b208a9..b0fc0976 100644 --- a/services/openxr-service/openxr_service.yaml +++ b/services/openxr-service/openxr_service.yaml @@ -3,3 +3,8 @@ endpoint: tcp://0.0.0.0:8330 cloudxr_env_file: ~/.cloudxr/run/cloudxr.env + +# Test hook: when true, set_sim_pose/clear_sim_pose RPC ops let eval drivers +# inject a simulated head pose that overrides real tracking for every client +# until cleared. Keep false outside eval runs. +allow_sim_pose: false diff --git a/services/openxr-service/openxr_service/__main__.py b/services/openxr-service/openxr_service/__main__.py index 07a4c878..37997831 100644 --- a/services/openxr-service/openxr_service/__main__.py +++ b/services/openxr-service/openxr_service/__main__.py @@ -24,6 +24,7 @@ class Config: endpoint: str cloudxr_env_file: Path | None + allow_sim_pose: bool def _load_config(path: Path) -> Config: @@ -35,6 +36,7 @@ def _load_config(path: Path) -> Config: return Config( endpoint=str(raw.get("endpoint", "tcp://0.0.0.0:8330")), cloudxr_env_file=env_file, + allow_sim_pose=bool(raw.get("allow_sim_pose", False)), ) @@ -48,7 +50,7 @@ async def _serve(config: Config, ready_file: Path | None) -> None: from .session import HardwarePoseSource source = HardwarePoseSource() - server = RPCServer(config.endpoint, OpenXRService(source).dispatch) + server = RPCServer(config.endpoint, OpenXRService(source, allow_sim_pose=config.allow_sim_pose).dispatch) logger.info("openxr-service rpc={}", config.endpoint) try: await server.serve(ready=ready_file.touch if ready_file else None) diff --git a/services/openxr-service/openxr_service/service.py b/services/openxr-service/openxr_service/service.py index be09d83c..b9db0401 100644 --- a/services/openxr-service/openxr_service/service.py +++ b/services/openxr-service/openxr_service/service.py @@ -6,9 +6,10 @@ import asyncio from typing import Any, Protocol +from loguru import logger from pydantic import ValidationError from xr_ai_nat.functions._service.rpc import RPCError -from xr_ai_nat.functions.xr_tracking._client import HeadPoseRequest, OpenXRHealthRequest +from xr_ai_nat.functions.xr_tracking._client import HeadPose, HeadPoseRequest, OpenXRHealthRequest class PoseSource(Protocol): @@ -22,13 +23,30 @@ def health(self) -> dict[str, Any]: class OpenXRService: """Validate service calls and delegate hardware access to one pose source.""" - def __init__(self, source: PoseSource) -> None: + def __init__(self, source: PoseSource, *, allow_sim_pose: bool = False) -> None: self._source = source + self._allow_sim_pose = allow_sim_pose + self._sim_pose: dict[str, Any] | None = None + if allow_sim_pose: + logger.warning("sim-pose test hook enabled: any RPC peer can override head tracking") async def dispatch(self, operation: str, arguments: dict[str, Any]) -> dict[str, Any]: if operation == "get_head_pose": self._validate(HeadPoseRequest, arguments) + if self._sim_pose is not None: + return self._sim_pose return await asyncio.to_thread(self._source.get_pose) + if operation == "set_sim_pose" and self._allow_sim_pose: + try: + pose = HeadPose.model_validate({**arguments, "is_valid": True, "error": None}) + except ValidationError as exc: + raise RPCError("invalid sim pose", code="invalid_request") from exc + self._sim_pose = pose.model_dump() + logger.warning("sim pose set; real head tracking is overridden until cleared") + return {"ok": True} + if operation == "clear_sim_pose" and self._allow_sim_pose: + self._sim_pose = None + return {"ok": True} if operation == "get_health": self._validate(OpenXRHealthRequest, arguments) return self._source.health() diff --git a/tests/pyproject.toml b/tests/pyproject.toml index c13137c8..9b690e51 100644 --- a/tests/pyproject.toml +++ b/tests/pyproject.toml @@ -28,6 +28,8 @@ dependencies = [ "vlm-mcp-server", "render-mcp", "xr-render-scene", + "xr-render-demo-worker", + "xr-render-demo-eval", "video-mcp-server", "xr-video-memory-service", "xr-rag-service", @@ -58,6 +60,8 @@ transcript-mcp-server = { path = "../agent-mcp-servers/transcript-mcp", editable vlm-mcp-server = { path = "../agent-mcp-servers/vlm-mcp", editable = true } render-mcp = { path = "../agent-mcp-servers/render-mcp", editable = true } xr-render-scene = { path = "../agent-samples/xr-render-demo/scene", editable = true } +xr-render-demo-worker = { path = "../agent-samples/xr-render-demo/worker", editable = true } +xr-render-demo-eval = { path = "../agent-samples/xr-render-demo/eval", editable = true } video-mcp-server = { path = "../agent-mcp-servers/video-mcp", editable = true } xr-video-memory-service = { path = "../services/video-memory-service", editable = true } xr-rag-service = { path = "../services/rag-service", editable = true } diff --git a/tests/test_langchain_content_recovery.py b/tests/test_langchain_content_recovery.py new file mode 100644 index 00000000..06b0d592 --- /dev/null +++ b/tests/test_langchain_content_recovery.py @@ -0,0 +1,93 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Pin the content-leak tool-call recovery in the shared LangChain adapter. + +Recovery converts model text into executed tool calls, so its rejection +branches matter as much as its acceptance branches: a lone argument echo +must never become a mutation. +""" + +from xr_ai_models import ToolDef +from xr_ai_nat.llm._langchain import _content_tool_call + + +def _tool(name, required, optional=()): + properties = {key: {"type": "string"} for key in (*required, *optional)} + return ToolDef( + name=name, + description="", + parameters={"type": "object", "properties": properties, "required": list(required)}, + ) + + +OFFERED = { + tool.name: tool + for tool in ( + _tool("object_ops__remove_object", ["object_words"]), + _tool("object_ops__resize_object", ["object_words", "factor"]), + _tool("placement_ops__nudge", ["object_words"], ["forward", "right", "up"]), + ) +} + + +def test_named_call_recovers(): + recovered = _content_tool_call( + '{"name": "object_ops__resize_object", "arguments": {"object_words": "the cone", "factor": 2}}', + OFFERED, + ) + assert recovered and recovered["name"] == "object_ops__resize_object" + assert recovered["args"] == {"object_words": "the cone", "factor": 2} + + +def test_wrapper_keys_unwrap(): + for wrapper in ("command", "function", "tool_call"): + recovered = _content_tool_call( + '{"%s": {"tool": "object_ops__resize_object", "args": {"object_words": "x", "factor": 2}}}' % wrapper, + OFFERED, + ) + assert recovered and recovered["name"] == "object_ops__resize_object" + + +def test_bare_name_suffix_match_requires_uniqueness(): + recovered = _content_tool_call('{"name": "nudge", "arguments": {"object_words": "x"}}', OFFERED) + assert recovered and recovered["name"] == "placement_ops__nudge" + two = dict(OFFERED) + two["other_ops__nudge"] = _tool("other_ops__nudge", ["object_words"]) + assert _content_tool_call('{"name": "nudge", "arguments": {"object_words": "x"}}', two) is None + + +def test_string_encoded_arguments_parse(): + recovered = _content_tool_call( + '{"name": "object_ops__remove_object", "arguments": "{\\"object_words\\": \\"the cone\\"}"}', + OFFERED, + ) + assert recovered and recovered["args"] == {"object_words": "the cone"} + + +def test_mixed_prose_is_not_recovered(): + text = 'I will not do that. {"name": "object_ops__remove_object", "arguments": {"object_words": "x"}}' + assert _content_tool_call(text, OFFERED) is None + + +def test_lone_argument_echo_never_recovers(): + # A one-key echo would uniquely match remove_object's schema and delete + # an object; shape inference requires at least two keys. + assert _content_tool_call('{"object_words": "the blue sphere"}', OFFERED) is None + + +def test_two_key_shape_inference_recovers_unique_tool(): + recovered = _content_tool_call('{"object_words": "the cone", "factor": 0.5}', OFFERED) + assert recovered and recovered["name"] == "object_ops__resize_object" + + +def test_ambiguous_shape_inference_rejected(): + two = dict(OFFERED) + two["other_ops__grow"] = _tool("other_ops__grow", ["object_words", "factor"]) + assert _content_tool_call('{"object_words": "x", "factor": 2}', two) is None + + +def test_recovery_ids_are_unique(): + first = _content_tool_call('{"name": "nudge", "arguments": {"object_words": "x"}}', OFFERED) + second = _content_tool_call('{"name": "nudge", "arguments": {"object_words": "x"}}', OFFERED) + assert first["id"] != second["id"] diff --git a/tests/test_render_eval_checkers.py b/tests/test_render_eval_checkers.py new file mode 100644 index 00000000..0b8057c3 --- /dev/null +++ b/tests/test_render_eval_checkers.py @@ -0,0 +1,21 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Pin the eval harness's scoring primitives; a silent checker bug voids every tier score.""" + +from xr_render_demo_eval.subagents import _args_match + + +def test_exact_values_match(): + assert _args_match({"obj_id": "cone-7", "x": 1.0}, {"obj_id": "cone-7"}) + assert not _args_match({"obj_id": "cone-7"}, {"obj_id": "ring-1"}) + + +def test_range_is_inclusive_at_both_ends(): + assert _args_match({"x": 0.5}, {"x": (0.5, 1.0)}) + assert _args_match({"x": 1.0}, {"x": (0.5, 1.0)}) + assert not _args_match({"x": 1.01}, {"x": (0.5, 1.0)}) + + +def test_missing_key_fails(): + assert not _args_match({"x": 0.5}, {"y": (0.0, 1.0)}) diff --git a/tests/test_render_spatial_ops.py b/tests/test_render_spatial_ops.py new file mode 100644 index 00000000..9bdc22e1 --- /dev/null +++ b/tests/test_render_spatial_ops.py @@ -0,0 +1,187 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Unit tests for the render worker's deterministic reference resolvers. + +The eval tiers exercise these only through a live LLM, where a resolver +regression reads as unexplained score variance; these tests pin the logic +directly. +""" + +import pytest +from xr_render_demo_worker.spatial_ops import CreationLedger, TurnGuard, _Leaves +from xr_render_scene import SceneObject, SceneState + + +class _Fn: + def __init__(self, result=None): + self.result = result + self.calls = [] + + async def ainvoke(self, request): + self.calls.append(request) + return self.result + + +def _obj(object_id, kind, color, position=(0.0, 1.6, -1.5), size=0.1): + r, g, b = color + x, y, z = position + return SceneObject.model_validate( + {"id": object_id, "type": kind, "position": {"x": x, "y": y, "z": z}, + "color": {"r": r, "g": g, "b": b}, "size": size} + ) + + +def _leaves(objects, guard=None, ledger=None): + state = SceneState(objects=objects) + functions = { + "scene_state__get_scene_state": _Fn(state), + "scene_updates__update_primitive": _Fn({}), + "scene_objects__remove_primitive": _Fn({}), + } + return _Leaves(functions, ledger=ledger, guard=guard), functions + + +async def test_exact_id_wins(): + leaves, _ = _leaves([_obj("sphere-0", "sphere", (0, 0, 1))]) + assert (await leaves.find("sphere-0")).id == "sphere-0" + + +async def test_unicode_dash_id_normalizes(): + leaves, _ = _leaves([_obj("sphere-0", "sphere", (0, 0, 1))]) + assert (await leaves.find("sphere‑0")).id == "sphere-0" + + +async def test_color_word_resolves_without_shape(): + leaves, _ = _leaves([ + _obj("sphere-0", "sphere", (1, 0, 0)), + _obj("box-0", "box", (0, 0, 1)), + ]) + assert (await leaves.find("the red one")).id == "sphere-0" + + +async def test_everyday_words_do_not_become_shapes(): + # "one"/"thing"/"right" sound like cone/ring; the color must still win. + leaves, _ = _leaves([ + _obj("sphere-0", "sphere", (1, 0, 0)), + _obj("sphere-1", "sphere", (0, 0.8, 0)), + ]) + assert (await leaves.find("the red one on the right")).id == "sphere-0" + + +async def test_mangled_shape_noun_resolves(): + leaves, _ = _leaves([ + _obj("sphere-0", "sphere", (0, 0.8, 0)), + _obj("box-0", "box", (0, 0.8, 0)), + ]) + assert (await leaves.find("green spear")).id == "sphere-0" + + +async def test_absent_color_reports_back_not_nearest(): + guard = TurnGuard() + leaves, _ = _leaves( + [_obj("sphere-0", "sphere", (1, 1, 1)), _obj("sphere-1", "sphere", (0, 0.8, 0))], + guard=guard, + ) + with pytest.raises(ValueError, match="blue sphere"): + await leaves.find("blue sphere") + assert guard.halted + + +async def test_color_tie_raises_ambiguous(): + leaves, _ = _leaves([ + _obj("sphere-0", "sphere", (0, 0.8, 0)), + _obj("sphere-1", "sphere", (0, 0.8, 0)), + ]) + with pytest.raises(ValueError, match="ambiguous"): + await leaves.find("green sphere") + + +async def test_capitalized_id_still_hits_id_branch(): + leaves, _ = _leaves([_obj("box-3", "box", (1, 0, 0)), _obj("box-4", "box", (0, 0.8, 0))]) + assert (await leaves.find("Box-3")).id == "box-3" + with pytest.raises(ValueError, match="box-3"): + await leaves.find("Box-9") + + +async def test_numeric_rgb_ignores_id_bearing_strings(): + leaves, _ = _leaves([_obj("capsule-0", "capsule", (0.25, 0.5, 0.75))]) + assert await leaves.color("same as capsule-0") == (0.25, 0.5, 0.75) + assert await leaves.color("RGB (1.0, 0.5, 0.0)") == (1.0, 0.5, 0.0) + # A sign-invalid triple has no letters either; it falls through to the + # standard default rather than a wrong saturated color. + assert await leaves.color("-0.5 0.2 0.3") == (0.2, 0.9, 1.0) + + +async def test_synonym_prefixed_id_resolves(): + leaves, _ = _leaves([_obj("box-39", "box", (1, 1, 0))]) + assert (await leaves.find("cube-39")).id == "box-39" + + +async def test_id_shaped_miss_lists_ids_without_halting(): + guard = TurnGuard() + leaves, _ = _leaves([_obj("sphere-0", "sphere", (0, 0, 1))], guard=guard) + with pytest.raises(ValueError, match="sphere-0"): + await leaves.find("sphere-9") + assert not guard.halted + + +async def test_halt_blocks_moves_but_not_creates(): + guard = TurnGuard() + ledger = CreationLedger() + leaves, functions = _leaves([_obj("sphere-0", "sphere", (0, 0, 1))], guard=guard, ledger=ledger) + functions["scene_objects__add_primitive"] = _Fn( + type("R", (), {"id": "box-0"})() + ) + guard.halted = True + with pytest.raises(ValueError, match="report that failure back"): + await leaves.write("sphere-0", (0, 0, 0)) + with pytest.raises(ValueError, match="report that failure back"): + leaves.check_writable() + created = await leaves.add("box", (0, 1, 0), (1, 0, 0), 0.1) + assert created.id == "box-0" + + +def test_shape_words_resolve_and_reject(): + leaves, _ = _leaves([]) + assert leaves.shape("spear") == "sphere" + assert leaves.shape("kube") == "box" + assert leaves.shape("cone") == "cone" + with pytest.raises(ValueError, match="Unknown shape"): + leaves.shape("xylophone") + + +async def test_color_words_resolve_fuzzy_default_and_copy(): + leaves, _ = _leaves([_obj("cone-3", "cone", (0.25, 0.5, 0.75))]) + assert await leaves.color("teal") == (0, 0.8, 0.8) + assert await leaves.color("blew") == (0, 0.4, 1) + assert await leaves.color("") == (0.2, 0.9, 1.0) + assert await leaves.color("same as cone-3") == (0.25, 0.5, 0.75) + assert await leaves.color("normalized RGB (1.0, 0.5, 0.0)") == (1.0, 0.5, 0.0) + with pytest.raises(ValueError, match="Unknown color"): + await leaves.color("wibble") + + +async def test_ledger_dedupes_identical_creates(): + ledger = CreationLedger() + leaves, functions = _leaves([], ledger=ledger) + results = iter([type("R", (), {"id": "box-0"})(), type("R", (), {"id": "box-1"})()]) + + class _Adder: + def __init__(self): + self.calls = [] + + async def ainvoke(self, request): + self.calls.append(request) + return next(results) + + adder = _Adder() + functions["scene_objects__add_primitive"] = adder + first = await leaves.add("box", (0.001, 1.0, 0.0), (1, 0, 0), 0.1) + second = await leaves.add("box", (0.004, 1.0, 0.0), (1, 0, 0), 0.1) + assert first.id == second.id == "box-0" + assert len(adder.calls) == 1 + assert first.created_this_turn == 1 + ledger.reset() + third = await leaves.add("box", (0.001, 1.0, 0.0), (1, 0, 0), 0.1) + assert third.id == "box-1" diff --git a/tests/test_xr_render_demo_wire.py b/tests/test_xr_render_demo_wire.py index d3bb9ecd..f6294d39 100644 --- a/tests/test_xr_render_demo_wire.py +++ b/tests/test_xr_render_demo_wire.py @@ -1,1153 +1,99 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -"""Wire-trace golden tests for the xr-render-demo model-service contracts. +"""Wire-level checks for the xr-render-demo worker package: it builds, its +prompts exist and stay within the eval audit's rules, and the worker config +round-trips.""" -Exercises the worker's direct model calls against ``StubOpenAI`` without a -real server or GPU. Asserts that the JSON bodies sent over the wire retain the -required fields and that ``ChatResponse`` fields are correctly extracted. - -GPU verification skipped — stub-server tests only. -""" -from __future__ import annotations - -import asyncio -import json -import sys -from contextlib import asynccontextmanager from pathlib import Path import pytest +from xr_render_demo_worker.config import load_config +from xr_render_demo_worker.models import SceneReply, SceneRequest, SubagentResult, SubagentTask -# Add the worker directory to sys.path so we can import its modules. -_WORKER_DIR = ( - Path(__file__).resolve().parent.parent - / "agent-samples" / "xr-render-demo" / "worker" -) -sys.path.insert(0, str(_WORKER_DIR)) - -from _stub_openai import StubOpenAI - -from pipecat.frames.frames import EndFrame, Frame, TextFrame -from pipecat.pipeline.pipeline import Pipeline -from pipecat.pipeline.worker import PipelineWorker -from pipecat.processors.frame_processor import FrameDirection, FrameProcessor -from pipecat.workers.runner import WorkerRunner - -from xr_ai_hub import DataMessage -from xr_ai_models import ( - ChatMessage, - OpenAICompatLLM, - ToolDef, - load_models_config, -) -from nat.builder.workflow_builder import WorkflowBuilder -from nat.plugin_api import FunctionGroupRef -from xr_ai_nat.functions.video_memory import VideoMemoryFunctionsConfig -from xr_ai_nat.functions.vision import VisionToolsConfig - -# ── helpers ──────────────────────────────────────────────────────────────────── - -_MODELS_YAML = ( - Path(__file__).resolve().parent.parent - / "agent-samples" / "xr-render-demo" / "yaml" / "models.yaml" -) - - -def _make_llm(stub: StubOpenAI, *, model_name: str = "llm", - reasoning_field: str | None = None, - default_extras: dict | None = None) -> OpenAICompatLLM: - """Build an LLM client wired to a StubOpenAI transport.""" - return OpenAICompatLLM( - "http://stub", - model_name, - reasoning_field=reasoning_field, - default_extras=default_extras, - client=stub.client(), - ) - - -def _make_spec_llm(stub: StubOpenAI, name: str) -> OpenAICompatLLM: - """Build an LLM client from the shipped models.yaml spec for *name*.""" - spec = load_models_config(_MODELS_YAML).llm(name) - return _make_llm( - stub, - model_name=spec.model_name, - reasoning_field=spec.reasoning_field, - default_extras=spec.default_extras, - ) - - -# ── models.yaml round-trip ──────────────────────────────────────────────────── - - -def test_models_yaml_loads() -> None: - """The bundled models.yaml parses without error and exposes expected names.""" - cfg = load_models_config(_MODELS_YAML) - llm_spec = cfg.llm("llm") - agent_llm_spec = cfg.llm("agent_llm") - stt_spec = cfg.stt("stt") - tts_spec = cfg.tts("tts") - vlm_spec = cfg.vlm("vlm") - - assert llm_spec.base_url == "http://localhost:8107" - assert agent_llm_spec.base_url == "http://localhost:8107" - assert stt_spec.base_url == "http://localhost:8103" - assert tts_spec.base_url == "http://localhost:8105" - assert vlm_spec.base_url == "http://localhost:8100" - - # nemotron3_nano preset must set reasoning_field so ChatResponse.reasoning - # is populated from the server's "reasoning" field. - assert agent_llm_spec.reasoning_field == "reasoning" - - # Both logical models share the nemotron3_nano server. The preset must pin - # thinking off at the wire level: Nemotron-3-Nano's template defaults - # thinking-on, which would burn the quick-ack's 40-token budget on hidden - # reasoning and return empty content with finish_reason="length". - for spec in (llm_spec, agent_llm_spec): - assert spec.model_name == "llm" - assert spec.default_extras["chat_template_kwargs"] == {"enable_thinking": False} - - -def test_worker_config_idle_timeout_disabled_by_default() -> None: - """The shipped worker YAML ships idle_timeout_secs: 0, which the loader - maps to None (disabled) so a quiet session is never auto-cancelled.""" - from config import load_config - - worker_yaml = ( - Path(__file__).resolve().parent.parent - / "agent-samples" / "xr-render-demo" / "yaml" / "xr_render_demo_worker.yaml" - ) - cfg = load_config(worker_yaml) - assert cfg.idle_timeout_secs is None - - -def test_worker_config_idle_timeout_opt_in(tmp_path) -> None: - """A positive idle_timeout_secs in the YAML is parsed to a float.""" - from config import load_config - - y = tmp_path / "w.yaml" - y.write_text("idle_timeout_secs: 300\n") - cfg = load_config(y) - assert cfg.idle_timeout_secs == 300.0 - - -# ── quick-ack wire golden ───────────────────────────────────────────────────── - - -async def test_quick_ack_wire_golden() -> None: - """quick-ack: max_tokens=40, temperature=0.0, no tools, thinking pinned off.""" - stub = StubOpenAI() - stub.set_chat_message(content='{"ack": "On it!", "think": false}') - llm = _make_spec_llm(stub, "llm") - - messages = [ - ChatMessage(role="system", content="You are a quick-ack classifier."), - ChatMessage(role="user", content="Add a red sphere in front of me"), - ] - resp = await llm.chat(messages, max_tokens=40, temperature=0.0) - - body = stub.last_json() - - assert body["model"] == "llm" - assert body["max_tokens"] == 40 - assert body["temperature"] == 0.0 - assert "tools" not in body - assert body["chat_template_kwargs"] == {"enable_thinking": False} - assert len(body["messages"]) == 2 - assert body["messages"][0]["role"] == "system" - assert body["messages"][1]["role"] == "user" - - assert resp.content == '{"ack": "On it!", "think": false}' - assert resp.reasoning is None - assert resp.tool_calls is None - - -# ── still-working wire golden ───────────────────────────────────────────────── - - -async def test_still_working_wire_golden() -> None: - """still-working: max_tokens=24, temperature=0.9, no tools, thinking pinned off.""" - stub = StubOpenAI() - stub.set_chat_message(content="Still calculating the position...") - llm = _make_spec_llm(stub, "llm") - - messages = [ - ChatMessage(role="system", content="Generate a short still-working message."), - ChatMessage(role="user", content="User request: Add a sphere to my left"), - ] - resp = await llm.chat(messages, max_tokens=24, temperature=0.9) - - body = stub.last_json() - - assert body["model"] == "llm" - assert body["max_tokens"] == 24 - assert body["temperature"] == 0.9 - assert "tools" not in body - assert body["chat_template_kwargs"] == {"enable_thinking": False} - - assert resp.content == "Still calculating the position..." - - -# ── agentic-loop wire golden ────────────────────────────────────────────────── - - -async def test_agentic_loop_wire_golden_thinking_on() -> None: - """agentic-loop with thinking enabled: tools, enable_thinking=True, thinking_budget=1024.""" - stub = StubOpenAI() - stub.set_chat_message(content="Done — sphere added in front of you.") - - agent_llm = _make_spec_llm(stub, "agent_llm") - - tools = [ - ToolDef( - name="add_primitive", - description="Add a primitive object to the scene.", - parameters={ - "type": "object", - "properties": { - "type": {"type": "string"}, - "x": {"type": "number"}, - "y": {"type": "number"}, - "z": {"type": "number"}, - "color": {"type": "string"}, - }, - }, - ), - ToolDef( - name="get_scene_state", - description="Return the current scene objects.", - parameters={"type": "object", "properties": {}}, - ), - ] - - messages = [ - ChatMessage(role="system", content="You are a spatial AI assistant."), - ChatMessage( - role="user", - content="[Pre-fetched context]\nSCENE OBJECTS: (empty)\n\n[Request]\nAdd a blue sphere", - ), - ] - - resp = await agent_llm.chat( - messages, - tools=tools, - max_tokens=2048, - temperature=0.0, - enable_thinking=True, - thinking_budget=1024, - ) - - body = stub.last_json() - - # Model name from nemotron3_nano preset. - assert body["model"] == "llm" - assert body["max_tokens"] == 2048 - assert body["temperature"] == 0.0 - - # Tools must be present in OpenAI wire format. - assert "tools" in body - assert len(body["tools"]) == 2 - tool_names = {t["function"]["name"] for t in body["tools"]} - assert tool_names == {"add_primitive", "get_scene_state"} +_SAMPLE = Path(__file__).resolve().parent.parent / "agent-samples/xr-render-demo" +_PACKAGE = _SAMPLE / "worker/xr_render_demo_worker" - # Thinking kwargs must be present. - assert body.get("chat_template_kwargs") == { - "enable_thinking": True, - "thinking_budget": 1024, - } - # Messages wired correctly. - assert body["messages"][0]["role"] == "system" - assert body["messages"][1]["role"] == "user" +def test_worker_config_loads_sample_yaml() -> None: + config = load_config(_SAMPLE / "yaml/xr_render_demo_worker.yaml") + assert Path(config.models_yaml).exists() - # Response parsing. - assert resp.content == "Done — sphere added in front of you." - assert resp.tool_calls is None +def test_all_agent_modules_export_descriptions() -> None: + from xr_render_demo_worker.agents import appearance, memory, object, placement, vision -async def test_agentic_loop_wire_golden_thinking_off() -> None: - """agentic-loop with thinking off: the preset's wire-level default applies.""" - stub = StubOpenAI() - stub.set_chat_message(content="Done.") - agent_llm = _make_spec_llm(stub, "agent_llm") - - messages = [ - ChatMessage(role="system", content="You are a spatial AI assistant."), - ChatMessage(role="user", content="[Pre-fetched context]\n\n[Request]\nAdd sphere"), + descriptions = [ + module.agent.DESCRIPTION + for module in (appearance, memory, object, placement, vision) ] - await agent_llm.chat( - messages, - tools=[ToolDef(name="add_primitive", description="Add.", parameters={})], - max_tokens=1024, - temperature=0.0, - enable_thinking=False, - ) - - body = stub.last_json() - assert body["max_tokens"] == 1024 - assert body["chat_template_kwargs"] == {"enable_thinking": False} - - -# ── reasoning-field normalization ───────────────────────────────────────────── - - -async def test_agentic_loop_reasoning_field_normalized() -> None: - """nemotron3_nano preset uses reasoning_field='reasoning'; SDK exposes it as ChatResponse.reasoning.""" - stub = StubOpenAI() - stub.set_chat_message( - content="I placed the sphere ahead of you.", - reasoning="RESOLVE: user said 'in front' → forward direction. COMPUTE: pos = head + fwd × 1.5", - reasoning_field="reasoning", # nano_v3 server writes to "reasoning" - ) - - agent_llm = _make_llm(stub, reasoning_field="reasoning") - - resp = await agent_llm.chat( - [ChatMessage(role="user", content="Add a sphere in front")], - ) - - assert resp.reasoning == ( - "RESOLVE: user said 'in front' → forward direction. COMPUTE: pos = head + fwd × 1.5" - ) - assert resp.content == "I placed the sphere ahead of you." - assert resp.tool_calls is None - - -async def test_agentic_loop_tool_calls_parsed() -> None: - """Tool calls in the agentic loop are parsed into ToolCall objects.""" - stub = StubOpenAI() - stub.set_chat_message( - content="", - tool_calls=[{ - "id": "call_abc123", - "type": "function", - "function": { - "name": "add_primitive", - "arguments": '{"type": "sphere", "x": 0.0, "y": 1.6, "z": -1.5}', - }, - }], - finish_reason="tool_calls", - ) - - agent_llm = _make_llm(stub, reasoning_field="reasoning") - resp = await agent_llm.chat( - [ChatMessage(role="user", content="Add sphere ahead")], - tools=[ToolDef(name="add_primitive", description="Add.", parameters={})], - ) - - assert resp.tool_calls is not None - assert len(resp.tool_calls) == 1 - tc = resp.tool_calls[0] - assert tc.id == "call_abc123" - assert tc.name == "add_primitive" - args = json.loads(tc.arguments) - assert args["type"] == "sphere" - assert args["x"] == 0.0 - - -# ── ToolDef.to_openai() round-trip ──────────────────────────────────────────── - - -def test_tool_def_to_openai_wire_shape() -> None: - """ToolDef.to_openai() must produce the exact OpenAI wire shape. - - The SDK re-produces the same shape the pre-migration hand-rolled dicts had - so the upstream server sees byte-identical tool definitions. - """ - td = ToolDef( - name="update_primitive", - description="Update an existing object.", - parameters={ - "type": "object", - "properties": { - "id": {"type": "string"}, - "x": {"type": "number"}, - }, - }, - ) - wire = td.to_openai() - assert wire == { - "type": "function", - "function": { - "name": "update_primitive", - "description": "Update an existing object.", - "parameters": { - "type": "object", - "properties": { - "id": {"type": "string"}, - "x": {"type": "number"}, - }, - }, - }, - } - - -# ── XR-launch-failure notice delivery (yield → TTS + _send → panel) ──────────── -# -# When start_xr / the LOVR-spawn poll fails, RenderDemoAgent calls -# RenderSceneProcessor.enqueue_notice(pid, msg). The notice must be delivered -# with the SAME shape as a normal final answer: spoken (yielded → TextFrame at -# the TTS-facing sink) AND on the agent.response data topic (panel). The notice -# path runs no LLM/MCP, so the brain is built with None clients and the real -# prompt files — only the transport is faked to capture _send. - -_PROMPTS_DIR = _WORKER_DIR / "prompts" -_SYSTEM_PROMPT = _PROMPTS_DIR / "system.txt" - -_LAUNCH_FAIL_MSG = "I couldn't start the XR session — try Launch XR again." - - -class _CaptureSink(FrameProcessor): - """Tail processor — collects every downstream frame it sees.""" - - def __init__(self) -> None: - super().__init__(enable_direct_mode=True) - self.frames: list[Frame] = [] - - async def process_frame(self, frame: Frame, direction: FrameDirection) -> None: - await super().process_frame(frame, direction) - self.frames.append(frame) - await self.push_frame(frame, direction) - - -class _CaptureTransport: - """XRMediaHubTransport double — records send_return_data and owns the - target participant. Only the surface the notice path touches.""" - - def __init__(self) -> None: - self.target_participant = "" - self.sent: list[DataMessage] = [] - - def set_target_participant(self, pid: str) -> None: - self.target_participant = pid - - async def send_return_data(self, msg: DataMessage) -> None: - self.sent.append(msg) - - -class _UnusedToolbox: - async def invoke(self, name: str, arguments: dict): - raise AssertionError(f"unexpected native tool invocation: {name} {arguments}") - - -def _make_brain(transport: _CaptureTransport, llm=None): - """Build a real RenderSceneProcessor whose service clients are unused. - - The notice path (enqueue_notice → handle_query short-circuit → - _emit_notice) never dereferences them. The constructor eagerly reads - the real prompt files, so point at the bundled prompts/ directory. - Pass ``llm`` to exercise the real _quick_ack parse paths against a stub. - """ - return _proc.RenderSceneProcessor( - transport = transport, - cfg = None, - toolbox = _UnusedToolbox(), - release_vision = lambda _pid: None, - text_memory = None, - prompt_path = _SYSTEM_PROMPT, - tools = [], - llm = llm, - agent_llm = None, - ) - - -async def _drive_notice(brain, transport: _CaptureTransport, - pid: str, msg: str) -> _CaptureSink: - """Run brain → sink in a PipelineWorker and call enqueue_notice once the - pipeline has started, then drain with EndFrame. Returns the sink.""" - sink = _CaptureSink() - pipeline = Pipeline([brain, sink]) - worker = PipelineWorker( - pipeline, cancel_on_idle_timeout=False, enable_rtvi=False, - ) - runner = WorkerRunner() - await runner.add_workers(worker) - - async def drive() -> None: - # Let StartFrame propagate before injecting the notice. - await asyncio.sleep(0.05) - await brain.enqueue_notice(pid, msg) - await asyncio.sleep(0.15) - await worker.queue_frame(EndFrame()) - - await asyncio.gather(runner.run(), drive()) - return sink - - -async def test_launch_failure_notice_spoken_and_paneled() -> None: - """enqueue_notice delivers BOTH: a yielded TextFrame (→ TTS) and an - agent.response data message (→ panel), routed to the originating pid. - - Because agent_llm is None, if the query had wrongly fallen through to - the agentic loop it would have returned "Done." — so seeing the exact - notice string at the sink proves the LLM loop was skipped. - """ - transport = _CaptureTransport() - transport.set_target_participant("pid-1") - brain = _make_brain(transport) - - sink = await _drive_notice(brain, transport, "pid-1", _LAUNCH_FAIL_MSG) - - # Spoken: exactly one TextFrame carrying the notice verbatim. - texts = [f.text for f in sink.frames if isinstance(f, TextFrame)] - assert texts == [_LAUNCH_FAIL_MSG] - - # Panel: exactly one agent.response send to the originating pid. - assert len(transport.sent) == 1 - sent = transport.sent[0] - assert sent.topic == "agent.response" - assert sent.participant_id == "pid-1" - assert sent.data.decode() == _LAUNCH_FAIL_MSG - # No brain.close() — that closes the (None) LLM clients; the notice - # path never opened them. - - -async def test_pending_notice_not_consumed_by_real_query() -> None: - """Exact-text match: a real query that interleaves before the notice - task runs must NOT be mistaken for the pending notice. The pending - entry survives a non-matching handle_query; the matching one drains it. - - handle_query returns the generator without iterating, so no LLM fires — - we only assert on _pending_notices bookkeeping here. - """ - transport = _CaptureTransport() - transport.set_target_participant("pid-1") - brain = _make_brain(transport) - - # Register a pending notice the way enqueue_notice does, without spawning. - brain._pending_notices.setdefault("pid-1", []).append(_LAUNCH_FAIL_MSG) - - # A real, different query for the same pid must not consume the notice. - await brain.handle_query("pid-1", "move the cube left", False) - assert brain._pending_notices.get("pid-1") == [_LAUNCH_FAIL_MSG] - - # The matching text drains it. - await brain.handle_query("pid-1", _LAUNCH_FAIL_MSG, False) - assert "pid-1" not in brain._pending_notices - - -async def test_quick_ack_spoken_on_non_thinking_turn() -> None: - """ACK-SPEAK POLICY: the quick-ack is yielded (→ TTS) on EVERY turn, - including a non-thinking one, so a tool-using turn is never silent until - the final reply. Pre-change the ack was spoken only when needs_thinking. - - _quick_ack and _agentic_loop are stubbed so no LLM/MCP client is touched. - """ - transport = _CaptureTransport() - transport.set_target_participant("pid-1") - brain = _make_brain(transport) - - async def _fake_quick_ack(_text): - return ("On it.", False) # ack present, needs_thinking = False - - async def _fake_loop(*_a, **_k): - return "All set." - - brain._quick_ack = _fake_quick_ack # noqa: SLF001 - brain._agentic_loop = _fake_loop # noqa: SLF001 - - gen = await brain.handle_query("pid-1", "place a cube", False) - spoken = [s async for s in gen] - - # Ack is spoken first (so the turn isn't silent), then the final reply. - assert spoken and spoken[0] == "On it." - assert "All set." in spoken - # Ack is also mirrored to the panel on agent.progress. - progress = [m for m in transport.sent if m.topic == "agent.progress"] - assert any(m.data.decode() == "On it." for m in progress) - - -def test_tool_result_json_is_sanitized() -> None: - """A final response that is nothing but a JSON object (e.g. an echoed - tool result) must be flagged so it never reaches TTS; prose that merely - contains JSON passes.""" - from tooling import looks_like_leaked_tool_call - - assert looks_like_leaked_tool_call('{"id": "box-1", "ok": true, "reason": null}') - assert looks_like_leaked_tool_call('[{"id": "box-1", "ok": true}]') - assert not looks_like_leaked_tool_call('Added box-1 ({"ok": true} from the scene).') - assert not looks_like_leaked_tool_call("Added a blue sphere.") - - -async def test_quick_ack_parses_wellformed_json() -> None: - """_quick_ack returns the ack string and a strict-bool think flag.""" - stub = StubOpenAI() - stub.set_chat_message(content='{"ack": "On it", "think": true}') - brain = _make_brain(_CaptureTransport(), llm=_make_spec_llm(stub, "llm")) - assert await brain._quick_ack("move the cube") == ("On it", True) # noqa: SLF001 - - -async def test_quick_ack_string_think_is_not_truthy() -> None: - """A model emitting "think": "false" (a string) must not enable thinking.""" - stub = StubOpenAI() - stub.set_chat_message(content='{"ack": "On it", "think": "false"}') - brain = _make_brain(_CaptureTransport(), llm=_make_spec_llm(stub, "llm")) - assert await brain._quick_ack("move the cube") == ("On it", False) # noqa: SLF001 - - -async def test_quick_ack_truncated_json_not_spoken() -> None: - """A truncated JSON payload has no closing brace; the raw fragment must - not be returned as a speakable ack.""" - stub = StubOpenAI() - stub.set_chat_message(content='{"ack": "Let me ta') - brain = _make_brain(_CaptureTransport(), llm=_make_spec_llm(stub, "llm")) - assert await brain._quick_ack("what am I holding") == ("", False) # noqa: SLF001 - - -async def test_quick_ack_bare_prose_fallback() -> None: - """Non-JSON prose is used as the ack with thinking off.""" - stub = StubOpenAI() - stub.set_chat_message(content="On it") - brain = _make_brain(_CaptureTransport(), llm=_make_spec_llm(stub, "llm")) - assert await brain._quick_ack("add a sphere") == ("On it", False) # noqa: SLF001 - - -async def test_quick_ack_transport_error_falls_back_silent_fast() -> None: - """LLM failure → no ack, thinking off (fail toward the tool-trusting mode).""" - - class _BoomLLM: - async def chat(self, *_a, **_k): - raise TimeoutError - - brain = _make_brain(_CaptureTransport(), llm=_BoomLLM()) - assert await brain._quick_ack("move it up") == ("", False) # noqa: SLF001 - - -async def test_already_punctuated_ack_not_doubled() -> None: - """An ack ending in !/? passes through unchanged (no "On it!.").""" - transport = _CaptureTransport() - transport.set_target_participant("pid-1") - brain = _make_brain(transport) - - async def _fake_quick_ack(_text): - return ("On it!", False) - - async def _fake_loop(*_a, **_k): - return "Done." - - brain._quick_ack = _fake_quick_ack # noqa: SLF001 - brain._agentic_loop = _fake_loop # noqa: SLF001 - - gen = await brain.handle_query("pid-1", "add a cube", False) - spoken = [s async for s in gen] - assert spoken == ["On it!", "Done."] - - -async def test_empty_ack_yields_no_spoken_line() -> None: - """The quick-ack failure fallback ("", False) must not yield an empty - ack line or post an empty progress message; the final reply still lands.""" - transport = _CaptureTransport() - transport.set_target_participant("pid-1") - brain = _make_brain(transport) - - async def _fake_quick_ack(_text): - return ("", False) - - async def _fake_loop(*_a, **_k): - return "All set." - - brain._quick_ack = _fake_quick_ack # noqa: SLF001 - brain._agentic_loop = _fake_loop # noqa: SLF001 - - gen = await brain.handle_query("pid-1", "add a cube", False) - spoken = [s async for s in gen] - assert spoken == ["All set."] - assert not [m for m in transport.sent if m.topic == "agent.progress"] - - -async def test_unpunctuated_ack_gets_terminal_period() -> None: - """Acks without terminal punctuation are normalized before being yielded - to TTS; the panel copy stays verbatim.""" - transport = _CaptureTransport() - transport.set_target_participant("pid-1") - brain = _make_brain(transport) - - async def _fake_quick_ack(_text): - return ("Let me take a look", True) - - async def _fake_loop(*_a, **_k): - return "Done." - - brain._quick_ack = _fake_quick_ack # noqa: SLF001 - brain._agentic_loop = _fake_loop # noqa: SLF001 - - gen = await brain.handle_query("pid-1", "what am I holding", False) - spoken = [s async for s in gen] - - assert spoken[0] == "Let me take a look." - progress = [m for m in transport.sent if m.topic == "agent.progress"] - assert any(m.data.decode() == "Let me take a look" for m in progress) - - -# ── live-frame perception routing (look_at_current_frame) ────────────────────── -# -# A real-world visual question — "what colour is this thing I'm holding?" — must -# reach the LIVE-FRAME VLM path, not stall in the reasoning loop. Before the fix -# the render-demo worker had no frame tracking and never turned the camera on, so -# a perception query looped on an unanswerable tool (get_frame_from_time, which -# isn't even registered when recording is disabled) and hung. These tests stub -# the VLM client + the hub frame path and assert the routing mechanically. - -from xr_ai_hub import FrameData, FrameSignal, PixelFormat # noqa: E402 -from xr_ai_models import ChatResponse, ToolCall # noqa: E402 - -import processors as _proc # noqa: E402 -import capabilities as _caps # noqa: E402 -from xr_ai_nat.functions.spatial_math import SpatialMathFunctionsConfig # noqa: E402 -from xr_ai_nat.functions.xr_tracking import XRTrackingFunctionsConfig # noqa: E402 - - -async def test_render_spatial_native_toolbox_builds() -> None: - """The sample's prompt-compatible spatial surface derives from NAT Functions.""" - async with WorkflowBuilder() as builder: - await builder.add_function_group( - "tracking", - XRTrackingFunctionsConfig(endpoint="tcp://127.0.0.1:65530", timeout_s=0.1), - ) - await builder.add_function_group("spatial_math", SpatialMathFunctionsConfig()) - await builder.add_function_group("render_spatial", _caps.RenderSpatialToolsConfig()) - group = await builder.get_function_group("render_spatial") - toolbox = _caps.NativeToolbox(await group.get_all_functions()) - - definitions = {tool.name: tool for tool in toolbox.definitions()} - expected_parameters = { - "along_direction": { - "origin_x", "origin_y", "origin_z", "target_x", "target_y", "target_z", "distance", - }, - "between_anchors": {"a_x", "a_y", "a_z", "b_x", "b_y", "b_z"}, - "displace_object": {"current_x", "current_y", "current_z", "right", "up", "forward"}, - "displace_objects": { - "object_ids", "current_xs", "current_ys", "current_zs", "right", "up", "forward", - }, - "get_head_pose": set(), - "place_inside_by_id": {"movee_id", "container_x", "container_y", "container_z"}, - "place_object_relative": {"origin_x", "origin_y", "origin_z", "direction", "distance"}, - "place_user_relative": {"direction", "distance"}, - "position_ahead": {"distance"}, - "position_relative": { - "forward", "right", "up", "origin_x", "origin_y", "origin_z", - }, - "scale_value": {"current", "factor"}, - "world_offset": {"origin_x", "origin_y", "origin_z", "dx", "dy", "dz"}, - } - assert set(definitions) == set(expected_parameters) - for name, parameters in expected_parameters.items(): - assert set(definitions[name].parameters["properties"]) == parameters - - -async def test_live_worker_and_eval_share_native_toolbox_assembly() -> None: - """The shared builder exposes the complete runtime tool surface without MCP discovery.""" - async with WorkflowBuilder() as builder: - toolbox, _vision_config = await _caps.build_native_toolbox( - builder, - scene_endpoint="tcp://127.0.0.1:65527", - openxr_endpoint="tcp://127.0.0.1:65528", - video_memory_endpoint="tcp://127.0.0.1:65529", - frame_endpoint=_FakeEndpoint(), - vlm=_FakeVLM(), - ) - names = {tool.name for tool in toolbox.definitions()} - - assert names == { - "add_primitive", - "along_direction", - "between_anchors", - "displace_object", - "displace_objects", - "get_frame_from_time", - "get_head_pose", - "get_health", - "get_scene_state", - "get_video_stats", - "list_recorded_participants", - "look_at_current_frame", - "look_at_past_frame", - "place_inside_by_id", - "place_object_relative", - "place_user_relative", - "position_ahead", - "position_relative", - "query_video", - "remove_primitive", - "scale_value", - "start_xr", - "update_primitive", - "world_offset", - } - - -async def test_model_facing_perception_schema_is_trimmed() -> None: - """The perception tools reach the model with participant/reference context - stripped. The worker injects ``participant_id`` (and ``reference_time_us`` - for recorded lookups); exposing them verbatim would tell the model to fill a - required ``participant_id`` it cannot know and whose value is discarded. - Guards the do-not-reverse of main's trimmed ``{question}`` contract.""" - from xr_render_demo_worker import _WORKER_MANAGED_TOOLS - - async with WorkflowBuilder() as builder: - toolbox, _vision_config = await _caps.build_native_toolbox( - builder, - scene_endpoint="tcp://127.0.0.1:65527", - openxr_endpoint="tcp://127.0.0.1:65528", - video_memory_endpoint="tcp://127.0.0.1:65529", - frame_endpoint=_FakeEndpoint(), - vlm=_FakeVLM(), - ) - # The raw native request schemas DO expose the injected context — which is - # exactly why they must not reach the model verbatim. - native = {tool.name: tool for tool in toolbox.definitions()} - assert "participant_id" in native["look_at_current_frame"].parameters["properties"] - assert "participant_id" in native["look_at_past_frame"].parameters["properties"] - - # Assemble the model-facing list exactly as the worker does. - tools = toolbox.definitions( - exclude=_WORKER_MANAGED_TOOLS - | {_proc._LIVE_PERCEPTION_TOOL, _proc._PAST_PERCEPTION_TOOL} - ) - tools.extend(_proc._PERCEPTION_TOOL_DEFS) - - model_facing = {tool.name: tool for tool in tools} - live = model_facing["look_at_current_frame"].parameters - past = model_facing["look_at_past_frame"].parameters - assert set(live["properties"]) == {"question"} - assert live["required"] == ["question"] - assert set(past["properties"]) == {"question", "second_ago"} - assert set(past["required"]) == {"question", "second_ago"} - # No injected context leaks to the model. - for schema in (live, past): - assert "participant_id" not in schema["properties"] - assert "reference_time_us" not in schema["properties"] - - -class _FakeEndpoint: - """Hub ProcessorEndpoint double — frame callback, pixel request, status, and - return-data send. Native vision functions acquire frames through this endpoint; - the transport delegates return-data sends to it, so camera-control messages are - recorded in the shared ``sent`` list.""" - - def __init__(self, sent: list[DataMessage] | None = None) -> None: - self.frame_cbs: list = [] - self.frame: FrameData | None = None - self.frame_requests: list[FrameSignal] = [] - self.statuses: list[tuple[str, str]] = [] - self.sent: list[DataMessage] = sent if sent is not None else [] - - def on_frame(self, cb) -> None: - self.frame_cbs.append(cb) - - def on_participant(self, _cb) -> None: - pass - - async def request_frame(self, sig: FrameSignal, timeout: float = 0.0): - self.frame_requests.append(sig) - return self.frame - - async def set_status(self, status: str, pid: str | None = None) -> None: - self.statuses.append((status, pid or "")) - - async def send_return_data(self, msg: DataMessage) -> None: - self.sent.append(msg) - - -class _CaptureTransportWithEndpoint(_CaptureTransport): - """Capture transport that also exposes a fake hub endpoint so the brain - can register its frame callback and pull pixels. The endpoint shares this - transport's ``sent`` list so endpoint sends show up in ``transport.sent``.""" - - def __init__(self) -> None: - super().__init__() - self.endpoint = _FakeEndpoint(sent=self.sent) - - -class _FakeVLM: - """VLMService double — records the ask_image call and returns a canned - ChatResponse so we can assert the perception path reached the VLM.""" - - def __init__(self, answer: str = "It's a red mug.") -> None: - self.answer = answer - self.calls: list[tuple[str, str]] = [] - - async def ask_image(self, image, question, *, system_prompt: str = "", - **_kw) -> ChatResponse: - self.calls.append((image, question)) - return ChatResponse( - content=self.answer, reasoning=None, tool_calls=None, - finish_reason="stop", raw={}, - ) - - async def close(self) -> None: - pass - - -def _rgb_frame(pid: str, *, w: int = 4, h: int = 4) -> tuple[FrameSignal, FrameData]: - """A tiny solid-colour RGB24 frame + its matching signal for *pid*.""" - pts = _now_us_test() - data = bytes([200, 30, 30]) * (w * h) # solid red - sig = FrameSignal( - slot=0, seq=1, pts_us=pts, width=w, height=h, - fmt=PixelFormat.RGB24, data_sz=len(data), participant_id=pid, - ) - fd = FrameData( - seq=1, pts_us=pts, width=w, height=h, - fmt=PixelFormat.RGB24, data=data, participant_id=pid, - ) - return sig, fd - - -def _now_us_test() -> int: - import time as _t - return _t.time_ns() // 1_000 - - -@asynccontextmanager -async def _perception_brain(transport, vlm: _FakeVLM): - config = VisionToolsConfig( - endpoint=transport.endpoint, - vlm=vlm, - video_memory=FunctionGroupRef("video_memory"), - frame_max_age_s=60.0, - frame_timeout_s=0.2, - ) - async with WorkflowBuilder() as builder: - # A resolvable (offline) video-memory group so the vision group builds; - # look_at_current_frame never touches it. - await builder.add_function_group( - "video_memory", - VideoMemoryFunctionsConfig(endpoint="tcp://127.0.0.1:65529"), - ) - await builder.add_function_group("vision", config) - vision = await builder.get_function_group("vision") - toolbox = _caps.NativeToolbox(await vision.get_all_functions()) - yield _proc.RenderSceneProcessor( - transport=transport, - cfg=None, - toolbox=toolbox, - release_vision=config.release, - text_memory=None, - prompt_path=_SYSTEM_PROMPT, - tools=[], - llm=None, - agent_llm=None, - ) - - -def test_perception_tool_def_in_prompt_and_classifier() -> None: - """The perception tool is named in the system prompt, and the quick-ack - classifier treats camera lookups as tool-settled (think=false territory): - thinking is reserved for requests no tool pattern covers.""" - prompt = _SYSTEM_PROMPT.read_text(encoding="utf-8") - assert "look_at_current_frame" in prompt - ack = (_PROMPTS_DIR / "quick_ack.txt").read_text(encoding="utf-8").lower() - assert "default is false" in ack and "camera" in ack - - -async def test_perception_query_reaches_vlm_frame_path() -> None: - """A vision question routed to look_at_current_frame pulls the current - always-on live frame and runs the VLM — returning the VLM answer to the - loop (NOT a generic reasoning-loop fallback).""" - transport = _CaptureTransportWithEndpoint() - transport.set_target_participant("pid-1") - vlm = _FakeVLM(answer="It's a red mug.") - async with _perception_brain(transport, vlm) as brain: - sig, fd = _rgb_frame("pid-1") - for cb in transport.endpoint.frame_cbs: - await cb(sig) - transport.endpoint.frame = fd - - result = await brain._execute_tool( # noqa: SLF001 - "look_at_current_frame", - {"question": "What colour is this thing I'm holding?"}, - pid="pid-1", - ) - - # Reached the VLM with the encoded frame + the question. - assert len(vlm.calls) == 1 - image, question = vlm.calls[0] - assert image.startswith("data:image/jpeg;base64,") - assert "colour" in question - # The pixel request used the seeded live frame. - assert transport.endpoint.frame_requests == [sig] - # The VLM answer is returned to the loop, not a generic fallback. - assert result == {"answer": "It's a red mug."} - - -async def test_perception_unavailable_frame_ends_turn_gracefully() -> None: - """A failed live-vision invocation ends the turn via the graceful no-frame path. - - The native ``look_at_current_frame`` raises (no frame / no VLM answer); the - processor converts any failure into a ``_PerceptionUnavailableError`` carrying - the short spoken message rather than feeding an error back to the model.""" - transport = _CaptureTransport() - brain = _make_brain(transport) # _UnusedToolbox.invoke raises on call - - with pytest.raises(_proc._PerceptionUnavailableError) as excinfo: - await brain._look_at_current_frame("pid-1", "What is shown?") # noqa: SLF001 - - assert excinfo.value.spoken == _proc._NO_FRAME_MSG - - -def _stub_turn(brain, loop) -> None: - """Stub the LLM-driven parts so _run_turn exercises only the status bracket.""" - async def _ack(_text): - return "", False - brain._quick_ack = _ack # noqa: SLF001 - brain._agentic_loop = loop # noqa: SLF001 - - -async def test_run_turn_brackets_client_status_processing_then_idle() -> None: - """The render turn owns the per-client UI status: 'processing' at entry and - 'idle' when it ends. (Native vision functions never emit status.)""" - transport = _CaptureTransportWithEndpoint() - transport.set_target_participant("pid-1") - brain = _make_brain(transport) - - async def _loop(_text, _pid, *, ref_us, needs_thinking, thinking_ctx): - return "All set." - - _stub_turn(brain, _loop) - async for _ in brain._run_turn("pid-1", "add a red sphere"): # noqa: SLF001 - pass - - assert transport.endpoint.statuses == [("processing", "pid-1"), ("idle", "pid-1")] - - -async def test_run_turn_status_clears_on_failure() -> None: - """'idle' still fires when the turn fails (finally path).""" - transport = _CaptureTransportWithEndpoint() - transport.set_target_participant("pid-1") - brain = _make_brain(transport) - - async def _boom(_text, _pid, *, ref_us, needs_thinking, thinking_ctx): - raise RuntimeError("loop failed") - - _stub_turn(brain, _boom) - async for _ in brain._run_turn("pid-1", "q"): # noqa: SLF001 - pass - - assert transport.endpoint.statuses == [("processing", "pid-1"), ("idle", "pid-1")] - - -async def test_run_turn_status_clears_on_barge_in_cancellation() -> None: - """A barge-in cancels the turn; 'idle' must still fire and the - CancelledError must propagate.""" - transport = _CaptureTransportWithEndpoint() - transport.set_target_participant("pid-1") - brain = _make_brain(transport) - - async def _cancel(_text, _pid, *, ref_us, needs_thinking, thinking_ctx): - raise asyncio.CancelledError - - _stub_turn(brain, _cancel) - with pytest.raises(asyncio.CancelledError): - async for _ in brain._run_turn("pid-1", "q"): # noqa: SLF001 - pass - - assert transport.endpoint.statuses == [("processing", "pid-1"), ("idle", "pid-1")] - - -async def test_run_turn_status_clears_when_cancelled_during_initial_publish() -> None: - """If a barge-in lands while the initial 'processing' publish is still in - flight, the turn must still clear to 'idle' — the publish sits inside the - protected region, so the finally always runs.""" - transport = _CaptureTransportWithEndpoint() - transport.set_target_participant("pid-1") - brain = _make_brain(transport) - - in_processing = asyncio.Event() - statuses = transport.endpoint.statuses - - async def _blocking_status(status, pid=None): - statuses.append((status, pid or "")) - if status == "processing": - in_processing.set() - await asyncio.sleep(3600) # hold the publish open until cancelled - - transport.endpoint.set_status = _blocking_status # type: ignore[assignment] - - async def _loop(_text, _pid, *, ref_us, needs_thinking, thinking_ctx): - return "unreached" + assert all(isinstance(text, str) and text for text in descriptions) + assert len(set(descriptions)) == len(descriptions) - _stub_turn(brain, _loop) - async def _run() -> None: - async for _ in brain._run_turn("pid-1", "q"): # noqa: SLF001 - pass +def test_prompt_files_exist_and_are_nonempty() -> None: + prompts = sorted(_PACKAGE.rglob("*prompt*.txt")) + assert len(prompts) == 6 # supervisor + five subagents + for prompt in prompts: + assert prompt.read_text(encoding="utf-8").strip(), prompt - task = asyncio.create_task(_run()) - await in_processing.wait() # cancellation now lands mid-'processing'-publish - task.cancel() - with pytest.raises(asyncio.CancelledError): - await task - assert ("idle", "pid-1") in statuses +def test_models_round_trip() -> None: + request = SceneRequest(transcript="hi", participant_id="p", timestamp_us=1) + assert SceneRequest.model_validate(request.model_dump()) == request + task = SubagentTask(instruction="do", participant_id="p", reference_time_us=1) + assert SubagentTask.model_validate(task.model_dump()) == task + assert SceneReply(response="ok").response == "ok" + assert SubagentResult(result="ok").result == "ok" -async def test_agentic_loop_reports_agent_llm_failure() -> None: - """A failed agent-LLM request must not fall through to a success reply.""" - transport = _CaptureTransport() - brain = _make_brain(transport) +def test_supervisor_ledger_field_is_declared() -> None: + # A supervisor-owned turn ledger silently dropped by pydantic is dead + # code; pin that the config actually declares the field. + from xr_render_demo_worker.agents.object.agent import ObjectAgentConfig - async def _context(_pid: str, *, ref_us: int) -> str: - return "scene context" + assert "ledger" in ObjectAgentConfig.model_fields - class _FailingAgentLLM: - async def chat(self, *_args, **_kwargs): - raise RuntimeError("backend unavailable") - brain._build_turn_context = _context # noqa: SLF001 - brain._agent_llm = _FailingAgentLLM() # noqa: SLF001 +def test_prompt_audit_is_clean() -> None: + import io + from contextlib import redirect_stdout - answer = await brain._agentic_loop("add a cube", "pid-1") # noqa: SLF001 + from xr_render_demo_eval import harness - assert answer == "Something went wrong — please try again." + buffer = io.StringIO() + with redirect_stdout(buffer): + harness.audit_prompts() + warnings = [line for line in buffer.getvalue().splitlines() if line.startswith("AUDIT WARNING")] + assert not warnings, warnings -async def test_perception_no_frame_yields_graceful_message() -> None: - """When no live camera frame can be obtained, the perception turn ends with - a short spoken+panel message — never a hang or a silent failure. +@pytest.mark.parametrize("entry", ["xr_render_demo_worker.__main__"]) +def test_entry_module_imports(entry: str) -> None: + __import__(entry) - Driven through _agentic_loop so the full graceful path is exercised: - look_at_current_frame → _PerceptionUnavailableError → the loop returns the - spoken message (which _run_turn then speaks and panels).""" - transport = _CaptureTransportWithEndpoint() - transport.set_target_participant("pid-1") - vlm = _FakeVLM() - async def _fake_call_tool(_tool, _args, *, silent=False): - return {} - call_count = {"n": 0} +def test_truncation_replies_resolve() -> None: + from xr_render_demo_worker.supervisor import _resolve_truncation_reply, _splice_completion - async def _fake_chat(messages, **kwargs): - call_count["n"] += 1 - return ChatResponse( - content="", - reasoning=None, - tool_calls=[ToolCall( - id="call_look", - name="look_at_current_frame", - arguments='{"question": "What colour is this?"}', - )], - finish_reason="tool_calls", - raw={}, - ) + assert _splice_completion("Put the sphere on the", "On the box.") == "Put the sphere on the box." + assert _splice_completion("Put the sphere on the", "The box.") == "Put the sphere on the box." + assert _splice_completion("Move it towards", "The window.") == "Move it towards The window." + assert _resolve_truncation_reply("Put the sphere on the", "Never mind.") is None + assert _resolve_truncation_reply("Put the sphere on the", "Cancel") is None + fresh = _resolve_truncation_reply("Put the sphere on the", "Put the sphere on the box.") + assert fresh == "Put the sphere on the box." - class _LLM: - async def chat(self, messages, **kw): - return await _fake_chat(messages, **kw) - async with _perception_brain(transport, vlm) as brain: - brain._call_tool = _fake_call_tool # noqa: SLF001 - brain._agent_llm = _LLM() # noqa: SLF001 - answer = await brain._agentic_loop( # noqa: SLF001 - "what colour is this thing I'm holding?", "pid-1", - ref_us=_now_us_test(), needs_thinking=True, thinking_ctx=[""], - ) +def test_truncated_transcripts_detected() -> None: + from xr_render_demo_worker.supervisor import _is_truncated, _truncated_reply - # Graceful spoken message, not a hang or a generic "Done." fallback. - assert answer == _proc._NO_FRAME_MSG - # Camera is always-on streaming — no startCamera/stopCamera messages sent. - controls = [m for m in transport.sent if m.topic == "clientControl"] - assert not any(b'"startCamera"' in m.data for m in controls) - # VLM was never reached — there was no frame to ask about. - assert vlm.calls == [] + assert _truncated_reply("Put the sphere on the").endswith("On the what?") + assert _truncated_reply("Move it towards").endswith("Towards what?") + assert _is_truncated("Put the sphere on the") + assert _is_truncated("Move it towards") + assert _is_truncated("Make it the same color as my") + assert not _is_truncated("Put the sphere on the box.") + assert not _is_truncated("Move it up") + assert not _is_truncated("Okay.") + assert not _is_truncated("") diff --git a/tests/test_xr_tracking_functions.py b/tests/test_xr_tracking_functions.py index ed1c30be..3d4227fe 100644 --- a/tests/test_xr_tracking_functions.py +++ b/tests/test_xr_tracking_functions.py @@ -187,3 +187,50 @@ def test_pose_conversion_preserves_openxr_axes( assert result["yaw_deg"] == yaw_deg assert result["pitch_deg"] == pitch_deg assert isinstance(result["ts"], int) + +_SIM_POSE = { + "position": {"x": 2.0, "y": 1.6, "z": 1.5}, + "forward": {"x": 0.0, "y": 0.0, "z": -1.0}, + "right": {"x": 1.0, "y": 0.0, "z": 0.0}, + "up": {"x": 0.0, "y": 1.0, "z": 0.0}, + "yaw_deg": 0.0, + "pitch_deg": 0.0, + "ts": 1, +} + + +@pytest.mark.asyncio +async def test_openxr_sim_pose_rejected_when_disabled() -> None: + class Source: + def get_pose(self) -> dict: + return {"is_valid": True, "ts": 10} + + def health(self) -> dict: + return {"status": "ok"} + + service = OpenXRService(Source()) + with pytest.raises(RPCError) as error: + await service.dispatch("set_sim_pose", _SIM_POSE) + assert error.value.code == "unknown_operation" + + +@pytest.mark.asyncio +async def test_openxr_sim_pose_overrides_and_clears() -> None: + hardware = {"is_valid": True, "ts": 10} + + class Source: + def get_pose(self) -> dict: + return hardware + + def health(self) -> dict: + return {"status": "ok"} + + service = OpenXRService(Source(), allow_sim_pose=True) + assert await service.dispatch("set_sim_pose", _SIM_POSE) == {"ok": True} + overridden = await service.dispatch("get_head_pose", {}) + assert overridden["position"] == _SIM_POSE["position"] + assert await service.dispatch("clear_sim_pose", {}) == {"ok": True} + assert await service.dispatch("get_head_pose", {}) is hardware + with pytest.raises(RPCError) as error: + await service.dispatch("set_sim_pose", {"position": "garbage"}) + assert error.value.code == "invalid_request"