Migrate XR render sample to NAT and improve eval

AGENTS.md

@@ -232,12 +232,18 @@ and new config files.
 When a sample's behaviour is driven by an LLM prompt (e.g.
 `agent-samples/xr-render-demo/`):
 
-- When you add or change a rule in `system.txt`, add or update a case
-  in the sample's `eval/` harness in the same edit. A rule without a
-  case is unverified.
+- When you add or change a rule in any agent prompt (xr-render-demo:
+  `supervisor_prompt.txt` and `agents/*/prompt.txt`), add or update a
+  case in the sample's `eval/` harness in the same edit. A rule without
+  a case is unverified.
+- **Run the cheap gate after every prompt or ops edit** (xr-render-demo:
+  `uv run xr_render_demo_eval basics`). Prompt edits contaminate
+  neighboring behaviors, and full-suite variance hides single-case
+  damage; debug at the lowest tier that reproduces a failure and rerun
+  the tiers above it before calling a fix done.
 - **Don't train on the test set.** Don't reuse a prompt's worked-
-  example specifics (coordinates, colors, shapes, trigger phrases) in
-  a case fixture, or vice versa — that makes the eval a memorization
+  example specifics (coordinates, colors, shapes, ids, trigger phrases)
+  in a case fixture, or vice versa — that makes the eval a memorization
   check. The harness audits this at startup and warns; clear the
   warning by changing the prompt example, not the case.
 

DEPENDENCIES.md

@@ -337,6 +337,8 @@ xr-ai-tests  (tests/)
     └── vlm-mcp-server          [editable: ../agent-mcp-servers/vlm-mcp]
     └── render-mcp              [editable: ../agent-mcp-servers/render-mcp]
     └── xr-render-scene         [editable: ../agent-samples/xr-render-demo/scene]
+    └── xr-render-demo-worker   [editable: ../agent-samples/xr-render-demo/worker]
+    └── xr-render-demo-eval     [editable: ../agent-samples/xr-render-demo/eval]
     └── video-mcp-server        [editable: ../agent-mcp-servers/video-mcp]
     └── vec-mcp-server          [editable: ../agent-mcp-servers/vec-mcp]
     └── pytest >=8.0
@@ -590,12 +592,13 @@ user-relative requests such as "to my left".
 |---|---|---|---|
 | Orchestrator | `xr-render-demo` | `xr-ai-launcher`, `xr-ai-logging` | loguru >=0.7 |
 | Scene | `xr-render-scene` | `xr-ai-launcher`, `xr-ai-logging`, `xr-ai-nat` | pyzmq >=27.0, msgpack >=1.0, pyyaml >=6.0 |
-| Worker | `xr-render-demo-worker` | `xr-ai-hub-client`, `xr-ai-models` [editable], `xr-ai-nat[services,vision]` [editable], `xr-ai-pipecat` [editable], `xr-ai-voicegate` [editable], `xr-ai-logging` [editable], `xr-render-scene` [editable] | pyyaml >=6.0, pipecat-ai >=1.3 (native scene, tracking, spatial-math, video-memory, vision, and text-memory functions replace capability MCP clients; silero-vad via xr-ai-pipecat → xr-ai-vad). |
+| Worker | `xr-render-demo-worker` | `xr-ai-hub-client`, `xr-ai-models` [editable], `xr-ai-nat[agents,services,vision,voice]` [editable], `xr-ai-voice` [editable], `xr-ai-voicegate` [editable], `xr-ai-logging` [editable], `xr-render-scene` [editable] | loguru >=0.7, pyyaml >=6.0 (native scene, tracking, spatial-math, video-memory, vision, and text-memory functions replace capability MCP clients; voice runtime via xr-ai-voice). |
+| Eval | `xr-render-demo-eval` | `xr-render-demo-worker` [editable], `xr-render-scene` [editable], `xr-ai-models` [editable], `xr-ai-nat[agents,services]` [editable] | pyyaml >=6.0 (offline tiers over faked leaf functions; live drivers join the running stack over the hub IPC endpoints). |
 
 Model endpoints (llm, agent_llm, stt, tts, vlm) are declared in
 `yaml/models.yaml` and loaded via `xr-ai-models` `load_models_config` /
 `make_llm` / `make_stt` / `make_tts` / `make_vlm`.  `httpx` is retained as
-a transitive dep of `xr-ai-pipecat` and `xr-ai-nat[vision]`.
+a transitive dep of `xr-ai-voice` and `xr-ai-nat[vision]`.
 
 Requires `model-servers` to be running first — model servers are declared as
 `launch_mode="reuse"` so the launcher skips spawning them but the dependency
@@ -627,6 +630,7 @@ updated in the same commit**.
 | cloudxr-runtime YAML config keys | `agent-samples/xr-render-demo/yaml/cloudxr_runtime.yaml`, `docs/adding-cloudxr.md` |
 | `utils/xr-ai-launcher/xr_ai_launcher/_cloudxr_env.py` API | xr-render-scene + oxr-mcp + cloudxr-runtime `__main__.py` imports, `agent-samples/xr-render-demo/main.py` (native-profile gate), `docs/adding-cloudxr.md`, `docs/xr-render-demo.md` (client-type section) |
 | scene service YAML config keys | `agent-samples/xr-render-demo/scene/scene_service.yaml`, orchestrator process declaration, `docs/xr-render-demo.md` |
+| openxr-service YAML config keys (`allow_sim_pose`, …) | `services/openxr-service/openxr_service.yaml` (reference copy), `agent-samples/xr-render-demo/yaml/openxr_service.yaml` |
 | render-mcp YAML config keys | `agent-mcp-servers/render-mcp/render_mcp.yaml`, worker URL constants |
 | oxr-mcp YAML config keys | `agent-mcp-servers/oxr-mcp/oxr_mcp_server.yaml`, sample copies, worker URL constants |
 | Any `pyproject.toml` dependency | `DEPENDENCIES.md` (this file) |

README.md

@@ -318,10 +318,11 @@ web client for desktop dev.
 Under the hood, the orchestrator launches the hub, CloudXR runtime, model
 endpoints, typed capability processes, and the worker. The worker calls those
 processes through native NAT functions; MCP adapters remain optional outward
-compatibility surfaces and are not in the sample's execution path. The Pipecat pipeline runs
-quick-acks and a Nemotron-30B agentic tool-calling loop over
-scene, XR tracking, spatial math, vision, and video-memory functions. Full process map,
-agentic-loop details, and the XR session lifecycle:
+compatibility surfaces and are not in the sample's execution path. The
+`xr-ai-voice` runtime feeds a Nemotron-30B supervisor that delegates to five
+focused subagents over scene, XR tracking, spatial math, vision, and
+video-memory functions, with reference and geometry resolution done in code.
+Full process map, agent details, and the XR session lifecycle:
 [`docs/xr-render-demo.md`](docs/xr-render-demo.md).
 
 **Requires `model-servers` to be running first** — the demo does not start
@@ -510,7 +511,7 @@ For engineers and agents working in the repo:
 | [`docs/architecture.md`](docs/architecture.md) | Hub ↔ transport ↔ agent boundaries; known limitations |
 | [`docs/process-model.md`](docs/process-model.md) | `Process` / `run_stack` mechanics; ready-file protocol |
 | [`docs/ai-services.md`](docs/ai-services.md) | VLM / STT / TTS / LLM server reference + worker call examples |
-| [`docs/xr-render-demo.md`](docs/xr-render-demo.md) | xr-render-demo architecture: native functions, agentic loop, XR lifecycle |
+| [`docs/xr-render-demo.md`](docs/xr-render-demo.md) | xr-render-demo architecture: native functions, supervisor + subagents, XR lifecycle |
 | [`docs/adding-a-sample.md`](docs/adding-a-sample.md) | Boilerplate for scaffolding a new sample |
 | [`docs/adding-cloudxr.md`](docs/adding-cloudxr.md) | Wiring CloudXR into a sample |
 | [`docs/credentials.md`](docs/credentials.md) | HF / NGC token management |

agent-samples/xr-render-demo/eval/README.md

@@ -5,136 +5,127 @@
 
 # xr-render-demo eval harness
 
-End-to-end test of the agent LLM's tool-calling against the live model. Each
-case feeds a synthetic scene and head pose into the model with the same system
-prompt and native NAT function schemas as the live worker, executes tool
-effects against deterministic fixtures, then checks the resulting scene
-mutations against a per-case expectation.
+Four tiers, cheapest first. Debug at the lowest tier that reproduces the
+problem; rerun the tiers above it as regression before calling a fix done.
 
-## Prerequisites
+| Tier | Command | Runs against | Cost |
+|---|---|---|---|
+| Supervisor routing | `xr_render_demo_eval_supervisor` | faked subagents that record delegations | ~15 s/case |
+| Subagent components | `xr_render_demo_eval_subagents` | one real agent over faked leaf functions | ~30 s/case |
+| End-to-end corpus + basics | `xr_render_demo_eval` | supervisor + agents over faked services | ~30 min full |
+| Live | `xr_render_demo_live_{smoke,pose_matrix,manip,garble,explore}` | the running demo stack | minutes |
 
-The agent LLM must be running:
+All commands run from the eval project:
 
 ```bash
-# weights resident in the background — start once, leave alone
-uv run --project ~/hub/xr-ai/agent-samples/model-servers model_servers
+cd agent-samples/xr-render-demo/eval && uv sync   # once
 
-```
+# Full corpus + native cases + basics battery
+uv run xr_render_demo_eval
 
-By default the harness calls the agent LLM at `http://localhost:8107`. It does
-not require the render-demo stack, capability services, MCP adapters, or LOVR.
+# The basics battery alone: the most common utterances, their perturbation
+# classes, and history-bearing variants. Run after EVERY prompt or ops
+# change; full-suite variance hides single-case damage.
+uv run xr_render_demo_eval basics
 
-## Run
+# Subset by case name (space-separated; unknown names error out)
+uv run xr_render_demo_eval move_left_one_meter between_two_spheres
 
-```bash
-# All built-in cases against the current system.txt
-uv run --project agent-samples/xr-render-demo/worker \
-  python agent-samples/xr-render-demo/eval/eval.py
-
-# Subset by case name — fast iteration on a single failing cluster.
-# Comma-separated; unknown names error out (mutually exclusive with the
-# positional query arg below).
-uv run --project agent-samples/xr-render-demo/worker \
-  python agent-samples/xr-render-demo/eval/eval.py \
-  --only move_left_one_meter,between_two_spheres
-
-# Watcher-friendly equivalent: write case names (newline- or
-# comma-separated; '#' comments OK) to eval/.only. Gitignored.
-# Active subset is echoed at startup.
-
-# One ad-hoc query (prints the raw LLM response)
-uv run --project agent-samples/xr-render-demo/worker \
-  python agent-samples/xr-render-demo/eval/eval.py "Move the cube up 30 cm"
-
-# Score a prompt file other than the live worker's system.txt — e.g.
-# main's version, a draft, or a checkout from another branch.
-uv run --project agent-samples/xr-render-demo/worker \
-  python agent-samples/xr-render-demo/eval/eval.py --prompt /tmp/alt-system.txt
-
-# Score against a hosted model (e.g. nvidia/nemotron-3-super-120b-a12b at
-# build.nvidia.com) instead of the local vLLM on 8107.  Set NVIDIA_API_KEY
-# in the env first (or pass --agent-api-key).
-export NVIDIA_API_KEY=nvapi-...
-uv run --project agent-samples/xr-render-demo/worker \
-  python agent-samples/xr-render-demo/eval/eval.py \
-  --agent-llm   https://integrate.api.nvidia.com/v1/chat/completions \
-  --agent-model nvidia/nemotron-3-super-120b-a12b
+# Routing and component tiers, optionally filtered by agent or case name
+uv run xr_render_demo_eval_supervisor
+uv run xr_render_demo_eval_subagents placement
 ```
 
-Run `uv sync` in `agent-samples/xr-render-demo/worker` before the first eval.
+The offline tiers need only the agent LLM (default `http://localhost:8107`);
+they do not require the demo stack, capability services, or LOVR.
+
+## Live drivers
+
+Live drivers join the running stack (`uv run python main.py` from the sample
+directory) as synthetic participants, inject typed text, set a simulated
+head pose, and score real scene state. They require `allow_sim_pose: true`
+in `yaml/openxr_service.yaml` (off by default; flip it for eval runs and
+restart the stack). Isolation rules:
+
+- Fresh participant id per case: transcript history otherwise bleeds between
+  cases and collapses supervisor behavior.
+- Clear the scene between cases through the scene RPC: leftovers make
+  referents ambiguous and invite anchoring on stale objects.
+- Vary prompt phrasing across cases: repeating one sentence builds a
+  self-history no real user produces.
+- Never filter a run's output in the run command; write the full log to a
+  file and filter the file.
+- Repeat runs (3x) before believing any single-run delta; near-tie decisions
+  flip run to run even at temperature 0.
+
+`xr_render_demo_live_garble` covers speech-to-text noise (homophones,
+truncations, corrections, stutters) with restraint scoring: wrong mutations
+fail, clarifying replies pass. `xr_render_demo_live_explore` sends novel
+conversational phrasings scored by intent invariants; promote any violation
+into a permanent tier case, then fix.
+
+## Prompt-tuning law
+
+The current agent model follows templates and contrast pairs; it ignores
+prohibitions. Fix behavior with worked examples, and pair every
+refuse-example with a proceed-example so it does not contaminate neighboring
+behaviors.
+
+When even worked examples fail (the model keeps resolving what it should
+copy) move the resolution into code and rename the tool parameter so the
+schema asks for exactly what the model does reliably. Anchor descriptors are
+the precedent: renaming the parameter to `anchor_words` with a copy-verbatim
+description fixed in one step what five prompt variants could not, with
+`spatial_ops` resolving shape synonyms, mangled nouns, and color words
+against the scene deterministically.
 
 ## Watcher
 
-`eval_watch.sh` polls `system.txt`'s sha1 once per second (hash, not
-mtime — editors and language servers re-save the file without
-changing bytes when you switch focus). This allows a coding agent to
-iterate on the prompt and read scores out of `/tmp/eval_loop.log`
-without the user re-launching `eval.py` between rounds. Any content
-change aborts the running eval and starts a new one once the file
-has been quiet for 10 seconds.
+`eval_watch.sh` polls a combined sha1 of every worker prompt file
+(`supervisor_prompt.txt` and `agents/*/prompt.txt`) once per second (hash,
+not mtime: editors re-save without changing bytes). Any content change
+aborts the running eval and starts a new corpus run once the prompts have
+been quiet for 10 seconds, so a prompt-tuning loop can read scores out of
+`/tmp/eval_loop.log` without relaunching between rounds.
 
 ```bash
 agent-samples/xr-render-demo/eval/eval_watch.sh
 tail -f /tmp/eval_loop.log
-
-agent-samples/xr-render-demo/eval/eval_watch.sh /path/to/alt.txt   # different prompt
-kill $(cat /tmp/eval_watch.pid)                                     # stop
+kill $(cat /tmp/eval_watch.pid)   # stop
 ```
 
-Only one watcher runs at a time. A second invocation refuses to
-start, exits non-zero, and prints the existing PID along with the
-two ways to handle it (`tail` the log of the running watcher, or
-`kill <pid>` and rerun). The script never kills processes it didn't
-spawn — that decision stays with the caller, which keeps the behavior
-predictable across users / sandboxes / CI runners.
-
-`eval_watch.sh` is Linux-only. The single-instance guard reads
-`/proc/<pid>/cmdline` to confirm the stored PID is the watcher (not
-some unrelated process that recycled the same PID); macOS has no
-`/proc`, so the script will not run there.
-
-Score history at a glance:
-
-```bash
-grep "passed$" /tmp/eval_loop.log | tail
-```
+Only one watcher runs at a time; a second invocation refuses to start and
+prints the existing PID. Linux-only (the single-instance guard reads
+`/proc/<pid>/cmdline`).
 
 ## Writing a case
 
-Read `eval.py`'s `CASES` list — every shape (single-turn, pose
-override, multi-turn `history`, undo `recent_moves`) is exemplified
-there. Copy the closest existing case and edit. The case dict is
-what the harness consumes directly; there's no case schema layer.
+The end-to-end corpus lives in `xr_render_demo_eval/cases.py` (dict-shaped;
+pose override, multi-turn `history`, and undo `recent_moves` are all
+exemplified). Native and basics cases are `Case` dataclasses in
+`xr_render_demo_eval/harness.py`; routing and component cases live in
+`supervisor.py` and `subagents.py`. Copy the closest existing case and edit.
 
 ## Don't train on the test set
 
-Prompt worked-examples and case fixtures share the same model. The
-harness audits at startup for four kinds of overlap and prints a
-warning for any it finds:
-
-1. Verbatim user utterance from a case appearing in `system.txt`.
-2. Concrete scene coordinates (formatted like `(0.50, 1.60, -1.50)`)
-   from a case appearing in `system.txt`.
-3. `recent_moves` coordinates from a case appearing in `system.txt`.
-4. **Reserved prompt vocabulary** — any colour or shape word from the
-   eval-case vocabulary (`_EVAL_VOCAB_COLORS` / `_EVAL_VOCAB_SHAPES`
-   in `eval.py`) appearing inside a worked-example section of
-   `system.txt`. Worked-example sections are triple-backtick blocks
-   and any block starting with `WORKED EXAMPLE`, `Example:`,
-   `iter N:`, or `tool_call N:`; the first blank line after the
-   marker ends the block. Rule narration outside those blocks may
-   still mention the eval vocabulary generically (e.g. the colour
-   table, anchor-routing rules) — the restriction is only on the
-   worked examples, which are the strings the model is most likely
-   to memorise as a template.
-
-Fix overlaps by changing the prompt example, not the case. For
-check #4, use colours and shapes outside the eval vocabulary
-(turquoise / teal / lavender / magenta / cone / cylinder / capsule)
-when reaching for a fixture word in a worked example.
+Prompt worked examples and case fixtures share the same model, so the
+harness audits every worker prompt at startup (all three offline tiers run
+it) and warns on:
+
+1. A case utterance from any tier appearing verbatim in a prompt.
+2. A case fixture id appearing in a prompt.
+3. A quoted prompt example pairing an eval-vocabulary color
+   (red/green/blue/yellow/cyan/orange/purple/white/black) with an
+   eval-vocabulary shape (sphere/cube/box/ball).
+
+Fix overlaps by changing the prompt, not the case; use colors and shapes
+outside the eval vocabulary (teal / lavender / magenta / turquoise, cone /
+cylinder / capsule / torus) in worked examples. A case that only passes
+while the prompt contains its vocabulary is scoring recall, not skill.
 
 ## What the harness does not cover
 
-- The live worker pipeline (VAD, STT, TTS, history bookkeeping).
+- The live worker pipeline (VAD, STT, TTS, history bookkeeping); the live
+  tier covers it.
 - Real scene-service / LOVR effects (fixture-succeeded).
-- Real visual queries (`look_at_current_frame`, `look_at_past_frame`) — stubbed.
+- Real visual queries (`look_at_current_frame`, `look_at_past_frame`): stubbed.