Add on-demand visual task guide

DEPENDENCIES.md

@@ -560,6 +560,23 @@ external NVIDIA NIM VLM, and `models.omni.json` reuses Nemotron-Omni on port
 credentials, and launcher ownership. Voice-gate knobs are configured via
 `yaml/voice_gate.yaml`.
 
+### visual-task-guide  (agent-samples/visual-task-guide/)
+
+On-demand hand-counting workflow with session-local task state, deterministic
+controls and validation, native current-frame vision, and a focused NAT guide
+agent over bundled task knowledge retrieved through `rag-service`.
+
+| Sub-project | Package | Internal deps | External deps |
+|---|---|---|---|
+| Orchestrator | `visual-task-guide` | `xr-ai-launcher` | — |
+| Worker | `visual-task-guide-worker` | `xr-ai-hub-client [editable]`, `xr-ai-logging [editable]`, `xr-ai-models [editable]`, `xr-ai-nat[agents,services,vision,voice] [editable]`, `xr-ai-voice [editable]`, `xr-ai-voicegate [editable]` | loguru >=0.7, pydantic >=2.10, pyyaml >=6.0 |
+| Eval | `visual-task-guide-eval` | `visual-task-guide-worker [editable]`, `xr-ai-models [editable]`, `xr-ai-nat[agents,services,vision] [editable]` | Pillow >=10.0 |
+
+The orchestrator reuses embedding-server (8109), launches `rag-service`
+(private ZMQ 8340) over the task knowledge directory, then starts the hub and
+worker. The worker composes the service through `RAGFunctionsConfig`; it has no
+sample-local retrieval implementation.
+
 ### model-servers  (agent-samples/model-servers/)
 
 Standalone launcher that starts the shared AI inference servers and keeps

README.md

@@ -307,6 +307,29 @@ for the full option list.
 
 ---
 
+### Visual task guide (on-demand hand-counting workflow)
+
+This sample guides a ten-step hand-counting task with session-local native NAT
+state. Start the shared model servers, then run the sample:
+
+```bash
+cd agent-samples/model-servers
+uv sync && uv run model_servers
+
+cd ../visual-task-guide
+uv sync
+cd worker && uv sync && cd ..
+uv run visual_task_guide
+```
+
+Use `start task`, `next step`, `task status`, and `reset task`. Asking “Did I
+do it correctly?” runs one fresh, target-neutral VLM count and compares the
+result with the trusted current step. Vision never advances progress. See the
+[`sample guide`](agent-samples/visual-task-guide/README.md) and
+[`system diagram`](agent-samples/visual-task-guide/SYSTEM_DIAGRAM.md).
+
+---
+
 ### XR render demo (voice-driven sphere in CloudXR)
 
 Speak to the web client and a sphere in the streamed scene tracks your

agent-samples/visual-task-guide/README.md

@@ -0,0 +1,90 @@
+<!--
+  SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+  SPDX-License-Identifier: Apache-2.0
+-->
+
+# Visual task guide
+
+This focused sample guides a ten-step hand-counting task with an explicit NAT
+state machine and on-demand current-frame vision. The web client's **Agent
+output** shows the current step and each requested validation, for example:
+
+```text
+Show three — Yes, I see 3 extended fingers.
+```
+
+Vision never advances the task. Only explicit `start task`, `next step`, and
+`reset task` commands change state. `task status` reads it. These controls,
+on-demand vision, RAG, and the root workflow are native NAT functions.
+
+## Bundled task
+
+`tasks/hand-counting/workflow.yaml` orders ten separate step YAML files. Each
+step declares its instruction, visible criterion, expected finger count, and
+expected visible-hand count. Deterministic validation reads those fields rather
+than inferring the answer from the step number. The RAG service is intentionally
+configured for this sample's bundled `knowledge/` directory; this PR does not
+claim a general copy-and-retarget task-folder contract.
+
+## Run
+
+```bash
+cd agent-samples/model-servers
+uv sync
+uv run model_servers
+```
+
+In another terminal:
+
+```bash
+cd agent-samples/visual-task-guide
+uv sync
+cd worker && uv sync && cd ..
+uv run visual_task_guide
+```
+
+The launcher requires the shared VLM, Nemotron-3-Nano guide LLM, STT, and
+embedding endpoints before it starts the hub, Piper TTS, RAG service, and
+worker. This sample does not launch video memory or record historical video.
+
+Open `https://localhost:8080`, connect, and start the camera. Start the
+microphone for voice interaction or use the text box:
+
+- `start task` starts at **Show one**.
+- `next step` advances exactly once.
+- “What’s the next step?” reports the following step without advancing.
+- `task status` prints the current step.
+- `reset task` returns to **Show one / not started**.
+- “Did I do the step correctly?” captures one fresh frame and compares its
+  reliable count with the current step without invoking RAG.
+- “How many fingers do you see?” captures one fresh frame.
+- “How should I position both hands?” uses dense retrieval over the bundled task documents.
+
+Voice and typed commands are both dispatched directly; this focused demo does
+not require a wake phrase, but accepts an optional “agent” or “hey agent”
+prefix. A vision request runs only when the user asks a visual question. The
+workflow uses a neutral count query with no target answer, then captures one
+latest frame. Validation parses the VLM's compact count contract and compares
+it deterministically with the trusted step. Direct count questions bypass the
+guide LLM; other questions combine the fresh visual result with bounded dense
+retrieval in one 128-token pass.
+
+The worker console logs task transitions, RAG citations, and total workflow
+latency. Model prompts and full payloads are not logged.
+
+The reusable boundaries are `StreamingVisionConfig`, `RAGFunctionsConfig`, and
+`ModelsLLMConfig`. The sample owns the session-local state machine, task
+workflow, and focused guide agent.
+Progress resets whenever the worker starts or the participant reconnects.
+
+## Evaluate deployed prompts
+
+With the shared model servers and this sample's RAG service running:
+
+```bash
+uv run --project eval visual_task_guide_eval
+```
+
+The harness calls both deployed models, checks concise output, verifies native
+dense RAG retrieval, and audits fixture leakage. See
+[`eval/README.md`](eval/README.md).

agent-samples/visual-task-guide/SYSTEM_DIAGRAM.md

@@ -0,0 +1,33 @@
+<!--
+  SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+  SPDX-License-Identifier: Apache-2.0
+-->
+
+# Visual task guide system
+
+```text
+voice/text -> VoiceSession -> TaskGuideWorkflowConfig
+                              | controls: start / next / reset / status
+                              | state queries: deterministic current/next step
+                              | validation: neutral visual count -> step check
+                              | other questions: neutral visual count + RAG
+                              v
+camera -> XR Media Hub -> StreamingVisionConfig -> current-frame VLM
+                                                      :8100 / reused
+                              v
+                       read-only NAT guide agent -> xr_rag NAT group
+                              |                     -> RAG service :8340
+                              |                     -> embedding :8109 / reused
+                              |              Nemotron-3-Nano :8107 / reused
+                              v
+                      voice + agent.response reply
+```
+
+Vision runs only on a user request. It receives no task target or expected
+answer. Validation parses the structured count and compares it with the trusted
+current step afterward. Direct count questions return that parsed observation;
+other questions pass it to the read-only guide agent alongside bounded RAG.
+
+Task state is `not_started`, `running`, or `completed`. Only the native task
+control functions mutate it. Visual results are question evidence only, so
+they never advance the task. Progress is session-local and resets on reconnect.

agent-samples/visual-task-guide/eval/README.md

@@ -0,0 +1,35 @@
+<!--
+  SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+  SPDX-License-Identifier: Apache-2.0
+-->
+
+# Visual task guide deployed-model eval
+
+The harness exercises both model-driven prompt paths against configured
+services:
+
+- The caption prompt evaluates generated two-finger and closed-fist fixtures
+  through the deployed VLM with the same 40-token ceiling as the worker.
+- `TaskGuideAgentConfig` performs bounded native dense retrieval, then uses
+  one deployed NAT agent pass with real task state and a latest observation.
+
+Start the shared model servers and the visual task guide stack, then run:
+
+```bash
+uv run --project agent-samples/visual-task-guide/eval visual_task_guide_eval
+```
+
+Run selected cases or save the complete report:
+
+```bash
+uv run --project agent-samples/visual-task-guide/eval visual_task_guide_eval \
+  --case rag_hand_presentation_answer \
+  --output agent-samples/visual-task-guide/eval/results/local.json
+```
+
+The harness parses exact structured count/hand/confidence fields, checks the
+30-word guide limit, verifies a distinctive RAG source and fact, and confirms
+the task revision stays immutable. Native workflow tests separately cover
+deterministic next-step and current-step validation queries. Before model calls,
+the harness audits distinctive fixture markers against both prompts to prevent
+test leakage.

agent-samples/visual-task-guide/eval/__init__.py

@@ -0,0 +1,4 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Visual task guide deployed-model evaluation."""

agent-samples/visual-task-guide/eval/benchmark.py

@@ -0,0 +1,178 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Evaluate the live-caption and guide prompts against deployed models."""
+
+import argparse
+import asyncio
+import json
+from pathlib import Path
+from typing import Any
+
+from nat.builder.workflow_builder import WorkflowBuilder
+from visual_task_guide_worker.agent import TaskGuideAgentConfig
+from visual_task_guide_worker.finger_count import parse_finger_count
+from visual_task_guide_worker.models import GuideAgentRequest
+from visual_task_guide_worker.task_functions import TaskStateFunctionsConfig
+from visual_task_guide_worker.task_store import TaskStore
+from xr_ai_models import load_models_config, make_llm, make_vlm
+from xr_ai_nat.functions.rag import RAGFunctionsConfig, RetrieveResult
+from xr_ai_nat.llm import ModelsLLMConfig
+
+try:
+    from .cases import GUIDE_CASES, LEAKAGE_MARKERS, VLM_CASES
+except ImportError:
+    from cases import GUIDE_CASES, LEAKAGE_MARKERS, VLM_CASES
+
+_HERE = Path(__file__).resolve().parent
+_SAMPLE = _HERE.parent
+_CAPTION_PROMPT = _SAMPLE / "worker/visual_task_guide_worker/prompts/caption.txt"
+_GUIDE_PROMPT = _SAMPLE / "worker/visual_task_guide_worker/prompts/guide_agent.txt"
+_FIXTURES = _HERE / "fixtures"
+
+
+def audit_fixture_leakage() -> None:
+    prompts = f"{_CAPTION_PROMPT.read_text()} {_GUIDE_PROMPT.read_text()}".casefold()
+    leaked = [marker for marker in LEAKAGE_MARKERS if marker.casefold() in prompts]
+    if leaked:
+        raise ValueError(f"eval fixture details leaked into prompts: {leaked}")
+
+
+async def run_eval(
+    models_path: Path,
+    selected: set[str] | None = None,
+    *,
+    rag_endpoint: str = "tcp://127.0.0.1:8340",
+) -> dict[str, Any]:
+    audit_fixture_leakage()
+    models = load_models_config(models_path)
+    llm = make_llm(models, "guide_llm")
+    vlm = make_vlm(models, "vlm")
+    results: list[dict[str, Any]] = []
+    store = TaskStore(_SAMPLE / "tasks/hand-counting")
+    try:
+        store.start("eval-user")
+
+        for case in VLM_CASES:
+            if selected and case["name"] not in selected:
+                continue
+            try:
+                response = await vlm.ask_image(
+                    _FIXTURES / case["fixture"],
+                    case["question"],
+                    system_prompt=_CAPTION_PROMPT.read_text(encoding="utf-8").strip(),
+                    max_tokens=40,
+                    temperature=0.0,
+                )
+                text = (response.content or "").strip()
+                parsed = parse_finger_count(text)
+                passed = (
+                    parsed is not None
+                    and parsed.count == case["expected_count"]
+                    and parsed.hands == case["expected_hands"]
+                    and parsed.confidence in {"high", "medium"}
+                )
+            except Exception as error:
+                text, passed = f"{type(error).__name__}: {error}", False
+            results.append(
+                {"stage": "live_caption", "name": case["name"], "passed": passed, "output": text}
+            )
+
+        async with WorkflowBuilder() as builder:
+            await builder.add_llm(
+                "guide_llm",
+                ModelsLLMConfig(
+                    service=llm,
+                    model_name="visual-task-guide-eval",
+                    temperature=0.0,
+                    max_tokens=128,
+                ),
+            )
+            await builder.add_function_group("task_state", TaskStateFunctionsConfig(store=store))
+            await builder.add_function_group(
+                "task_knowledge",
+                RAGFunctionsConfig(endpoint=rag_endpoint),
+            )
+            knowledge_group = await builder.get_function_group("task_knowledge")
+            knowledge_functions = await knowledge_group.get_all_functions()
+            retrieve = knowledge_functions["task_knowledge__retrieve"]
+            guide = await builder.add_function("task_guide_agent", TaskGuideAgentConfig())
+
+            for index, case in enumerate(GUIDE_CASES):
+                if selected and case["name"] not in selected:
+                    continue
+                before_revision = store.progress("eval-user").revision
+                try:
+                    retrieval = RetrieveResult.model_validate(
+                        await retrieve.ainvoke({"query": case["question"], "top_k": 2})
+                    )
+                    expected_source = case.get("knowledge_source")
+                    expected_term = case.get("knowledge_term", "").casefold()
+                    retrieval_passed = expected_source is None or any(
+                        result.source == expected_source
+                        and expected_term in result.text.casefold()
+                        for result in retrieval.results
+                    )
+                    reply = await guide.ainvoke(
+                        GuideAgentRequest(
+                            participant_id="eval-user",
+                            user_text=case["question"],
+                            latest_observation=case["observation"],
+                        )
+                    )
+                    text = reply.response.casefold()
+                    passed = (
+                        all(term in text for term in case["required_terms"])
+                        and len(reply.response.split()) <= case["max_words"]
+                        and store.progress("eval-user").revision == before_revision
+                        and retrieval_passed
+                    )
+                    output: Any = {
+                        "response": reply.response,
+                        "retrieved_sources": [result.source for result in retrieval.results],
+                    }
+                except Exception as error:
+                    passed = False
+                    output = f"{type(error).__name__}: {error}"
+                results.append({"stage": "guide", "name": case["name"], "passed": passed, "output": output})
+
+    finally:
+        await llm.close()
+        await vlm.close()
+    return {
+        "profile": str(models_path),
+        "passed": all(item["passed"] for item in results),
+        "results": results,
+    }
+
+
+def run() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--models",
+        type=Path,
+        default=_SAMPLE / "yaml/models.local.json",
+        help="xr-ai-models deployment profile with reachable guide_llm and vlm endpoints.",
+    )
+    parser.add_argument("--case", action="append", default=[])
+    parser.add_argument("--rag-endpoint", default="tcp://127.0.0.1:8340")
+    parser.add_argument("--output", type=Path)
+    args = parser.parse_args()
+    report = asyncio.run(
+        run_eval(
+            args.models.resolve(),
+            set(args.case) or None,
+            rag_endpoint=args.rag_endpoint,
+        )
+    )
+    rendered = json.dumps(report, indent=2)
+    print(rendered)
+    if args.output:
+        args.output.parent.mkdir(parents=True, exist_ok=True)
+        args.output.write_text(f"{rendered}\n", encoding="utf-8")
+    if not report["passed"]:
+        raise SystemExit(1)
+
+
+if __name__ == "__main__":
+    run()