Add on-demand visual task guide

[nvddr]

What changed

• adds a focused visual task guide sample for a ten-step hand-counting workflow
• composes session-local task state, explicit controls, on-demand native NAT vision, and a focused guide agent
• retrieves bundled task documentation through the service-backed xr_rag NAT function group
• reuses the shared VLM, LLM, STT, and embedding model servers through deployment profiles
• documents the sample architecture, quickstart, dependencies, and prompt-evaluation harness

Why

Continuous monitoring and target-bearing visual prompts made the prototype stale, unnecessarily complex, and susceptible to confirmation bias. This sample captures a fresh frame only when requested. The VLM receives a target-neutral count query, and deterministic code compares its structured result with trusted task state afterward.

Task progress resets on each participant connection. Vision never advances state; only explicit start task, next step, and reset task controls mutate it. Retrieval is no longer implemented inside the sample.

Dependency

Stacked on #335, which adds the reusable embedding server, typed RAG service, and native RAGFunctionsConfig boundary. After #335 merges, this PR can be retargeted to main.

Impact

Users get a NAT-native example demonstrating structured tasks, on-demand visual validation, voice/text controls, and dense task-document retrieval without background VLM loops or persisted participant progress.

Validation

• ruff check on the visual task guide Python paths
• focused unit tests: 8 passed
• worker and eval dependency locks resolve
• git diff --check

[wenxind-nvidia]

Review of current head 14b3baca.

Blockers

1. The PR is no longer focused on the visual task guide. Commits cd23e2fa and 8a936879 remove the shared Llama model, change GPU placement for VLM/STT/embedding, and modify xr-render-demo, its tests, and its documentation. Those changes affect existing samples and deployment topology independently of this new sample. Please move them to a separate PR (and keep the delegated xr-render-demo refactor out of this PR), leaving Add on-demand visual task guide #334 focused on the visual task guide.

2. Read-only state functions expose mutable store-owned objects. TaskStore.progress() returns the stored TaskProgress instance directly, and TaskStatusResult retains the same nested Pydantic object. A caller can mutate state, step_index, or transitions without invoking start_task, advance_task, or reset_task, violating the central invariant that only explicit controls mutate trusted state. Make the state/result immutable (frozen=True, immutable collections) and/or return deep snapshots; add a test proving a status result cannot mutate the store.

3. Participant state is never released. participant_left() only calls vision_config.release(), while TaskStore._progress retains every participant indefinitely. Reconnection calls reset(), which preserves and extends the old revision/history rather than creating a fresh session. Add a locked release()/drop() operation, invoke it on disconnect, and test disconnect/reconnect behavior.

4. The documented task-folder contract is not implemented. The README says another workflow can be created by copying the task folder and changing task_directory, but validation hardcodes expected = step_index + 1; visual_completion_criteria and step-level knowledge_files are never used; and rag_service.yaml independently hardcodes tasks/hand-counting/knowledge. Either make expected results and knowledge selection data-driven, or document this as a fixed one-to-ten hand-counting sample and remove the misleading/dead generalization.

Suggestions

• Use the packaged caption prompt by default, as the existing packaged-worker pattern does. Both the shipped YAML and load_config() reach into ../worker/visual_task_guide_worker/prompts/caption.txt, which fails when the worker wheel is installed without the source checkout. Resolve the default from Path(__file__) and reserve a YAML path for explicit overrides.
• Normalize idle_timeout_secs: 0 to None and reject negative values. VoiceSession documents None as disabled and positive values as enabled; passing 0.0 currently relies on Pipecat's downstream falsey check.
• Make the eval semantic. The VLM cases use substring checks (count=2 also matches count=20, and any confidence= passes) instead of the production parser. The requires_knowledge flag is unused, so the RAG case does not prove that a relevant chunk was retrieved. Parse and compare exact fields, and assert a distinctive retrieved source/fact.
• Cross-validate FingerCount: contradictory outputs such as COUNT=7; HANDS=1 or COUNT=3; HANDS=0 currently pass parsing, and validation ignores hands.
• Add configuration and lifecycle coverage: non-mapping YAML, packaged prompt fallback, idle-timeout normalization, immutable status snapshots, and participant disconnect/reconnect. The current tests do not compose the shipped worker configuration or run_app().

All current CI jobs pass, but they do not exercise these runtime/package/lifecycle contracts.

[nvddr]

Review outcome: changes requested

I’m taking over shepherding from the current head and agree with the blockers in Wenxin’s review:

• Rebase onto current main and keep the PR focused on visual-task-guide; the stacked RAG/embedding work and unrelated model/xr-render changes have since landed independently.
• Prevent read-only task status from exposing mutable store-owned state.
• Release participant task state on disconnect so reconnect starts a genuinely fresh session.
• Make the task contract honest and data-driven: validation must consume step completion criteria, and retrieval/document configuration must follow the selected task rather than a second hard-coded path.

I’ll also fold in the review’s packaging/configuration/evaluation hardening: packaged prompt defaults, strict idle-timeout normalization, semantic eval parsing and retrieval assertions, cross-field finger-count validation, and lifecycle/config tests.

The branch currently conflicts with main; historical CI cannot establish the result after the required adaptation. I’m resolving these items on the author branch and will update this same consolidated comment with validation and final status.