Skip to content

refactor(simple-vlm): separate agentic and voice vision paths - #349

Merged
nvddr merged 3 commits into
mainfrom
agent/relay-simple-vlm-native-vision
Aug 12, 2026
Merged

refactor(simple-vlm): separate agentic and voice vision paths#349
nvddr merged 3 commits into
mainfrom
agent/relay-simple-vlm-native-vision

Conversation

@nvddr

@nvddr nvddr commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add a finite Relay-managed look_at_current_frame tool for normal agentic planning
  • keep direct voice token streaming in a separate LiveVisionResponder that shares frame acquisition and the VLM service boundary
  • forward controlled Relay request headers and redact inline camera frames from telemetry
  • remove the unsupported streaming-tool lifecycle and legacy NeMo Agent Toolkit extras

Validation

  • focused tests: 59 passed
  • full CPU suite: 833 passed, 1 skipped, 12 deselected
  • Ruff passed
  • targeted Pyright passed with 0 errors

@nvddr
nvddr deployed to github-pages August 11, 2026 21:05 — with GitHub Actions Active
github-actions Bot added a commit that referenced this pull request Aug 11, 2026
@nvddr
nvddr force-pushed the agent/relay-runtime-foundation branch from 14f5a35 to 94b76b8 Compare August 11, 2026 21:18
@nvddr
nvddr force-pushed the agent/relay-simple-vlm-native-vision branch from 41b75b9 to 0abbb52 Compare August 11, 2026 21:27
@nvddr nvddr changed the title refactor(simple-vlm): move streaming vision off NAT refactor(simple-vlm): use native live vision tool Aug 11, 2026
@nvddr
nvddr deployed to github-pages August 11, 2026 21:27 — with GitHub Actions Active
github-actions Bot added a commit that referenced this pull request Aug 11, 2026
@nvddr
nvddr force-pushed the agent/relay-simple-vlm-native-vision branch from 0abbb52 to ad1d15c Compare August 11, 2026 22:09
@nvddr
nvddr deployed to github-pages August 11, 2026 22:10 — with GitHub Actions Active
github-actions Bot added a commit that referenced this pull request Aug 11, 2026
@nvddr
nvddr force-pushed the agent/relay-simple-vlm-native-vision branch from ad1d15c to 05c2619 Compare August 11, 2026 22:16
@nvddr
nvddr deployed to github-pages August 11, 2026 22:17 — with GitHub Actions Active
github-actions Bot added a commit that referenced this pull request Aug 11, 2026
Base automatically changed from agent/relay-runtime-foundation to main August 11, 2026 22:40
Signed-off-by: Devdeep Ray <devdeepr@nvidia.com>
@nvddr
nvddr force-pushed the agent/relay-simple-vlm-native-vision branch from 05c2619 to 3dc056e Compare August 11, 2026 22:43
@nvddr
nvddr deployed to github-pages August 11, 2026 22:44 — with GitHub Actions Active
@nvddr
nvddr marked this pull request as ready for review August 11, 2026 22:44
github-actions Bot added a commit that referenced this pull request Aug 11, 2026
@wenxind-nvidia

Copy link
Copy Markdown
Collaborator

Blockers

  1. agent-sdk/xr-ai-nat/xr_ai_nat/live_vision.py:139-143 drops request.headers when invoking VLMService.stream(). Relay request intercepts can inject authentication, routing, and tracing headers, but none reach the HTTP request. Please extend the VLM protocol/client with controlled per-call headers and add coverage equivalent to the existing agent header-forwarding test.

  2. agent-sdk/xr-ai-nat/xr_ai_nat/streaming.py:58-64 manually opens a Relay span and then calls the handler directly. nemo_relay.tools.call() does not run tool execution intercepts, so retry, short-circuit, timing, and result-shaping middleware is bypassed. I reproduced this with a registered execution intercept: StreamingTool never invoked it, while tools.execute() did. This conflicts with the documented architecture that Relay owns tool middleware and guardrails.

Suggestion

  • agent-sdk/xr-ai-nat/xr_ai_nat/live_vision.py:127 places the complete base64 camera frame in the Relay request. Relay's start event records it in both raw and annotated request data, exposing potentially sensitive frames to subscribers/exporters and substantially increasing telemetry volume. Please sanitize or replace the image payload in emitted telemetry.

The migration itself is cleanly scoped and preserves participant routing and streaming behavior. The 21 focused tests pass locally, and the current GitHub checks are green.

Signed-off-by: Devdeep Ray <devdeepr@nvidia.com>
@nvddr
nvddr deployed to github-pages August 12, 2026 00:08 — with GitHub Actions Active
github-actions Bot added a commit that referenced this pull request Aug 12, 2026
@nvddr nvddr changed the title refactor(simple-vlm): use native live vision tool refactor(simple-vlm): use native live vision responder Aug 12, 2026
Signed-off-by: Devdeep Ray <devdeepr@nvidia.com>
@nvddr
nvddr deployed to github-pages August 12, 2026 00:33 — with GitHub Actions Active
github-actions Bot added a commit that referenced this pull request Aug 12, 2026
@nvddr nvddr changed the title refactor(simple-vlm): use native live vision responder refactor(simple-vlm): separate agentic and voice vision paths Aug 12, 2026
@nvddr
nvddr merged commit 6de6ca7 into main Aug 12, 2026
14 checks passed
@nvddr
nvddr deployed to github-pages August 12, 2026 01:40 — with GitHub Actions Active
@nvddr
nvddr deleted the agent/relay-simple-vlm-native-vision branch August 12, 2026 01:40
github-actions Bot added a commit that referenced this pull request Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants