Skip to content

refactor(native-tools): add Relay-managed tool layer - #348

Merged
nvddr merged 3 commits into
mainfrom
agent/relay-runtime-foundation
Aug 11, 2026
Merged

refactor(native-tools): add Relay-managed tool layer#348
nvddr merged 3 commits into
mainfrom
agent/relay-runtime-foundation

Conversation

@nvddr

@nvddr nvddr commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • make xr-ai-nat the toolkit-independent native tools layer
  • run new typed tools through NeMo Relay and expose model-backed agents as ordinary tools
  • add the generic AgentRunner seam; the bundled Agent remains a bounded default loop, while custom or future framework-backed runners stay behind a normal tool
  • forward Relay-rewritten request content and controlled lineage headers to xr-ai-models, supply request/response OpenAI codecs, omit empty tools, and accept tool-call-only content: null responses
  • retain existing NeMo Agent Toolkit function groups only behind legacy extras

Validation

  • uv run --project tests pytest tests/test_native_tools.py tests/test_models_openai_compat.py tests/test_nat_models_llm.py tests/test_functions_types.py tests/test_docs_versions.py -q
  • uvx ruff check agent-sdk/xr-ai-nat/xr_ai_nat/agent_runner.py agent-sdk/xr-ai-nat/xr_ai_nat/agents.py agent-sdk/xr-ai-models/xr_ai_models/_openai_compat.py tests/test_native_tools.py tests/test_models_openai_compat.py
  • uv run --project tests --with pyright pyright agent-sdk/xr-ai-nat/xr_ai_nat/agent_runner.py agent-sdk/xr-ai-nat/xr_ai_nat/agents.py agent-sdk/xr-ai-models/xr_ai_models/_openai_compat.py tests/test_native_tools.py tests/test_models_openai_compat.py
  • isolated base-install check verified no installed nvidia-nat-* distribution
  • uv build --wheel agent-sdk/xr-ai-nat

Follow-up PRs add streaming vision as a normal native tool and route voice plus background triggers through registered tools.

github-actions Bot added a commit that referenced this pull request Aug 11, 2026
@nvddr
nvddr marked this pull request as ready for review August 11, 2026 20:56
@wenxind-nvidia

Copy link
Copy Markdown
Collaborator

Blocker

  • agent.py:135 drops request.headers when calling LLMService.chat(). Relay automatically injects Dynamo session-lineage headers and allows intercepts to add headers, but none reach the actual model HTTP request. Extend xr-ai-models with controlled per-call headers and forward them here. Relay's managed-call contract explicitly requires the provider callback to consume the mutated LLMRequest.

Suggestions

  • agent.py:147 supplies only response_codec. Consequently, request intercepts always receive annotated=None, preventing provider-neutral middleware from inspecting or rewriting messages and tools. Pass OpenAIChatCodec as codec too, and omit tools when empty because the codec rejects tools: null.
  • agent.py:265 rejects content: null. That is valid for tool-call-only OpenAI responses, so a Relay execution intercept returning such a response crashes before the tool loop. Normalize None to "", matching xr-ai-models.

Nit

  • AGENTS.md:61 calls out public worker SDKs but omits the new xr_ai_runtime surface.

Assumption: Relay is intended to instrument and modify the real provider request, not merely emit local lifecycle events. The PR documentation supports that interpretation.

Focused verification: 7 tests passed, Ruff passed, and the package built successfully. Full test-project type checking could not run locally because PyNvVideoCodec has no macOS ARM wheel; PR CI is green.

Signed-off-by: Devdeep Ray <devdeepr@nvidia.com>
@nvddr
nvddr force-pushed the agent/relay-runtime-foundation branch from 14f5a35 to 94b76b8 Compare August 11, 2026 21:18
@nvddr nvddr changed the title feat(agent-runtime): add Relay native tool loop refactor(native-tools): add Relay-managed tool layer Aug 11, 2026
github-actions Bot added a commit that referenced this pull request Aug 11, 2026
Signed-off-by: Devdeep Ray <devdeepr@nvidia.com>
github-actions Bot added a commit that referenced this pull request Aug 11, 2026
Comment thread agent-sdk/xr-ai-nat/xr_ai_nat/agent_runner.py Fixed
@nvddr

nvddr commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed in caca425: Relay now receives request and response OpenAI codecs; its rewritten content and controlled headers reach the model service; empty tools are omitted; and tool-call-only null content normalizes to an empty native message. Added focused coverage for the Relay boundary and a custom AgentRunner tool.

Signed-off-by: Devdeep Ray <devdeepr@nvidia.com>
github-actions Bot added a commit that referenced this pull request Aug 11, 2026
@nvddr
nvddr merged commit 8e07456 into main Aug 11, 2026
14 checks passed
@nvddr
nvddr deleted the agent/relay-runtime-foundation branch August 11, 2026 22:40
github-actions Bot added a commit that referenced this pull request Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants