ux: improve clarity and use of skills and sandbox in config - #284
Conversation
Signed-off-by: Chantal D Gama Rose <cdgamarose@nvidia.com>
WalkthroughDeep-research config now uses named skills and sandbox runtime blocks, with resolved runtime configs flowing into agent construction, graph wiring, prompts, and tests. Citation verification now normalizes ChangesDeep Research runtime and citation changes
Dask job cancellation
Sequence Diagram(s)sequenceDiagram
participant run_agent_job
participant resolve_deep_research_runtime_config
participant DeepResearcherAgent
participant DeepAgentsRuntime
participant build_deep_research_graph
run_agent_job->>resolve_deep_research_runtime_config: resolve skills and sandbox refs
resolve_deep_research_runtime_config-->>run_agent_job: concrete runtime configs
run_agent_job->>DeepResearcherAgent: construct with resolved config
DeepResearcherAgent->>DeepAgentsRuntime: build routed runtime backend
DeepResearcherAgent->>build_deep_research_graph: compose context, middleware, permissions
DeepResearcherAgent->>DeepResearcherAgent: run citation verification when enabled
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Possibly related PRs
Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error, 2 warnings)
✅ Passed checks (2 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
frontends/aiq_api/src/aiq_api/jobs/runner.py (1)
595-619: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueStale docstring after adding the explicit DeepResearch path.
The new explicit
DeepResearchAgentConfigbranch (Lines 602-619) is the correct fix — it stops swallowing real constructorTypeErrors into the fallback chain. But the docstring still only documents the "Tries in order" fallback patterns and omits the deep-research keyword-arg path, which is now the primary branch.📝 Docstring update
""" Create an agent instance, supporting different constructor patterns. - Tries in order: - 1. llm_provider + tools pattern (DeepResearcherAgent style) - 2. llm + tools pattern (simpler agents) + DeepResearchAgentConfig uses an explicit keyword-argument constructor. + All other configs try, in order: + 1. llm_provider + tools + verbose pattern + 2. llm_provider + tools + max_tool_iterations pattern + 3. llm + tools pattern + 4. callbacks-only fallback """🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@frontends/aiq_api/src/aiq_api/jobs/runner.py` around lines 595 - 619, Update the docstring for the agent creation helper in runner.py so it matches the new control flow: the explicit DeepResearchAgentConfig branch should be documented as the primary path before the fallback constructor patterns. Keep the wording aligned with the symbols already in the function (DeepResearchAgentConfig, agent_cls, llm_provider, tools) and revise the “Tries in order” description to include the deep-research keyword-arg branch plus the existing llm_provider + tools and llm + tools fallbacks.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@configs/config_domain_routing_and_skills.yml`:
- Line 207: The public skills/sandbox config is disabling citation verification
by default, which weakens source-attribution guarantees. Update the config entry
in the skills/sandbox section to keep citation verification enabled unless this
profile is explicitly documented as a special performance/debug mode, and verify
the change in the relevant config block that controls report-generation
behavior.
In `@docs/source/examples/skills-sandbox/index.md`:
- Around line 56-88: The embedded YAML in this example does not match the
referenced config, so update the snippet to use the same values as the source
config and keep it in sync with the documented setup. In particular, verify the
deep_research_sandbox settings and the deep_research_agent wiring, including the
image and packages fields, and make sure the example reflects the actual public
skill collection names used by deep_research_skills. If the snippet is
intentionally simplified or illustrative, clearly label it as an example rather
than the relevant section.
In `@src/aiq_agent/agents/deep_researcher/deepagents_runtime.py`:
- Around line 187-216: The validation in deepagents_runtime’s
_resolve_agent_skill_sources and _validate_sandbox_requirements is too
permissive: unknown keys in skills.agents can slip through, and require_sandbox
collection names are skipped whenever a sandbox is present. Update
_resolve_agent_skill_sources to reject agent names that are not among the known
DeepAgents agent names, and make _validate_sandbox_requirements always call
resolve_skill_collections on skills.require_sandbox before any sandbox-related
early return so invalid collection names are caught consistently.
In `@src/aiq_agent/common/citation_verification.py`:
- Around line 598-609: The source-section delimiter is too broad because
_REFERENCE_SECTION_RE matches a bare Sources/References line and can terminate
parsing before the actual citations block. Tighten the shared heading-matching
logic used by _REFERENCE_SECTION_RE and _REFERENCE_HEADING_LINE_RE so they only
recognize the intended references section, and define both regexes from one
shared pattern string to keep verify_citations and sanitize_report consistent.
- Around line 687-692: The normalization in citation_verification’s reference
cleanup is too broad and can split valid single-line citations when bracketed
numbers appear in titles. Update the final re.sub logic in the citation section
handling so it only inserts newlines for truly collapsed source entries, or add
stricter context checks before rewriting text in
_normalize_ordered_reference_lines / the ref_section normalization path.
Preserve single-line citations like entries with bracketed years in titles while
still normalizing only malformed source lines.
In `@tests/aiq_agent/agents/deep_researcher/test_custom_middleware.py`:
- Around line 128-147: The sync path in ToolVisibilityMiddleware is not covered,
so add a parallel test for wrap_model_call alongside
test_awrap_model_call_filters_hidden_tools. Reuse the same setup with
ToolVisibilityMiddleware, mock_request.override, and the same tool list to
verify hidden tools are filtered, then assert wrap_model_call returns the
handler result and calls override with only the visible tools.
In `@tests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.py`:
- Around line 123-139: Add regression coverage in DeepAgentsRuntime tests for
invalid mapping keys and sandbox-required skill names. Update the
DeepAgentsRuntime/DeepResearchSkillsConfig test suite to assert that typoed
agent subagent keys in agents are not silently ignored, and that require_sandbox
still validates skill names even when a DeepResearchSandboxConfig is present.
Place the new cases near the existing sandbox validation tests and use the
existing DeepAgentsRuntime, DeepResearchSkillsConfig, and
DeepResearchSandboxConfig symbols to keep the new tests aligned with the current
validation behavior.
---
Outside diff comments:
In `@frontends/aiq_api/src/aiq_api/jobs/runner.py`:
- Around line 595-619: Update the docstring for the agent creation helper in
runner.py so it matches the new control flow: the explicit
DeepResearchAgentConfig branch should be documented as the primary path before
the fallback constructor patterns. Keep the wording aligned with the symbols
already in the function (DeepResearchAgentConfig, agent_cls, llm_provider,
tools) and revise the “Tries in order” description to include the deep-research
keyword-arg branch plus the existing llm_provider + tools and llm + tools
fallbacks.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 0ddfa1e8-eac8-4905-b0d3-a5b0c41cad5e
📒 Files selected for processing (27)
configs/config_domain_routing_and_skills.ymldocs/source/architecture/agents/deep-researcher.mddocs/source/examples/index.mddocs/source/examples/skills-sandbox/index.mdfrontends/aiq_api/src/aiq_api/jobs/runner.pyfrontends/aiq_api/src/aiq_api/routes/jobs.pyfrontends/aiq_api/tests/test_periodic_cleanup.pyskills/aiq-deploy/references/configs.mdsrc/aiq_agent/agents/deep_researcher/agent.pysrc/aiq_agent/agents/deep_researcher/custom_middleware.pysrc/aiq_agent/agents/deep_researcher/deepagents_runtime.pysrc/aiq_agent/agents/deep_researcher/factory.pysrc/aiq_agent/agents/deep_researcher/prompts/orchestrator.j2src/aiq_agent/agents/deep_researcher/prompts/researcher.j2src/aiq_agent/agents/deep_researcher/prompts/writer.j2src/aiq_agent/agents/deep_researcher/register.pysrc/aiq_agent/agents/deep_researcher/skills/research/data-table-analysis/SKILL.mdsrc/aiq_agent/agents/deep_researcher/skills/research/forecast-analysis/SKILL.mdsrc/aiq_agent/agents/deep_researcher/skills/research/lightweight-calculation/SKILL.mdsrc/aiq_agent/agents/deep_researcher/skills/synthesis/long-form-report-writer/SKILL.mdsrc/aiq_agent/common/citation_verification.pytests/aiq_agent/agents/deep_researcher/test_agent.pytests/aiq_agent/agents/deep_researcher/test_custom_middleware.pytests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.pytests/aiq_agent/agents/deep_researcher/test_factory.pytests/aiq_agent/common/test_citation_verification.pytests/aiq_agent/jobs/test_runner.py
📜 Review details
⏰ Context from checks skipped due to timeout. (1)
- GitHub Check: Run Harbor skill eval
🧰 Additional context used
📓 Path-based instructions (11)
**
⚙️ CodeRabbit configuration file
**:AI-Q Agent Guidance
Repository-global instructions for coding agents and for humans reviewing
agent-authored changes. These rules apply to every task in this repository.
Task-specific runbooks live in.agents/skills/— load the
relevant skill before starting a workflow it covers.Project overview
AI-Q is an NVIDIA AI Blueprint: an enterprise research agent built on the
NeMo Agent Toolkit (NAT). The deployed product is a research blueprint, not
a general skill runtime. New retrieval sources and tools are NAT functions;
agent behavior is driven by workflow YAML, Jinja2 prompts, and a data-source
registry — not by hard-coded logic.Primary boundaries:
- Backend Python package:
src/aiq_agent/.- Data-source and tool packages:
sources/(each is its own package).- Frontends and tooling:
frontends/(web UI infrontends/ui/, eval harnesses
infrontends/benchmarks/).- Configs, deployment, docs:
configs/,deploy/,docs/.Stay inside this repository. If your workspace also contains adjacent repos
(for example a sibling NeMo-Relay checkout), do not edit them as part of an AI-Q
change. Treatsources/*as independent packages: prefer the smallest change
scoped to the package you are touching.Repository structure
Path Purpose src/aiq_agent/Backend agent, FastAPI extensions, auth, observability, knowledge sources/Data-source / tool packages (e.g. tavily_web_search,google_scholar_paper_search)configs/Workflow YAML configs (e.g. config_cli_default.yml)frontends/ui/Next.js / React / TypeScript / Tailwind / KUI web UI frontends/benchmarks/Eval harnesses: freshqa,deepsearch_qa,deepresearch_benchdeploy/Docker Compose and Helm/Kubernetes assets; deploy/.envfor secretsdocs/source/...
Files:
src/aiq_agent/agents/deep_researcher/prompts/orchestrator.j2docs/source/examples/index.mdsrc/aiq_agent/agents/deep_researcher/skills/synthesis/long-form-report-writer/SKILL.mdskills/aiq-deploy/references/configs.mdfrontends/aiq_api/tests/test_periodic_cleanup.pydocs/source/architecture/agents/deep-researcher.mdtests/aiq_agent/agents/deep_researcher/test_custom_middleware.pytests/aiq_agent/common/test_citation_verification.pyfrontends/aiq_api/src/aiq_api/routes/jobs.pysrc/aiq_agent/agents/deep_researcher/prompts/writer.j2frontends/aiq_api/src/aiq_api/jobs/runner.pyconfigs/config_domain_routing_and_skills.ymlsrc/aiq_agent/agents/deep_researcher/prompts/researcher.j2src/aiq_agent/agents/deep_researcher/custom_middleware.pytests/aiq_agent/agents/deep_researcher/test_factory.pysrc/aiq_agent/agents/deep_researcher/register.pydocs/source/examples/skills-sandbox/index.mdsrc/aiq_agent/agents/deep_researcher/factory.pysrc/aiq_agent/agents/deep_researcher/agent.pysrc/aiq_agent/common/citation_verification.pytests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.pysrc/aiq_agent/agents/deep_researcher/deepagents_runtime.pytests/aiq_agent/jobs/test_runner.pytests/aiq_agent/agents/deep_researcher/test_agent.py
src/aiq_agent/agents/**/*
⚙️ CodeRabbit configuration file
src/aiq_agent/agents/**/*: Review agent changes for research workflow correctness, graph state transitions, prompt/tool contracts,
HITL behavior, and failure handling. Flag changes that weaken source attribution, report generation,
async cancellation, checkpointing, or data-source selection without focused tests and docs.
Files:
src/aiq_agent/agents/deep_researcher/prompts/orchestrator.j2src/aiq_agent/agents/deep_researcher/skills/synthesis/long-form-report-writer/SKILL.mdsrc/aiq_agent/agents/deep_researcher/prompts/writer.j2src/aiq_agent/agents/deep_researcher/prompts/researcher.j2src/aiq_agent/agents/deep_researcher/custom_middleware.pysrc/aiq_agent/agents/deep_researcher/register.pysrc/aiq_agent/agents/deep_researcher/factory.pysrc/aiq_agent/agents/deep_researcher/agent.pysrc/aiq_agent/agents/deep_researcher/deepagents_runtime.py
docs/source/**/*
📄 CodeRabbit inference engine (AGENTS.md)
Update the docs under docs/source/ when behavior, configuration, or workflows change
Files:
docs/source/examples/index.mddocs/source/architecture/agents/deep-researcher.mddocs/source/examples/skills-sandbox/index.md
{docs/**,README.md,CONTRIBUTING.md,SECURITY.md,CODE-OF-CONDUCT.md}
⚙️ CodeRabbit configuration file
{docs/**,README.md,CONTRIBUTING.md,SECURITY.md,CODE-OF-CONDUCT.md}: Review documentation for command accuracy, branch-name consistency, current CI and copy-pr-bot behavior, public
vs internal boundary clarity, stale examples, and links that no longer match the repository layout.
Files:
docs/source/examples/index.mddocs/source/architecture/agents/deep-researcher.mddocs/source/examples/skills-sandbox/index.md
skills/aiq-deploy/**
⚙️ CodeRabbit configuration file
skills/aiq-deploy/**: ---
name: aiq-deploy
description: |
Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.
license: Apache-2.0
compatibility: |
Designed for Claude Code, OpenCode, Codex, and Agent Skills-compatible tools. Requires Git, network
access to GitHub, and one selected runtime path: Docker Compose v2 for the default local deployment,
Python 3.11+ and uv for local process or CLI mode, Node.js 20+ and npm for local web UI mode, or
kubectl 1.28+ and Helm 3.12+ for Kubernetes and Helm mode.
metadata:
version: "2.1.0"
author: "NVIDIA AI-Q Blueprint Team aiq-blueprint@nvidia.com"
github-url: "https://github.com/NVIDIA-AI-Blueprints/aiq"
tags:
- nvidia
- aiq
- blueprint
- deploy
- operations
- agent-skills
allowed-tools: Read BashAIQ Deploy Skill
Purpose
Use this skill to get a local or self-hosted NVIDIA AI-Q Blueprint server running and verified for use by
aiq-research.This skill owns setup, deployment, operational checks, troubleshooting, and shutdown. It does not run deep
research itself. After deployment is healthy, hand off the verified server URL toaiq-research.
The workflow stays explicit so deployment validation and handoff are repeatable across supported agent clients.Prerequisites
Users need:
- Access to clone or update
https://github.com/NVIDIA-AI-Blueprints/aiq.- Git available in the shell.
- One deployment runtime:
- Docker Engine with Docker Compose v2 for the default durable local deployment.
- Python 3.11+ and
uvfor local process or CLI mode.- Node.js 20+ and
npmfor local browser UI development mode.kubectl1.28+, Helm 3.12+, and access to a Kubernetes cluster for Helm mode.- Network access to GitHub, NVIDIA-hosted model endpoints, and any selected search provider.
- Credentials stored outside chat. Hosted-model usage requires
NVIDIA_API_KEY; web research requires at least
one supported s...
Files:
skills/aiq-deploy/references/configs.md
{skills/**,.agents/skills/**,.claude/skills/**,.github/skill-eval/**}
⚙️ CodeRabbit configuration file
{skills/**,.agents/skills/**,.claude/skills/**,.github/skill-eval/**}: Review Agent Skill and skill-eval changes for valid skill metadata, deterministic eval specs, safe handling of
credentials, and clear generated-output boundaries. Do not flag SKILL.md files for missing SPDX headers when the
entrypoint intentionally starts with YAML frontmatter.
Files:
skills/aiq-deploy/references/configs.md
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Run ruff check and ruff format validation for Python code changes
**/*.py: Python code must be linted and formatted with Ruff using line length 120, target Python 3.11, rule sets E,F,W,I,PL,UP, and isort force-single-line configuration
Never commit secrets, tokens, or environment-specific hostnames; use environment variables and SecretStr instead, resolving API keys at runtime
Never print or log secret values, including in tool output or error messages
Missing-secret paths must degrade gracefully (stub/skip), not crash or leak
Do not hand-reformat unrelated code when making changes; match the existing import and formatting style
Files:
frontends/aiq_api/tests/test_periodic_cleanup.pytests/aiq_agent/agents/deep_researcher/test_custom_middleware.pytests/aiq_agent/common/test_citation_verification.pyfrontends/aiq_api/src/aiq_api/routes/jobs.pyfrontends/aiq_api/src/aiq_api/jobs/runner.pysrc/aiq_agent/agents/deep_researcher/custom_middleware.pytests/aiq_agent/agents/deep_researcher/test_factory.pysrc/aiq_agent/agents/deep_researcher/register.pysrc/aiq_agent/agents/deep_researcher/factory.pysrc/aiq_agent/agents/deep_researcher/agent.pysrc/aiq_agent/common/citation_verification.pytests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.pysrc/aiq_agent/agents/deep_researcher/deepagents_runtime.pytests/aiq_agent/jobs/test_runner.pytests/aiq_agent/agents/deep_researcher/test_agent.py
**/*test*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Run pytest for all behavior changes in Python code
Files:
frontends/aiq_api/tests/test_periodic_cleanup.pytests/aiq_agent/agents/deep_researcher/test_custom_middleware.pytests/aiq_agent/common/test_citation_verification.pytests/aiq_agent/agents/deep_researcher/test_factory.pytests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.pytests/aiq_agent/jobs/test_runner.pytests/aiq_agent/agents/deep_researcher/test_agent.py
{src/aiq_agent/fastapi_extensions/**,frontends/aiq_api/src/aiq_api/**}
⚙️ CodeRabbit configuration file
{src/aiq_agent/fastapi_extensions/**,frontends/aiq_api/src/aiq_api/**}: Treat API, auth, and job-runner changes as externally visible contracts. Check authorization boundaries,
request tracing, async job lifecycle, websocket reconnect behavior, error responses, and cross-user data isolation.
Require tests for route behavior, access decisions, and job state transitions when those surfaces change.
Files:
frontends/aiq_api/src/aiq_api/routes/jobs.pyfrontends/aiq_api/src/aiq_api/jobs/runner.py
{deploy/**,configs/**}
⚙️ CodeRabbit configuration file
{deploy/**,configs/**}: Review deployment and config changes for secret separation, safe defaults, local-vs-production behavior, Helm and
Docker portability, and documentation parity. Flag committed credentials, environment-specific NVIDIA internals in
public defaults, and changes that make examples diverge from CI-tested paths.
Files:
configs/config_domain_routing_and_skills.yml
src/aiq_agent/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
src/aiq_agent/**/*.py: Respect authenticated data sources by honoring requires_auth, per-user token pass-through, and backend token validators; apply owner guardrails before loading protected report or artifact context into an agent
Do not weaken or bypass AuthMiddleware, validators, or auth gating without a prior design discussion
Files:
src/aiq_agent/agents/deep_researcher/custom_middleware.pysrc/aiq_agent/agents/deep_researcher/register.pysrc/aiq_agent/agents/deep_researcher/factory.pysrc/aiq_agent/agents/deep_researcher/agent.pysrc/aiq_agent/common/citation_verification.pysrc/aiq_agent/agents/deep_researcher/deepagents_runtime.py
🧠 Learnings (1)
📚 Learning: 2026-06-14T17:49:00.640Z
Learnt from: torkian
Repo: NVIDIA-AI-Blueprints/aiq PR: 273
File: frontends/aiq_api/tests/test_sse_reconnect_cursor.py:384-401
Timestamp: 2026-06-14T17:49:00.640Z
Learning: When using `unittest.mock.patch` for code that imports dependencies inside functions/generators (e.g., inside `aiq_api.routes.jobs`), don’t patch via an attribute that doesn’t exist on the consuming module. If the generator does `from ..jobs.event_store import EventStore` inside the generator body, then `aiq_api.routes.jobs` will not have an `EventStore` attribute; patch the source class/method in its defining module instead (e.g., `aiq_api.jobs.event_store.EventStore.get_events_async`). Patching `aiq_api.routes.jobs.EventStore...` would raise `AttributeError` because that symbol is not present at module scope.
Applied to files:
frontends/aiq_api/tests/test_periodic_cleanup.py
🪛 GitHub Actions: AIQ CI / 1_Pytest and Coverage.txt
src/aiq_agent/common/citation_verification.py
[warning] 796-796: [CitationVerify] No source section found in report; skipping
🪛 GitHub Actions: AIQ CI / Pytest and Coverage
src/aiq_agent/common/citation_verification.py
[warning] 796-796: [CitationVerify] No source section found in report; skipping
🔇 Additional comments (27)
src/aiq_agent/agents/deep_researcher/custom_middleware.py (2)
164-178: LGTM!
181-198: 🩺 Stability & AvailabilityNo base initializer call needed here.
ToolVisibilityMiddlewarematches the other middleware classes in this file, and this code path doesn’t rely onAgentMiddleware.__init__()state.> Likely an incorrect or invalid review comment.src/aiq_agent/agents/deep_researcher/factory.py (3)
226-273: LGTM!
284-397: LGTM!
417-471: LGTM!tests/aiq_agent/agents/deep_researcher/test_factory.py (1)
97-120: LGTM!Also applies to: 155-296
frontends/aiq_api/src/aiq_api/routes/jobs.py (1)
934-946: LGTM!frontends/aiq_api/tests/test_periodic_cleanup.py (1)
479-519: LGTM!src/aiq_agent/agents/deep_researcher/deepagents_runtime.py (1)
20-160: LGTM!Also applies to: 228-250, 253-366
src/aiq_agent/agents/deep_researcher/register.py (1)
19-24: LGTM!Also applies to: 47-60, 83-93, 111-174, 229-231, 243-263
configs/config_domain_routing_and_skills.yml (1)
175-195: LGTM!Also applies to: 218-219
tests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.py (1)
16-68: LGTM!Also applies to: 72-121, 141-247
skills/aiq-deploy/references/configs.md (1)
30-30: LGTM!frontends/aiq_api/src/aiq_api/jobs/runner.py (1)
343-349: LGTM!tests/aiq_agent/jobs/test_runner.py (1)
1376-1449: LGTM!Also applies to: 1631-1674
docs/source/architecture/agents/deep-researcher.md (1)
144-146: LGTM!docs/source/examples/index.md (1)
17-17: LGTM!docs/source/examples/skills-sandbox/index.md (1)
90-104: LGTM!Also applies to: 189-197
src/aiq_agent/agents/deep_researcher/agent.py (2)
37-38: LGTM!Also applies to: 68-70, 104-104
212-249: LGTM!src/aiq_agent/agents/deep_researcher/prompts/orchestrator.j2 (1)
17-18: LGTM!src/aiq_agent/agents/deep_researcher/prompts/researcher.j2 (1)
11-11: LGTM!Also applies to: 49-51, 81-83
src/aiq_agent/agents/deep_researcher/prompts/writer.j2 (1)
61-69: LGTM!src/aiq_agent/common/citation_verification.py (1)
16-21: LGTM!Also applies to: 792-832, 912-925
src/aiq_agent/agents/deep_researcher/skills/synthesis/long-form-report-writer/SKILL.md (1)
68-70: LGTM!tests/aiq_agent/agents/deep_researcher/test_agent.py (1)
185-282: LGTM!Also applies to: 344-374, 401-401, 432-435, 524-548, 942-1016
tests/aiq_agent/common/test_citation_verification.py (1)
674-686: LGTM!Note: none of these cases cover a source line whose title contains a bracketed number (see the
_normalize_source_section_layoutcomment incitation_verification.py); adding such a case would lock in the expected behavior once that split is fixed.Also applies to: 933-933, 1065-1075
Signed-off-by: Chantal D Gama Rose <cdgamarose@nvidia.com>
Signed-off-by: Chantal D Gama Rose <cdgamarose@nvidia.com>
Signed-off-by: Chantal D Gama Rose <cdgamarose@nvidia.com>
There was a problem hiding this comment.
♻️ Duplicate comments (1)
src/aiq_agent/common/citation_verification.py (1)
607-608: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winBody prose can still be mistaken for the source section.
Line 608 still accepts any post-heading line that starts with
[N]as a source entry. A report body likeSources\n[1] Evidence shows...will still split before the real later## Sourcesblock. Please strengthen the boundary check here (or choose the last viable heading) and add a regression test for that shape.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/aiq_agent/common/citation_verification.py` around lines 607 - 608, The reference-section detection in citation_verification.py is still too permissive because _REFERENCE_SECTION_RE treats any immediate [N] line after a heading as a valid source block, which can cause body prose like “Sources\n[1] ...” to be split incorrectly. Tighten the boundary logic in _REFERENCE_SECTION_RE (or update the selection logic to prefer the last valid heading in the document) so it only matches an actual sources section, and add a regression test covering a report body that contains a false “Sources” heading before a later real “## Sources” block.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In `@src/aiq_agent/common/citation_verification.py`:
- Around line 607-608: The reference-section detection in
citation_verification.py is still too permissive because _REFERENCE_SECTION_RE
treats any immediate [N] line after a heading as a valid source block, which can
cause body prose like “Sources\n[1] ...” to be split incorrectly. Tighten the
boundary logic in _REFERENCE_SECTION_RE (or update the selection logic to prefer
the last valid heading in the document) so it only matches an actual sources
section, and add a regression test covering a report body that contains a false
“Sources” heading before a later real “## Sources” block.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 28359917-ee3c-46f4-9263-c69cee99cb8c
📒 Files selected for processing (6)
src/aiq_agent/agents/deep_researcher/deepagents_runtime.pysrc/aiq_agent/common/citation_verification.pytests/aiq_agent/agents/deep_researcher/test_custom_middleware.pytests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.pytests/aiq_agent/agents/shallow_researcher/test_agent.pytests/aiq_agent/common/test_citation_verification.py
📜 Review details
⏰ Context from checks skipped due to timeout. (1)
- GitHub Check: Run Harbor skill eval
🧰 Additional context used
📓 Path-based instructions (5)
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Run ruff check and ruff format validation for Python code changes
**/*.py: Python code must be linted and formatted with Ruff using line length 120, target Python 3.11, rule sets E,F,W,I,PL,UP, and isort force-single-line configuration
Never commit secrets, tokens, or environment-specific hostnames; use environment variables and SecretStr instead, resolving API keys at runtime
Never print or log secret values, including in tool output or error messages
Missing-secret paths must degrade gracefully (stub/skip), not crash or leak
Do not hand-reformat unrelated code when making changes; match the existing import and formatting style
Files:
tests/aiq_agent/agents/deep_researcher/test_custom_middleware.pytests/aiq_agent/agents/shallow_researcher/test_agent.pytests/aiq_agent/common/test_citation_verification.pytests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.pysrc/aiq_agent/agents/deep_researcher/deepagents_runtime.pysrc/aiq_agent/common/citation_verification.py
**/*test*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Run pytest for all behavior changes in Python code
Files:
tests/aiq_agent/agents/deep_researcher/test_custom_middleware.pytests/aiq_agent/agents/shallow_researcher/test_agent.pytests/aiq_agent/common/test_citation_verification.pytests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.py
**
⚙️ CodeRabbit configuration file
**:AI-Q Agent Guidance
Repository-global instructions for coding agents and for humans reviewing
agent-authored changes. These rules apply to every task in this repository.
Task-specific runbooks live in.agents/skills/— load the
relevant skill before starting a workflow it covers.Project overview
AI-Q is an NVIDIA AI Blueprint: an enterprise research agent built on the
NeMo Agent Toolkit (NAT). The deployed product is a research blueprint, not
a general skill runtime. New retrieval sources and tools are NAT functions;
agent behavior is driven by workflow YAML, Jinja2 prompts, and a data-source
registry — not by hard-coded logic.Primary boundaries:
- Backend Python package:
src/aiq_agent/.- Data-source and tool packages:
sources/(each is its own package).- Frontends and tooling:
frontends/(web UI infrontends/ui/, eval harnesses
infrontends/benchmarks/).- Configs, deployment, docs:
configs/,deploy/,docs/.Stay inside this repository. If your workspace also contains adjacent repos
(for example a sibling NeMo-Relay checkout), do not edit them as part of an AI-Q
change. Treatsources/*as independent packages: prefer the smallest change
scoped to the package you are touching.Repository structure
Path Purpose src/aiq_agent/Backend agent, FastAPI extensions, auth, observability, knowledge sources/Data-source / tool packages (e.g. tavily_web_search,google_scholar_paper_search)configs/Workflow YAML configs (e.g. config_cli_default.yml)frontends/ui/Next.js / React / TypeScript / Tailwind / KUI web UI frontends/benchmarks/Eval harnesses: freshqa,deepsearch_qa,deepresearch_benchdeploy/Docker Compose and Helm/Kubernetes assets; deploy/.envfor secretsdocs/source/...
Files:
tests/aiq_agent/agents/deep_researcher/test_custom_middleware.pytests/aiq_agent/agents/shallow_researcher/test_agent.pytests/aiq_agent/common/test_citation_verification.pytests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.pysrc/aiq_agent/agents/deep_researcher/deepagents_runtime.pysrc/aiq_agent/common/citation_verification.py
src/aiq_agent/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
src/aiq_agent/**/*.py: Respect authenticated data sources by honoring requires_auth, per-user token pass-through, and backend token validators; apply owner guardrails before loading protected report or artifact context into an agent
Do not weaken or bypass AuthMiddleware, validators, or auth gating without a prior design discussion
Files:
src/aiq_agent/agents/deep_researcher/deepagents_runtime.pysrc/aiq_agent/common/citation_verification.py
src/aiq_agent/agents/**/*
⚙️ CodeRabbit configuration file
src/aiq_agent/agents/**/*: Review agent changes for research workflow correctness, graph state transitions, prompt/tool contracts,
HITL behavior, and failure handling. Flag changes that weaken source attribution, report generation,
async cancellation, checkpointing, or data-source selection without focused tests and docs.
Files:
src/aiq_agent/agents/deep_researcher/deepagents_runtime.py
🔇 Additional comments (3)
src/aiq_agent/agents/deep_researcher/deepagents_runtime.py (1)
39-39: LGTM!Also applies to: 48-48, 65-74, 214-220
tests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.py (1)
132-135: LGTM!Also applies to: 145-152
tests/aiq_agent/agents/deep_researcher/test_custom_middleware.py (1)
131-145: LGTM!
AjayThorve
left a comment
AjayThorve
left a comment
There was a problem hiding this comment.
Local run works really well, good to go
Signed-off-by: Chantal D Gama Rose <cdgamarose@nvidia.com>
There was a problem hiding this comment.
♻️ Duplicate comments (1)
docs/source/examples/skills-sandbox/index.md (1)
56-87: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winKeep this YAML block exactly aligned with
configs/config_domain_routing_and_skills.yml.The section says “the relevant section is,” but this snippet appears to remain a simplified variant (prior review flagged
image/packagesdrift). Either make it byte-for-byte consistent with the referenced config or label it explicitly as illustrative. As per path instructions, “Review documentation for command accuracy, branch-name consistency, current CI and copy-pr-bot behavior, public vs internal boundary clarity, stale examples, and links that no longer match the repository layout.”🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/source/examples/skills-sandbox/index.md` around lines 56 - 87, The YAML example in the skills-sandbox docs is still drifting from configs/config_domain_routing_and_skills.yml, so update the snippet in the relevant section to match the referenced config exactly or clearly mark it as illustrative. Keep the deep_research_skills, deep_research_sandbox, and deep_research_agent entries aligned with the source config, including the sandbox image/packages/network details, so readers see the same command and config structure.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In `@docs/source/examples/skills-sandbox/index.md`:
- Around line 56-87: The YAML example in the skills-sandbox docs is still
drifting from configs/config_domain_routing_and_skills.yml, so update the
snippet in the relevant section to match the referenced config exactly or
clearly mark it as illustrative. Keep the deep_research_skills,
deep_research_sandbox, and deep_research_agent entries aligned with the source
config, including the sandbox image/packages/network details, so readers see the
same command and config structure.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 43b7d22e-ec17-4bb5-9f1e-7b21cea374b0
📒 Files selected for processing (5)
configs/config_cli_default.ymlconfigs/config_frontier_models.ymlconfigs/config_web_default_llamaindex.ymlconfigs/config_web_frag.ymldocs/source/examples/skills-sandbox/index.md
📜 Review details
⏰ Context from checks skipped due to timeout. (1)
- GitHub Check: Run Harbor skill eval
🧰 Additional context used
📓 Path-based instructions (4)
**
⚙️ CodeRabbit configuration file
**:AI-Q Agent Guidance
Repository-global instructions for coding agents and for humans reviewing
agent-authored changes. These rules apply to every task in this repository.
Task-specific runbooks live in.agents/skills/— load the
relevant skill before starting a workflow it covers.Project overview
AI-Q is an NVIDIA AI Blueprint: an enterprise research agent built on the
NeMo Agent Toolkit (NAT). The deployed product is a research blueprint, not
a general skill runtime. New retrieval sources and tools are NAT functions;
agent behavior is driven by workflow YAML, Jinja2 prompts, and a data-source
registry — not by hard-coded logic.Primary boundaries:
- Backend Python package:
src/aiq_agent/.- Data-source and tool packages:
sources/(each is its own package).- Frontends and tooling:
frontends/(web UI infrontends/ui/, eval harnesses
infrontends/benchmarks/).- Configs, deployment, docs:
configs/,deploy/,docs/.Stay inside this repository. If your workspace also contains adjacent repos
(for example a sibling NeMo-Relay checkout), do not edit them as part of an AI-Q
change. Treatsources/*as independent packages: prefer the smallest change
scoped to the package you are touching.Repository structure
Path Purpose src/aiq_agent/Backend agent, FastAPI extensions, auth, observability, knowledge sources/Data-source / tool packages (e.g. tavily_web_search,google_scholar_paper_search)configs/Workflow YAML configs (e.g. config_cli_default.yml)frontends/ui/Next.js / React / TypeScript / Tailwind / KUI web UI frontends/benchmarks/Eval harnesses: freshqa,deepsearch_qa,deepresearch_benchdeploy/Docker Compose and Helm/Kubernetes assets; deploy/.envfor secretsdocs/source/...
Files:
configs/config_web_default_llamaindex.ymlconfigs/config_cli_default.ymlconfigs/config_frontier_models.ymlconfigs/config_web_frag.ymldocs/source/examples/skills-sandbox/index.md
{deploy/**,configs/**}
⚙️ CodeRabbit configuration file
{deploy/**,configs/**}: Review deployment and config changes for secret separation, safe defaults, local-vs-production behavior, Helm and
Docker portability, and documentation parity. Flag committed credentials, environment-specific NVIDIA internals in
public defaults, and changes that make examples diverge from CI-tested paths.
Files:
configs/config_web_default_llamaindex.ymlconfigs/config_cli_default.ymlconfigs/config_frontier_models.ymlconfigs/config_web_frag.yml
docs/source/**/*
📄 CodeRabbit inference engine (AGENTS.md)
Update the docs under docs/source/ when behavior, configuration, or workflows change
Files:
docs/source/examples/skills-sandbox/index.md
{docs/**,README.md,CONTRIBUTING.md,SECURITY.md,CODE-OF-CONDUCT.md}
⚙️ CodeRabbit configuration file
{docs/**,README.md,CONTRIBUTING.md,SECURITY.md,CODE-OF-CONDUCT.md}: Review documentation for command accuracy, branch-name consistency, current CI and copy-pr-bot behavior, public
vs internal boundary clarity, stale examples, and links that no longer match the repository layout.
Files:
docs/source/examples/skills-sandbox/index.md
🔇 Additional comments (4)
configs/config_cli_default.yml (1)
131-131: LGTM!configs/config_frontier_models.yml (1)
151-151: LGTM!configs/config_web_default_llamaindex.yml (1)
205-205: LGTM!configs/config_web_frag.yml (1)
173-173: LGTM!
Add the image-processing reference skill under the deep researcher's research/ skill collection (Pillow: metadata, format conversion, resize/thumbnail, crop, rotate, grayscale, basic color/brightness), mirroring the existing research skills: process in /workspace, then RETURN text-survivable results (JSON/Markdown metadata, optional small base64 thumbnail) in the researcher's ResearchNotes, which run_research_batch persists to /shared. The researcher must not call write_file (research.py), so the skill uses the return-in-ResearchNotes pattern. Rebased onto develop after NVIDIA-AI-Blueprints#284 restructured the skills layout (research-sandbox -> research) and rewrote the runtime/test. Placed at skills/research/image-processing/; no test change needed -- the post-NVIDIA-AI-Blueprints#284 test enumerates only synthesis skills, so research skills (incl. this one) are auto-discovered. Covers SKILLS-REF-1's image-processing category. No durable binary-artifact capture on develop, so outputs are text. Validation: uv run pytest tests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.py (18 passed). Signed-off-by: Zhongxuan Wang <daniewang@nvidia.com>
Add the image-processing reference skill under the deep researcher's research/ skill collection (Pillow: metadata, format conversion, resize/thumbnail, crop, rotate, grayscale, basic color/brightness), mirroring the existing research skills: process in /workspace, then RETURN text-survivable results (JSON/Markdown metadata, optional small base64 thumbnail) in the researcher's ResearchNotes, which run_research_batch persists to /shared. The researcher must not call write_file (research.py), so the skill uses the return-in-ResearchNotes pattern. Rebased onto develop after NVIDIA-AI-Blueprints#284 restructured the skills layout (research-sandbox -> research) and rewrote the runtime/test. Placed at skills/research/image-processing/; no test change needed -- the post-NVIDIA-AI-Blueprints#284 test enumerates only synthesis skills, so research skills (incl. this one) are auto-discovered. Covers SKILLS-REF-1's image-processing category. No durable binary-artifact capture on develop, so outputs are text. Validation: uv run pytest tests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.py (18 passed). Signed-off-by: Zhongxuan Wang <daniewang@nvidia.com>
2 participants
Overview
This PR rewrites the deep research DeepAgents runtime configuration around explicit, named skill and sandbox configs. Instead of exposing DeepAgents virtual paths in user YAML, skills are assigned by stable collection names per subagent, and sandbox requirements are declared by collection name. Sandbox configuration is now a separate
deep_research_sandboxfunction reference, keeping the deep research agent config focused on wiring rather than provider details.The runtime now uses the simplest backend needed for each run: plain
StateBackendwhen no skills or sandbox are configured, a read-onlyFilesystemBackendmounted at/skills/when built-in skills are enabled, and sandbox execution only when sandbox config is explicitly present./shared/is only added for sandbox-backed runs. This avoids enabling execution accidentally and makes missing Modal dependencies fail only when sandbox is actually configured.The PR also makes citation verification optional and keeps report post-processing responsibilities clearer: citation verification checks source identity, while sanitization handles output hygiene and source-section display normalization.
Validation
git commit -sor an equivalent sign-off.Where should reviewers start?
Use
configs/config_domain_routing_and_skills.ymlto test with citation verification disabled and with per agent skills and sandbox.Related Issues
Summary by CodeRabbit
## Sourceshandling.