feat: enable Nemotron 3.5 Lightning defaults - #430
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (1)
📜 Recent review details⏰ Context from checks skipped due to timeout. (7)
🧰 Additional context used📓 Path-based instructions (4)**/*📄 CodeRabbit inference engine (CONTRIBUTING.md)
Files:
**/*.{py,pyi,js,jsx,ts,tsx,yml,yaml,json,env,md}📄 CodeRabbit inference engine (AGENTS.md)
Files:
docs/source/**/*📄 CodeRabbit inference engine (AGENTS.md)
Files:
{docs/**,README.md,CONTRIBUTING.md,SECURITY.md,CODE-OF-CONDUCT.md}⚙️ CodeRabbit configuration file
Files:
🔇 Additional comments (1)
WalkthroughThe pull request migrates intent classification and shallow research to Nemotron Lightning profiles. It also enforces research-tool use before shallow-research answers and suppresses intermediate output artifacts. ChangesModel and shallow research updates
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant ShallowResearcher
participant ResearchLLM
participant ResearchTool
participant OutputCallbacks
ShallowResearcher->>ResearchLLM: Request initial research response
alt No tool call
ShallowResearcher->>ResearchLLM: Retry with one allowed tool call
ResearchLLM->>ResearchTool: Execute research query
ResearchTool-->>ResearchLLM: Return research results
ResearchLLM-->>ShallowResearcher: Return evidence-backed answer
else Invalid retry response
ShallowResearcher-->>ShallowResearcher: Raise RuntimeError
end
ShallowResearcher->>OutputCallbacks: Suppress tagged tokens and artifacts
Possibly related PRs
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@configs/config_web_default_llamaindex.yml`:
- Line 72: ShallowResearcherAgent.agent_node must pass the configured
parallel_tool_calls value to llm.bind_tools instead of hard-coding true, and add
an agent-level regression test verifying the binding. The anchor
configs/config_web_default_llamaindex.yml:72, configs/config_web_frag.yml:74,
configs/config_web_frag_mcp_auth.yml:81, and
configs/config_web_opensearch.yml:73 require no direct change;
docs/notebooks/0_Getting_Started_with_AIQ.ipynb:328 and
docs/source/customization/configuration-reference.md:122, 132, 613 likewise
require no direct change. Update
tests/aiq_agent/test_default_model_profiles.py:104-113 as needed to cover the
runtime binding behavior.
In `@docs/notebooks/0_Getting_Started_with_AIQ.ipynb`:
- Around line 807-813: Update the Together.ai guidance text in the notebook to
refer to the defined “Lightning or Ultra LLM block” profiles instead of “Nano or
Ultra LLM block,” keeping the surrounding provider-change example unchanged.
In `@docs/source/examples/minimal-shallow-only.md`:
- Around line 28-29: Update the description near the model configuration to
identify the role-specific model as Nemotron 3.5 Lightning instead of Nano,
while preserving the existing explanation that the NIM type connects to NVIDIA
API Catalog.
In `@src/aiq_agent/agents/shallow_researcher/agent.py`:
- Around line 252-260: Update the response flow around the initial model
invocation and the tool-use check so callbacks cannot publish either rejected
no-tool response, including when the retry also lacks tool calls and run()
raises. Buffer or suppress final-answer callbacks until a valid tool call
succeeds, while preserving normal callback delivery afterward, and add a
regression test covering the callback behavior for this failure path.
- Around line 254-264: Update the retry path in
src/aiq_agent/agents/shallow_researcher/agent.py around the
llm_with_tools.ainvoke call to use a retry-specific tool binding with
parallel_tool_calls=False, then reject responses unless they contain exactly one
call to an allowed research tool before ToolNode execution. Preserve the
existing failure behavior for invalid retries. Add the requested two-call retry
regression test in tests/aiq_agent/agents/shallow_researcher/test_agent.py at
the cited range, asserting the retry fails closed without executing multiple
calls or exceeding max_tool_iterations.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 27d47083-c4db-4832-b183-1215d187b30f
📒 Files selected for processing (33)
.agents/skills/aiq-configure-workflow/assets/config-scaffold.yml.agents/skills/aiq-configure-workflow/references/config-profiles.md.secrets.baselineREADME.mdconfigs/config_cli_default.ymlconfigs/config_frontier_models.ymlconfigs/config_mcp.ymlconfigs/config_openshell.ymlconfigs/config_web_azure_ai_search.ymlconfigs/config_web_default_guardrails.ymlconfigs/config_web_default_llamaindex.ymlconfigs/config_web_frag.ymlconfigs/config_web_frag_mcp_auth.ymlconfigs/config_web_opensearch.ymldocs/notebooks/0_Getting_Started_with_AIQ.ipynbdocs/source/customization/configuration-reference.mddocs/source/customization/mcp-tools.mddocs/source/customization/swapping-models.mddocs/source/customization/tools-and-sources.mddocs/source/examples/full-pipeline-llamaindex.mddocs/source/examples/full-pipeline-web.mddocs/source/examples/hybrid-frontier-model.mddocs/source/examples/minimal-shallow-only.mddocs/source/extending/adding-a-tool.mddocs/source/get-started/installation.mddocs/source/resources/troubleshooting.mdfrontends/benchmarks/freshqa/configs/config_full_workflow.ymlfrontends/benchmarks/freshqa/configs/config_shallow_research_only.ymlmcp/tests/test_preclassification_integration.pysrc/aiq_agent/agents/shallow_researcher/agent.pysrc/aiq_agent/agents/shallow_researcher/prompts/researcher.j2tests/aiq_agent/agents/shallow_researcher/test_agent.pytests/aiq_agent/test_default_model_profiles.py
📜 Review details
⏰ Context from checks skipped due to timeout. (2)
- GitHub Check: Lint and Hooks
- GitHub Check: Pytest and Coverage
🧰 Additional context used
📓 Path-based instructions (15)
**/*
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*: Do not include secrets, credentials, private hostnames, internal-only logs, customer data, or generated local artifacts.
Add or update tests for behavior changes.
**/*: For substantial behavior, authentication, UI, or architecture changes, open a design discussion before coding rather than landing a large unreviewed change.
Keep changes scoped to this repository and avoid editing adjacent repositories; treat eachsources/*package independently and prefer the smallest package-scoped change.
Keep pull requests scoped, avoid unrelated files and generated artifacts, provide validation evidence, and ensure every commit has DCO sign-off.
Files:
docs/source/customization/mcp-tools.mdsrc/aiq_agent/agents/shallow_researcher/prompts/researcher.j2src/aiq_agent/agents/shallow_researcher/agent.pydocs/source/examples/minimal-shallow-only.mdconfigs/config_web_default_guardrails.ymldocs/source/examples/full-pipeline-llamaindex.mddocs/source/get-started/installation.mdconfigs/config_openshell.ymlconfigs/config_web_azure_ai_search.ymlconfigs/config_frontier_models.ymlmcp/tests/test_preclassification_integration.pydocs/source/customization/tools-and-sources.mdconfigs/config_cli_default.ymldocs/source/resources/troubleshooting.mddocs/source/customization/swapping-models.mddocs/source/extending/adding-a-tool.mddocs/source/examples/hybrid-frontier-model.mdconfigs/config_mcp.ymlconfigs/config_web_frag_mcp_auth.ymlfrontends/benchmarks/freshqa/configs/config_full_workflow.ymltests/aiq_agent/agents/shallow_researcher/test_agent.pydocs/notebooks/0_Getting_Started_with_AIQ.ipynbconfigs/config_web_default_llamaindex.ymlfrontends/benchmarks/freshqa/configs/config_shallow_research_only.ymldocs/source/examples/full-pipeline-web.mdtests/aiq_agent/test_default_model_profiles.pydocs/source/customization/configuration-reference.mdconfigs/config_web_frag.ymlREADME.mdconfigs/config_web_opensearch.yml
**/*.{py,pyi,js,jsx,ts,tsx,yml,yaml,json,env,md}
📄 CodeRabbit inference engine (AGENTS.md)
Never commit secrets, tokens, or environment-specific hostnames; use environment variables and
SecretStr, and resolve API keys at runtime.
Files:
docs/source/customization/mcp-tools.mdsrc/aiq_agent/agents/shallow_researcher/agent.pydocs/source/examples/minimal-shallow-only.mdconfigs/config_web_default_guardrails.ymldocs/source/examples/full-pipeline-llamaindex.mddocs/source/get-started/installation.mdconfigs/config_openshell.ymlconfigs/config_web_azure_ai_search.ymlconfigs/config_frontier_models.ymlmcp/tests/test_preclassification_integration.pydocs/source/customization/tools-and-sources.mdconfigs/config_cli_default.ymldocs/source/resources/troubleshooting.mddocs/source/customization/swapping-models.mddocs/source/extending/adding-a-tool.mddocs/source/examples/hybrid-frontier-model.mdconfigs/config_mcp.ymlconfigs/config_web_frag_mcp_auth.ymlfrontends/benchmarks/freshqa/configs/config_full_workflow.ymltests/aiq_agent/agents/shallow_researcher/test_agent.pyconfigs/config_web_default_llamaindex.ymlfrontends/benchmarks/freshqa/configs/config_shallow_research_only.ymldocs/source/examples/full-pipeline-web.mdtests/aiq_agent/test_default_model_profiles.pydocs/source/customization/configuration-reference.mdconfigs/config_web_frag.ymlREADME.mdconfigs/config_web_opensearch.yml
docs/source/**/*
📄 CodeRabbit inference engine (AGENTS.md)
Update canonical documentation under
docs/source/when behavior, configuration, or workflows change; do not duplicate full documentation pages in skills.
Files:
docs/source/customization/mcp-tools.mddocs/source/examples/minimal-shallow-only.mddocs/source/examples/full-pipeline-llamaindex.mddocs/source/get-started/installation.mddocs/source/customization/tools-and-sources.mddocs/source/resources/troubleshooting.mddocs/source/customization/swapping-models.mddocs/source/extending/adding-a-tool.mddocs/source/examples/hybrid-frontier-model.mddocs/source/examples/full-pipeline-web.mddocs/source/customization/configuration-reference.md
{docs/**,README.md,CONTRIBUTING.md,SECURITY.md,CODE-OF-CONDUCT.md}
⚙️ CodeRabbit configuration file
{docs/**,README.md,CONTRIBUTING.md,SECURITY.md,CODE-OF-CONDUCT.md}: Review documentation for command accuracy, branch-name consistency, current CI and copy-pr-bot behavior, public
vs internal boundary clarity, stale examples, and links that no longer match the repository layout.
Files:
docs/source/customization/mcp-tools.mddocs/source/examples/minimal-shallow-only.mddocs/source/examples/full-pipeline-llamaindex.mddocs/source/get-started/installation.mddocs/source/customization/tools-and-sources.mddocs/source/resources/troubleshooting.mddocs/source/customization/swapping-models.mddocs/source/extending/adding-a-tool.mddocs/source/examples/hybrid-frontier-model.mddocs/notebooks/0_Getting_Started_with_AIQ.ipynbdocs/source/examples/full-pipeline-web.mddocs/source/customization/configuration-reference.mdREADME.md
{skills/**,.agents/skills/**,.claude/skills/**,.github/skill-eval/**}
⚙️ CodeRabbit configuration file
{skills/**,.agents/skills/**,.claude/skills/**,.github/skill-eval/**}: Review Agent Skill and skill-eval changes for valid skill metadata, deterministic eval specs, safe handling of
credentials, and clear generated-output boundaries. Do not flag SKILL.md files for missing SPDX headers when the
entrypoint intentionally starts with YAML frontmatter.
Files:
.agents/skills/aiq-configure-workflow/references/config-profiles.md.agents/skills/aiq-configure-workflow/assets/config-scaffold.yml
src/aiq_agent/agents/**/*
⚙️ CodeRabbit configuration file
src/aiq_agent/agents/**/*: Review agent changes for research workflow correctness, graph state transitions, prompt/tool contracts,
HITL behavior, and failure handling. Flag changes that weaken source attribution, report generation,
async cancellation, checkpointing, or data-source selection without focused tests and docs.
Files:
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2src/aiq_agent/agents/shallow_researcher/agent.py
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.py: Runuv run ruff check .anduv run ruff format --check .for root Python changes.
Runuv run pytestfor root project Python changes.
Files:
src/aiq_agent/agents/shallow_researcher/agent.pymcp/tests/test_preclassification_integration.pytests/aiq_agent/agents/shallow_researcher/test_agent.pytests/aiq_agent/test_default_model_profiles.py
**/*.{py,pyi}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{py,pyi}: Format and lint Python code with Ruff using line length 120, Python 3.11 targeting, rules E, F, W, I, PL, and UP, with single-line imports; do not reformat unrelated code.
Missing-secret paths must degrade gracefully by stubbing or skipping rather than crashing or leaking secrets.
Files:
src/aiq_agent/agents/shallow_researcher/agent.pymcp/tests/test_preclassification_integration.pytests/aiq_agent/agents/shallow_researcher/test_agent.pytests/aiq_agent/test_default_model_profiles.py
**/*.{py,pyi,js,jsx,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Never print or log secret values, including in tool output or error messages.
Files:
src/aiq_agent/agents/shallow_researcher/agent.pymcp/tests/test_preclassification_integration.pytests/aiq_agent/agents/shallow_researcher/test_agent.pytests/aiq_agent/test_default_model_profiles.py
src/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
src/**/*.py: Respect authenticated data sources by honoringrequires_auth, passing through per-user tokens, and using backend token validators; apply owner guardrails before loading protected report or artifact context into an agent.
Do not weaken or bypassAuthMiddleware, authentication validators, or authentication gating without prior design discussion.
Files:
src/aiq_agent/agents/shallow_researcher/agent.py
**/*.{yaml,yml}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
For deployment changes, run the relevant Helm or Compose validation and describe the environment used.
Files:
configs/config_web_default_guardrails.ymlconfigs/config_openshell.ymlconfigs/config_web_azure_ai_search.ymlconfigs/config_frontier_models.ymlconfigs/config_cli_default.ymlconfigs/config_mcp.ymlconfigs/config_web_frag_mcp_auth.ymlfrontends/benchmarks/freshqa/configs/config_full_workflow.ymlconfigs/config_web_default_llamaindex.ymlfrontends/benchmarks/freshqa/configs/config_shallow_research_only.ymlconfigs/config_web_frag.ymlconfigs/config_web_opensearch.yml
configs/**/*.{yml,yaml}
📄 CodeRabbit inference engine (AGENTS.md)
Use
_typenames that come from the registered configuration class.
Files:
configs/config_web_default_guardrails.ymlconfigs/config_openshell.ymlconfigs/config_web_azure_ai_search.ymlconfigs/config_frontier_models.ymlconfigs/config_cli_default.ymlconfigs/config_mcp.ymlconfigs/config_web_frag_mcp_auth.ymlconfigs/config_web_default_llamaindex.ymlconfigs/config_web_frag.ymlconfigs/config_web_opensearch.yml
{deploy/**,configs/**}
⚙️ CodeRabbit configuration file
{deploy/**,configs/**}: Review deployment and config changes for secret separation, safe defaults, local-vs-production behavior, Helm and
Docker portability, and documentation parity. Flag committed credentials, environment-specific NVIDIA internals in
public defaults, and changes that make examples diverge from CI-tested paths.
Files:
configs/config_web_default_guardrails.ymlconfigs/config_openshell.ymlconfigs/config_web_azure_ai_search.ymlconfigs/config_frontier_models.ymlconfigs/config_cli_default.ymlconfigs/config_mcp.ymlconfigs/config_web_frag_mcp_auth.ymlconfigs/config_web_default_llamaindex.ymlconfigs/config_web_frag.ymlconfigs/config_web_opensearch.yml
mcp/**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
For MCP changes, run the MCP development dependency setup and
uv run --project mcp --extra dev pytest mcp/tests.
Files:
mcp/tests/test_preclassification_integration.py
mcp/**/*
📄 CodeRabbit inference engine (AGENTS.md)
Treat
mcp/as an independent uv project with its own lockfile, tests, and release assets; do not include it in the root workspace.
Files:
mcp/tests/test_preclassification_integration.py
🧠 Learnings (2)
📚 Learning: 2026-08-11T06:34:44.687Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 429
File: src/aiq_agent/agents/deep_researcher/register.py:302-307
Timestamp: 2026-08-11T06:34:44.687Z
Learning: In Python logging code that handles potentially sensitive exceptions, do not add `exc_info=True` solely to restore stack traces, because standard traceback formatting includes `str(exception)` and may expose provider, customer, or credential-bearing content. When sensitive-content redaction is required, log the exception type together with `log_content_metadata(exception)` instead.
Applied to files:
src/aiq_agent/agents/shallow_researcher/agent.pymcp/tests/test_preclassification_integration.pytests/aiq_agent/agents/shallow_researcher/test_agent.pytests/aiq_agent/test_default_model_profiles.py
📚 Learning: 2026-07-06T23:55:42.908Z
Learnt from: cdgamarose-nv
Repo: NVIDIA-AI-Blueprints/aiq PR: 311
File: src/aiq_agent/agents/deep_researcher/prompts/orchestrator.j2:74-81
Timestamp: 2026-07-06T23:55:42.908Z
Learning: In agent test files (e.g., tests/aiq_agent/agents/*/test_agent.py), avoid brittle assertions that match exact substrings from prompt template files (such as *.j2 prompt wording). Prompt wording can change frequently, so instead assert structural/behavioral properties (e.g., that the prompt builder is called, that required sections/fields are present via stable markers, that the model output/agent behavior conforms to an expected schema, or that key actions are taken) rather than matching literal prompt text.
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
🪛 LanguageTool
README.md
[style] ~295-~295: Consider a different adjective to strengthen your wording.
Context: ...amaIndex profile using GPT Sol/Luna for deep research and Nemotron 3.5 Lightning for...
(DEEP_PROFOUND)
🔇 Additional comments (32)
mcp/tests/test_preclassification_integration.py (1)
30-30: 📐 Maintainability & Code QualityRun the required MCP validation.
The renamed target matches
nemotron_lightning_intent_llminconfigs/config_mcp.ymland the role-profile regression test. Run the MCP development dependency setup, thenuv run --project mcp --extra dev pytest mcp/tests.Source: Coding guidelines
.agents/skills/aiq-configure-workflow/assets/config-scaffold.yml (1)
20-42: LGTM!Also applies to: 75-79
configs/config_cli_default.yml (1)
19-41: LGTM!Also applies to: 122-122, 138-138
configs/config_frontier_models.yml (1)
53-75: LGTM!Also applies to: 143-143, 162-162
configs/config_mcp.yml (1)
29-51: LGTM!Also applies to: 102-102, 114-114
configs/config_openshell.yml (1)
31-53: LGTM!Also applies to: 127-127, 139-139
configs/config_web_azure_ai_search.yml (1)
50-72: LGTM!Also applies to: 194-194, 212-212
configs/config_web_default_guardrails.yml (1)
31-53: LGTM!Also applies to: 231-231, 243-243
docs/source/customization/mcp-tools.md (1)
117-117: LGTM!docs/source/customization/swapping-models.md (1)
13-24: LGTM!Also applies to: 43-43, 79-79, 88-96
docs/source/customization/tools-and-sources.md (1)
140-140: LGTM!docs/source/examples/full-pipeline-llamaindex.md (1)
46-68: LGTM!Also applies to: 144-144, 163-163
configs/config_web_default_llamaindex.yml (1)
195-195: LGTM!Also applies to: 211-211
configs/config_web_frag.yml (1)
165-165: LGTM!Also applies to: 180-180
configs/config_web_frag_mcp_auth.yml (1)
214-214: LGTM!Also applies to: 229-229
configs/config_web_opensearch.yml (1)
172-172: LGTM!Also applies to: 187-187
.agents/skills/aiq-configure-workflow/references/config-profiles.md (1)
20-20: LGTM!.secrets.baseline (1)
358-358: LGTM!tests/aiq_agent/test_default_model_profiles.py (1)
12-79: LGTM!Also applies to: 128-153
README.md (1)
111-112: LGTM!Also applies to: 153-153, 290-300
docs/notebooks/0_Getting_Started_with_AIQ.ipynb (1)
74-74: LGTM!Also applies to: 97-97, 369-375, 408-408
docs/source/customization/configuration-reference.md (1)
131-131: LGTM!Also applies to: 382-382, 660-660, 675-675
docs/source/examples/full-pipeline-web.md (1)
64-92: LGTM!Also applies to: 160-160, 187-187
docs/source/examples/hybrid-frontier-model.md (1)
22-41: LGTM!Also applies to: 80-80, 95-95
docs/source/examples/minimal-shallow-only.md (1)
30-40: LGTM!docs/source/extending/adding-a-tool.md (1)
282-282: LGTM!docs/source/get-started/installation.md (1)
28-28: LGTM!docs/source/resources/troubleshooting.md (1)
47-53: LGTM!frontends/benchmarks/freshqa/configs/config_full_workflow.yml (1)
12-34: LGTM!Also applies to: 70-89
frontends/benchmarks/freshqa/configs/config_shallow_research_only.yml (1)
2-2: LGTM!Also applies to: 16-25, 36-51
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2 (1)
3-10: LGTM!tests/aiq_agent/agents/shallow_researcher/test_agent.py (1)
203-203: LGTM!Also applies to: 222-222, 242-242, 263-263, 319-329, 408-408, 824-824, 851-851, 878-878, 959-959
c3c0545 to
93f9e99
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
configs/config_web_default_guardrails.yml (1)
182-182: 🔒 Security & Privacy | 🟡 Minor | ⚡ Quick winAdd regression cases for the missing guardrail forms.
Extend
test_default_deep_guardrails_block_document_exfiltration_requestswithdump documents,dump all documents, andexfiltrate the knowledge base. The chained form is already covered.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@configs/config_web_default_guardrails.yml` at line 182, Add regression inputs to test_default_deep_guardrails_block_document_exfiltration_requests for “dump documents,” “dump all documents,” and “exfiltrate the knowledge base,” while retaining the existing chained-form coverage.Source: Coding guidelines
README.md (1)
502-502: 🔒 Security & Privacy | 🟡 Minor | ⚡ Quick winNarrow the logging-redaction claim or sanitize all built-in exception logs.
callbacks.py:942logsedirectly, and workflow paths use unredactedexc_info=Truetracebacks. Replace raw exception logging with the exception type andlog_content_metadata(e), then add tests that sensitive exception text is absent from emitted logs.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@README.md` at line 502, Update built-in exception logging, including callbacks.py around the callback error handler and workflow paths using exc_info=True, to avoid emitting raw exception text or traceback contents. Log only the exception type together with log_content_metadata(e), and add tests asserting sensitive exception text is absent from emitted logs.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/source/customization/configuration-reference.md`:
- Around line 591-613: Update the configuration-reference aliases
nano_intent_llm and nano_agent_llm to the Lightning model-family names, and
change the corresponding model descriptions in the table, including the entry
around line 713 and the referenced entries around lines 660 and 675, from
Nemotron 3 Super to Lightning. Keep the configured model IDs unchanged.
---
Outside diff comments:
In `@configs/config_web_default_guardrails.yml`:
- Line 182: Add regression inputs to
test_default_deep_guardrails_block_document_exfiltration_requests for “dump
documents,” “dump all documents,” and “exfiltrate the knowledge base,” while
retaining the existing chained-form coverage.
In `@README.md`:
- Line 502: Update built-in exception logging, including callbacks.py around the
callback error handler and workflow paths using exc_info=True, to avoid emitting
raw exception text or traceback contents. Log only the exception type together
with log_content_metadata(e), and add tests asserting sensitive exception text
is absent from emitted logs.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 4243ae75-3673-49bc-8dcc-a00a4c043397
📒 Files selected for processing (14)
README.mdconfigs/config_frontier_models.ymlconfigs/config_web_default_guardrails.ymldocs/notebooks/0_Getting_Started_with_AIQ.ipynbdocs/source/customization/configuration-reference.mddocs/source/customization/swapping-models.mddocs/source/examples/full-pipeline-llamaindex.mddocs/source/examples/hybrid-frontier-model.mddocs/source/examples/minimal-shallow-only.mdfrontends/aiq_api/src/aiq_api/jobs/callbacks.pysrc/aiq_agent/agents/shallow_researcher/agent.pysrc/aiq_agent/common/callbacks.pytests/aiq_agent/agents/shallow_researcher/test_agent.pytests/aiq_agent/jobs/test_runner.py
📜 Review details
⏰ Context from checks skipped due to timeout. (3)
- GitHub Check: Script Validation
- GitHub Check: Lint and Hooks
- GitHub Check: Pytest and Coverage
⚠️ CI failures not shown inline (2)
GitHub Actions: AIQ UI CI / 4_Install Dependencies.txt: feat: enable Nemotron 3.5 Lightning defaults
Conclusion: failure
##[group]Run npm audit --omit=dev --audit-level=high
�[36;1mnpm audit --omit=dev --audit-level=high�[0m
shell: /usr/bin/bash -e {0}
##[endgroup]
# npm audit report
dompurify <=3.4.12
Severity: moderate
DOMPurify: IN_PLACE hook removal leaves a detached subtree executable, causing XSS - https://github.com/advisories/GHSA-55q2-fjhq-7xh7
fix available via `npm audit fix`
node_modules/dompurify
nanoid <3.3.17
Severity: high
nanoid: custom generators can loop indefinitely when size is zero - https://github.com/advisories/GHSA-2v37-7h3g-55p8
fix available via `npm audit fix`
node_modules/nanoid
2 vulnerabilities (1 moderate, 1 high)
To address all issues, run:
npm audit fix
##[error]Process completed with exit code 1.
GitHub Actions: AIQ UI CI / Install Dependencies: feat: enable Nemotron 3.5 Lightning defaults
Conclusion: failure
##[group]Run npm audit --omit=dev --audit-level=high
�[36;1mnpm audit --omit=dev --audit-level=high�[0m
shell: /usr/bin/bash -e {0}
##[endgroup]
# npm audit report
dompurify <=3.4.12
Severity: moderate
DOMPurify: IN_PLACE hook removal leaves a detached subtree executable, causing XSS - https://github.com/advisories/GHSA-55q2-fjhq-7xh7
fix available via `npm audit fix`
node_modules/dompurify
nanoid <3.3.17
Severity: high
nanoid: custom generators can loop indefinitely when size is zero - https://github.com/advisories/GHSA-2v37-7h3g-55p8
fix available via `npm audit fix`
node_modules/nanoid
2 vulnerabilities (1 moderate, 1 high)
To address all issues, run:
npm audit fix
##[error]Process completed with exit code 1.
🧰 Additional context used
📓 Path-based instructions (13)
**/*
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*: Do not include secrets, credentials, private hostnames, internal-only logs, customer data, or generated local artifacts.
Add or update tests for behavior changes.
**/*: For substantial behavior, authentication, UI, or architecture changes, open a design discussion before coding rather than landing a large unreviewed change.
Keep changes scoped to this repository and avoid editing adjacent repositories; treat eachsources/*package independently and prefer the smallest package-scoped change.
Keep pull requests scoped, avoid unrelated files and generated artifacts, provide validation evidence, and ensure every commit has DCO sign-off.
Files:
frontends/aiq_api/src/aiq_api/jobs/callbacks.pydocs/source/examples/hybrid-frontier-model.mddocs/source/examples/full-pipeline-llamaindex.mdtests/aiq_agent/jobs/test_runner.pyREADME.mdsrc/aiq_agent/common/callbacks.pyconfigs/config_web_default_guardrails.ymldocs/notebooks/0_Getting_Started_with_AIQ.ipynbdocs/source/customization/swapping-models.mdsrc/aiq_agent/agents/shallow_researcher/agent.pydocs/source/customization/configuration-reference.mdconfigs/config_frontier_models.ymltests/aiq_agent/agents/shallow_researcher/test_agent.pydocs/source/examples/minimal-shallow-only.md
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.py: Runuv run ruff check .anduv run ruff format --check .for root Python changes.
Runuv run pytestfor root project Python changes.
Files:
frontends/aiq_api/src/aiq_api/jobs/callbacks.pytests/aiq_agent/jobs/test_runner.pysrc/aiq_agent/common/callbacks.pysrc/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
**/*.{py,pyi}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{py,pyi}: Format and lint Python code with Ruff using line length 120, Python 3.11 targeting, rules E, F, W, I, PL, and UP, with single-line imports; do not reformat unrelated code.
Missing-secret paths must degrade gracefully by stubbing or skipping rather than crashing or leaking secrets.
Files:
frontends/aiq_api/src/aiq_api/jobs/callbacks.pytests/aiq_agent/jobs/test_runner.pysrc/aiq_agent/common/callbacks.pysrc/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
**/*.{py,pyi,js,jsx,ts,tsx,yml,yaml,json,env,md}
📄 CodeRabbit inference engine (AGENTS.md)
Never commit secrets, tokens, or environment-specific hostnames; use environment variables and
SecretStr, and resolve API keys at runtime.
Files:
frontends/aiq_api/src/aiq_api/jobs/callbacks.pydocs/source/examples/hybrid-frontier-model.mddocs/source/examples/full-pipeline-llamaindex.mdtests/aiq_agent/jobs/test_runner.pyREADME.mdsrc/aiq_agent/common/callbacks.pyconfigs/config_web_default_guardrails.ymldocs/source/customization/swapping-models.mdsrc/aiq_agent/agents/shallow_researcher/agent.pydocs/source/customization/configuration-reference.mdconfigs/config_frontier_models.ymltests/aiq_agent/agents/shallow_researcher/test_agent.pydocs/source/examples/minimal-shallow-only.md
**/*.{py,pyi,js,jsx,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Never print or log secret values, including in tool output or error messages.
Files:
frontends/aiq_api/src/aiq_api/jobs/callbacks.pytests/aiq_agent/jobs/test_runner.pysrc/aiq_agent/common/callbacks.pysrc/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
{src/aiq_agent/fastapi_extensions/**,frontends/aiq_api/src/aiq_api/**}
⚙️ CodeRabbit configuration file
{src/aiq_agent/fastapi_extensions/**,frontends/aiq_api/src/aiq_api/**}: Treat API, auth, and job-runner changes as externally visible contracts. Check authorization boundaries,
request tracing, async job lifecycle, websocket reconnect behavior, error responses, and cross-user data isolation.
Require tests for route behavior, access decisions, and job state transitions when those surfaces change.
Files:
frontends/aiq_api/src/aiq_api/jobs/callbacks.py
docs/source/**/*
📄 CodeRabbit inference engine (AGENTS.md)
Update canonical documentation under
docs/source/when behavior, configuration, or workflows change; do not duplicate full documentation pages in skills.
Files:
docs/source/examples/hybrid-frontier-model.mddocs/source/examples/full-pipeline-llamaindex.mddocs/source/customization/swapping-models.mddocs/source/customization/configuration-reference.mddocs/source/examples/minimal-shallow-only.md
{docs/**,README.md,CONTRIBUTING.md,SECURITY.md,CODE-OF-CONDUCT.md}
⚙️ CodeRabbit configuration file
{docs/**,README.md,CONTRIBUTING.md,SECURITY.md,CODE-OF-CONDUCT.md}: Review documentation for command accuracy, branch-name consistency, current CI and copy-pr-bot behavior, public
vs internal boundary clarity, stale examples, and links that no longer match the repository layout.
Files:
docs/source/examples/hybrid-frontier-model.mddocs/source/examples/full-pipeline-llamaindex.mdREADME.mddocs/notebooks/0_Getting_Started_with_AIQ.ipynbdocs/source/customization/swapping-models.mddocs/source/customization/configuration-reference.mddocs/source/examples/minimal-shallow-only.md
src/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
src/**/*.py: Respect authenticated data sources by honoringrequires_auth, passing through per-user tokens, and using backend token validators; apply owner guardrails before loading protected report or artifact context into an agent.
Do not weaken or bypassAuthMiddleware, authentication validators, or authentication gating without prior design discussion.
Files:
src/aiq_agent/common/callbacks.pysrc/aiq_agent/agents/shallow_researcher/agent.py
**/*.{yaml,yml}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
For deployment changes, run the relevant Helm or Compose validation and describe the environment used.
Files:
configs/config_web_default_guardrails.ymlconfigs/config_frontier_models.yml
configs/**/*.{yml,yaml}
📄 CodeRabbit inference engine (AGENTS.md)
Use
_typenames that come from the registered configuration class.
Files:
configs/config_web_default_guardrails.ymlconfigs/config_frontier_models.yml
{deploy/**,configs/**}
⚙️ CodeRabbit configuration file
{deploy/**,configs/**}: Review deployment and config changes for secret separation, safe defaults, local-vs-production behavior, Helm and
Docker portability, and documentation parity. Flag committed credentials, environment-specific NVIDIA internals in
public defaults, and changes that make examples diverge from CI-tested paths.
Files:
configs/config_web_default_guardrails.ymlconfigs/config_frontier_models.yml
src/aiq_agent/agents/**/*
⚙️ CodeRabbit configuration file
src/aiq_agent/agents/**/*: Review agent changes for research workflow correctness, graph state transitions, prompt/tool contracts,
HITL behavior, and failure handling. Flag changes that weaken source attribution, report generation,
async cancellation, checkpointing, or data-source selection without focused tests and docs.
Files:
src/aiq_agent/agents/shallow_researcher/agent.py
🧠 Learnings (2)
📚 Learning: 2026-08-11T06:34:44.687Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 429
File: src/aiq_agent/agents/deep_researcher/register.py:302-307
Timestamp: 2026-08-11T06:34:44.687Z
Learning: In Python logging code that handles potentially sensitive exceptions, do not add `exc_info=True` solely to restore stack traces, because standard traceback formatting includes `str(exception)` and may expose provider, customer, or credential-bearing content. When sensitive-content redaction is required, log the exception type together with `log_content_metadata(exception)` instead.
Applied to files:
frontends/aiq_api/src/aiq_api/jobs/callbacks.pytests/aiq_agent/jobs/test_runner.pysrc/aiq_agent/common/callbacks.pysrc/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-07-06T23:55:42.908Z
Learnt from: cdgamarose-nv
Repo: NVIDIA-AI-Blueprints/aiq PR: 311
File: src/aiq_agent/agents/deep_researcher/prompts/orchestrator.j2:74-81
Timestamp: 2026-07-06T23:55:42.908Z
Learning: In agent test files (e.g., tests/aiq_agent/agents/*/test_agent.py), avoid brittle assertions that match exact substrings from prompt template files (such as *.j2 prompt wording). Prompt wording can change frequently, so instead assert structural/behavioral properties (e.g., that the prompt builder is called, that required sections/fields are present via stable markers, that the model output/agent behavior conforms to an expected schema, or that key actions are taken) rather than matching literal prompt text.
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
🪛 LanguageTool
docs/source/examples/hybrid-frontier-model.md
[style] ~9-~9: Consider a different adjective to strengthen your wording.
Context: ...with GPT Sol/Luna for clarification and deep research. ## Shipped Configuration | ...
(DEEP_PROFOUND)
README.md
[style] ~297-~297: Consider a different adjective to strengthen your wording.
Context: ...amaIndex profile using GPT Sol/Luna for deep research and Nemotron 3.5 Lightning for...
(DEEP_PROFOUND)
🔇 Additional comments (16)
src/aiq_agent/common/callbacks.py (1)
23-29: LGTM!Also applies to: 83-114, 148-148, 173-173, 186-186, 197-210, 229-254
src/aiq_agent/agents/shallow_researcher/agent.py (2)
250-273: 📐 Maintainability & Code QualityConfirm required root Python validation.
Run the required Ruff and pytest commands for this behavior change. Include the results in the PR validation evidence.
As per coding guidelines, “Run
uv run ruff check .anduv run ruff format --check .” and “Runuv run pytestfor root project Python changes.”Source: Coding guidelines
41-49: LGTM!Also applies to: 194-195, 223-226, 284-288, 344-347, 430-443
frontends/aiq_api/src/aiq_api/jobs/callbacks.py (1)
33-33: LGTM!Also applies to: 45-46, 132-144, 395-415, 466-478, 744-744, 788-788
tests/aiq_agent/agents/shallow_researcher/test_agent.py (1)
20-20: LGTM!Also applies to: 33-33, 205-205, 224-224, 244-244, 265-265, 304-374, 453-453, 642-675, 707-722, 828-863, 944-944, 971-971, 998-998, 1079-1079
tests/aiq_agent/jobs/test_runner.py (1)
66-66: LGTM!Also applies to: 76-76, 377-389, 481-513, 671-803, 923-1036, 1614-1776, 1944-1983, 2096-2123, 2126-2177, 2198-2280
configs/config_frontier_models.yml (1)
54-76: LGTM!Also applies to: 144-144, 163-163
configs/config_web_default_guardrails.yml (1)
31-53: LGTM!Also applies to: 232-232, 244-244
README.md (1)
55-55: LGTM!Also applies to: 111-123, 155-155, 292-302, 450-450
docs/notebooks/0_Getting_Started_with_AIQ.ipynb (1)
74-74: LGTM!Also applies to: 97-97, 306-328, 369-375, 408-408, 807-813
docs/source/customization/configuration-reference.md (1)
122-122: LGTM!Also applies to: 131-132, 382-382
docs/source/customization/swapping-models.md (1)
9-23: LGTM!Also applies to: 29-40, 59-59, 95-95, 104-112
docs/source/examples/full-pipeline-llamaindex.md (2)
46-68: LGTM!Also applies to: 144-144, 163-163
243-251: 🗄️ Data Integrity & IntegrationThe
conversation-idcontract is correctly implemented. The async route passes the header to the worker, which sets it before tool creation.knowledge_retrievaluses it as the collection and falls back tocollection_namewhen absent.docs/source/examples/hybrid-frontier-model.md (1)
8-29: LGTM!Also applies to: 32-59
docs/source/examples/minimal-shallow-only.md (1)
28-40: LGTM!
Signed-off-by: Ajay Thorve <athorve@nvidia.com>
93f9e99 to
9243641
Compare
|
/merge |
#### Overview - Keep the optional frontier profile role-specific instead of inheriting the repository Lightning defaults. - Use GPT-5.6 Luna for intent classification, shallow research, source routing, and research. - Keep GPT-5.6 Sol for clarification, orchestration, planning, and writing. - Update profile documentation and add regression coverage for the exact frontier assignments. PR #430 correctly moved the ordinary default profiles to Nemotron 3.5 Lightning, but it also applied that default to `config_frontier_models.yml`. The frontier profile is an explicit GPT Sol/Luna topology and should remain independently validated. Restoring its historical Super shallow assignment would reintroduce a deprecated model, so Luna is used for the bounded shallow role. #### DCO sign-off for the squash commit Signed-off-by: Ajay Thorve <athorve@nvidia.com> #### Validation - [x] `uv run pytest -q` — 2142 passed, 13 skipped. - [x] Focused frontier/default model tests — 60 passed. - [x] All changed-file pre-commit hooks, including Markdown link check and detect-secrets. - [x] `nat validate --config_file configs/config_frontier_models.yml`. - [x] Added regression coverage for the exact frontier model assignments. - [x] Updated user-facing configuration documentation. - [x] Confirmed the PR contains no secrets, credentials, or internal-only data. - [x] Commits are GPG signed and DCO signed off. Historical held-out FreshQA TEST validation for GPT-5.6 Luna used the November 24, 2025 dataset revision: 500/500 generation and judge coverage, zero judge errors, and 70.8% accuracy. This validates the profile wiring; it is not a claim that Luna is the strongest shallow-research model. The runtime config and source are copied into the release image. A Git tag alone does not update an already-built RC container; a container intended to carry this correction must be rebuilt from the final corrected tag. #### Where should reviewers start? Start with `configs/config_frontier_models.yml`, then review the exact role assertions in `tests/aiq_agent/test_default_model_profiles.py` and the matching documentation. #### Related Issues - Relates to the August 2026 model-endpoint deprecation migration tracked by PR #430. ## Summary by CodeRabbit * **New Features** * Updated the frontier model profile to use GPT-5.6 Luna for intent classification, shallow research, source routing, and deep-research execution. * GPT-5.6 Sol now supports clarification, orchestration, planning, and writing workflows. * Added clearer API-key requirements, including Tavily access. * **Documentation** * Updated configuration guides, onboarding materials, and hybrid-model examples to reflect the revised model assignments and token settings. * **Tests** * Expanded validation for frontier model selection, credentials, limits, retries, and tool-call settings. Authors: - Ajay Thorve (https://github.com/AjayThorve) Approvers: - Kyle Zheng (https://github.com/KyleZheng1284) URL: #433
2 participants
Summary
nvidia/nemotron-3.5-lightning-30b-a3bthe default model for intent classification and shallow researchhttps://integrate.api.nvidia.com/v1endpointtavily_web_searchand registering it as a citable data sourceThis replaces the closed placeholder PR #419. The branch contains no preview model name or Inference Hub base URL.
Live model validation
NVIDIA Build
nvidia/nemotron-3.5-lightning-30b-a3bconfig_cli_default.ymlcompleted two AIQ end-to-end prompts with live Tavily calls and source captureInference Hub
The provider-specific evaluation overlay used
nvidia/nvidia/nemotron-3.5-lightning; that alias and endpoint are not checked into the repository.The TEST comparison is useful launch evidence, but the Lightning run used the current release's Tavily adapter and source-registry contract, so historical deltas should not be interpreted as a model-only causal estimate.
Validation
uv run pytest -q: 2,142 passed, 13 skippedScope
This changes only intent/shallow defaults. Nemotron 3 Ultra remains unchanged for clarification and all deep-research roles.
Summary by CodeRabbit
New Features
Bug Fixes
Documentation