fix: enforce shallow citation integrity - #439
Conversation
Signed-off-by: Ajay Thorve <athorve@nvidia.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThe shallow researcher now enforces citation integrity. It detects citation formats, repairs incomplete multi-source reports within a timeout, re-verifies output, and suppresses publication when integrity remains incomplete. ChangesCitation integrity enforcement
Estimated code review effort: 4 (Complex) | ~45 minutes Mergeability Score: 🟡 Moderate · up to The citation gate can currently remove valid marker-first answer text after a source block, causing researched answers to be rejected or truncated; the timeout test also does not prove repair is bounded. The PR is not merge-ready until the parsing boundary is fixed and timeout behavior is explicitly verified. Sequence Diagram(s)sequenceDiagram
participant ShallowResearcher
participant CitationVerifier
participant RepairLLM
participant FinalReportCallback
ShallowResearcher->>CitationVerifier: Verify report citations
CitationVerifier-->>ShallowResearcher: Return citation validity and source metadata
ShallowResearcher->>RepairLLM: Repair uncited multi-source report without tools
RepairLLM-->>ShallowResearcher: Return inline citations and References section
ShallowResearcher->>CitationVerifier: Re-verify repaired report
ShallowResearcher->>FinalReportCallback: Publish integrity-checked report
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/aiq_agent/agents/shallow_researcher/agent.py`:
- Around line 247-251: Bound the provider call in _repair_missing_citations by
adding the required asyncio import and a constructor-configurable
citation_repair_timeout defaulting to 60.0, then wrap _get_llm().ainvoke with
asyncio.wait_for using that timeout. Preserve the existing exception handling so
TimeoutError follows the current CitationIntegrityError fail-closed path.
- Around line 62-66: The citation heading matcher is too narrow and integrity
validation incorrectly requires citations before the heading. Update
_SOURCE_SECTION_HEADING_RE to support Markdown headings from # through ######,
optional trailing heading text, plain Sources:/References: lines, and bold
headings with trailing text; update _has_citation_integrity to accept an inline
marker anywhere in the report while remaining fail-closed when none exists. Add
focused cases for these variants in TestAppendMinimalCitation and the integrity
tests.
In `@src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2`:
- Around line 13-14: Update Rule 7 in the researcher prompt to define the
required citation format for tool sources without URLs, using the source’s
citation_key and a clear non-URL fallback line. Preserve the existing exact-URL
format for sources that provide URLs and ensure the rule still requires the
final References section.
In `@tests/aiq_agent/agents/shallow_researcher/test_agent.py`:
- Around line 296-305: Replace the brittle literal prompt assertions in
tests/aiq_agent/agents/shallow_researcher/test_agent.py lines 296-305 with
structural checks for an inline citation marker example and a references
heading; at lines 733-737, replace the “do not call tools” text match with a
behavioral assertion that the citation-repair invocation uses an unbound model,
verifying bind_tools was not called for that invocation.
In `@tests/aiq_agent/test_default_model_profiles.py`:
- Around line 150-157: Replace the non-discriminating system_prompt absence
assertion in test_shallow_profiles_use_the_shared_citation_prompt with checks
that each loaded profile’s shallow_research_agent resolves to the same expected
_type and that the loaded prompt content matches prompts/researcher.j2. Preserve
the parameterized coverage for every path in SHALLOW_PROFILE_PATHS.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: ddb16b3f-5439-4364-a6e8-66ca97d42941
📒 Files selected for processing (4)
src/aiq_agent/agents/shallow_researcher/agent.pysrc/aiq_agent/agents/shallow_researcher/prompts/researcher.j2tests/aiq_agent/agents/shallow_researcher/test_agent.pytests/aiq_agent/test_default_model_profiles.py
📜 Review details
⏰ Context from checks skipped due to timeout. (3)
- GitHub Check: Script Validation
- GitHub Check: Pytest and Coverage
- GitHub Check: Lint and Hooks
🧰 Additional context used
📓 Path-based instructions (7)
**/*
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*: Do not include secrets, credentials, private hostnames, internal-only logs, customer data, or generated local artifacts.
Add or update tests for behavior changes.
**/*: For substantial behavior, authentication, UI, or architecture changes, open a design discussion before coding rather than landing a large unreviewed change.
Keep changes scoped to this repository and avoid editing adjacent repositories; treat eachsources/*package independently and prefer the smallest package-scoped change.
Keep pull requests scoped, avoid unrelated files and generated artifacts, provide validation evidence, and ensure every commit has DCO sign-off.
Files:
tests/aiq_agent/test_default_model_profiles.pysrc/aiq_agent/agents/shallow_researcher/prompts/researcher.j2src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.py: Runuv run ruff check .anduv run ruff format --check .for root Python changes.
Runuv run pytestfor root project Python changes.
Files:
tests/aiq_agent/test_default_model_profiles.pysrc/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
**/*.{py,pyi}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{py,pyi}: Format and lint Python code with Ruff using line length 120, Python 3.11 targeting, rules E, F, W, I, PL, and UP, with single-line imports; do not reformat unrelated code.
Missing-secret paths must degrade gracefully by stubbing or skipping rather than crashing or leaking secrets.
Files:
tests/aiq_agent/test_default_model_profiles.pysrc/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
**/*.{py,pyi,js,jsx,ts,tsx,yml,yaml,json,env,md}
📄 CodeRabbit inference engine (AGENTS.md)
Never commit secrets, tokens, or environment-specific hostnames; use environment variables and
SecretStr, and resolve API keys at runtime.
Files:
tests/aiq_agent/test_default_model_profiles.pysrc/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
**/*.{py,pyi,js,jsx,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Never print or log secret values, including in tool output or error messages.
Files:
tests/aiq_agent/test_default_model_profiles.pysrc/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
src/aiq_agent/agents/**/*
⚙️ CodeRabbit configuration file
src/aiq_agent/agents/**/*: Review agent changes for research workflow correctness, graph state transitions, prompt/tool contracts,
HITL behavior, and failure handling. Flag changes that weaken source attribution, report generation,
async cancellation, checkpointing, or data-source selection without focused tests and docs.
Files:
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2src/aiq_agent/agents/shallow_researcher/agent.py
src/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
src/**/*.py: Respect authenticated data sources by honoringrequires_auth, passing through per-user tokens, and using backend token validators; apply owner guardrails before loading protected report or artifact context into an agent.
Do not weaken or bypassAuthMiddleware, authentication validators, or authentication gating without prior design discussion.
Files:
src/aiq_agent/agents/shallow_researcher/agent.py
🧠 Learnings (20)
📓 Common learnings
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: skills/aiq-research/SKILL.md:0-0
Timestamp: 2026-07-23T22:20:18.400Z
Learning: Applies to skills/aiq-research/**/SKILL.md : Present returned reports with citations and source URLs intact; do not truncate citations or source URLs.
📚 Learning: 2026-07-06T23:55:46.952Z
Learnt from: cdgamarose-nv
Repo: NVIDIA-AI-Blueprints/aiq PR: 311
File: src/aiq_agent/agents/deep_researcher/prompts/orchestrator.j2:74-81
Timestamp: 2026-07-06T23:55:46.952Z
Learning: In `tests/aiq_agent/agents/deep_researcher/test_agent.py`, avoid asserting exact substrings from the orchestrator/writer prompt templates (e.g., `src/aiq_agent/agents/deep_researcher/prompts/orchestrator.j2`) since prompt wording changes frequently. Prefer testing structural/behavioral properties instead.
Applied to files:
tests/aiq_agent/test_default_model_profiles.pysrc/aiq_agent/agents/shallow_researcher/prompts/researcher.j2src/aiq_agent/agents/shallow_researcher/agent.py
📚 Learning: 2026-08-04T22:37:11.331Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 414
File: tests/aiq_agent/test_default_model_profiles.py:72-75
Timestamp: 2026-08-04T22:37:11.331Z
Learning: For the AIQ default-model migration, PR `#414` intentionally retains Nemotron 3 Ultra expectations for intent classification and shallow research because the Nemotron Nano 3.5 Preview public endpoint and GA slug are unavailable. The Nano intent alias and corresponding profile tests are deferred to stacked draft PR `#419` after the availability gate.
Applied to files:
tests/aiq_agent/test_default_model_profiles.pysrc/aiq_agent/agents/shallow_researcher/prompts/researcher.j2tests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-06-11T21:21:11.314Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: docs/source/contributing/code-organization.md:0-0
Timestamp: 2026-06-11T21:21:11.314Z
Learning: Applies to docs/source/contributing/src/aiq_agent/agents/shallow_researcher/** : Shallow researcher agent should be organized in `src/aiq_agent/agents/shallow_researcher/`
Applied to files:
tests/aiq_agent/test_default_model_profiles.pysrc/aiq_agent/agents/shallow_researcher/prompts/researcher.j2src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-08-11T06:34:44.687Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 429
File: src/aiq_agent/agents/deep_researcher/register.py:302-307
Timestamp: 2026-08-11T06:34:44.687Z
Learning: In Python logging code that handles potentially sensitive exceptions, do not add `exc_info=True` solely to restore stack traces, because standard traceback formatting includes `str(exception)` and may expose provider, customer, or credential-bearing content. When sensitive-content redaction is required, log the exception type together with `log_content_metadata(exception)` instead.
Applied to files:
tests/aiq_agent/test_default_model_profiles.pysrc/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-06-11T21:21:11.314Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: docs/source/contributing/code-organization.md:0-0
Timestamp: 2026-06-11T21:21:11.314Z
Learning: Applies to docs/source/contributing/src/aiq_agent/agents/deep_researcher/** : Deep researcher agent should be organized in `src/aiq_agent/agents/deep_researcher/` with implementation details documented in README.md
Applied to files:
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2
📚 Learning: 2026-07-23T22:20:18.400Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: skills/aiq-research/SKILL.md:0-0
Timestamp: 2026-07-23T22:20:18.400Z
Learning: Applies to skills/aiq-research/**/SKILL.md : Treat report editing as cosmetic only and use `report_edit <JOB_ID> <EDIT_INSTRUCTIONS>`; use `research` for a new or refined investigation.
Applied to files:
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2
📚 Learning: 2026-06-11T21:21:11.314Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: docs/source/contributing/code-organization.md:0-0
Timestamp: 2026-06-11T21:21:11.314Z
Learning: Applies to docs/source/contributing/src/aiq_agent/agents/chat_researcher/** : Chat researcher agent should be organized in `src/aiq_agent/agents/chat_researcher/` and serve as orchestrator with orchestration node containing intent, meta, and depth information
Applied to files:
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2
📚 Learning: 2026-07-23T22:20:18.400Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: skills/aiq-research/SKILL.md:0-0
Timestamp: 2026-07-23T22:20:18.400Z
Learning: Applies to skills/aiq-research/**/SKILL.md : Present returned reports with citations and source URLs intact; do not truncate citations or source URLs.
Applied to files:
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-07-23T22:20:09.213Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: skills/aiq-deploy/SKILL.md:0-0
Timestamp: 2026-07-23T22:20:09.213Z
Learning: Applies to skills/aiq-deploy/**/* : Do not continue into deep research or deep-research completion validation unless the user asks for it or confirms the post-deployment validation prompt.
Applied to files:
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2
📚 Learning: 2026-08-11T06:35:46.104Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 429
File: src/aiq_agent/agents/report_rewriter/agent.py:197-200
Timestamp: 2026-08-11T06:35:46.104Z
Learning: In `src/aiq_agent/agents/report_rewriter/agent.py`, `rewrite_report` is shared by async and inline rewrite paths. It must derive its citation source allowlist only from the canonical `original_report` and durable `parent_context`. Do not widen its API to accept caller-supplied sources or change its string-only return contract solely to remove bounded duplicate parsing.
Applied to files:
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-07-23T22:20:18.400Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: skills/aiq-research/SKILL.md:0-0
Timestamp: 2026-07-23T22:20:18.400Z
Learning: Applies to skills/aiq-research/**/SKILL.md : Use this skill for research-shaped requests, including deep research, AI-Q research, or requests to ask AI-Q; do not use it for installation, deployment, lifecycle, UI, CLI, Docker, Helm, or troubleshooting requests.
Applied to files:
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2
📚 Learning: 2026-07-23T22:20:18.400Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: skills/aiq-research/SKILL.md:0-0
Timestamp: 2026-07-23T22:20:18.400Z
Learning: Applies to skills/aiq-research/**/SKILL.md : For a report question answerable from the report already in hand, answer directly without calling the backend; otherwise send a fresh request carrying relevant prior context.
Applied to files:
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2
📚 Learning: 2026-07-15T01:23:44.115Z
Learnt from: tanleach
Repo: NVIDIA-AI-Blueprints/aiq PR: 319
File: mcp/REFERENCE_PARITY.md:26-26
Timestamp: 2026-07-15T01:23:44.115Z
Learning: In the AI-Q MCP server (mcp/ directory), the public tool surface is intentionally frozen to exactly three tools for Phase 1: `submit_query`, `poll_query`, and `get_final_report` (documented in `mcp/REFERENCE_PARITY.md` and `docs/source/integration/mcp-server.md`). Report-aware Q&A (`report_ask`) and report editing (`report_edit`) capabilities are intentionally excluded from this MCP tool surface and require a future contract change/release to add.
Applied to files:
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2
📚 Learning: 2026-08-11T06:34:42.948Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 429
File: src/aiq_agent/agents/chat_researcher/agent.py:183-183
Timestamp: 2026-08-11T06:34:42.948Z
Learning: In `src/aiq_agent/agents/chat_researcher/agent.py`, `ChatResearcherAgent.validate_deep_research_tools_fn` is an injected validator contract whose returned `error_msg` can contain arbitrary sensitive text. Return `error_msg` to the caller when required, but use `log_content_metadata(error_msg)` for production logging so centralized logs do not contain the diagnostic content.
Applied to files:
src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-06-11T21:21:11.314Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: docs/source/contributing/code-organization.md:0-0
Timestamp: 2026-06-11T21:21:11.314Z
Learning: Applies to docs/source/contributing/src/aiq_agent/common/** : Common utilities including LLM provider, callbacks, prompt utils, and data_sources should be organized in `src/aiq_agent/common/`
Applied to files:
src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-07-06T23:55:42.908Z
Learnt from: cdgamarose-nv
Repo: NVIDIA-AI-Blueprints/aiq PR: 311
File: src/aiq_agent/agents/deep_researcher/prompts/orchestrator.j2:74-81
Timestamp: 2026-07-06T23:55:42.908Z
Learning: In agent test files (e.g., tests/aiq_agent/agents/*/test_agent.py), avoid brittle assertions that match exact substrings from prompt template files (such as *.j2 prompt wording). Prompt wording can change frequently, so instead assert structural/behavioral properties (e.g., that the prompt builder is called, that required sections/fields are present via stable markers, that the model output/agent behavior conforms to an expected schema, or that key actions are taken) rather than matching literal prompt text.
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-07-23T22:19:55.347Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: docs/source/contributing/pr-workflow.md:0-0
Timestamp: 2026-07-23T22:19:55.347Z
Learning: Add or update tests for changes and run the narrowest relevant local checks, recording the exact commands in the pull request description.
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-07-23T22:19:59.759Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: docs/source/contributing/testing.md:0-0
Timestamp: 2026-07-23T22:19:59.759Z
Learning: Applies to docs/source/contributing/**/test_*.py : Use mocks for external services in pytest tests.
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-06-14T17:49:10.515Z
Learnt from: torkian
Repo: NVIDIA-AI-Blueprints/aiq PR: 273
File: frontends/aiq_api/tests/test_sse_reconnect_cursor.py:384-401
Timestamp: 2026-06-14T17:49:10.515Z
Learning: In `frontends/aiq_api/tests/test_sse_reconnect_cursor.py` (and the generators in `frontends/aiq_api/src/aiq_api/routes/jobs.py`), `EventStore` and `get_connection_manager` are imported **inside** the generator functions at call time (`from ..jobs.event_store import EventStore`), not at module level. Therefore, `aiq_api.routes.jobs` has no `EventStore` attribute. The correct `unittest.mock.patch` targets are on the source class itself (e.g., `aiq_api.jobs.event_store.EventStore.get_events_async`), not `aiq_api.routes.jobs.EventStore`. Suggesting the latter would raise `AttributeError`.
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
🪛 ast-grep (0.45.1)
src/aiq_agent/agents/shallow_researcher/agent.py
[warning] 108-108: XPath query is request-/variable-derived; use parameterized XPath to prevent injection.
Context: _INLINE_CITATION_RE.findall(body)
Note: [CWE-643] Improper Neutralization of Data within XPath Expressions ('XPath Injection').
(xpath-injection-python)
🔇 Additional comments (5)
src/aiq_agent/agents/shallow_researcher/agent.py (2)
42-42: LGTM!Also applies to: 215-246, 499-532
112-120: 🗄️ Data Integrity & IntegrationNo change is needed.
SourceRegistry.addstores entries only whenurlorcitation_keyis present.runpassesregistry.all_sources()to_format_citation_repair_sources, so numbering cannot drift.> Likely an incorrect or invalid review comment.src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2 (1)
1-12: LGTM!Also applies to: 15-16, 35-40
tests/aiq_agent/agents/shallow_researcher/test_agent.py (1)
26-26: LGTM!Also applies to: 35-35, 71-71, 681-682, 708-711, 739-786, 897-897, 911-916, 929-959, 1230-1236
tests/aiq_agent/test_default_model_profiles.py (1)
22-25: LGTM!
Signed-off-by: Ajay Thorve <athorve@nvidia.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/aiq_agent/agents/shallow_researcher/agent.py`:
- Around line 115-120: Update the citation validation logic around the
source-heading handling and _REFERENCE_ENTRY_LINE_RE so it removes only the
contiguous reference-entry block belonging to the matched heading, not matching
lines throughout the report. Preserve valid prefix citations such as “[1] CUDA
is ...” outside that block, and add a focused regression test covering that
case.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 794c24ff-054d-4994-9b94-e230569ae724
📒 Files selected for processing (4)
src/aiq_agent/agents/shallow_researcher/agent.pysrc/aiq_agent/agents/shallow_researcher/prompts/researcher.j2tests/aiq_agent/agents/shallow_researcher/test_agent.pytests/aiq_agent/test_default_model_profiles.py
📜 Review details
⏰ Context from checks skipped due to timeout. (3)
- GitHub Check: Lint and Hooks
- GitHub Check: Pytest and Coverage
- GitHub Check: Script Validation
🧰 Additional context used
📓 Path-based instructions (7)
**/*
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*: Do not include secrets, credentials, private hostnames, internal-only logs, customer data, or generated local artifacts.
Add or update tests for behavior changes.
**/*: For substantial behavior, authentication, UI, or architecture changes, open a design discussion before coding rather than landing a large unreviewed change.
Keep changes scoped to this repository and avoid editing adjacent repositories; treat eachsources/*package independently and prefer the smallest package-scoped change.
Keep pull requests scoped, avoid unrelated files and generated artifacts, provide validation evidence, and ensure every commit has DCO sign-off.
Files:
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.pytests/aiq_agent/test_default_model_profiles.py
src/aiq_agent/agents/**/*
⚙️ CodeRabbit configuration file
src/aiq_agent/agents/**/*: Review agent changes for research workflow correctness, graph state transitions, prompt/tool contracts,
HITL behavior, and failure handling. Flag changes that weaken source attribution, report generation,
async cancellation, checkpointing, or data-source selection without focused tests and docs.
Files:
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2src/aiq_agent/agents/shallow_researcher/agent.py
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.py: Runuv run ruff check .anduv run ruff format --check .for root Python changes.
Runuv run pytestfor root project Python changes.
Files:
src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.pytests/aiq_agent/test_default_model_profiles.py
**/*.{py,pyi}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{py,pyi}: Format and lint Python code with Ruff using line length 120, Python 3.11 targeting, rules E, F, W, I, PL, and UP, with single-line imports; do not reformat unrelated code.
Missing-secret paths must degrade gracefully by stubbing or skipping rather than crashing or leaking secrets.
Files:
src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.pytests/aiq_agent/test_default_model_profiles.py
**/*.{py,pyi,js,jsx,ts,tsx,yml,yaml,json,env,md}
📄 CodeRabbit inference engine (AGENTS.md)
Never commit secrets, tokens, or environment-specific hostnames; use environment variables and
SecretStr, and resolve API keys at runtime.
Files:
src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.pytests/aiq_agent/test_default_model_profiles.py
**/*.{py,pyi,js,jsx,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Never print or log secret values, including in tool output or error messages.
Files:
src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.pytests/aiq_agent/test_default_model_profiles.py
src/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
src/**/*.py: Respect authenticated data sources by honoringrequires_auth, passing through per-user tokens, and using backend token validators; apply owner guardrails before loading protected report or artifact context into an agent.
Do not weaken or bypassAuthMiddleware, authentication validators, or authentication gating without prior design discussion.
Files:
src/aiq_agent/agents/shallow_researcher/agent.py
🧠 Learnings (17)
📓 Common learnings
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: skills/aiq-research/SKILL.md:0-0
Timestamp: 2026-07-23T22:20:18.400Z
Learning: Applies to skills/aiq-research/**/SKILL.md : Present returned reports with citations and source URLs intact; do not truncate citations or source URLs.
📚 Learning: 2026-07-23T22:20:18.400Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: skills/aiq-research/SKILL.md:0-0
Timestamp: 2026-07-23T22:20:18.400Z
Learning: Applies to skills/aiq-research/**/SKILL.md : Present returned reports with citations and source URLs intact; do not truncate citations or source URLs.
Applied to files:
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-08-11T06:35:46.104Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 429
File: src/aiq_agent/agents/report_rewriter/agent.py:197-200
Timestamp: 2026-08-11T06:35:46.104Z
Learning: In `src/aiq_agent/agents/report_rewriter/agent.py`, `rewrite_report` is shared by async and inline rewrite paths. It must derive its citation source allowlist only from the canonical `original_report` and durable `parent_context`. Do not widen its API to accept caller-supplied sources or change its string-only return contract solely to remove bounded duplicate parsing.
Applied to files:
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-07-23T22:20:18.400Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: skills/aiq-research/SKILL.md:0-0
Timestamp: 2026-07-23T22:20:18.400Z
Learning: Applies to skills/aiq-research/**/SKILL.md : Treat report editing as cosmetic only and use `report_edit <JOB_ID> <EDIT_INSTRUCTIONS>`; use `research` for a new or refined investigation.
Applied to files:
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2
📚 Learning: 2026-07-06T23:55:46.952Z
Learnt from: cdgamarose-nv
Repo: NVIDIA-AI-Blueprints/aiq PR: 311
File: src/aiq_agent/agents/deep_researcher/prompts/orchestrator.j2:74-81
Timestamp: 2026-07-06T23:55:46.952Z
Learning: In `tests/aiq_agent/agents/deep_researcher/test_agent.py`, avoid asserting exact substrings from the orchestrator/writer prompt templates (e.g., `src/aiq_agent/agents/deep_researcher/prompts/orchestrator.j2`) since prompt wording changes frequently. Prefer testing structural/behavioral properties instead.
Applied to files:
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/test_default_model_profiles.py
📚 Learning: 2026-06-11T21:21:11.314Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: docs/source/contributing/code-organization.md:0-0
Timestamp: 2026-06-11T21:21:11.314Z
Learning: Applies to docs/source/contributing/src/aiq_agent/agents/shallow_researcher/** : Shallow researcher agent should be organized in `src/aiq_agent/agents/shallow_researcher/`
Applied to files:
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.pytests/aiq_agent/test_default_model_profiles.py
📚 Learning: 2026-07-23T22:20:09.213Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: skills/aiq-deploy/SKILL.md:0-0
Timestamp: 2026-07-23T22:20:09.213Z
Learning: Applies to skills/aiq-deploy/**/* : Do not continue into deep research or deep-research completion validation unless the user asks for it or confirms the post-deployment validation prompt.
Applied to files:
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2
📚 Learning: 2026-07-23T22:20:18.400Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: skills/aiq-research/SKILL.md:0-0
Timestamp: 2026-07-23T22:20:18.400Z
Learning: Applies to skills/aiq-research/**/SKILL.md : Use this skill for research-shaped requests, including deep research, AI-Q research, or requests to ask AI-Q; do not use it for installation, deployment, lifecycle, UI, CLI, Docker, Helm, or troubleshooting requests.
Applied to files:
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2
📚 Learning: 2026-07-15T01:23:44.115Z
Learnt from: tanleach
Repo: NVIDIA-AI-Blueprints/aiq PR: 319
File: mcp/REFERENCE_PARITY.md:26-26
Timestamp: 2026-07-15T01:23:44.115Z
Learning: In the AI-Q MCP server (mcp/ directory), the public tool surface is intentionally frozen to exactly three tools for Phase 1: `submit_query`, `poll_query`, and `get_final_report` (documented in `mcp/REFERENCE_PARITY.md` and `docs/source/integration/mcp-server.md`). Report-aware Q&A (`report_ask`) and report editing (`report_edit`) capabilities are intentionally excluded from this MCP tool surface and require a future contract change/release to add.
Applied to files:
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2
📚 Learning: 2026-07-23T22:20:18.400Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: skills/aiq-research/SKILL.md:0-0
Timestamp: 2026-07-23T22:20:18.400Z
Learning: Applies to skills/aiq-research/**/SKILL.md : For a report question answerable from the report already in hand, answer directly without calling the backend; otherwise send a fresh request carrying relevant prior context.
Applied to files:
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2
📚 Learning: 2026-08-11T06:34:42.948Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 429
File: src/aiq_agent/agents/chat_researcher/agent.py:183-183
Timestamp: 2026-08-11T06:34:42.948Z
Learning: In `src/aiq_agent/agents/chat_researcher/agent.py`, `ChatResearcherAgent.validate_deep_research_tools_fn` is an injected validator contract whose returned `error_msg` can contain arbitrary sensitive text. Return `error_msg` to the caller when required, but use `log_content_metadata(error_msg)` for production logging so centralized logs do not contain the diagnostic content.
Applied to files:
src/aiq_agent/agents/shallow_researcher/agent.py
📚 Learning: 2026-06-29T20:51:55.634Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 293
File: skills/aiq-research/scripts/aiq.py:2-3
Timestamp: 2026-06-29T20:51:55.634Z
Learning: For `skills/aiq-research/scripts/aiq.py`, when using the OSRB `CC-BY-4.0 AND Apache-2.0` dual-license SPDX template, the `SPDX-FileCopyrightText` format intentionally omits the comma after the year; do not flag that punctuation as a repo-style issue when the header is matching the source template exactly.
Applied to files:
src/aiq_agent/agents/shallow_researcher/agent.py
📚 Learning: 2026-08-11T06:34:44.687Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 429
File: src/aiq_agent/agents/deep_researcher/register.py:302-307
Timestamp: 2026-08-11T06:34:44.687Z
Learning: In Python logging code that handles potentially sensitive exceptions, do not add `exc_info=True` solely to restore stack traces, because standard traceback formatting includes `str(exception)` and may expose provider, customer, or credential-bearing content. When sensitive-content redaction is required, log the exception type together with `log_content_metadata(exception)` instead.
Applied to files:
src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.pytests/aiq_agent/test_default_model_profiles.py
📚 Learning: 2026-07-06T23:55:42.908Z
Learnt from: cdgamarose-nv
Repo: NVIDIA-AI-Blueprints/aiq PR: 311
File: src/aiq_agent/agents/deep_researcher/prompts/orchestrator.j2:74-81
Timestamp: 2026-07-06T23:55:42.908Z
Learning: In agent test files (e.g., tests/aiq_agent/agents/*/test_agent.py), avoid brittle assertions that match exact substrings from prompt template files (such as *.j2 prompt wording). Prompt wording can change frequently, so instead assert structural/behavioral properties (e.g., that the prompt builder is called, that required sections/fields are present via stable markers, that the model output/agent behavior conforms to an expected schema, or that key actions are taken) rather than matching literal prompt text.
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-06-11T21:21:11.314Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: docs/source/contributing/code-organization.md:0-0
Timestamp: 2026-06-11T21:21:11.314Z
Learning: Applies to docs/source/contributing/src/aiq_agent/agents/deep_researcher/** : Deep researcher agent should be organized in `src/aiq_agent/agents/deep_researcher/` with implementation details documented in README.md
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-06-11T21:21:11.314Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: docs/source/contributing/code-organization.md:0-0
Timestamp: 2026-06-11T21:21:11.314Z
Learning: Applies to docs/source/contributing/src/aiq_agent/agents/chat_researcher/** : Chat researcher agent should be organized in `src/aiq_agent/agents/chat_researcher/` and serve as orchestrator with orchestration node containing intent, meta, and depth information
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-08-04T22:37:11.331Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 414
File: tests/aiq_agent/test_default_model_profiles.py:72-75
Timestamp: 2026-08-04T22:37:11.331Z
Learning: For the AIQ default-model migration, PR `#414` intentionally retains Nemotron 3 Ultra expectations for intent classification and shallow research because the Nemotron Nano 3.5 Preview public endpoint and GA slug are unavailable. The Nano intent alias and corresponding profile tests are deferred to stacked draft PR `#419` after the availability gate.
Applied to files:
tests/aiq_agent/test_default_model_profiles.py
🪛 ast-grep (0.45.1)
src/aiq_agent/agents/shallow_researcher/agent.py
[warning] 119-119: XPath query is request-/variable-derived; use parameterized XPath to prevent injection.
Context: _INLINE_CITATION_RE.findall(prose)
Note: [CWE-643] Improper Neutralization of Data within XPath Expressions ('XPath Injection').
(xpath-injection-python)
🔇 Additional comments (3)
src/aiq_agent/agents/shallow_researcher/prompts/researcher.j2 (1)
30-37: LGTM!tests/aiq_agent/agents/shallow_researcher/test_agent.py (1)
311-330: LGTM!Also applies to: 735-765, 767-814
tests/aiq_agent/test_default_model_profiles.py (1)
156-164: LGTM!
Signed-off-by: Ajay Thorve <athorve@nvidia.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/aiq_agent/agents/shallow_researcher/test_agent.py`:
- Around line 682-719: Update
test_missing_uploaded_document_citation_is_repaired_before_publication to cover
the bounded repair path by registering a second source and configuring the
mocked repaired response to cite report.pdf, p.15; ensure the assertions verify
the repaired citation and publication output. If retaining the single-source
setup instead, rename the test to identify fallback behavior and explicitly
assert the **References:** section.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 054ee50a-02f4-4a19-8865-20a17cedcb6e
📒 Files selected for processing (1)
tests/aiq_agent/agents/shallow_researcher/test_agent.py
📜 Review details
⏰ Context from checks skipped due to timeout. (3)
- GitHub Check: Script Validation
- GitHub Check: Lint and Hooks
- GitHub Check: Pytest and Coverage
🧰 Additional context used
📓 Path-based instructions (5)
**/*
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*: Do not include secrets, credentials, private hostnames, internal-only logs, customer data, or generated local artifacts.
Add or update tests for behavior changes.
**/*: For substantial behavior, authentication, UI, or architecture changes, open a design discussion before coding rather than landing a large unreviewed change.
Keep changes scoped to this repository and avoid editing adjacent repositories; treat eachsources/*package independently and prefer the smallest package-scoped change.
Keep pull requests scoped, avoid unrelated files and generated artifacts, provide validation evidence, and ensure every commit has DCO sign-off.
Files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.py: Runuv run ruff check .anduv run ruff format --check .for root Python changes.
Runuv run pytestfor root project Python changes.
Files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
**/*.{py,pyi}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{py,pyi}: Format and lint Python code with Ruff using line length 120, Python 3.11 targeting, rules E, F, W, I, PL, and UP, with single-line imports; do not reformat unrelated code.
Missing-secret paths must degrade gracefully by stubbing or skipping rather than crashing or leaking secrets.
Files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
**/*.{py,pyi,js,jsx,ts,tsx,yml,yaml,json,env,md}
📄 CodeRabbit inference engine (AGENTS.md)
Never commit secrets, tokens, or environment-specific hostnames; use environment variables and
SecretStr, and resolve API keys at runtime.
Files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
**/*.{py,pyi,js,jsx,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Never print or log secret values, including in tool output or error messages.
Files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
🧠 Learnings (6)
📓 Common learnings
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 429
File: src/aiq_agent/agents/report_rewriter/agent.py:197-200
Timestamp: 2026-08-11T06:35:46.104Z
Learning: In `src/aiq_agent/agents/report_rewriter/agent.py`, `rewrite_report` is shared by async and inline rewrite paths. It must derive its citation source allowlist only from the canonical `original_report` and durable `parent_context`. Do not widen its API to accept caller-supplied sources or change its string-only return contract solely to remove bounded duplicate parsing.
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: skills/aiq-research/SKILL.md:0-0
Timestamp: 2026-07-23T22:20:18.400Z
Learning: Applies to skills/aiq-research/**/SKILL.md : Present returned reports with citations and source URLs intact; do not truncate citations or source URLs.
📚 Learning: 2026-07-06T23:55:42.908Z
Learnt from: cdgamarose-nv
Repo: NVIDIA-AI-Blueprints/aiq PR: 311
File: src/aiq_agent/agents/deep_researcher/prompts/orchestrator.j2:74-81
Timestamp: 2026-07-06T23:55:42.908Z
Learning: In agent test files (e.g., tests/aiq_agent/agents/*/test_agent.py), avoid brittle assertions that match exact substrings from prompt template files (such as *.j2 prompt wording). Prompt wording can change frequently, so instead assert structural/behavioral properties (e.g., that the prompt builder is called, that required sections/fields are present via stable markers, that the model output/agent behavior conforms to an expected schema, or that key actions are taken) rather than matching literal prompt text.
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-08-11T06:35:46.104Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 429
File: src/aiq_agent/agents/report_rewriter/agent.py:197-200
Timestamp: 2026-08-11T06:35:46.104Z
Learning: In `src/aiq_agent/agents/report_rewriter/agent.py`, `rewrite_report` is shared by async and inline rewrite paths. It must derive its citation source allowlist only from the canonical `original_report` and durable `parent_context`. Do not widen its API to accept caller-supplied sources or change its string-only return contract solely to remove bounded duplicate parsing.
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-06-11T21:21:11.314Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: docs/source/contributing/code-organization.md:0-0
Timestamp: 2026-06-11T21:21:11.314Z
Learning: Applies to docs/source/contributing/src/aiq_agent/agents/shallow_researcher/** : Shallow researcher agent should be organized in `src/aiq_agent/agents/shallow_researcher/`
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-07-23T22:20:18.400Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: skills/aiq-research/SKILL.md:0-0
Timestamp: 2026-07-23T22:20:18.400Z
Learning: Applies to skills/aiq-research/**/SKILL.md : Present returned reports with citations and source URLs intact; do not truncate citations or source URLs.
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-08-11T06:34:44.687Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 429
File: src/aiq_agent/agents/deep_researcher/register.py:302-307
Timestamp: 2026-08-11T06:34:44.687Z
Learning: In Python logging code that handles potentially sensitive exceptions, do not add `exc_info=True` solely to restore stack traces, because standard traceback formatting includes `str(exception)` and may expose provider, customer, or credential-bearing content. When sensitive-content redaction is required, log the exception type together with `log_content_metadata(exception)` instead.
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
🔇 Additional comments (1)
tests/aiq_agent/agents/shallow_researcher/test_agent.py (1)
554-566: LGTM!
Signed-off-by: Ajay Thorve <athorve@nvidia.com>
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (4)
src/aiq_agent/agents/shallow_researcher/agent.py (3)
102-128: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winScan every recognized source section before checking inline markers.
The current scan removes only the first heading's reference block. A report such as
Sources: [1] ...followed byReferences: [2] ...leaves the second[2]definition inprose. If citation2is valid, a source-only report can pass without any inline citation. (raw.githubusercontent.com)Process every recognized source section when excluding reference-definition lines. Add a regression with two source headings and no inline marker.
As per path instructions: “Review agent changes for research workflow correctness, graph state transitions, prompt/tool contracts, HITL behavior, and failure handling. Flag changes that weaken source attribution, report generation, async cancellation, checkpointing, or data-source selection without focused tests and docs.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/aiq_agent/agents/shallow_researcher/agent.py` around lines 102 - 128, Update _has_citation_integrity to identify and exclude contiguous reference-definition blocks after every recognized source heading, not only the first _SOURCE_SECTION_HEADING_RE match, before applying _INLINE_CITATION_RE. Preserve valid inline markers elsewhere in the report, and add a regression covering multiple source headings with definitions but no inline citation.Source: Path instructions
63-99: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winPreserve answer text after a source heading.
_append_minimal_citationtruncates the report atsource_heading.start(). If a model places a source section before trailing answer text, the single-source fallback deletes that answer before adding[1]. The integrity helper already accepts inline markers after the source section, so this layout is part of the supported contract. (raw.githubusercontent.com)Remove only the source heading and its contiguous reference-definition block. Preserve the remaining report text. Add a regression with answer text after
## Sources.As per path instructions: “Review agent changes for research workflow correctness, graph state transitions, prompt/tool contracts, HITL behavior, and failure handling. Flag changes that weaken source attribution, report generation, async cancellation, checkpointing, or data-source selection without focused tests and docs.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/aiq_agent/agents/shallow_researcher/agent.py` around lines 63 - 99, Update _append_minimal_citation so replacing an existing source section removes only the matched heading and its contiguous reference-definition lines identified by _REFERENCE_ENTRY_LINE_RE, preserving any trailing report text. Reinsert the verified inline citation and references without deleting answer content that follows the source section. Add a regression test covering answer text after a “## Sources” heading.Source: Path instructions
525-558: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winSuppress draft output artifacts until final citation verification.
AgentEventCallback.on_llm_endemitsArtifactType.OUTPUTfor unsuppressed post-tool LLM responses. A failed repair can therefore publish an uncited draft beforeCitationIntegrityError. ApplySUPPRESS_OUTPUT_ARTIFACT_TAGto all pre-verification synthesis calls and add a regression for failed repair publication.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/aiq_agent/agents/shallow_researcher/agent.py` around lines 525 - 558, Apply SUPPRESS_OUTPUT_ARTIFACT_TAG to every pre-verification synthesis LLM call in the shallow researcher flow, including the initial generation and _repair_missing_citations, so AgentEventCallback.on_llm_end cannot publish draft OUTPUT artifacts before final citation verification. Keep final verified publication behavior unchanged, and add a regression test covering a failed citation repair that raises CitationIntegrityError without emitting an OUTPUT artifact.Source: Path instructions
tests/aiq_agent/agents/shallow_researcher/test_agent.py (1)
770-771: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winAssert the repaired multi-source output includes a source section.
repaired_responsecontains inline[1]and[2]markers but no references heading or reference lines. The post-repair verification can synthesize a source section fromsources, so this test can pass even when the tool-free repair violates its own output contract. (raw.githubusercontent.com)Assert the final content includes the source heading and both allowed reference entries, or include the required section in
repaired_response.As per coding guidelines: “Add or update tests for behavior changes.”
Also applies to: 797-879
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/aiq_agent/agents/shallow_researcher/test_agent.py` around lines 770 - 771, Update test_missing_citations_are_repaired_once_for_multiple_sources and the related assertions in the 797–879 range to verify the repaired response itself contains the required source section, including the source heading and both allowed reference entries. Do not rely solely on post-repair verification synthesizing references from sources; alternatively, update repaired_response to include that complete section while preserving the existing bounded-repair behavior.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@src/aiq_agent/agents/shallow_researcher/agent.py`:
- Around line 102-128: Update _has_citation_integrity to identify and exclude
contiguous reference-definition blocks after every recognized source heading,
not only the first _SOURCE_SECTION_HEADING_RE match, before applying
_INLINE_CITATION_RE. Preserve valid inline markers elsewhere in the report, and
add a regression covering multiple source headings with definitions but no
inline citation.
- Around line 63-99: Update _append_minimal_citation so replacing an existing
source section removes only the matched heading and its contiguous
reference-definition lines identified by _REFERENCE_ENTRY_LINE_RE, preserving
any trailing report text. Reinsert the verified inline citation and references
without deleting answer content that follows the source section. Add a
regression test covering answer text after a “## Sources” heading.
- Around line 525-558: Apply SUPPRESS_OUTPUT_ARTIFACT_TAG to every
pre-verification synthesis LLM call in the shallow researcher flow, including
the initial generation and _repair_missing_citations, so
AgentEventCallback.on_llm_end cannot publish draft OUTPUT artifacts before final
citation verification. Keep final verified publication behavior unchanged, and
add a regression test covering a failed citation repair that raises
CitationIntegrityError without emitting an OUTPUT artifact.
In `@tests/aiq_agent/agents/shallow_researcher/test_agent.py`:
- Around line 770-771: Update
test_missing_citations_are_repaired_once_for_multiple_sources and the related
assertions in the 797–879 range to verify the repaired response itself contains
the required source section, including the source heading and both allowed
reference entries. Do not rely solely on post-repair verification synthesizing
references from sources; alternatively, update repaired_response to include that
complete section while preserving the existing bounded-repair behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 415c7510-d9ca-4ddf-8dd3-80ce58194a3d
📒 Files selected for processing (2)
src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
📜 Review details
🧰 Additional context used
📓 Path-based instructions (7)
**/*
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*: Do not include secrets, credentials, private hostnames, internal-only logs, customer data, or generated local artifacts.
Add or update tests for behavior changes.
**/*: For substantial behavior, authentication, UI, or architecture changes, open a design discussion before coding rather than landing a large unreviewed change.
Keep changes scoped to this repository and avoid editing adjacent repositories; treat eachsources/*package independently and prefer the smallest package-scoped change.
Keep pull requests scoped, avoid unrelated files and generated artifacts, provide validation evidence, and ensure every commit has DCO sign-off.
Files:
tests/aiq_agent/agents/shallow_researcher/test_agent.pysrc/aiq_agent/agents/shallow_researcher/agent.py
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.py: Runuv run ruff check .anduv run ruff format --check .for root Python changes.
Runuv run pytestfor root project Python changes.
Files:
tests/aiq_agent/agents/shallow_researcher/test_agent.pysrc/aiq_agent/agents/shallow_researcher/agent.py
**/*.{py,pyi}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{py,pyi}: Format and lint Python code with Ruff using line length 120, Python 3.11 targeting, rules E, F, W, I, PL, and UP, with single-line imports; do not reformat unrelated code.
Missing-secret paths must degrade gracefully by stubbing or skipping rather than crashing or leaking secrets.
Files:
tests/aiq_agent/agents/shallow_researcher/test_agent.pysrc/aiq_agent/agents/shallow_researcher/agent.py
**/*.{py,pyi,js,jsx,ts,tsx,yml,yaml,json,env,md}
📄 CodeRabbit inference engine (AGENTS.md)
Never commit secrets, tokens, or environment-specific hostnames; use environment variables and
SecretStr, and resolve API keys at runtime.
Files:
tests/aiq_agent/agents/shallow_researcher/test_agent.pysrc/aiq_agent/agents/shallow_researcher/agent.py
**/*.{py,pyi,js,jsx,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Never print or log secret values, including in tool output or error messages.
Files:
tests/aiq_agent/agents/shallow_researcher/test_agent.pysrc/aiq_agent/agents/shallow_researcher/agent.py
src/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
src/**/*.py: Respect authenticated data sources by honoringrequires_auth, passing through per-user tokens, and using backend token validators; apply owner guardrails before loading protected report or artifact context into an agent.
Do not weaken or bypassAuthMiddleware, authentication validators, or authentication gating without prior design discussion.
Files:
src/aiq_agent/agents/shallow_researcher/agent.py
src/aiq_agent/agents/**/*
⚙️ CodeRabbit configuration file
src/aiq_agent/agents/**/*: Review agent changes for research workflow correctness, graph state transitions, prompt/tool contracts,
HITL behavior, and failure handling. Flag changes that weaken source attribution, report generation,
async cancellation, checkpointing, or data-source selection without focused tests and docs.
Files:
src/aiq_agent/agents/shallow_researcher/agent.py
🧠 Learnings (10)
📓 Common learnings
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: skills/aiq-research/SKILL.md:0-0
Timestamp: 2026-07-23T22:20:18.400Z
Learning: Applies to skills/aiq-research/**/SKILL.md : Present returned reports with citations and source URLs intact; do not truncate citations or source URLs.
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 429
File: src/aiq_agent/agents/report_rewriter/agent.py:197-200
Timestamp: 2026-08-11T06:35:46.104Z
Learning: In `src/aiq_agent/agents/report_rewriter/agent.py`, `rewrite_report` is shared by async and inline rewrite paths. It must derive its citation source allowlist only from the canonical `original_report` and durable `parent_context`. Do not widen its API to accept caller-supplied sources or change its string-only return contract solely to remove bounded duplicate parsing.
📚 Learning: 2026-08-11T06:35:46.104Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 429
File: src/aiq_agent/agents/report_rewriter/agent.py:197-200
Timestamp: 2026-08-11T06:35:46.104Z
Learning: In `src/aiq_agent/agents/report_rewriter/agent.py`, `rewrite_report` is shared by async and inline rewrite paths. It must derive its citation source allowlist only from the canonical `original_report` and durable `parent_context`. Do not widen its API to accept caller-supplied sources or change its string-only return contract solely to remove bounded duplicate parsing.
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.pysrc/aiq_agent/agents/shallow_researcher/agent.py
📚 Learning: 2026-07-06T23:55:42.908Z
Learnt from: cdgamarose-nv
Repo: NVIDIA-AI-Blueprints/aiq PR: 311
File: src/aiq_agent/agents/deep_researcher/prompts/orchestrator.j2:74-81
Timestamp: 2026-07-06T23:55:42.908Z
Learning: In agent test files (e.g., tests/aiq_agent/agents/*/test_agent.py), avoid brittle assertions that match exact substrings from prompt template files (such as *.j2 prompt wording). Prompt wording can change frequently, so instead assert structural/behavioral properties (e.g., that the prompt builder is called, that required sections/fields are present via stable markers, that the model output/agent behavior conforms to an expected schema, or that key actions are taken) rather than matching literal prompt text.
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-06-11T21:21:11.314Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: docs/source/contributing/code-organization.md:0-0
Timestamp: 2026-06-11T21:21:11.314Z
Learning: Applies to docs/source/contributing/src/aiq_agent/agents/shallow_researcher/** : Shallow researcher agent should be organized in `src/aiq_agent/agents/shallow_researcher/`
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.pysrc/aiq_agent/agents/shallow_researcher/agent.py
📚 Learning: 2026-07-23T22:20:18.400Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: skills/aiq-research/SKILL.md:0-0
Timestamp: 2026-07-23T22:20:18.400Z
Learning: Applies to skills/aiq-research/**/SKILL.md : Present returned reports with citations and source URLs intact; do not truncate citations or source URLs.
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.pysrc/aiq_agent/agents/shallow_researcher/agent.py
📚 Learning: 2026-06-11T21:21:11.314Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: docs/source/contributing/code-organization.md:0-0
Timestamp: 2026-06-11T21:21:11.314Z
Learning: Applies to docs/source/contributing/src/aiq_agent/agents/deep_researcher/** : Deep researcher agent should be organized in `src/aiq_agent/agents/deep_researcher/` with implementation details documented in README.md
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-08-04T22:37:11.331Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 414
File: tests/aiq_agent/test_default_model_profiles.py:72-75
Timestamp: 2026-08-04T22:37:11.331Z
Learning: For the AIQ default-model migration, PR `#414` intentionally retains Nemotron 3 Ultra expectations for intent classification and shallow research because the Nemotron Nano 3.5 Preview public endpoint and GA slug are unavailable. The Nano intent alias and corresponding profile tests are deferred to stacked draft PR `#419` after the availability gate.
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-08-11T06:34:42.948Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 429
File: src/aiq_agent/agents/chat_researcher/agent.py:183-183
Timestamp: 2026-08-11T06:34:42.948Z
Learning: In `src/aiq_agent/agents/chat_researcher/agent.py`, `ChatResearcherAgent.validate_deep_research_tools_fn` is an injected validator contract whose returned `error_msg` can contain arbitrary sensitive text. Return `error_msg` to the caller when required, but use `log_content_metadata(error_msg)` for production logging so centralized logs do not contain the diagnostic content.
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.pysrc/aiq_agent/agents/shallow_researcher/agent.py
📚 Learning: 2026-08-11T06:34:44.687Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 429
File: src/aiq_agent/agents/deep_researcher/register.py:302-307
Timestamp: 2026-08-11T06:34:44.687Z
Learning: In Python logging code that handles potentially sensitive exceptions, do not add `exc_info=True` solely to restore stack traces, because standard traceback formatting includes `str(exception)` and may expose provider, customer, or credential-bearing content. When sensitive-content redaction is required, log the exception type together with `log_content_metadata(exception)` instead.
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.pysrc/aiq_agent/agents/shallow_researcher/agent.py
📚 Learning: 2026-07-06T23:55:46.952Z
Learnt from: cdgamarose-nv
Repo: NVIDIA-AI-Blueprints/aiq PR: 311
File: src/aiq_agent/agents/deep_researcher/prompts/orchestrator.j2:74-81
Timestamp: 2026-07-06T23:55:46.952Z
Learning: In `tests/aiq_agent/agents/deep_researcher/test_agent.py`, avoid asserting exact substrings from the orchestrator/writer prompt templates (e.g., `src/aiq_agent/agents/deep_researcher/prompts/orchestrator.j2`) since prompt wording changes frequently. Prefer testing structural/behavioral properties instead.
Applied to files:
src/aiq_agent/agents/shallow_researcher/agent.py
🪛 ast-grep (0.45.1)
src/aiq_agent/agents/shallow_researcher/agent.py
[warning] 127-127: XPath query is request-/variable-derived; use parameterized XPath to prevent injection.
Context: _INLINE_CITATION_RE.findall(prose)
Note: [CWE-643] Improper Neutralization of Data within XPath Expressions ('XPath Injection').
(xpath-injection-python)
🔇 Additional comments (5)
src/aiq_agent/agents/shallow_researcher/agent.py (4)
20-20: LGTM!Also applies to: 43-43
129-141: LGTM!
175-197: LGTM!
238-290: LGTM!tests/aiq_agent/agents/shallow_researcher/test_agent.py (1)
18-19: LGTM!Also applies to: 28-38, 74-74, 111-111, 300-330, 554-571, 687-733, 990-990, 1004-1009, 1022-1052, 1323-1345, 1348-1372
|
after going back and forth with my review agent I found the following: _append_minimal_citation() then discards everything from the first match onward. I reproduced this input: ## Sources of renewable energy
Solar is renewable.The fallback produced: [1]
**References:**
- [1] Article - https://example.comThe complete answer was deleted, and the resulting bare citation can still satisfy the final structural check. Similar losses occur with: Sources: market revenue
The business grew.and with a real source block followed by trailing answer text. Required correction A source section should be recognized only when:
When replacing a source section, remove only the heading and associated reference definitions. Preserve all surrounding Add regressions for:
|
Signed-off-by: Ajay Thorve <athorve@nvidia.com>
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
tests/aiq_agent/agents/shallow_researcher/test_agent.py (1)
315-329: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winMake the timeout assertion distinguish bounded and unbounded repair.
If
asyncio.wait_foris removed,stalled_repaircompletes after one second and this test still receivesCitationIntegrityErrorfrom the empty response path. The test does not prove thatcitation_repair_timeout=0.001bounded the invocation.Record elapsed monotonic time and assert completion well before the one-second stall, or apply an outer test deadline that fails when the internal timeout is absent.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/aiq_agent/agents/shallow_researcher/test_agent.py` around lines 315 - 329, The timeout test around ShallowResearcherAgent._repair_missing_citations must verify bounded execution rather than merely receiving CitationIntegrityError. Measure elapsed time with a monotonic clock around the invocation, or impose an outer deadline, and assert completion occurs well before stalled_repair’s one-second delay while retaining the existing exception assertion.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/aiq_agent/agents/shallow_researcher/agent.py`:
- Around line 97-100: Update the reference-block parsing in
_remove_source_sections so marker-first answer text immediately following a
source definition block is preserved rather than consumed as another reference
definition; establish an unambiguous boundary using the surrounding report
structure, and add focused regression coverage for a marker-first answer
directly after a source block, including any required documentation.
---
Outside diff comments:
In `@tests/aiq_agent/agents/shallow_researcher/test_agent.py`:
- Around line 315-329: The timeout test around
ShallowResearcherAgent._repair_missing_citations must verify bounded execution
rather than merely receiving CitationIntegrityError. Measure elapsed time with a
monotonic clock around the invocation, or impose an outer deadline, and assert
completion occurs well before stalled_repair’s one-second delay while retaining
the existing exception assertion.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 04d83eda-d935-4c64-86c1-ff88b0ecd726
📒 Files selected for processing (2)
src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
📜 Review details
⏰ Context from checks skipped due to timeout. (3)
- GitHub Check: Script Validation
- GitHub Check: Lint and Hooks
- GitHub Check: Pytest and Coverage
🧰 Additional context used
📓 Path-based instructions (7)
**/*
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*: Do not include secrets, credentials, private hostnames, internal-only logs, customer data, or generated local artifacts.
Add or update tests for behavior changes.
**/*: For substantial behavior, authentication, UI, or architecture changes, open a design discussion before coding rather than landing a large unreviewed change.
Keep changes scoped to this repository and avoid editing adjacent repositories; treat eachsources/*package independently and prefer the smallest package-scoped change.
Keep pull requests scoped, avoid unrelated files and generated artifacts, provide validation evidence, and ensure every commit has DCO sign-off.
Files:
src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.py: Runuv run ruff check .anduv run ruff format --check .for root Python changes.
Runuv run pytestfor root project Python changes.
Files:
src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
**/*.{py,pyi}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{py,pyi}: Format and lint Python code with Ruff using line length 120, Python 3.11 targeting, rules E, F, W, I, PL, and UP, with single-line imports; do not reformat unrelated code.
Missing-secret paths must degrade gracefully by stubbing or skipping rather than crashing or leaking secrets.
Files:
src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
**/*.{py,pyi,js,jsx,ts,tsx,yml,yaml,json,env,md}
📄 CodeRabbit inference engine (AGENTS.md)
Never commit secrets, tokens, or environment-specific hostnames; use environment variables and
SecretStr, and resolve API keys at runtime.
Files:
src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
**/*.{py,pyi,js,jsx,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Never print or log secret values, including in tool output or error messages.
Files:
src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
src/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
src/**/*.py: Respect authenticated data sources by honoringrequires_auth, passing through per-user tokens, and using backend token validators; apply owner guardrails before loading protected report or artifact context into an agent.
Do not weaken or bypassAuthMiddleware, authentication validators, or authentication gating without prior design discussion.
Files:
src/aiq_agent/agents/shallow_researcher/agent.py
src/aiq_agent/agents/**/*
⚙️ CodeRabbit configuration file
src/aiq_agent/agents/**/*: Review agent changes for research workflow correctness, graph state transitions, prompt/tool contracts,
HITL behavior, and failure handling. Flag changes that weaken source attribution, report generation,
async cancellation, checkpointing, or data-source selection without focused tests and docs.
Files:
src/aiq_agent/agents/shallow_researcher/agent.py
🧠 Learnings (10)
📓 Common learnings
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 429
File: src/aiq_agent/agents/report_rewriter/agent.py:197-200
Timestamp: 2026-08-11T06:35:46.104Z
Learning: In `src/aiq_agent/agents/report_rewriter/agent.py`, `rewrite_report` is shared by async and inline rewrite paths. It must derive its citation source allowlist only from the canonical `original_report` and durable `parent_context`. Do not widen its API to accept caller-supplied sources or change its string-only return contract solely to remove bounded duplicate parsing.
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: AGENTS.md:0-0
Timestamp: 2026-08-07T22:08:51.443Z
Learning: Applies to **/* : Keep pull requests scoped, avoid unrelated files and generated artifacts, provide validation evidence, and ensure every commit has DCO sign-off.
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: skills/aiq-research/SKILL.md:0-0
Timestamp: 2026-07-23T22:20:18.400Z
Learning: Applies to skills/aiq-research/**/SKILL.md : Present returned reports with citations and source URLs intact; do not truncate citations or source URLs.
📚 Learning: 2026-08-11T06:35:46.104Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 429
File: src/aiq_agent/agents/report_rewriter/agent.py:197-200
Timestamp: 2026-08-11T06:35:46.104Z
Learning: In `src/aiq_agent/agents/report_rewriter/agent.py`, `rewrite_report` is shared by async and inline rewrite paths. It must derive its citation source allowlist only from the canonical `original_report` and durable `parent_context`. Do not widen its API to accept caller-supplied sources or change its string-only return contract solely to remove bounded duplicate parsing.
Applied to files:
src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-06-11T21:21:11.314Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: docs/source/contributing/code-organization.md:0-0
Timestamp: 2026-06-11T21:21:11.314Z
Learning: Applies to docs/source/contributing/src/aiq_agent/agents/shallow_researcher/** : Shallow researcher agent should be organized in `src/aiq_agent/agents/shallow_researcher/`
Applied to files:
src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-08-11T06:34:42.948Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 429
File: src/aiq_agent/agents/chat_researcher/agent.py:183-183
Timestamp: 2026-08-11T06:34:42.948Z
Learning: In `src/aiq_agent/agents/chat_researcher/agent.py`, `ChatResearcherAgent.validate_deep_research_tools_fn` is an injected validator contract whose returned `error_msg` can contain arbitrary sensitive text. Return `error_msg` to the caller when required, but use `log_content_metadata(error_msg)` for production logging so centralized logs do not contain the diagnostic content.
Applied to files:
src/aiq_agent/agents/shallow_researcher/agent.py
📚 Learning: 2026-07-23T22:20:18.400Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: skills/aiq-research/SKILL.md:0-0
Timestamp: 2026-07-23T22:20:18.400Z
Learning: Applies to skills/aiq-research/**/SKILL.md : Present returned reports with citations and source URLs intact; do not truncate citations or source URLs.
Applied to files:
src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-07-06T23:55:46.952Z
Learnt from: cdgamarose-nv
Repo: NVIDIA-AI-Blueprints/aiq PR: 311
File: src/aiq_agent/agents/deep_researcher/prompts/orchestrator.j2:74-81
Timestamp: 2026-07-06T23:55:46.952Z
Learning: In `tests/aiq_agent/agents/deep_researcher/test_agent.py`, avoid asserting exact substrings from the orchestrator/writer prompt templates (e.g., `src/aiq_agent/agents/deep_researcher/prompts/orchestrator.j2`) since prompt wording changes frequently. Prefer testing structural/behavioral properties instead.
Applied to files:
src/aiq_agent/agents/shallow_researcher/agent.py
📚 Learning: 2026-08-11T06:34:44.687Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 429
File: src/aiq_agent/agents/deep_researcher/register.py:302-307
Timestamp: 2026-08-11T06:34:44.687Z
Learning: In Python logging code that handles potentially sensitive exceptions, do not add `exc_info=True` solely to restore stack traces, because standard traceback formatting includes `str(exception)` and may expose provider, customer, or credential-bearing content. When sensitive-content redaction is required, log the exception type together with `log_content_metadata(exception)` instead.
Applied to files:
src/aiq_agent/agents/shallow_researcher/agent.pytests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-07-06T23:55:42.908Z
Learnt from: cdgamarose-nv
Repo: NVIDIA-AI-Blueprints/aiq PR: 311
File: src/aiq_agent/agents/deep_researcher/prompts/orchestrator.j2:74-81
Timestamp: 2026-07-06T23:55:42.908Z
Learning: In agent test files (e.g., tests/aiq_agent/agents/*/test_agent.py), avoid brittle assertions that match exact substrings from prompt template files (such as *.j2 prompt wording). Prompt wording can change frequently, so instead assert structural/behavioral properties (e.g., that the prompt builder is called, that required sections/fields are present via stable markers, that the model output/agent behavior conforms to an expected schema, or that key actions are taken) rather than matching literal prompt text.
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-06-11T21:21:11.314Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: docs/source/contributing/code-organization.md:0-0
Timestamp: 2026-06-11T21:21:11.314Z
Learning: Applies to docs/source/contributing/src/aiq_agent/agents/deep_researcher/** : Deep researcher agent should be organized in `src/aiq_agent/agents/deep_researcher/` with implementation details documented in README.md
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
📚 Learning: 2026-08-04T22:37:11.331Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 414
File: tests/aiq_agent/test_default_model_profiles.py:72-75
Timestamp: 2026-08-04T22:37:11.331Z
Learning: For the AIQ default-model migration, PR `#414` intentionally retains Nemotron 3 Ultra expectations for intent classification and shallow research because the Nemotron Nano 3.5 Preview public endpoint and GA slug are unavailable. The Nano intent alias and corresponding profile tests are deferred to stacked draft PR `#419` after the availability gate.
Applied to files:
tests/aiq_agent/agents/shallow_researcher/test_agent.py
🪛 ast-grep (0.45.1)
src/aiq_agent/agents/shallow_researcher/agent.py
[warning] 174-174: XPath query is request-/variable-derived; use parameterized XPath to prevent injection.
Context: _INLINE_CITATION_RE.findall(prose)
Note: [CWE-643] Improper Neutralization of Data within XPath Expressions ('XPath Injection').
(xpath-injection-python)
Signed-off-by: Ajay Thorve <athorve@nvidia.com>
|
@tanleach Thanks — I reproduced this and the finding was valid. Fixed in
The requested regressions are included. Validation is green: 224 citation-focused tests, the full suite at 2165 passed / 13 skipped, and all PR checks including CodeRabbit and DCO. |
|
/merge |
2 participants
Overview
release/2.2shallow-research prompt and add one targeted citation preflight reminderreport.pdf, p.15Nemotron 3.5 Lightning can intermittently return a correct answer after successful research without inline citation markers or a source section. Prompting can reduce that failure rate, but it is not a product invariant. This change keeps the current research behavior intact and makes an uncited researched answer unpublishable.
DCO sign-off for the squash commit
Signed-off-by: Ajay Thorve athorve@nvidia.com
Validation
uv run ruff check .uv run ruff format --check .uv run pytest -q tests/ sources/: 2157 passed, 13 skippedrelease/2.2is one citation-preflight line; query rewriting, tool selection, source hierarchy, and answer style are unchangedknowledge_searchsummary invokes the bounded repair and publishes only with an inline marker plus verified filename/page citation keyThe live smoke is a provider compatibility and citation-reliability check, not a replacement for the existing FreshQA quality benchmark.
Where should reviewers start?
src/aiq_agent/agents/shallow_researcher/agent.py: the final publication invariant, deterministic single-source fallback, and bounded multi-source repairtests/aiq_agent/agents/shallow_researcher/test_agent.py: fail-closed, timeout, uploaded-document, source-heading, prefix-citation, and repair-path regression coveragetests/aiq_agent/test_default_model_profiles.py: default Lightning and frontier Luna sharing the same hardened pathsrc/aiq_agent/agents/shallow_researcher/prompts/researcher.j2: the single targeted prompt additionRelated Issues
Summary by CodeRabbit