fix: make shallow citation enforcement configurable - #456
Conversation
Signed-off-by: Shubhadeep Das <shubhadeepd@nvidia.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: 📒 Files selected for processing (7)
Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review. 📜 Recent review details⏰ Context from checks skipped due to timeout. (3)
🧰 Additional context used📓 Path-based instructions (10)**/*📄 CodeRabbit inference engine (CONTRIBUTING.md)
Files:
**/*.py📄 CodeRabbit inference engine (CONTRIBUTING.md)
Files:
**/*.{py,pyi}📄 CodeRabbit inference engine (AGENTS.md)
Files:
**/*.{py,pyi,js,jsx,ts,tsx,yml,yaml,json,env,md}📄 CodeRabbit inference engine (AGENTS.md)
Files:
**/*.{py,pyi,js,jsx,ts,tsx}📄 CodeRabbit inference engine (AGENTS.md)
Files:
src/**/*.py📄 CodeRabbit inference engine (AGENTS.md)
Files:
src/aiq_agent/agents/**/*⚙️ CodeRabbit configuration file
Files:
{src/aiq_agent/fastapi_extensions/**,frontends/aiq_api/src/aiq_api/**}⚙️ CodeRabbit configuration file
Files:
docs/source/**/*📄 CodeRabbit inference engine (AGENTS.md)
Files:
{docs/**,README.md,CONTRIBUTING.md,SECURITY.md,CODE-OF-CONDUCT.md}⚙️ CodeRabbit configuration file
Files:
🔇 Additional comments (8)
WalkthroughThe shallow researcher now supports optional citation enforcement. The default mode returns sanitized answers when citation integrity fails. Strict mode preserves citation repair and typed errors. Configuration, construction wiring, documentation, and tests were updated. ChangesCitation enforcement
Estimated code review effort: 3 (Moderate) | ~20 minutes Merge Risk: 🔵 Low · up to The change makes citation enforcement configurable and defaults shallow research to returning a sanitized answer when citation integrity fails. The PR is mergeable with owner awareness, but broader repository validation results should be supplied as a bounded follow-up. Sequence Diagram(s)sequenceDiagram
participant ShallowResearchAgentConfig
participant _create_agent_instance
participant ShallowResearcherAgent
participant Callbacks
ShallowResearchAgentConfig->>_create_agent_instance: provide enforce_citations
_create_agent_instance->>ShallowResearcherAgent: construct with enforce_citations
ShallowResearcherAgent->>ShallowResearcherAgent: validate sources and citation integrity
alt enforcement disabled
ShallowResearcherAgent->>Callbacks: publish uncited report
ShallowResearcherAgent-->>_create_agent_instance: return sanitized answer
else enforcement enabled
ShallowResearcherAgent-->>_create_agent_instance: raise citation error after failed repair or integrity check
end
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
The flag to skip enforcing works, thank you!
But there's something weird going on with citation checking. I get the sources back but the logs still show that it failed the integrity check since [N] wasn't added to the content in the answer. We should definitely see if we can do anything to fix the log failures in a different PR. Can you please add it to the github issue to track it?
2026-08-20 11:48:52 - INFO - aiq_agent.common.callbacks:188 - ------------------------------
2026-08-20 11:48:52 - INFO - aiq_agent.common.callbacks:158 - [Chain End] agent
2026-08-20 11:48:52 - INFO - aiq_agent.common.citation_verification:1108 - [CitationVerify] Starting verification against 5 registered source(s)
2026-08-20 11:48:52 - INFO - aiq_agent.agents.shallow_researcher.agent:656 - Shallow report is missing citation integrity; returning generated answer because enforce_citations is false (registered_sources=5)
2026-08-20 11:48:52 - INFO - aiq_agent.common.citation_verification:1108 - [CitationVerify] Starting verification against 5 registered source(s)
2026-08-20 11:48:52 - WARNING - aiq_agent.agents.shallow_researcher.agent:669 - Shallow report failed final citation integrity check (registered_sources=5 verified_sources=2)
Yes created github issue #458 |
…pty-source-registry Brings in upstream 2.2.1 hardening (NVIDIA-AI-Blueprints#454, NVIDIA-AI-Blueprints#455) and configurable shallow citation enforcement (NVIDIA-AI-Blueprints#456). No textual conflicts: upstream's change to _extract_title_for_url (HTML-escaped URL matching) is disjoint from this branch's citation-verification disposition helpers in the same file. Signed-off-by: Ranit Karmakar <karmakarranit6@gmail.com>
#### Overview Manually backports the remaining v2.2.1 follow-ups to `release/2.2` in one release-scoped change: - PR #456: makes shallow citation enforcement configurable, defaulting to returning the sanitized generated answer while preserving strict fail-closed behavior when `enforce_citations: true`. - PR #461: raises the Nemotron 3.5 Lightning shallow-agent output cap to 32,768 tokens across every matching config shipped on `release/2.2`, with regression coverage. - Thanh Phan commit `f64e80d`: corrects stale references across 10 deployment, integration, profiling, architecture, and contributor guides, with a focused review follow-up clarifying the report-routing contract. - Refreshes the unchanged MCP production lock from pip 26.1.2 to 26.2.1 after the newly published `PYSEC-2026-3721` advisory began failing the required vulnerability gate. No develop-only feature surface is introduced. #### Release-branch adaptations - Retains the existing `release/2.2` final-report formatting. The `_format_chat_references` helper used by #456 already existed on `develop` via unrelated PR #448 and is not part of this release line. - Preserves deletion of `configs/nemo_relay/config_web_default_with_pricing.yml`, which is not shipped on `release/2.2`. - Folds #461 output-cap coverage into the broader existing release model-profile invariant instead of retaining a redundant develop-side test. Shipped/default profiles require 32,768 tokens; benchmark profiles intentionally remain at 8,192. All three backport commits preserve the original authors, DCO sign-offs, and `cherry-pick -x` provenance. The CI-unblock and review-follow-up changes are separate DCO-signed commits. #### DCO sign-off for the squash commit Signed-off-by: Ajay Thorve <athorve@nvidia.com> Signed-off-by: Ajay Thorve <AjayThorve@users.noreply.github.com> #### Validation - `uv run --frozen pytest -q tests/aiq_agent/test_default_model_profiles.py tests/aiq_agent/agents/shallow_researcher/test_agent.py tests/aiq_agent/jobs/test_runner.py` — 234 passed. - `uv run pre-commit run --all-files` — passed, including Ruff, lock validation, secrets scan, agent-skill validation, YAML checks, and Markdown link checks. - Exact MCP production audit command from CI — zero unignored vulnerabilities after the pip lock refresh. - `uv run pre-commit run --files docs/source/architecture/overview.md` — passed. - `git diff --check origin/release/2.2...HEAD` — passed. - [x] I ran the relevant local checks or explained why they are not applicable. - [x] I added or updated tests for behavior changes. - [x] I updated documentation for user-facing or contributor-facing changes. - [x] I confirmed this PR does not include secrets, credentials, or internal-only data. - [x] I certify this contribution under the Developer Certificate of Origin (DCO) and signed my commits with `git commit -s` or an equivalent sign-off. - [x] I replaced the DCO sign-off placeholder with my GitHub commit identity and kept the required angle brackets around the email address. #### Where should reviewers start? Start with `src/aiq_agent/agents/shallow_researcher/agent.py`, `tests/aiq_agent/test_default_model_profiles.py`, `docs/source/architecture/overview.md`, and the release-branch adaptations above. #### Related Issues - Backports #456 - Backports #461 - Includes thanhpt1110@f64e80d ## Summary by CodeRabbit - **New Features** - Increased the maximum response length for standard Lightning model configurations to 32,768 tokens. - Added optional citation enforcement for shallow research, supporting strict errors or sanitized fallback responses. - Expanded report-oriented workflow routing guidance. - **Documentation** - Updated configuration, deployment, testing, troubleshooting, profiling, and evaluation guidance. - Added Claude Code compatibility guidance for agent skills. - Clarified workflow defaults, citation behavior, expected installation checks, and benchmark-specific response limits. Authors: - Ajay Thorve (https://github.com/AjayThorve) - Shubhadeep Das (https://github.com/shubhadeepd) - Thanh Phan (https://github.com/thanhpt1110) Approvers: - Tanner Leach (https://github.com/tanleach) URL: #462
Brings the shallow researcher up to date with develop, which is what this merge was for: NVIDIA-AI-Blueprints#456 makes citation enforcement configurable (`enforce_citations`, default false) and adds `_format_chat_references`, and NVIDIA-AI-Blueprints#452 routes the agent through NeMo Relay. The relay package does not exist on this branch, so shallow_researcher could not be synced on its own. Conflict resolutions: * deep_researcher/tools/research.py - both sides rewrote the researcher worker. Kept the union: this branch's per-worker guard state and invocation-id logging, wrapped in develop's `agent_scope` lifecycle, with develop's budget-exhaustion handling and its redacted log messages (no raw query text or exception strings interpolated into the message). * deep_researcher/custom_middleware.py - purely additive on both sides; StructuredOutputRetryGuardMiddleware and ResearcherFinalizationMiddleware both kept. * aiq_api/jobs/runner.py - relay bootstrap from develop, this branch's widened agent-type check (deep/adaptive/autonomous) rather than develop's DeepResearchAgentConfig isinstance guard, which the other two configs would not satisfy. * knowledge_layer/src/register.py - develop's try/finally ingestor release, keeping this branch's extra `query` parameter note in the description. Follow-on fixes the merge required: * Restore `is_verbose` in aiq_agent.common. develop deleted it when its own agents moved to relay; adaptive and autonomous still thread `verbose` through to their agents, and it defaults to True, so dropping it would have silently changed eval log output. * Pass `max_researcher_model_calls` (new required field) to DeepResearchGraphContext and build_researcher_runnable from both the adaptive and autonomous factories, using the deep researcher's default. * runner.py `_create_agent_instance` no longer takes `verbose`; the two branch-added call sites read it off fn_config instead. Validation: uv run pytest -> 3234 passed, 110 skipped, 9 failed. All 9 are tests/deploy/test_helm_deployment_k8s.py failing on a missing `helm` binary (6 failed the same way before this merge; develop adds 3 more). Agent suites: 1234 passed, 97 skipped, 0 failed. ruff check and format clean. Signed-off-by: smasurekar <smasurekar@nvidia.com>
2 participants
Overview
Adds
enforce_citationsto the shallow research agent configuration.By default,
enforce_citationsisfalse, so AI-Q returns the sanitized generated answer when shallow citation integrity fails. When set totrue, the existing strict behavior is preserved: citation integrity failures trigger the bounded repair flow and still raise if the contract cannot be restored.This change updates:
DCO sign-off for the squash commit
Signed-off-by: Shubhadeep Das shubhadeepd@nvidia.com
Validation
UV_CACHE_DIR=/tmp/aiq-uv-cache uv run pytest tests/aiq_agent/agents/shallow_researcher/test_agent.py tests/aiq_agent/jobs/test_runner.py::TestAsyncJobRunnerAgentFactory::test_create_agent_instance_passes_shallow_research_config -qUV_CACHE_DIR=/tmp/aiq-uv-cache uv run ruff check src/aiq_agent/agents/shallow_researcher/agent.py src/aiq_agent/agents/shallow_researcher/register.py frontends/aiq_api/src/aiq_api/jobs/runner.py tests/aiq_agent/agents/shallow_researcher/test_agent.py tests/aiq_agent/jobs/test_runner.pyUV_CACHE_DIR=/tmp/aiq-uv-cache uv run ruff format --check src/aiq_agent/agents/shallow_researcher/agent.py src/aiq_agent/agents/shallow_researcher/register.py frontends/aiq_api/src/aiq_api/jobs/runner.py tests/aiq_agent/agents/shallow_researcher/test_agent.py tests/aiq_agent/jobs/test_runner.pygit diff --checkI ran the relevant local checks or explained why they are not applicable.
I added or updated tests for behavior changes.
I updated documentation for user-facing or contributor-facing changes.
I confirmed this PR does not include secrets, credentials, or internal-only data.
I certify this contribution under the Developer Certificate of Origin (DCO) and signed my commits with
git commit -sor an equivalent sign-off.I replaced the DCO sign-off placeholder with my GitHub commit identity and kept the required angle brackets around the email address.
Where should reviewers start?
Start with
src/aiq_agent/agents/shallow_researcher/agent.py, especially the strict-vs-default citation handling inShallowResearcherAgent.run().Then review:
src/aiq_agent/agents/shallow_researcher/register.pyfor config plumbingfrontends/aiq_api/src/aiq_api/jobs/runner.pyfor async job constructiontests/aiq_agent/agents/shallow_researcher/test_agent.pyfor strict and default behavior coverageRelated Issues
NA
Summary by CodeRabbit
New Features
enforce_citationsoption for shallow research.Documentation
Tests