fix: preserve GPT frontier model roles - #433
Conversation
Signed-off-by: Ajay Thorve <athorve@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
Important Review skippedReview was skipped due to path filters ⛔ Files ignored due to path filters (1)
CodeRabbit blocks several paths by default. You can override this behavior by explicitly including those paths in the path filters. For example, including ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: 📒 Files selected for processing (1)
📜 Recent review details⏰ Context from checks skipped due to timeout. (3)
🧰 Additional context used📓 Path-based instructions (3)**/*📄 CodeRabbit inference engine (CONTRIBUTING.md)
Files:
**/*.{py,pyi,js,jsx,ts,tsx,yml,yaml,json,env,md}📄 CodeRabbit inference engine (AGENTS.md)
Files:
{docs/**,README.md,CONTRIBUTING.md,SECURITY.md,CODE-OF-CONDUCT.md}⚙️ CodeRabbit configuration file
Files:
🧠 Learnings (4)📓 Common learnings📚 Learning: 2026-08-04T22:37:11.331ZApplied to files:
📚 Learning: 2026-08-04T22:37:10.303ZApplied to files:
📚 Learning: 2026-07-21T19:40:54.204ZApplied to files:
🔇 Additional comments (1)
WalkthroughThe frontier model profile now uses GPT-5.6 Luna for intent classification and shallow research. Runtime tests and documentation reflect the updated Luna and Sol role assignments. ChangesFrontier model migration
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@README.md`:
- Line 298: Update the config_frontier_models.yml entry in the README
configuration table to include TAVILY_API_KEY alongside OPENAI_API_KEY and
NVIDIA_API_KEY in the listed required environment variables.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 02e48dce-bf74-4ce3-ab78-e8970d86fc18
📒 Files selected for processing (10)
.secrets.baselineREADME.mdconfigs/config_frontier_models.ymldocs/notebooks/0_Getting_Started_with_AIQ.ipynbdocs/source/customization/configuration-reference.mddocs/source/customization/swapping-models.mddocs/source/examples/hybrid-frontier-model.mddocs/source/examples/index.mdtests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.pytests/aiq_agent/test_default_model_profiles.py
📜 Review details
⏰ Context from checks skipped due to timeout. (3)
- GitHub Check: Script Validation
- GitHub Check: Pytest and Coverage
- GitHub Check: Lint and Hooks
🧰 Additional context used
📓 Path-based instructions (10)
**/*
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*: Do not include secrets, credentials, private hostnames, internal-only logs, customer data, or generated local artifacts.
Add or update tests for behavior changes.
**/*: For substantial behavior, authentication, UI, or architecture changes, open a design discussion before coding rather than landing a large unreviewed change.
Keep changes scoped to this repository and avoid editing adjacent repositories; treat eachsources/*package independently and prefer the smallest package-scoped change.
Keep pull requests scoped, avoid unrelated files and generated artifacts, provide validation evidence, and ensure every commit has DCO sign-off.
Files:
docs/source/customization/swapping-models.mddocs/source/examples/index.mddocs/source/examples/hybrid-frontier-model.mddocs/source/customization/configuration-reference.mddocs/notebooks/0_Getting_Started_with_AIQ.ipynbREADME.mdconfigs/config_frontier_models.ymltests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.pytests/aiq_agent/test_default_model_profiles.py
**/*.{py,pyi,js,jsx,ts,tsx,yml,yaml,json,env,md}
📄 CodeRabbit inference engine (AGENTS.md)
Never commit secrets, tokens, or environment-specific hostnames; use environment variables and
SecretStr, and resolve API keys at runtime.
Files:
docs/source/customization/swapping-models.mddocs/source/examples/index.mddocs/source/examples/hybrid-frontier-model.mddocs/source/customization/configuration-reference.mdREADME.mdconfigs/config_frontier_models.ymltests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.pytests/aiq_agent/test_default_model_profiles.py
docs/source/**/*
📄 CodeRabbit inference engine (AGENTS.md)
Update canonical documentation under
docs/source/when behavior, configuration, or workflows change; do not duplicate full documentation pages in skills.
Files:
docs/source/customization/swapping-models.mddocs/source/examples/index.mddocs/source/examples/hybrid-frontier-model.mddocs/source/customization/configuration-reference.md
{docs/**,README.md,CONTRIBUTING.md,SECURITY.md,CODE-OF-CONDUCT.md}
⚙️ CodeRabbit configuration file
{docs/**,README.md,CONTRIBUTING.md,SECURITY.md,CODE-OF-CONDUCT.md}: Review documentation for command accuracy, branch-name consistency, current CI and copy-pr-bot behavior, public
vs internal boundary clarity, stale examples, and links that no longer match the repository layout.
Files:
docs/source/customization/swapping-models.mddocs/source/examples/index.mddocs/source/examples/hybrid-frontier-model.mddocs/source/customization/configuration-reference.mddocs/notebooks/0_Getting_Started_with_AIQ.ipynbREADME.md
**/*.{yaml,yml}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
For deployment changes, run the relevant Helm or Compose validation and describe the environment used.
Files:
configs/config_frontier_models.yml
configs/**/*.{yml,yaml}
📄 CodeRabbit inference engine (AGENTS.md)
Use
_typenames that come from the registered configuration class.
Files:
configs/config_frontier_models.yml
{deploy/**,configs/**}
⚙️ CodeRabbit configuration file
{deploy/**,configs/**}: Review deployment and config changes for secret separation, safe defaults, local-vs-production behavior, Helm and
Docker portability, and documentation parity. Flag committed credentials, environment-specific NVIDIA internals in
public defaults, and changes that make examples diverge from CI-tested paths.
Files:
configs/config_frontier_models.yml
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.py: Runuv run ruff check .anduv run ruff format --check .for root Python changes.
Runuv run pytestfor root project Python changes.
Files:
tests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.pytests/aiq_agent/test_default_model_profiles.py
**/*.{py,pyi}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{py,pyi}: Format and lint Python code with Ruff using line length 120, Python 3.11 targeting, rules E, F, W, I, PL, and UP, with single-line imports; do not reformat unrelated code.
Missing-secret paths must degrade gracefully by stubbing or skipping rather than crashing or leaking secrets.
Files:
tests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.pytests/aiq_agent/test_default_model_profiles.py
**/*.{py,pyi,js,jsx,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Never print or log secret values, including in tool output or error messages.
Files:
tests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.pytests/aiq_agent/test_default_model_profiles.py
🧠 Learnings (14)
📓 Common learnings
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 414
File: tests/aiq_agent/test_default_model_profiles.py:72-75
Timestamp: 2026-08-04T22:37:11.331Z
Learning: For the AIQ default-model migration, PR `#414` intentionally retains Nemotron 3 Ultra expectations for intent classification and shallow research because the Nemotron Nano 3.5 Preview public endpoint and GA slug are unavailable. The Nano intent alias and corresponding profile tests are deferred to stacked draft PR `#419` after the availability gate.
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 414
File: frontends/benchmarks/freshqa/configs/config_shallow_research_only.yml:16-27
Timestamp: 2026-08-04T22:37:10.303Z
Learning: For the August 2026 model migration, PR `#414` intentionally uses only currently available non-Nano successors because Nemotron Nano 3.5 Preview is not expected on the public Build endpoint until August 11. Draft PR `#419` owns the Nano 3.5 Preview intent-classification and shallow-research model profiles, role bindings, documentation, frozen profile, and tool-first runtime guard, with an endpoint, slug, and smoke-test merge gate.
Learnt from: soumilinandi
Repo: NVIDIA-AI-Blueprints/aiq PR: 374
File: deploy/helm/README.md:366-374
Timestamp: 2026-07-21T19:40:54.204Z
Learning: In `deploy/helm/README.md`, the `aiq-credentials` update example recreates only the required keys and consequently omits any optional credentials (including SERPER, JINA, WANDB, Modal, and artifact-storage/S3 keys). This is pre-existing behavior, not introduced by PR `#374`’s optional-key documentation additions; treat improvements to preserve optional keys as separate follow-up work.
📚 Learning: 2026-08-04T22:37:11.331Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 414
File: tests/aiq_agent/test_default_model_profiles.py:72-75
Timestamp: 2026-08-04T22:37:11.331Z
Learning: For the AIQ default-model migration, PR `#414` intentionally retains Nemotron 3 Ultra expectations for intent classification and shallow research because the Nemotron Nano 3.5 Preview public endpoint and GA slug are unavailable. The Nano intent alias and corresponding profile tests are deferred to stacked draft PR `#419` after the availability gate.
Applied to files:
docs/source/customization/swapping-models.mddocs/source/examples/index.mddocs/source/examples/hybrid-frontier-model.mddocs/source/customization/configuration-reference.mddocs/notebooks/0_Getting_Started_with_AIQ.ipynbREADME.mdconfigs/config_frontier_models.ymltests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.pytests/aiq_agent/test_default_model_profiles.py
📚 Learning: 2026-08-04T22:37:10.303Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 414
File: frontends/benchmarks/freshqa/configs/config_shallow_research_only.yml:16-27
Timestamp: 2026-08-04T22:37:10.303Z
Learning: For the August 2026 model migration, PR `#414` intentionally uses only currently available non-Nano successors because Nemotron Nano 3.5 Preview is not expected on the public Build endpoint until August 11. Draft PR `#419` owns the Nano 3.5 Preview intent-classification and shallow-research model profiles, role bindings, documentation, frozen profile, and tool-first runtime guard, with an endpoint, slug, and smoke-test merge gate.
Applied to files:
docs/source/customization/swapping-models.mddocs/source/examples/index.mddocs/source/examples/hybrid-frontier-model.mddocs/source/customization/configuration-reference.mdREADME.mdconfigs/config_frontier_models.yml
📚 Learning: 2026-07-23T23:04:39.419Z
Learnt from: KyleZheng1284
Repo: NVIDIA-AI-Blueprints/aiq PR: 387
File: src/aiq_agent/agents/deep_researcher/sandbox/README.md:554-555
Timestamp: 2026-07-23T23:04:39.419Z
Learning: For `src/aiq_agent/agents/deep_researcher/sandbox/README.md`, the OpenShell live acceptance suite is intentionally opt-in because required CI execution needs a provisioned Linux/OpenShell environment and separate CI/governance work. The canonical OpenShell deployment guide requires a live hard-Landlock run before production acceptance is recorded.
Applied to files:
docs/source/customization/swapping-models.mddocs/source/examples/index.mddocs/notebooks/0_Getting_Started_with_AIQ.ipynb
📚 Learning: 2026-07-23T22:20:18.400Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: skills/aiq-research/SKILL.md:0-0
Timestamp: 2026-07-23T22:20:18.400Z
Learning: Applies to skills/aiq-research/**/SKILL.md : Use this skill for research-shaped requests, including deep research, AI-Q research, or requests to ask AI-Q; do not use it for installation, deployment, lifecycle, UI, CLI, Docker, Helm, or troubleshooting requests.
Applied to files:
docs/source/examples/index.mddocs/source/customization/configuration-reference.mddocs/notebooks/0_Getting_Started_with_AIQ.ipynbREADME.md
📚 Learning: 2026-07-23T22:20:18.400Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: skills/aiq-research/SKILL.md:0-0
Timestamp: 2026-07-23T22:20:18.400Z
Learning: Applies to skills/aiq-research/**/SKILL.md : If the backend is reachable but `/chat` or async-agent routes fail, report that it is incompatible with the public research flow and offer `aiq-deploy` validation rather than fabricating an answer.
Applied to files:
docs/source/examples/index.mddocs/notebooks/0_Getting_Started_with_AIQ.ipynb
📚 Learning: 2026-07-23T22:20:09.213Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: skills/aiq-deploy/SKILL.md:0-0
Timestamp: 2026-07-23T22:20:09.213Z
Learning: Applies to skills/aiq-deploy/**/* : Read the relevant reference documentation before acting on repository location, environment configuration, workflow configuration, deployment mode, validation, troubleshooting, or shutdown tasks.
Applied to files:
docs/source/customization/configuration-reference.md
📚 Learning: 2026-07-21T19:40:54.204Z
Learnt from: soumilinandi
Repo: NVIDIA-AI-Blueprints/aiq PR: 374
File: deploy/helm/README.md:366-374
Timestamp: 2026-07-21T19:40:54.204Z
Learning: In `deploy/helm/README.md`, the `aiq-credentials` update example recreates only the required keys and consequently omits any optional credentials (including SERPER, JINA, WANDB, Modal, and artifact-storage/S3 keys). This is pre-existing behavior, not introduced by PR `#374`’s optional-key documentation additions; treat improvements to preserve optional keys as separate follow-up work.
Applied to files:
docs/notebooks/0_Getting_Started_with_AIQ.ipynbtests/aiq_agent/test_default_model_profiles.py
📚 Learning: 2026-08-04T02:15:52.352Z
Learnt from: soumilinandi
Repo: NVIDIA-AI-Blueprints/aiq PR: 412
File: docs/notebooks/0_Getting_Started_with_AIQ.ipynb:650-650
Timestamp: 2026-08-04T02:15:52.352Z
Learning: For the OpenShell profile in `docs/notebooks/0_Getting_Started_with_AIQ.ipynb`, keep the getting-started guidance concise and refer users to `docs/source/deployment/openshell.md`. That document is the authoritative procedure for validating and starting AI-Q with `configs/config_openshell.yml`.
Applied to files:
docs/notebooks/0_Getting_Started_with_AIQ.ipynb
📚 Learning: 2026-06-11T21:21:11.314Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: docs/source/contributing/code-organization.md:0-0
Timestamp: 2026-06-11T21:21:11.314Z
Learning: Applies to docs/source/contributing/src/aiq_agent/agents/deep_researcher/** : Deep researcher agent should be organized in `src/aiq_agent/agents/deep_researcher/` with implementation details documented in README.md
Applied to files:
docs/notebooks/0_Getting_Started_with_AIQ.ipynbtests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.py
📚 Learning: 2026-06-11T21:21:11.314Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: docs/source/contributing/code-organization.md:0-0
Timestamp: 2026-06-11T21:21:11.314Z
Learning: Applies to docs/source/contributing/src/aiq_agent/agents/chat_researcher/** : Chat researcher agent should be organized in `src/aiq_agent/agents/chat_researcher/` and serve as orchestrator with orchestration node containing intent, meta, and depth information
Applied to files:
docs/notebooks/0_Getting_Started_with_AIQ.ipynb
📚 Learning: 2026-06-11T21:21:11.314Z
Learnt from: CR
Repo: NVIDIA-AI-Blueprints/aiq PR: 0
File: docs/source/contributing/code-organization.md:0-0
Timestamp: 2026-06-11T21:21:11.314Z
Learning: Applies to docs/source/contributing/src/aiq_agent/agents/shallow_researcher/** : Shallow researcher agent should be organized in `src/aiq_agent/agents/shallow_researcher/`
Applied to files:
docs/notebooks/0_Getting_Started_with_AIQ.ipynbconfigs/config_frontier_models.ymltests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.py
📚 Learning: 2026-07-06T23:55:46.952Z
Learnt from: cdgamarose-nv
Repo: NVIDIA-AI-Blueprints/aiq PR: 311
File: src/aiq_agent/agents/deep_researcher/prompts/orchestrator.j2:74-81
Timestamp: 2026-07-06T23:55:46.952Z
Learning: In `tests/aiq_agent/agents/deep_researcher/test_agent.py`, avoid asserting exact substrings from the orchestrator/writer prompt templates (e.g., `src/aiq_agent/agents/deep_researcher/prompts/orchestrator.j2`) since prompt wording changes frequently. Prefer testing structural/behavioral properties instead.
Applied to files:
docs/notebooks/0_Getting_Started_with_AIQ.ipynbtests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.pytests/aiq_agent/test_default_model_profiles.py
📚 Learning: 2026-08-11T06:34:44.687Z
Learnt from: AjayThorve
Repo: NVIDIA-AI-Blueprints/aiq PR: 429
File: src/aiq_agent/agents/deep_researcher/register.py:302-307
Timestamp: 2026-08-11T06:34:44.687Z
Learning: In Python logging code that handles potentially sensitive exceptions, do not add `exc_info=True` solely to restore stack traces, because standard traceback formatting includes `str(exception)` and may expose provider, customer, or credential-bearing content. When sensitive-content redaction is required, log the exception type together with `log_content_metadata(exception)` instead.
Applied to files:
tests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.pytests/aiq_agent/test_default_model_profiles.py
🔇 Additional comments (11)
README.md (1)
119-123: LGTM!docs/notebooks/0_Getting_Started_with_AIQ.ipynb (1)
421-421: LGTM!docs/source/customization/configuration-reference.md (1)
713-713: LGTM!docs/source/customization/swapping-models.md (1)
16-16: LGTM!docs/source/examples/hybrid-frontier-model.md (1)
8-15: LGTM!Also applies to: 34-36
docs/source/examples/index.md (1)
17-17: LGTM!.secrets.baseline (1)
136-136: LGTM!Also applies to: 358-358
configs/config_frontier_models.yml (1)
54-67: LGTM!Also applies to: 134-134, 153-153
tests/aiq_agent/agents/deep_researcher/test_deepagents_runtime.py (2)
53-64: LGTM!Also applies to: 81-82
53-64: 📐 Maintainability & Code QualityAttach the required Python validation evidence.
The supplied context does not include results for
uv run ruff check .,uv run ruff format --check ., oruv run pytest. Run these commands for the root Python changes before merge.As per coding guidelines: "
**/*.py: Runuv run ruff check .anduv run ruff format --check .for root Python changes. Runuv run pytestfor root project Python changes."Also applies to: 81-82
Source: Coding guidelines
tests/aiq_agent/test_default_model_profiles.py (1)
86-95: LGTM!Also applies to: 107-108, 119-121
Signed-off-by: Ajay Thorve <athorve@nvidia.com>
Signed-off-by: Ajay Thorve <athorve@nvidia.com>
ea1c39e to
6ab12dc
Compare
|
/merge |
2 participants
Overview
PR #430 correctly moved the ordinary default profiles to Nemotron 3.5 Lightning, but it also applied that default to
config_frontier_models.yml. The frontier profile is an explicit GPT Sol/Luna topology and should remain independently validated. Restoring its historical Super shallow assignment would reintroduce a deprecated model, so Luna is used for the bounded shallow role.DCO sign-off for the squash commit
Signed-off-by: Ajay Thorve athorve@nvidia.com
Validation
uv run pytest -q— 2142 passed, 13 skipped.nat validate --config_file configs/config_frontier_models.yml.Historical held-out FreshQA TEST validation for GPT-5.6 Luna used the November 24, 2025 dataset revision: 500/500 generation and judge coverage, zero judge errors, and 70.8% accuracy. This validates the profile wiring; it is not a claim that Luna is the strongest shallow-research model.
The runtime config and source are copied into the release image. A Git tag alone does not update an already-built RC container; a container intended to carry this correction must be rebuilt from the final corrected tag.
Where should reviewers start?
Start with
configs/config_frontier_models.yml, then review the exact role assertions intests/aiq_agent/test_default_model_profiles.pyand the matching documentation.Related Issues
Summary by CodeRabbit
New Features
Documentation
Tests