Forward-merge release/2.2 into develop - #346
Merged
Merged
Conversation
## Summary - Adds `sources/nimble_web_search`, a NAT data source that wraps `langchain-nimble`'s `NimbleSearchRetriever`, mirroring the existing `exa_web_search` and `tavily_web_search` packages (typed config, stub-on-missing-key, retries, content truncation, XML-tagged output). - Exposes `lite` / `fast` / `deep` `search_depth`, typed as a `Literal` so invalid values fail at config-parse time. `lite` is the default — metadata-only, token-cheap, works on any account. `fast` is enterprise-tier and surfaces a clear 403 entitlement message on non-enterprise keys. - Adds a typed `focus` mode (default `general`, validated `Literal`), `country` / `locale` regional controls, and an optional `max_content_length` per-result cap. - Wires the plugin into the workspace, `deploy/Dockerfile`, and `scripts/setup.sh` so it installs in dev, Docker, and container builds. The Docker layer also installs Nimble's runtime deps (`langchain-nimble`, `nimble-python`, lockfile-pinned) so `_type: nimble_web_search` resolves in built images, where the `--no-dev` sync would otherwise omit them. - Documentation across the config reference, extending guides, installation, quick-start, deployment (docker-build, docker-compose, kubernetes), FAQ, and troubleshooting (with 401 and 403-enterprise rows). #### DCO sign-off for the squash commit Signed-off-by: Kobi Kadosh <kobi.kadosh@gmail.com> ## Motivation AI-Q ships Tavily- and Exa-backed web search today. [Nimble](https://nimbleway.com/) provides web search and content extraction for AI agents; this adds it as a first-class alternative with the same ergonomics and config surface — handy for users who already have a Nimble subscription, prefer its regional coverage, or want to test across multiple search backends. The default provider is unchanged (Tavily stays the documented default). It wraps the official `langchain-nimble` package (maintained by Nimble) rather than calling the HTTP API directly, so retry, auth, and response normalization come from the upstream integration — the same rationale as the Exa source (#181). ## Configuration ```yaml functions: web_search_tool: _type: nimble_web_search max_results: 5 search_depth: lite # lite (default) | fast (enterprise) | deep focus: general # general (default) | news | location | shopping | geo | social country: US locale: en ``` ```bash NIMBLE_API_KEY=... # or set api_key: in the YAML ``` ## How it works A real `lite` query with `NIMBLE_API_KEY` set, trimmed: ```text <Document href="https://docs.nvidia.com/aiq-blueprint/1.2.1/index.html"> <title> NVIDIA AI-Q Blueprint </title> AI-Q combines intelligent query routing, multi-agent research pipelines, and pluggable knowledge retrieval to deliver comprehensive, citation-backed answers. </Document> --- <Document href="https://build.nvidia.com/nvidia/aiq"> <title> NVIDIA AI-Q Blueprint for intelligent agents </title> The NVIDIA AI-Q Blueprint enables developers to build fully customizable AI agents that they own, inspect and control. Built on LangChain… </Document> ``` Each result renders as an XML `<Document>` block — the same shape the Tavily and Exa sources produce — so existing AI-Q agents consume it with no changes. To try it, point any existing web-search config at `_type: nimble_web_search` and run `nat run` (swap `advanced_search: true` → `search_depth: deep`). ## How this was tested - [x] `uv run pytest sources/nimble_web_search` — **32 passed**, credential-free (the SDK is mocked; no live network in CI). - [x] `uv run pytest sources/exa_web_search sources/nimble_web_search` — **46 passed**, confirming the new package co-runs cleanly with a sibling source. The test module has a unique name and no `tests/__init__.py`, so there's no pytest collection collision when sources are collected together. - [x] `ruff check` and `ruff format --check` — clean (whole repo). `uv lock --check` — no drift. - [x] Repo pre-commit hooks pass on the changed files: `detect-secrets`, `markdown-link-check` (all README/docs links resolve), `end-of-file-fixer`, `trailing-whitespace`, `check-added-large-files`, and `uv-lock` — matching the `AIQ CI` lint job. - [x] `nat info components --types function` lists `nimble_web_search` (1.0.0) next to `exa_web_search` and `tavily_web_search`, so `_type: nimble_web_search` resolves in a workflow. - [x] Container runtime: `langchain-nimble==3.0.0` + `nimble-python==0.18.0` install and import cleanly in a fresh environment the same way `deploy/Dockerfile` installs them, so `_type: nimble_web_search` resolves in built images — not only in editable dev installs. - [x] Live smoke with a real `NIMBLE_API_KEY` across `lite` and `deep`, plus the non-enterprise `fast` path (returns the friendly 403 entitlement message). Output is redacted; no key is logged by construction. Coverage: config defaults / all fields / invalid-enum rejection (incl. `focus`) / out-of-range numeric fields rejected / `focus` defaults to `general` and reaches the SDK / non-default `focus` passthrough / `include_answer` absent from config and kwargs / `FunctionBaseConfig` inheritance, the missing-key stub + warn-once, key-from-config env hydration, result rendering + description fallback, markup escaping of untrusted fields, `search_depth` and `country`/`locale` passthrough, query and content truncation (incl. small-limit hard-cut), empty-result handling, retry-then-succeed, non-transient (401/403) errors short-circuiting without retry, final-retry failure, and the 401 / 403 branches. ## How this was reviewed - Diffed against the merged Exa source to keep structure, retry loop, truncation, and output format at parity; the deviations below are deliberate. - Confirmed credential-free CI behavior and co-run safety with a sibling source. - Scanned the package for secrets and for hardcoded search-endpoint names — none. ## Deviations from the Exa source (all deliberate) 1. `search_depth` (3-value enum), a typed `focus` mode (default `general`), plus `country` / `locale`, mirroring `langchain-nimble`'s surface, where Exa exposes `search_type` / `full_text` / `highlights`. `focus` is a workflow-config setting, not an agent parameter, so general research queries cannot drift to `news`. 2. Falls back to the result's `description` when `page_content` is empty — Nimble's `lite` mode returns metadata only. 3. A 403 branch that turns Nimble's enterprise-tier gating into a clear, actionable message. Exa has no tier gating, so no equivalent. 4. `include_answer` (answer generation) is intentionally not exposed in this initial integration. It can be added in a follow-up. 5. Untrusted result fields (`url`, `title`, body) are HTML-escaped before rendering into the `<Document>` markup, so a result can't break the block or inject into downstream parsers. 6. Numeric config fields are bounded: `max_results` `1-100` (matching `langchain-nimble`'s own `ge=1, le=100`), `max_retries` `ge=1`, `max_content_length` `ge=1` (use `None` to disable truncation). Invalid values fail at config-parse time, and content truncation hard-cuts safely for very small limits. ## Known limitations - `max_results` is a soft cap — Nimble may return up to N+2 documents for N. The provider returns them all; downstream consumers can slice. - `lite` mode returns empty `page_content`; the provider renders the `description` (~150 chars, organic-result quality). - The non-enterprise `fast` path is characterized via its 403 message; the enterprise `fast` behavior itself isn't exercised here. ## Scope **In:** the `nimble_web_search` provider, config/docs/deploy wiring, 32 unit tests, README, troubleshooting rows. **Not in (easy follow-ups):** Nimble Extract / Map / Crawl / Agents; `include_answer`; framework integrations beyond AI-Q's data-source mechanism; any change to the default provider. ## Security - No secrets committed — `deploy/.env.example` carries a commented `NIMBLE_API_KEY=` placeholder only. - Key read from env or a `SecretStr` config field; never logged. - Unit tests need no credentials; the live smoke uses an inline env var and redacted output. ## Summary by CodeRabbit * **New Features** * Added Nimble Web Search as a supported web search option with configurable depth, focus, country/locale targeting, result limits, retries, and optional content truncation. * Provides structured `<Document>` formatted output with safer escaping and improved result URL handling. * **Documentation** * Added Nimble Web Search tool documentation and examples. * Updated installation, quick start, deployment key tables, FAQ, and troubleshooting for `NIMBLE_API_KEY`. * **Deployment/Setup** * Updated Docker/build, compose, Kubernetes, setup scripts, and the `.env.example` template for Nimble support. * **Tests** * Added credential-free recorded replay and opt-in live integration coverage. * **Chores** * Refreshed the secrets baseline metadata. Signed-off-by: Kobi Kadosh <kobi.kadosh@gmail.com> Authors: - Kobi Kadosh (https://github.com/wildcard) - Kyle Zheng (https://github.com/KyleZheng1284) - Ajay Thorve (https://github.com/AjayThorve) Approvers: - Kyle Zheng (https://github.com/KyleZheng1284) - Ajay Thorve (https://github.com/AjayThorve) URL: #261
Author
|
FAILURE - Unable to forward-merge due to an error, manual merge is necessary. Do not use the IMPORTANT: When merging this PR, do not use the auto-merger (i.e. the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Forward-merge triggered by push to release/2.2 that creates a PR to keep develop up-to-date. If this PR is unable to be immediately merged due to conflicts, it will remain open for the team to manually merge. See forward-merger docs for more info.