From 2b93fe7b5efc3601d5a354986d338253f69cc7ca Mon Sep 17 00:00:00 2001 From: Kobi Kadosh Date: Sun, 31 May 2026 17:24:34 -0700 Subject: [PATCH 01/14] feat: add nimble_web_search data source Adds a Nimble web search integration mirroring exa_web_search and tavily_web_search. The new sources/nimble_web_search package wraps langchain-nimble's NimbleSearchRetriever, supports NIMBLE_API_KEY via env or config, and exposes lite/fast/deep search depths (fast is an enterprise-tier feature that surfaces a clear error on non-enterprise keys; lite is the default). It is wired into the workspace, deploy/Dockerfile (with --no-deps to preserve the frozen lockfile), and scripts/setup.sh. Includes unit tests and documentation updates across the configuration reference, extending guides, installation, deployment, faq, and troubleshooting. Co-Authored-By: Claude Sonnet 4.6 Signed-off-by: Kobi Kadosh --- .secrets.baseline | 4 +- deploy/.env.example | 4 +- deploy/Dockerfile | 2 + .../customization/configuration-reference.md | 33 ++ docs/source/deployment/docker-build.md | 1 + docs/source/deployment/docker-compose.md | 1 + docs/source/deployment/kubernetes.md | 1 + docs/source/extending/adding-a-data-source.md | 1 + docs/source/extending/adding-a-tool.md | 1 + docs/source/get-started/installation.md | 6 +- docs/source/get-started/quick-start.md | 2 + docs/source/resources/faq.md | 1 + docs/source/resources/troubleshooting.md | 2 + pyproject.toml | 2 + scripts/setup.sh | 1 + sources/nimble_web_search/README.md | 118 +++++ sources/nimble_web_search/pyproject.toml | 37 ++ sources/nimble_web_search/src/__init__.py | 22 + sources/nimble_web_search/src/register.py | 212 +++++++++ .../tests/test_nimble_register.py | 438 ++++++++++++++++++ uv.lock | 48 ++ 21 files changed, 932 insertions(+), 5 deletions(-) create mode 100644 sources/nimble_web_search/README.md create mode 100644 sources/nimble_web_search/pyproject.toml create mode 100644 sources/nimble_web_search/src/__init__.py create mode 100644 sources/nimble_web_search/src/register.py create mode 100644 sources/nimble_web_search/tests/test_nimble_register.py diff --git a/.secrets.baseline b/.secrets.baseline index 0e6f9882d..addd26020 100644 --- a/.secrets.baseline +++ b/.secrets.baseline @@ -142,7 +142,7 @@ "filename": "deploy/.env.example", "hashed_secret": "9d4e1e23bd5b727046a9e3b4b7db57bd8d6ee684", "is_verified": false, - "line_number": 30 + "line_number": 32 } ], "deploy/compose/README.md": [ @@ -290,5 +290,5 @@ } ] }, - "generated_at": "2026-05-22T20:01:44Z" + "generated_at": "2026-06-01T05:45:53Z" } diff --git a/deploy/.env.example b/deploy/.env.example index 03b5d7130..53e12aff2 100644 --- a/deploy/.env.example +++ b/deploy/.env.example @@ -17,8 +17,10 @@ AIQ_DEV_ENV=cli NVIDIA_API_KEY= -# Web search (Required) +# Web search (Required — set at least one of TAVILY_API_KEY, EXA_API_KEY, or NIMBLE_API_KEY) TAVILY_API_KEY= +# EXA_API_KEY= +# NIMBLE_API_KEY= # Paper search (Optional) # SERPER_API_KEY= # to enable, set API key and update the relevant config in configs/ directory diff --git a/deploy/Dockerfile b/deploy/Dockerfile index 0c5eb3916..b0bb2bca1 100644 --- a/deploy/Dockerfile +++ b/deploy/Dockerfile @@ -77,6 +77,8 @@ RUN uv pip install --no-deps -e . \ && uv pip install --no-deps -e ./sources/google_scholar_paper_search \ && uv pip install --no-deps -e ./sources/tavily_web_search \ && uv pip install --no-deps -e ./sources/exa_web_search \ + && uv pip install --no-deps -e ./sources/nimble_web_search \ + && uv pip install langchain-nimble==3.0.0 nimble-python==0.18.0 \ && uv pip install --no-deps -e "./sources/knowledge_layer[all]" \ && uv pip install --no-deps -e ./frontends/aiq_api \ && uv pip install "psycopg[binary]>=3.0.0" diff --git a/docs/source/customization/configuration-reference.md b/docs/source/customization/configuration-reference.md index a47a21f0d..be825c6be 100644 --- a/docs/source/customization/configuration-reference.md +++ b/docs/source/customization/configuration-reference.md @@ -197,6 +197,39 @@ functions: - **`fast`** -- Optimized for low latency. Returns results quickly at the cost of recall and semantic depth. Use for interactive UIs, high-volume calls, or when the query is narrow and keyword-like. - **`deep`** -- Optimized for thoroughness. Runs a more expensive semantic search with broader retrieval. Use for research-quality queries where completeness matters more than speed. +### `nimble_web_search` + +Web search powered by the [Nimble API](https://nimbleway.com/) via `langchain-nimble`. + +```yaml +functions: + web_search_tool: + _type: nimble_web_search + max_results: 5 + max_content_length: 10000 + + deep_web_search_tool: + _type: nimble_web_search + max_results: 5 + search_depth: deep +``` + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `max_results` | `int` | `5` | Maximum number of search results to return. | +| `api_key` | `str` | `None` | Nimble API key. Falls back to `NIMBLE_API_KEY` environment variable. | +| `max_retries` | `int` | `3` | Number of retry attempts on search failure. | +| `search_depth` | `str` | `"lite"` | Nimble search depth. See options below. | +| `country` | `str` | `"US"` | ISO country code passed to Nimble (e.g. `US`, `UK`, `FR`). | +| `locale` | `str` | `"en"` | Language/locale passed to Nimble (e.g. `en`, `fr`, `es`). | +| `max_content_length` | `int` | `10000` | Max characters per result's page content. Set to `None` to disable truncation. | + +**`search_depth` options:** + +- **`lite`** (default) -- Returns metadata only (title, URL, description). Fastest, lowest token cost, safe default for general lookups. +- **`fast`** -- Returns rich content at low latency. **Enterprise-tier only**; non-enterprise accounts receive a 403 with a clear entitlement message. +- **`deep`** -- Returns full page content for each result. Use for research workflows that need the body text, not just URLs. + ### `paper_search` Academic paper search through Google Scholar (using the [Serper API](https://serper.dev/)). diff --git a/docs/source/deployment/docker-build.md b/docs/source/deployment/docker-build.md index 52b5712ab..a2a7e9e4c 100644 --- a/docs/source/deployment/docker-build.md +++ b/docs/source/deployment/docker-build.md @@ -48,6 +48,7 @@ The builder stage handles all compilation and package installation: - `sources/google_scholar_paper_search` -- Google Scholar search - `sources/tavily_web_search` -- Tavily web search - `sources/exa_web_search` -- Exa web search + - `sources/nimble_web_search` -- Nimble web search - `sources/knowledge_layer[all]` -- Knowledge layer with all extras - `frontends/aiq_api` -- [FastAPI](https://fastapi.tiangolo.com/) frontend - `psycopg[binary]>=3.0.0` -- PostgreSQL driver (psycopg v3, installed non-editable) diff --git a/docs/source/deployment/docker-compose.md b/docs/source/deployment/docker-compose.md index 535f61ce7..558a45ed4 100644 --- a/docs/source/deployment/docker-compose.md +++ b/docs/source/deployment/docker-compose.md @@ -44,6 +44,7 @@ The sections below explain each group of variables. | `NVIDIA_API_KEY` | Yes | NVIDIA API key for NIM model access. | | `TAVILY_API_KEY` | Conditional | Web search provider key (required if using `tavily_web_search`). | | `EXA_API_KEY` | Conditional | Web search provider key (required if using `exa_web_search`). | +| `NIMBLE_API_KEY` | Conditional | Web search provider key (required if using `nimble_web_search`). | | `SERPER_API_KEY` | No | Google Scholar paper search key (optional). | ### API keys (optional) diff --git a/docs/source/deployment/kubernetes.md b/docs/source/deployment/kubernetes.md index 008ddeef4..115b09509 100644 --- a/docs/source/deployment/kubernetes.md +++ b/docs/source/deployment/kubernetes.md @@ -263,6 +263,7 @@ For complete examples with NGC-specific flags, see `deploy/helm/README.md` in th | Key | Description | |-----|-------------| | `EXA_API_KEY` | Exa API key for web search | +| `NIMBLE_API_KEY` | Nimble API key for web search | | `SERPER_API_KEY` | Serper API key for Google search | | `JINA_API_KEY` | Jina API key | | `WANDB_API_KEY` | Weights & Biases API key | diff --git a/docs/source/extending/adding-a-data-source.md b/docs/source/extending/adding-a-data-source.md index b829bd045..74e76218d 100644 --- a/docs/source/extending/adding-a-data-source.md +++ b/docs/source/extending/adding-a-data-source.md @@ -460,6 +460,7 @@ async def search(self, query: str) -> str: |---|---|---|---| | Tavily Web Search | `tavily_web_search` | `sources/tavily_web_search` | General web search through Tavily API | | Exa Web Search | `exa_web_search` | `sources/exa_web_search` | General web search through Exa API | +| Nimble Web Search | `nimble_web_search` | `sources/nimble_web_search` | General web search through Nimble API (`langchain-nimble`) | | Google Scholar | `paper_search` | `sources/google_scholar_paper_search` | Academic papers through Serper/Google Scholar | | Knowledge Layer | `knowledge_retrieval` | `sources/knowledge_layer` | Document retrieval through pluggable backends | diff --git a/docs/source/extending/adding-a-tool.md b/docs/source/extending/adding-a-tool.md index af00a06d7..08d05798b 100644 --- a/docs/source/extending/adding-a-tool.md +++ b/docs/source/extending/adding-a-tool.md @@ -420,6 +420,7 @@ f'\n\n{title}\n\n{content}\n' |---|---|---|---| | Tavily Web Search | `tavily_web_search` | `sources/tavily_web_search` | `TAVILY_API_KEY` | | Exa Web Search | `exa_web_search` | `sources/exa_web_search` | `EXA_API_KEY` | +| Nimble Web Search | `nimble_web_search` | `sources/nimble_web_search` | `NIMBLE_API_KEY` | | Google Scholar | `paper_search` | `sources/google_scholar_paper_search` | `SERPER_API_KEY` | | Knowledge Layer | `knowledge_retrieval` | `sources/knowledge_layer` | (varies by backend) | diff --git a/docs/source/get-started/installation.md b/docs/source/get-started/installation.md index d5fee400f..2d9e77e57 100644 --- a/docs/source/get-started/installation.md +++ b/docs/source/get-started/installation.md @@ -51,7 +51,7 @@ The script performs the following steps: 3. Installs the core package with dev dependencies 4. Installs all frontends (CLI, debug console, API server) 5. Installs benchmark packages (freshqa, deepsearch_qa) -6. Installs all data source plugins (Tavily, Exa, Google Scholar, knowledge layer) +6. Installs all data source plugins (Tavily, Exa, Nimble, Google Scholar, knowledge layer) 7. Sets up pre-commit hooks 8. Copies `deploy/.env.example` to `deploy/.env` if no `.env` file exists 9. Installs UI npm dependencies (if Node.js is available) @@ -96,6 +96,7 @@ uv pip install -e ./frontends/aiq_api # Unified API server (includes debug) # Data sources (pick what you need) uv pip install -e ./sources/tavily_web_search uv pip install -e ./sources/exa_web_search +uv pip install -e ./sources/nimble_web_search uv pip install -e ./sources/google_scholar_paper_search uv pip install -e "./sources/knowledge_layer[llamaindex,foundational_rag]" @@ -132,9 +133,10 @@ Then edit `deploy/.env` and fill in your keys. |----------|----------|---------| | `TAVILY_API_KEY` | [Tavily](https://tavily.com/) | Web search (Tavily provider) | | `EXA_API_KEY` | [Exa](https://exa.ai/) | Web search (Exa provider) | +| `NIMBLE_API_KEY` | [Nimble](https://nimbleway.com/) | Web search (Nimble provider) | | `SERPER_API_KEY` | [Serper](https://serper.dev/) | Academic paper search (Google Scholar). To enable, uncomment `paper_search_tool` in your config file | -At minimum, you need `NVIDIA_API_KEY` for LLM inference and one of `TAVILY_API_KEY` or `EXA_API_KEY` for web search. Paper search (`SERPER_API_KEY`) is disabled by default in the shipped configs -- refer to the comments in your config file to enable it. +At minimum, you need `NVIDIA_API_KEY` for LLM inference and one of `TAVILY_API_KEY`, `EXA_API_KEY`, or `NIMBLE_API_KEY` for web search. Paper search (`SERPER_API_KEY`) is disabled by default in the shipped configs -- refer to the comments in your config file to enable it. ## Verify Installation diff --git a/docs/source/get-started/quick-start.md b/docs/source/get-started/quick-start.md index fbb59bac6..4917a13df 100644 --- a/docs/source/get-started/quick-start.md +++ b/docs/source/get-started/quick-start.md @@ -22,6 +22,8 @@ NVIDIA_API_KEY=nvapi-... TAVILY_API_KEY=tvly-... # Or, to use Exa instead of Tavily for web search: # EXA_API_KEY=... +# Or, to use Nimble instead of Tavily for web search: +# NIMBLE_API_KEY=... ``` ## Step 2: Choose a Mode diff --git a/docs/source/resources/faq.md b/docs/source/resources/faq.md index 20916c418..a90d57a27 100644 --- a/docs/source/resources/faq.md +++ b/docs/source/resources/faq.md @@ -44,6 +44,7 @@ If `enable_escalation: true` in the workflow config, the orchestrator evaluates - **Tavily Web Search** — General web search (requires `TAVILY_API_KEY`) - **Exa Web Search** — General web search via Exa (requires `EXA_API_KEY`) +- **Nimble Web Search** — General web search via Nimble (requires `NIMBLE_API_KEY`) - **Google Scholar Paper Search** — Academic paper search (requires `SERPER_API_KEY`) - **Knowledge Layer** — Document retrieval from local or hosted vector stores diff --git a/docs/source/resources/troubleshooting.md b/docs/source/resources/troubleshooting.md index c4dbedbda..73ac2c7d2 100644 --- a/docs/source/resources/troubleshooting.md +++ b/docs/source/resources/troubleshooting.md @@ -25,6 +25,8 @@ Common issues and solutions for the AI-Q blueprint. | `Gateway timeout (504)` | Model endpoint overloaded or unavailable | Retry, or switch to a different model in config | | Tavily search returns empty | Invalid `TAVILY_API_KEY` | Verify key at [tavily.com](https://tavily.com) | | Exa search returns empty or 401 | Invalid or missing `EXA_API_KEY` | Verify key at [exa.ai](https://exa.ai) | +| Nimble search returns empty or 401 | Invalid or missing `NIMBLE_API_KEY` | Verify key at [nimbleway.com](https://nimbleway.com) | +| Nimble search returns 403 with "enterprise" | `search_depth: fast` requires an Enterprise plan | Switch to `search_depth: lite` (default) or `deep`, or upgrade your Nimble plan | | Serper search fails | Missing `SERPER_API_KEY` | Set key or remove `paper_search_tool` from config | ## Runtime Issues diff --git a/pyproject.toml b/pyproject.toml index e08479b3d..f2873d6da 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -190,6 +190,7 @@ dev = [ "google-scholar-paper-search", "tavily-web-search", "exa-web-search", + "nimble-web-search", "knowledge-layer[all]", "aiq-api", "aiq-research-cli", @@ -231,6 +232,7 @@ aiq-agent = { workspace = true } google-scholar-paper-search = { workspace = true } tavily-web-search = { workspace = true } exa-web-search = { workspace = true } +nimble-web-search = { workspace = true } knowledge-layer = { workspace = true } aiq-api = { workspace = true } aiq-research-cli = { workspace = true } diff --git a/scripts/setup.sh b/scripts/setup.sh index b0992871c..f053af15d 100755 --- a/scripts/setup.sh +++ b/scripts/setup.sh @@ -66,6 +66,7 @@ echo "" echo "Installing data sources..." "${UV_BIN}" pip install -e ./sources/tavily_web_search "${UV_BIN}" pip install -e ./sources/exa_web_search +"${UV_BIN}" pip install -e ./sources/nimble_web_search "${UV_BIN}" pip install -e ./sources/google_scholar_paper_search "${UV_BIN}" pip install -e "./sources/knowledge_layer[llamaindex,foundational_rag]" echo "Data Sources installed" diff --git a/sources/nimble_web_search/README.md b/sources/nimble_web_search/README.md new file mode 100644 index 000000000..d5a9de601 --- /dev/null +++ b/sources/nimble_web_search/README.md @@ -0,0 +1,118 @@ +# Nimble Web Search + +NAT-based [Nimble](https://nimbleway.com/) web search tool for agentic search workflows that need live web context. Requires a `NIMBLE_API_KEY` environment variable or `api_key` config. + +## When to use + +Use `nimble_web_search` when your AI-Q agent needs fresh web context from Nimble's real-time web intelligence infrastructure. The provider is designed for agentic search workflows that benefit from live web discovery, structured results, and reliable retrieval through Nimble's search layer. + +Choose `nimble_web_search` when Tavily or Exa are not the right fit for your workflow, or when you want to standardize web-search retrieval through Nimble. The provider follows the same integration pattern as `exa_web_search` and `tavily_web_search`, making it straightforward to configure as an alternative search backend in AI-Q. + +## Install + +This package is installed automatically by `scripts/setup.sh` alongside the other data-source plugins. Manual install: + +```bash +uv pip install -e ./sources/nimble_web_search +``` + +## Configure + +Add a `nimble_web_search` function to your workflow YAML: + +```yaml +functions: + web_search_tool: + _type: nimble_web_search + max_results: 5 + search_depth: lite + country: US + locale: en +``` + +See [`docs/source/customization/configuration-reference.md`](../../docs/source/customization/configuration-reference.md) for the full parameter table. + +## Environment + +```bash +NIMBLE_API_KEY=... +``` + +You can alternatively set `api_key` directly in the YAML (as a string). Both paths use the standard `nat.data_models.function.FunctionBaseConfig` `SecretStr` handling — the key is not logged. + +## Test + +```bash +# Unit tests (credential-free, mocked) +uv run pytest sources/nimble_web_search -v + +# Lint +uv run ruff check sources/nimble_web_search +uv run ruff format --check sources/nimble_web_search +``` + +The unit tests cover: config defaults / all fields / invalid `search_depth` rejection, missing-key stub + warn-once, key-from-config env hydration, successful render + description fallback, deep depth passthrough, query/content truncation, empty/error handling, retry-then-success, final-retry failure, 401, 403 enterprise-tier, non-default country/locale passthrough, and renderer behavior on titles containing special characters. + +## Verification + +Beyond the mocked unit tests above, verify the provider is fully integrated with AI-Q by walking the [adding-a-data-source checklist](../../docs/source/extending/adding-a-data-source.md): + +```bash +# 1. Mocked unit tests pass (CI-safe, no credentials) +uv run pytest sources/nimble_web_search -q +# → 21 passed in <1s + +# 2. NAT discovers the registered function +nat info components --types function | grep nimble_web_search +# → nimble_web_search 1.0.0 function + +# 3. Lint, format, and dependency lock all clean +uv run ruff check sources/nimble_web_search +uv run ruff format --check sources/nimble_web_search +uv lock --check + +# 4. Live smoke through any workflow that names `_type: nimble_web_search` (requires NIMBLE_API_KEY) +export NIMBLE_API_KEY=... +nat run --config_file --input "your test query" +``` + +Step 4 satisfies the checklist's "Installed and tested with `nat run`" item. Any of the existing AI-Q web search configs (`configs/config_cli_default.yml`, `configs/config_web_default_llamaindex.yml`, etc.) becomes a Nimble-backed test by swapping `_type: tavily_web_search` → `_type: nimble_web_search` and translating `advanced_search: true` → `search_depth: deep`. + +## Native capabilities + +The provider exposes the following Nimble-specific surface. Defaults are tuned for the common AI-Q research workflow (lite-mode SERP for a few results, US/English regional bias): + +| Capability | Field | Default | Notes | +|---|---|---|---| +| Result count | `max_results` | `5` | Range `1-100` (Nimble's documented cap). Soft cap (Nimble may return up to N+2; see Known limitations). | +| Search depth | `search_depth` | `lite` | See the dedicated [Search depth](#search-depth) section below. | +| Localization — country | `country` | `US` | Two-letter country code (e.g. `FR`, `JP`, `UK`). Reaches the SDK constructor verbatim. | +| Localization — language | `locale` | `en` | ISO 639-1 language code (e.g. `fr`, `ja`). | +| Per-result content size | `max_content_length` | `10000` chars | Truncates each result's body to N chars (3-char ellipsis included). Minimum `1`; set to `null` to disable truncation; omit to use default. | +| Retries | `max_retries` | `3` | Exponential backoff on transient errors. Final failure surfaces a friendly per-status message (401, 403, generic). | +| Auth | `api_key` / `NIMBLE_API_KEY` | env or config | `pydantic.SecretStr`; never logged. Config-side `api_key` hydrates the env var so the underlying SDK can read it. | + +Each `` block in the rendered output carries an `entity_type` (`"OrganicResult"` for SERP results). The `include_answer=True` capability — which would produce an `entity_type="answer"` block first — is intentionally **not exposed** in v1; the SDK surfaces it as a 403 entitlement gate for non-enterprise accounts. See [Known limitations](#known-limitations). + +## Search depth + +| Value | Behavior | Account requirement | +|---|---|---| +| `lite` (default) | Metadata only — URL, title, description. Token-efficient. | Any | +| `fast` | Enterprise tier. Lower latency, richer content. | **Enterprise account required.** Returns a 403 ToolException with `"search_depth='fast' is not enabled for this account. Contact sales for access."` on non-enterprise accounts. | +| `deep` | Higher token cost. May return short page-content snippets in addition to metadata. | Any (in this account; behavior may vary by tier) | + +If you do not know your tier, leave `search_depth: lite` and let the description field carry the content. + +## Known limitations + +- `max_results` is a **soft cap**. The Nimble API may return up to N+2 documents when asked for N. The provider returns them all; AI-Q's downstream consumers can slice if they need a hard cap. +- `lite` mode returns `page_content == ""` per result. The provider falls back to `description` (~150 chars per result, organic-result quality). +- `include_answer` is **not exposed** in v1 because the langchain-nimble retriever surfaces it as a 403 enterprise gate for non-enterprise accounts. It can be added in a follow-up once a non-gated path is available. + +## Security + +- API key handling follows the existing `EXA_API_KEY` / `TAVILY_API_KEY` pattern: env var or `SecretStr` config; never logged. +- Untrusted API fields (`url`, `title`, body) are HTML-escaped before being rendered into the `` markup, so a result can't break the block or inject into downstream parsers. +- Tests are mocked; no live network in CI. +- The optional live smoke is documented in the PR description and uses a redacted output pattern. diff --git a/sources/nimble_web_search/pyproject.toml b/sources/nimble_web_search/pyproject.toml new file mode 100644 index 000000000..a40e0d773 --- /dev/null +++ b/sources/nimble_web_search/pyproject.toml @@ -0,0 +1,37 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[build-system] +build-backend = "setuptools.build_meta" +requires = ["setuptools >= 64", "setuptools-scm>=8"] + +[tool.setuptools] +packages = ["nimble_web_search"] +package-dir = {"nimble_web_search" = "src"} + +[project] +name = "nimble-web-search" +version = "1.0.0" +description = "NAT-based Nimble web search tool" +readme = "README.md" +requires-python = ">=3.11,<3.14" +license = {text = "Apache-2.0"} +dependencies = [ + "pydantic>=2.0.0", + "langchain-nimble>=3.0.0,<4.0.0", +] + +[project.entry-points."nat.plugins"] +nimble_web_search = "nimble_web_search.register" diff --git a/sources/nimble_web_search/src/__init__.py b/sources/nimble_web_search/src/__init__.py new file mode 100644 index 000000000..98e8bbb9d --- /dev/null +++ b/sources/nimble_web_search/src/__init__.py @@ -0,0 +1,22 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Nimble web search tool for NAT.""" + +from .register import nimble_web_search # noqa: F401 + +__all__ = [ + "nimble_web_search", +] diff --git a/sources/nimble_web_search/src/register.py b/sources/nimble_web_search/src/register.py new file mode 100644 index 000000000..60a244c51 --- /dev/null +++ b/sources/nimble_web_search/src/register.py @@ -0,0 +1,212 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import asyncio +import html +import logging +import os +from collections.abc import AsyncGenerator +from typing import Literal + +from pydantic import Field +from pydantic import SecretStr + +from nat.builder.builder import Builder +from nat.builder.function_info import FunctionInfo +from nat.cli.register_workflow import register_function +from nat.data_models.function import FunctionBaseConfig + +logger = logging.getLogger(__name__) + +_missing_key_warned = False + + +class NimbleWebSearchToolConfig(FunctionBaseConfig, name="nimble_web_search"): + """ + Tool that retrieves relevant contexts from web search (using Nimble) for the given question. + Requires a NIMBLE_API_KEY environment variable or api_key config. + """ + + max_results: int = Field( + default=5, + ge=1, + le=100, + description="Maximum number of search results to return (Nimble accepts 1-100).", + ) + api_key: SecretStr | None = Field(default=None, description="The API key for the Nimble service") + max_retries: int = Field(default=3, ge=1, description="Maximum number of retries for the search request") + search_depth: Literal["lite", "fast", "deep"] = Field( + default="lite", + description=( + "Nimble search depth. 'lite' returns metadata only and is the safe default. " + "'fast' is an Enterprise-tier feature and raises a 403 ToolException on " + "non-enterprise accounts. 'deep' returns full page content." + ), + ) + country: str = Field( + default="US", + description="ISO country code passed to Nimble (e.g. 'US', 'UK', 'FR').", + ) + locale: str = Field( + default="en", + description="Language/locale passed to Nimble (e.g. 'en', 'fr', 'es').", + ) + max_content_length: int | None = Field( + default=10000, + ge=1, + description=( + "Max characters per result's page content. Truncates each result to reduce " + "token usage. Set to None to disable truncation." + ), + ) + + +@register_function(config_type=NimbleWebSearchToolConfig) +async def nimble_web_search( + tool_config: NimbleWebSearchToolConfig, + builder: Builder, +) -> AsyncGenerator[FunctionInfo, None]: + """Register the Nimble web search tool with NAT. + + Wraps ``langchain_nimble.NimbleSearchRetriever`` in a NAT function so agents + can query the Nimble Search API. If ``NIMBLE_API_KEY`` is not available + (via environment or ``tool_config.api_key``), a stub function is registered + that returns an informative error instead of failing at import time. + + Args: + tool_config: Configuration controlling result count, retries, search + depth, geographic filters, and optional content truncation. + builder: NAT builder handle (unused; accepted for interface parity). + + Yields: + A ``FunctionInfo`` wrapping either the live Nimble search callable or + the missing-key stub. + """ + from langchain_nimble import NimbleSearchRetriever + + if not os.environ.get("NIMBLE_API_KEY") and tool_config.api_key: + os.environ["NIMBLE_API_KEY"] = tool_config.api_key.get_secret_value() + + if not os.environ.get("NIMBLE_API_KEY"): + global _missing_key_warned + if not _missing_key_warned: + logger.warning( + "NIMBLE_API_KEY not found. The web search tool will be registered but will " + "return an error when called. To enable: set NIMBLE_API_KEY in your environment, " + ".env file, or specify api_key in your workflow config." + ) + _missing_key_warned = True + + async def _nimble_web_search_stub(question: str) -> str: + """Web search tool (unavailable - missing NIMBLE_API_KEY).""" + return ( + "Error: Nimble web search is unavailable because NIMBLE_API_KEY is not set.\n" + "To enable this tool:\n" + "1. Get an API key from https://nimbleway.com/\n" + "2. Set the API key in your environment or in your .env file\n" + "3. Restart the application" + ) + + yield FunctionInfo.from_fn( + _nimble_web_search_stub, + description=_nimble_web_search_stub.__doc__, + ) + return + + # The constructor kwargs below (max_results / search_depth / country / locale) + # match langchain-nimble>=3.0.0,<4.0.0; this signature contract is exercised by + # the live smoke (see PR description), since the unit tests mock the retriever. + retriever = NimbleSearchRetriever( + max_results=tool_config.max_results, + search_depth=tool_config.search_depth, + country=tool_config.country, + locale=tool_config.locale, + ) + + async def _nimble_web_search(question: str) -> str: + """Retrieves relevant contexts from web search (using Nimble) for the given question. + + Args: + question (str): The question to be answered. Will be truncated to 400 characters if longer. + + Returns: + str: The web search results containing relevant documents and their URLs. + """ + if len(question) > 400: + question = question[:397] + "..." + + def _truncate_content(content: str) -> str: + limit = tool_config.max_content_length + if limit is not None and len(content) > limit: + # For very small limits there is no room for the ellipsis; hard-cut + # so the result never exceeds the configured budget. + if limit <= 3: + return content[:limit] + return content[: limit - 3] + "..." + return content + + for attempt in range(tool_config.max_retries): + try: + docs = await retriever.ainvoke(question) + + if not docs: + raise ValueError("Search returned no results") + + def _render(doc) -> str: + metadata = getattr(doc, "metadata", {}) or {} + url = metadata.get("url", "") or "" + title = metadata.get("title", "") or "" + page_content = getattr(doc, "page_content", "") or "" + description = metadata.get("description", "") or "" + body = _truncate_content(page_content if page_content else description) + # Escape untrusted API fields so they can't break the + # markup or inject into downstream renderers/parsers. + return ( + f'\n' + f"\n{html.escape(title)}\n\n" + f"{html.escape(body)}\n" + ) + + web_search_results = "\n\n---\n\n".join(_render(doc) for doc in docs) + return web_search_results if web_search_results else "Search returned no results" + + except Exception as e: + error_msg = str(e) + # Non-transient errors can't be resolved by retrying, so return + # immediately instead of sleeping through the remaining attempts. + if isinstance(e, ValueError): + return error_msg + if "401" in error_msg or "Unauthorized" in error_msg: + return ( + "Error: Web search failed due to invalid API key (401 Unauthorized).\n" + "Please check your NIMBLE_API_KEY and ensure it is valid.\n" + ) + if "403" in error_msg: + return ( + "Error: Web search failed due to a Nimble entitlement restriction (403).\n" + "The configured `search_depth` may require an enterprise Nimble account. " + "Try `search_depth: lite` or contact Nimble.\n" + ) + # Transient error: retry with backoff, or give up on the last attempt. + if attempt == tool_config.max_retries - 1: + return f"Error: Web search failed - {error_msg}" + await asyncio.sleep(2**attempt) + + return "Error: Search failed after all retries" + + yield FunctionInfo.from_fn( + _nimble_web_search, + description=_nimble_web_search.__doc__, + ) diff --git a/sources/nimble_web_search/tests/test_nimble_register.py b/sources/nimble_web_search/tests/test_nimble_register.py new file mode 100644 index 000000000..ec53ea601 --- /dev/null +++ b/sources/nimble_web_search/tests/test_nimble_register.py @@ -0,0 +1,438 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for the nimble_web_search NAT registration.""" + +import os +import sys +import types +from unittest.mock import AsyncMock +from unittest.mock import MagicMock + +import pytest +from nimble_web_search.register import NimbleWebSearchToolConfig +from nimble_web_search.register import nimble_web_search +from pydantic import SecretStr + + +class _FakeDoc: + """Mimic langchain Document just enough for the renderer.""" + + def __init__(self, url="", title="", page_content="", description=""): + self.page_content = page_content + self.metadata = { + "url": url, + "title": title, + "description": description, + "position": 1, + "entity_type": "OrganicResult", + } + + +@pytest.fixture +def fake_langchain_nimble(monkeypatch): + """Install a fake `langchain_nimble` module so tests never hit the network. + + Returns the shared NimbleSearchRetriever instance the registration will create. + """ + + module = types.ModuleType("langchain_nimble") + instance = MagicMock() + instance.ainvoke = AsyncMock() + + module.NimbleSearchRetriever = MagicMock(return_value=instance) + monkeypatch.setitem(sys.modules, "langchain_nimble", module) + return instance + + +@pytest.fixture(autouse=True) +def _reset_warn_flag(): + import nimble_web_search.register as reg + + reg._missing_key_warned = False + yield + reg._missing_key_warned = False + + +@pytest.fixture(autouse=True) +def _clear_env(monkeypatch): + monkeypatch.delenv("NIMBLE_API_KEY", raising=False) + + +async def _no_sleep(_): + return None + + +class TestNimbleWebSearchToolConfig: + def test_defaults(self): + config = NimbleWebSearchToolConfig() + assert config.max_results == 5 + assert config.api_key is None + assert config.max_retries == 3 + assert config.search_depth == "lite" + assert config.country == "US" + assert config.locale == "en" + assert config.max_content_length == 10000 + + def test_all_fields(self): + config = NimbleWebSearchToolConfig( + max_results=10, + api_key=SecretStr("sk-test"), + max_retries=1, + search_depth="deep", + country="UK", + locale="fr", + max_content_length=50, + ) + assert config.max_results == 10 + assert config.api_key.get_secret_value() == "sk-test" + assert config.max_retries == 1 + assert config.search_depth == "deep" + assert config.country == "UK" + assert config.locale == "fr" + assert config.max_content_length == 50 + + def test_invalid_search_depth_rejected(self): + from pydantic import ValidationError + + with pytest.raises(ValidationError): + NimbleWebSearchToolConfig(search_depth="ultra") + + @pytest.mark.parametrize( + "field,value", + [ + ("max_results", 0), # below ge=1 + ("max_results", 101), # above le=100 (Nimble's documented cap) + ("max_retries", 0), # below ge=1 + ("max_content_length", 0), # below ge=1 (use None to disable truncation) + ], + ) + def test_out_of_range_numeric_fields_rejected(self, field, value): + from pydantic import ValidationError + + with pytest.raises(ValidationError): + NimbleWebSearchToolConfig(**{field: value}) + + def test_max_content_length_none_allowed(self): + config = NimbleWebSearchToolConfig(max_content_length=None) + assert config.max_content_length is None + + def test_inherits_from_function_base_config(self): + from nat.data_models.function import FunctionBaseConfig + + assert issubclass(NimbleWebSearchToolConfig, FunctionBaseConfig) + + +class TestNimbleWebSearchStub: + async def test_stub_when_no_api_key(self): + config = NimbleWebSearchToolConfig() + builder = MagicMock() + + async with nimble_web_search(config, builder) as info: + result = await info.single_fn("anything") + + assert "NIMBLE_API_KEY" in result + assert "unavailable" in result.lower() + + async def test_warn_once_when_key_missing(self, caplog): + import logging + + config = NimbleWebSearchToolConfig() + builder = MagicMock() + with caplog.at_level(logging.WARNING, logger="nimble_web_search.register"): + async with nimble_web_search(config, builder): + pass + async with nimble_web_search(config, builder): + pass + + warnings = [r for r in caplog.records if "NIMBLE_API_KEY not found" in r.message] + assert len(warnings) == 1 + + +class TestNimbleWebSearchLive: + async def test_api_key_from_config_sets_env(self, fake_langchain_nimble): + fake_langchain_nimble.ainvoke.return_value = [ + _FakeDoc(url="https://a.example", title="A", page_content="body a") + ] + config = NimbleWebSearchToolConfig(api_key=SecretStr("sk-from-config")) + builder = MagicMock() + + async with nimble_web_search(config, builder) as info: + out = await info.single_fn("question") + + assert os.environ.get("NIMBLE_API_KEY") == "sk-from-config" + assert "https://a.example" in out + assert "body a" in out + + async def test_successful_search_formats_documents(self, fake_langchain_nimble, monkeypatch): + monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") + fake_langchain_nimble.ainvoke.return_value = [ + _FakeDoc(url="https://a.example", title="Title A", page_content="Body A"), + _FakeDoc(url="https://b.example", title="Title B", page_content="Body B"), + ] + config = NimbleWebSearchToolConfig(max_results=2) + builder = MagicMock() + + async with nimble_web_search(config, builder) as info: + out = await info.single_fn("query") + + assert "Title A" in out + assert "Title B" in out + assert "Body A" in out + assert "Body B" in out + assert "---" in out + assert "https://a.example" in out + + async def test_description_used_when_page_content_empty(self, fake_langchain_nimble, monkeypatch): + monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") + fake_langchain_nimble.ainvoke.return_value = [ + _FakeDoc(url="https://a.example", title="A", page_content="", description="metadata-only description"), + ] + config = NimbleWebSearchToolConfig() + builder = MagicMock() + + async with nimble_web_search(config, builder) as info: + out = await info.single_fn("q") + + assert "metadata-only description" in out + + async def test_search_depth_deep_passes_through(self, fake_langchain_nimble, monkeypatch): + monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") + fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="u", title="t", page_content="body")] + + config = NimbleWebSearchToolConfig(search_depth="deep") + builder = MagicMock() + async with nimble_web_search(config, builder): + pass + + # Verify the constructor received the expected kwargs via the module mock + ctor = sys.modules["langchain_nimble"].NimbleSearchRetriever + ctor.assert_called() + kwargs = ctor.call_args.kwargs + assert kwargs["search_depth"] == "deep" + assert kwargs["max_results"] == 5 + assert kwargs["country"] == "US" + assert kwargs["locale"] == "en" + # include_answer is intentionally omitted in v1 — the upstream retriever + # surfaces it as a 403 enterprise gate for non-enterprise accounts. + assert "include_answer" not in kwargs + + async def test_truncates_long_query(self, fake_langchain_nimble, monkeypatch): + monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") + fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="u", title="t", page_content="body")] + + config = NimbleWebSearchToolConfig() + builder = MagicMock() + long_q = "x" * 500 + async with nimble_web_search(config, builder) as info: + await info.single_fn(long_q) + + (passed_q,), _ = fake_langchain_nimble.ainvoke.call_args + assert len(passed_q) == 400 + assert passed_q.endswith("...") + + async def test_truncates_content(self, fake_langchain_nimble, monkeypatch): + monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") + fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="u", title="t", page_content="abcdefghijklmnop")] + + config = NimbleWebSearchToolConfig(max_content_length=8) + builder = MagicMock() + async with nimble_web_search(config, builder) as info: + out = await info.single_fn("q") + + assert "abcde..." in out + assert "abcdefghi" not in out + + async def test_truncates_content_small_limit_no_negative_slice(self, fake_langchain_nimble, monkeypatch): + """VERIFIES: a max_content_length below 4 hard-cuts without a negative slice, and the + result never exceeds the configured budget (the ellipsis needs 3 chars of headroom). + """ + monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") + fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="u", title="t", page_content="abcdefghij")] + + config = NimbleWebSearchToolConfig(max_content_length=2) + builder = MagicMock() + async with nimble_web_search(config, builder) as info: + out = await info.single_fn("q") + + # body hard-cut to exactly 2 chars ("ab"), no "..." appended, no over-run + assert "ab\n" in out + assert "abc" not in out + + async def test_empty_results_returns_error(self, fake_langchain_nimble, monkeypatch): + monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") + fake_langchain_nimble.ainvoke.return_value = [] + + config = NimbleWebSearchToolConfig(max_retries=1) + builder = MagicMock() + async with nimble_web_search(config, builder) as info: + out = await info.single_fn("q") + + assert "no results" in out.lower() + + async def test_retries_then_succeeds(self, fake_langchain_nimble, monkeypatch): + monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") + monkeypatch.setattr("nimble_web_search.register.asyncio.sleep", _no_sleep) + + fake_langchain_nimble.ainvoke.side_effect = [ + RuntimeError("transient"), + [_FakeDoc(url="u", title="t", page_content="ok")], + ] + + config = NimbleWebSearchToolConfig(max_retries=3) + builder = MagicMock() + async with nimble_web_search(config, builder) as info: + out = await info.single_fn("q") + + assert "ok" in out + assert fake_langchain_nimble.ainvoke.call_count == 2 + + async def test_final_retry_failure_returns_error(self, fake_langchain_nimble, monkeypatch): + monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") + monkeypatch.setattr("nimble_web_search.register.asyncio.sleep", _no_sleep) + fake_langchain_nimble.ainvoke.side_effect = RuntimeError("upstream broken") + + config = NimbleWebSearchToolConfig(max_retries=2) + builder = MagicMock() + async with nimble_web_search(config, builder) as info: + out = await info.single_fn("q") + + assert "Web search failed" in out + assert "upstream broken" in out + + async def test_401_returns_friendly_message(self, fake_langchain_nimble, monkeypatch): + monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") + monkeypatch.setattr("nimble_web_search.register.asyncio.sleep", _no_sleep) + fake_langchain_nimble.ainvoke.side_effect = RuntimeError("401 Unauthorized") + + config = NimbleWebSearchToolConfig(max_retries=2) + builder = MagicMock() + async with nimble_web_search(config, builder) as info: + out = await info.single_fn("q") + + assert "401" in out + assert "NIMBLE_API_KEY" in out + + async def test_403_returns_friendly_entitlement_message(self, fake_langchain_nimble, monkeypatch): + monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") + monkeypatch.setattr("nimble_web_search.register.asyncio.sleep", _no_sleep) + fake_langchain_nimble.ainvoke.side_effect = RuntimeError( + "403 - {'detail': 'This feature is only available for enterprise accounts.'}" + ) + + config = NimbleWebSearchToolConfig(max_retries=1, search_depth="fast") + builder = MagicMock() + async with nimble_web_search(config, builder) as info: + out = await info.single_fn("q") + + assert "403" in out + assert "enterprise" in out.lower() + + async def test_non_transient_errors_short_circuit_without_retry(self, fake_langchain_nimble, monkeypatch): + """VERIFIES: 401/403 (and other non-transient) errors return immediately after a single + attempt — no retry, no backoff sleep — so the caller isn't made to wait through retries + for an error that can't be resolved by retrying. + """ + monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") + slept = [] + monkeypatch.setattr( + "nimble_web_search.register.asyncio.sleep", + lambda d: slept.append(d) or _no_sleep(d), + ) + fake_langchain_nimble.ainvoke.side_effect = RuntimeError("401 Unauthorized") + + config = NimbleWebSearchToolConfig(max_retries=3) + builder = MagicMock() + async with nimble_web_search(config, builder) as info: + out = await info.single_fn("q") + + assert "401" in out + # short-circuited: exactly one upstream call, and no backoff sleep happened + assert fake_langchain_nimble.ainvoke.call_count == 1 + assert slept == [] + + # --- Field-passthrough tests (defaults are covered by test_search_depth_deep_passes_through; + # the four tests below verify that *non-default* values flow into the SDK constructor and + # that the rendered output stays well-formed for unusual title content) ------------------- + + async def test_non_default_country_passthrough(self, fake_langchain_nimble, monkeypatch): + """VERIFIES: NimbleWebSearchToolConfig(country="FR") forwards country='FR' to the SDK.""" + monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") + fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="u", title="t", page_content="b")] + + config = NimbleWebSearchToolConfig(country="FR") + builder = MagicMock() + async with nimble_web_search(config, builder): + pass + + kwargs = sys.modules["langchain_nimble"].NimbleSearchRetriever.call_args.kwargs + assert kwargs["country"] == "FR" + + async def test_non_default_locale_passthrough(self, fake_langchain_nimble, monkeypatch): + """VERIFIES: NimbleWebSearchToolConfig(locale="fr") forwards locale='fr' to the SDK.""" + monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") + fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="u", title="t", page_content="b")] + + config = NimbleWebSearchToolConfig(locale="fr") + builder = MagicMock() + async with nimble_web_search(config, builder): + pass + + kwargs = sys.modules["langchain_nimble"].NimbleSearchRetriever.call_args.kwargs + assert kwargs["locale"] == "fr" + + async def test_country_locale_combined_passthrough(self, fake_langchain_nimble, monkeypatch): + """VERIFIES: Both country and locale non-defaults are forwarded together to the SDK.""" + monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") + fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="u", title="t", page_content="b")] + + config = NimbleWebSearchToolConfig(country="JP", locale="ja") + builder = MagicMock() + async with nimble_web_search(config, builder): + pass + + kwargs = sys.modules["langchain_nimble"].NimbleSearchRetriever.call_args.kwargs + assert kwargs["country"] == "JP" + assert kwargs["locale"] == "ja" + + async def test_renderer_escapes_special_characters(self, fake_langchain_nimble, monkeypatch): + """VERIFIES: untrusted API fields containing <, >, &, " are HTML-escaped in the + rendered block, so they can't break the markup or inject into downstream + renderers/parsers. + """ + monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") + weird_title = "Apple Q4 < Microsoft Q4 > Forecast & Analysis" + fake_langchain_nimble.ainvoke.return_value = [ + _FakeDoc( + url="https://example.com/q4?a=1&b=2", + title=weird_title, + page_content='snippet with and "quotes"', + ) + ] + + config = NimbleWebSearchToolConfig() + builder = MagicMock() + async with nimble_web_search(config, builder) as info: + out = await info.single_fn("q") + + # Title special chars are escaped (raw < > & no longer present in the title text) + assert "Apple Q4 < Microsoft Q4 > Forecast & Analysis" in out + assert weird_title not in out + # The href attribute is escaped (& → &) so it can't break the attribute + assert 'href="https://example.com/q4?a=1&b=2"' in out + # A body that contains a literal can't terminate the block early + assert "</Document>" in out + # The block is still well-formed and terminates correctly + assert out.endswith("") diff --git a/uv.lock b/uv.lock index bbffebe98..1ea58d240 100644 --- a/uv.lock +++ b/uv.lock @@ -24,6 +24,7 @@ members = [ "freshqa-eval", "google-scholar-paper-search", "knowledge-layer", + "nimble-web-search", "tavily-web-search", ] overrides = [ @@ -276,6 +277,7 @@ dev = [ { name = "google-scholar-paper-search" }, { name = "knowledge-layer", extra = ["all"] }, { name = "mypy" }, + { name = "nimble-web-search" }, { name = "pre-commit" }, { name = "pytest" }, { name = "pytest-asyncio" }, @@ -336,6 +338,7 @@ dev = [ { name = "google-scholar-paper-search", editable = "sources/google_scholar_paper_search" }, { name = "knowledge-layer", extras = ["all"], editable = "sources/knowledge_layer" }, { name = "mypy", specifier = ">=1.5.0" }, + { name = "nimble-web-search", editable = "sources/nimble_web_search" }, { name = "pre-commit", specifier = ">=3.5.0" }, { name = "pytest", specifier = ">=7.4.0" }, { name = "pytest-asyncio", specifier = ">=0.21.0" }, @@ -2546,6 +2549,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/35/68/fc3d079108948f873de0b4aacc7e1a0f764a0f71ef664a2ed1afda2975b9/langchain_modal-0.0.3-py3-none-any.whl", hash = "sha256:9deeaa3c9bbbee49731d3a1bd0e579cc624b6f2d5540787f6bea749c89dd7c41", size = 4291, upload-time = "2026-04-07T18:02:21.166Z" }, ] +[[package]] +name = "langchain-nimble" +version = "3.0.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "langchain-core" }, + { name = "nimble-python" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/2f/e7/a457a7fd1291ec4f2009316255f3367788fe46aa7877ceaae37c0d3801ff/langchain_nimble-3.0.0.tar.gz", hash = "sha256:59c98ac6def2930aa2cb7dfd2db3a7e5549ca4dc5204aa9fd5b44c4781054507", size = 195257, upload-time = "2026-03-23T07:30:59.834Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/28/ad/417dea0df11bc8081808a48c11a8d7e0837a6acb412e0a34e8233aeb3ab2/langchain_nimble-3.0.0-py3-none-any.whl", hash = "sha256:2e53e965716aa3af6af609312a0aa08889355ba9527e04dff914f408286fe852", size = 23901, upload-time = "2026-03-23T07:30:58.759Z" }, +] + [[package]] name = "langchain-nvidia-ai-endpoints" version = "1.0.3" @@ -3590,6 +3606,38 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/9e/c9/b2622292ea83fbb4ec318f5b9ab867d0a28ab43c5717bb85b0a5f6b3b0a4/networkx-3.6.1-py3-none-any.whl", hash = "sha256:d47fbf302e7d9cbbb9e2555a0d267983d2aa476bac30e90dfbe5669bd57f3762", size = 2068504, upload-time = "2025-12-08T17:02:38.159Z" }, ] +[[package]] +name = "nimble-python" +version = "0.18.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "distro" }, + { name = "httpx" }, + { name = "pydantic" }, + { name = "sniffio" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/78/a8/7e6a131b181376e6c1acc69de86a7af9dbeab0653e7350234a12baf56c09/nimble_python-0.18.0.tar.gz", hash = "sha256:0e291944463f9fea85d70b8a5037f36aaa21a0d02ffbcaf9fc05fa67e9d7ce6c", size = 291230, upload-time = "2026-05-10T13:40:09.461Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/59/2a/4342e5eae52c926c56431a5024f3b472cf4e3cef2c6796293824cf9c13df/nimble_python-0.18.0-py3-none-any.whl", hash = "sha256:f27249d5c93a3d403468e04c074777da46300cbd76b872cc9312ef753206b781", size = 193584, upload-time = "2026-05-10T13:40:10.787Z" }, +] + +[[package]] +name = "nimble-web-search" +version = "1.0.0" +source = { editable = "sources/nimble_web_search" } +dependencies = [ + { name = "langchain-nimble" }, + { name = "pydantic" }, +] + +[package.metadata] +requires-dist = [ + { name = "langchain-nimble", specifier = ">=3.0.0,<4.0.0" }, + { name = "pydantic", specifier = ">=2.0.0" }, +] + [[package]] name = "nltk" version = "3.9.4" From 30699d9efb3a4f02dd5198577e43b19655435663 Mon Sep 17 00:00:00 2001 From: Kobi Kadosh Date: Mon, 1 Jun 2026 00:20:14 -0700 Subject: [PATCH 02/14] fix(nimble_web_search): add typed focus config defaulting to general and clarify tool description Expose `focus` as a typed Literal config (general, news, location, shopping, geo, social) defaulting to "general", and pass it explicitly to NimbleSearchRetriever. The upstream SDK field is an unvalidated str defaulting to general; the Literal adds parse-time validation and makes the general default explicit. focus is a workflow-config setting, not an agent-chosen parameter, so general research queries cannot silently switch to news. Tighten the tool description the agent sees to state it is a general-purpose web/research search. include_answer remains unexposed in this initial integration. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Kobi Kadosh --- .../customization/configuration-reference.md | 9 ++++ sources/nimble_web_search/README.md | 5 +- sources/nimble_web_search/src/register.py | 32 ++++++++++--- .../tests/test_nimble_register.py | 47 +++++++++++++++++++ 4 files changed, 85 insertions(+), 8 deletions(-) diff --git a/docs/source/customization/configuration-reference.md b/docs/source/customization/configuration-reference.md index be825c6be..60d782f9e 100644 --- a/docs/source/customization/configuration-reference.md +++ b/docs/source/customization/configuration-reference.md @@ -220,6 +220,7 @@ functions: | `api_key` | `str` | `None` | Nimble API key. Falls back to `NIMBLE_API_KEY` environment variable. | | `max_retries` | `int` | `3` | Number of retry attempts on search failure. | | `search_depth` | `str` | `"lite"` | Nimble search depth. See options below. | +| `focus` | `str` | `"general"` | Nimble focus mode. See options below. | | `country` | `str` | `"US"` | ISO country code passed to Nimble (e.g. `US`, `UK`, `FR`). | | `locale` | `str` | `"en"` | Language/locale passed to Nimble (e.g. `en`, `fr`, `es`). | | `max_content_length` | `int` | `10000` | Max characters per result's page content. Set to `None` to disable truncation. | @@ -230,6 +231,14 @@ functions: - **`fast`** -- Returns rich content at low latency. **Enterprise-tier only**; non-enterprise accounts receive a 403 with a clear entitlement message. - **`deep`** -- Returns full page content for each result. Use for research workflows that need the body text, not just URLs. +**`focus` options:** + +- **`general`** (default) -- Broad web/research queries. The right choice for almost all agent use. +- **`news`** -- Current events. Use only for a tool dedicated to news/recency. +- **`location`**, **`shopping`**, **`geo`**, **`social`** -- Domain-specific routing; set only when the tool targets that domain. + +`focus` is a workflow-config setting, not an agent-chosen parameter -- the model only passes a query, so general research queries cannot silently switch to `news`. Answer generation (`include_answer`) is **not exposed** in this initial integration. + ### `paper_search` Academic paper search through Google Scholar (using the [Serper API](https://serper.dev/)). diff --git a/sources/nimble_web_search/README.md b/sources/nimble_web_search/README.md index d5a9de601..8cb9968cb 100644 --- a/sources/nimble_web_search/README.md +++ b/sources/nimble_web_search/README.md @@ -86,13 +86,14 @@ The provider exposes the following Nimble-specific surface. Defaults are tuned f |---|---|---|---| | Result count | `max_results` | `5` | Range `1-100` (Nimble's documented cap). Soft cap (Nimble may return up to N+2; see Known limitations). | | Search depth | `search_depth` | `lite` | See the dedicated [Search depth](#search-depth) section below. | +| Search focus | `focus` | `general` | Nimble focus mode: `general` (default, broad web/research), `news` (current events), or domain-specific `location` / `shopping` / `geo` / `social`. Leave `general` for normal research; the LLM never selects focus, so general queries can't drift to `news`. | | Localization — country | `country` | `US` | Two-letter country code (e.g. `FR`, `JP`, `UK`). Reaches the SDK constructor verbatim. | | Localization — language | `locale` | `en` | ISO 639-1 language code (e.g. `fr`, `ja`). | | Per-result content size | `max_content_length` | `10000` chars | Truncates each result's body to N chars (3-char ellipsis included). Minimum `1`; set to `null` to disable truncation; omit to use default. | | Retries | `max_retries` | `3` | Exponential backoff on transient errors. Final failure surfaces a friendly per-status message (401, 403, generic). | | Auth | `api_key` / `NIMBLE_API_KEY` | env or config | `pydantic.SecretStr`; never logged. Config-side `api_key` hydrates the env var so the underlying SDK can read it. | -Each `` block in the rendered output carries an `entity_type` (`"OrganicResult"` for SERP results). The `include_answer=True` capability — which would produce an `entity_type="answer"` block first — is intentionally **not exposed** in v1; the SDK surfaces it as a 403 entitlement gate for non-enterprise accounts. See [Known limitations](#known-limitations). +Each `` block in the rendered output carries an `entity_type` (`"OrganicResult"` for SERP results). The `include_answer=True` capability — which would produce an `entity_type="answer"` block first — is intentionally **not exposed** in this initial integration. See [Known limitations](#known-limitations). ## Search depth @@ -108,7 +109,7 @@ If you do not know your tier, leave `search_depth: lite` and let the description - `max_results` is a **soft cap**. The Nimble API may return up to N+2 documents when asked for N. The provider returns them all; AI-Q's downstream consumers can slice if they need a hard cap. - `lite` mode returns `page_content == ""` per result. The provider falls back to `description` (~150 chars per result, organic-result quality). -- `include_answer` is **not exposed** in v1 because the langchain-nimble retriever surfaces it as a 403 enterprise gate for non-enterprise accounts. It can be added in a follow-up once a non-gated path is available. +- `include_answer` is **not exposed** in this initial integration. It can be added in a follow-up. ## Security diff --git a/sources/nimble_web_search/src/register.py b/sources/nimble_web_search/src/register.py index 60a244c51..381e710fd 100644 --- a/sources/nimble_web_search/src/register.py +++ b/sources/nimble_web_search/src/register.py @@ -55,6 +55,18 @@ class NimbleWebSearchToolConfig(FunctionBaseConfig, name="nimble_web_search"): "non-enterprise accounts. 'deep' returns full page content." ), ) + # Mirrors langchain-nimble's SearchFocus modes (general, news, location, + # shopping, geo, social). Declared locally because the SDK does not export + # the enum publicly; swap for a direct import if it becomes public. + focus: Literal["general", "news", "location", "shopping", "geo", "social"] = Field( + default="general", + description=( + "Nimble search focus mode. 'general' (default) covers broad web/research " + "queries and is the right choice for almost all agent use. 'news' targets " + "current events; the rest are domain-specific (location, shopping, geo, " + "social). Leave as 'general' unless the tool is dedicated to one of those." + ), + ) country: str = Field( default="US", description="ISO country code passed to Nimble (e.g. 'US', 'UK', 'FR').", @@ -125,24 +137,32 @@ async def _nimble_web_search_stub(question: str) -> str: ) return - # The constructor kwargs below (max_results / search_depth / country / locale) - # match langchain-nimble>=3.0.0,<4.0.0; this signature contract is exercised by - # the live smoke (see PR description), since the unit tests mock the retriever. + # The constructor kwargs below (max_results / search_depth / focus / country / + # locale) match langchain-nimble>=3.0.0,<4.0.0; this signature contract is + # exercised by the live smoke (see PR description), since the unit tests mock + # the retriever. `focus` is passed explicitly so the default is provably + # "general" rather than relying on the SDK's (unvalidated) field default. + # `include_answer` is intentionally not exposed in this initial integration. retriever = NimbleSearchRetriever( max_results=tool_config.max_results, search_depth=tool_config.search_depth, + focus=tool_config.focus, country=tool_config.country, locale=tool_config.locale, ) async def _nimble_web_search(question: str) -> str: - """Retrieves relevant contexts from web search (using Nimble) for the given question. + """Search the web with Nimble and return relevant sources for a question. + + General-purpose web/research search: pass a natural-language question and + get back the most relevant pages with their URLs and content. Use it for + broad informational and technical research queries. Args: - question (str): The question to be answered. Will be truncated to 400 characters if longer. + question (str): The question to answer. Truncated to 400 characters if longer. Returns: - str: The web search results containing relevant documents and their URLs. + str: Relevant documents and their URLs, rendered as XML blocks. """ if len(question) > 400: question = question[:397] + "..." diff --git a/sources/nimble_web_search/tests/test_nimble_register.py b/sources/nimble_web_search/tests/test_nimble_register.py index ec53ea601..77ad75871 100644 --- a/sources/nimble_web_search/tests/test_nimble_register.py +++ b/sources/nimble_web_search/tests/test_nimble_register.py @@ -82,6 +82,7 @@ def test_defaults(self): assert config.api_key is None assert config.max_retries == 3 assert config.search_depth == "lite" + assert config.focus == "general" assert config.country == "US" assert config.locale == "en" assert config.max_content_length == 10000 @@ -92,6 +93,7 @@ def test_all_fields(self): api_key=SecretStr("sk-test"), max_retries=1, search_depth="deep", + focus="news", country="UK", locale="fr", max_content_length=50, @@ -100,6 +102,7 @@ def test_all_fields(self): assert config.api_key.get_secret_value() == "sk-test" assert config.max_retries == 1 assert config.search_depth == "deep" + assert config.focus == "news" assert config.country == "UK" assert config.locale == "fr" assert config.max_content_length == 50 @@ -110,6 +113,18 @@ def test_invalid_search_depth_rejected(self): with pytest.raises(ValidationError): NimbleWebSearchToolConfig(search_depth="ultra") + def test_invalid_focus_rejected(self): + from pydantic import ValidationError + + with pytest.raises(ValidationError): + NimbleWebSearchToolConfig(focus="newsy") + + def test_no_include_answer_field(self): + # include_answer is intentionally not exposed as AI-Q-facing config in this + # initial integration (see register.py / README). + assert "include_answer" not in NimbleWebSearchToolConfig.model_fields + assert "include_answers" not in NimbleWebSearchToolConfig.model_fields + @pytest.mark.parametrize( "field,value", [ @@ -223,12 +238,44 @@ async def test_search_depth_deep_passes_through(self, fake_langchain_nimble, mon kwargs = ctor.call_args.kwargs assert kwargs["search_depth"] == "deep" assert kwargs["max_results"] == 5 + assert kwargs["focus"] == "general" # default is general, passed explicitly assert kwargs["country"] == "US" assert kwargs["locale"] == "en" # include_answer is intentionally omitted in v1 — the upstream retriever # surfaces it as a 403 enterprise gate for non-enterprise accounts. assert "include_answer" not in kwargs + async def test_focus_defaults_to_general(self, fake_langchain_nimble, monkeypatch): + """VERIFIES: with no focus configured, the retriever is built with focus='general' — + so general research queries never silently use a news/other focus. + """ + monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") + fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="u", title="t", page_content="b")] + + config = NimbleWebSearchToolConfig() + builder = MagicMock() + async with nimble_web_search(config, builder): + pass + + kwargs = sys.modules["langchain_nimble"].NimbleSearchRetriever.call_args.kwargs + assert kwargs["focus"] == "general" + + async def test_focus_news_passes_through_when_configured(self, fake_langchain_nimble, monkeypatch): + """VERIFIES: an operator can deliberately configure focus='news'; it reaches the SDK. + (The LLM never chooses focus — it's not a tool parameter — so general queries can't + drift to news on their own.) + """ + monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") + fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="u", title="t", page_content="b")] + + config = NimbleWebSearchToolConfig(focus="news") + builder = MagicMock() + async with nimble_web_search(config, builder): + pass + + kwargs = sys.modules["langchain_nimble"].NimbleSearchRetriever.call_args.kwargs + assert kwargs["focus"] == "news" + async def test_truncates_long_query(self, fake_langchain_nimble, monkeypatch): monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="u", title="t", page_content="body")] From 18b6f376257b19aaa2815ff42e3ef215e26740e7 Mon Sep 17 00:00:00 2001 From: Kobi Kadosh Date: Wed, 3 Jun 2026 23:48:08 -0700 Subject: [PATCH 03/14] docs(nimble_web_search): clarify focus=news semantics (no recency threshold) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `news` restricts results to news-publisher sources ordered by recency; it does not apply a recency threshold, so older articles still appear — it changes the source mix, not the time window. Recency windowing is a separate Nimble `time_range` capability that also works with `focus=general`. Reword the config reference, README, and field description to remove the misleading "current events" phrasing. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Kobi Kadosh --- docs/source/customization/configuration-reference.md | 2 +- sources/nimble_web_search/README.md | 2 +- sources/nimble_web_search/src/register.py | 8 +++++--- 3 files changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/source/customization/configuration-reference.md b/docs/source/customization/configuration-reference.md index 60d782f9e..87ec76009 100644 --- a/docs/source/customization/configuration-reference.md +++ b/docs/source/customization/configuration-reference.md @@ -234,7 +234,7 @@ functions: **`focus` options:** - **`general`** (default) -- Broad web/research queries. The right choice for almost all agent use. -- **`news`** -- Current events. Use only for a tool dedicated to news/recency. +- **`news`** -- Restricts results to news-publisher sources, ordered by recency. There is no recency threshold -- older articles still appear; it changes the source mix, not the time window. (Recency windowing is a separate Nimble `time_range` capability that also works with `focus=general`; not exposed in this initial integration.) - **`location`**, **`shopping`**, **`geo`**, **`social`** -- Domain-specific routing; set only when the tool targets that domain. `focus` is a workflow-config setting, not an agent-chosen parameter -- the model only passes a query, so general research queries cannot silently switch to `news`. Answer generation (`include_answer`) is **not exposed** in this initial integration. diff --git a/sources/nimble_web_search/README.md b/sources/nimble_web_search/README.md index 8cb9968cb..be70c022d 100644 --- a/sources/nimble_web_search/README.md +++ b/sources/nimble_web_search/README.md @@ -86,7 +86,7 @@ The provider exposes the following Nimble-specific surface. Defaults are tuned f |---|---|---|---| | Result count | `max_results` | `5` | Range `1-100` (Nimble's documented cap). Soft cap (Nimble may return up to N+2; see Known limitations). | | Search depth | `search_depth` | `lite` | See the dedicated [Search depth](#search-depth) section below. | -| Search focus | `focus` | `general` | Nimble focus mode: `general` (default, broad web/research), `news` (current events), or domain-specific `location` / `shopping` / `geo` / `social`. Leave `general` for normal research; the LLM never selects focus, so general queries can't drift to `news`. | +| Search focus | `focus` | `general` | Nimble focus mode: `general` (default, broad web/research), `news` (news-publisher sources ordered by recency — not a recency filter; older articles still appear), or domain-specific `location` / `shopping` / `geo` / `social`. Leave `general` for normal research; the LLM never selects focus, so general queries can't drift to `news`. | | Localization — country | `country` | `US` | Two-letter country code (e.g. `FR`, `JP`, `UK`). Reaches the SDK constructor verbatim. | | Localization — language | `locale` | `en` | ISO 639-1 language code (e.g. `fr`, `ja`). | | Per-result content size | `max_content_length` | `10000` chars | Truncates each result's body to N chars (3-char ellipsis included). Minimum `1`; set to `null` to disable truncation; omit to use default. | diff --git a/sources/nimble_web_search/src/register.py b/sources/nimble_web_search/src/register.py index 381e710fd..67c49f538 100644 --- a/sources/nimble_web_search/src/register.py +++ b/sources/nimble_web_search/src/register.py @@ -62,9 +62,11 @@ class NimbleWebSearchToolConfig(FunctionBaseConfig, name="nimble_web_search"): default="general", description=( "Nimble search focus mode. 'general' (default) covers broad web/research " - "queries and is the right choice for almost all agent use. 'news' targets " - "current events; the rest are domain-specific (location, shopping, geo, " - "social). Leave as 'general' unless the tool is dedicated to one of those." + "queries and is the right choice for almost all agent use. 'news' restricts " + "results to news-publisher sources ordered by recency (it is not a recency " + "filter -- older articles still appear); the rest are domain-specific " + "(location, shopping, geo, social). Leave as 'general' unless the tool is " + "dedicated to one of those." ), ) country: str = Field( From d6cbc2a3bd85b0d09038f16d39ff189dd6729e14 Mon Sep 17 00:00:00 2001 From: Kobi Kadosh Date: Thu, 9 Jul 2026 23:12:59 -0700 Subject: [PATCH 04/14] test(nimble_web_search): add shared test fixtures and output-contract helpers Adds tests/__init__.py, aligning the package layout with the other data sources, and a conftest.py exposing a fake langchain_nimble module fixture plus a structural output-contract assertion shared by the recorded-replay and live integration tests, so both layers certify identical success criteria. Document blocks are extracted as whole spans rather than by splitting on the joiner sequence, which deep-mode markdown content can legitimately contain. Co-Authored-By: Claude Fable 5 Signed-off-by: Kobi Kadosh --- sources/nimble_web_search/tests/__init__.py | 16 ++ sources/nimble_web_search/tests/conftest.py | 154 ++++++++++++++++++++ 2 files changed, 170 insertions(+) create mode 100644 sources/nimble_web_search/tests/__init__.py create mode 100644 sources/nimble_web_search/tests/conftest.py diff --git a/sources/nimble_web_search/tests/__init__.py b/sources/nimble_web_search/tests/__init__.py new file mode 100644 index 000000000..82a311c6a --- /dev/null +++ b/sources/nimble_web_search/tests/__init__.py @@ -0,0 +1,16 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for the nimble_web_search data source.""" diff --git a/sources/nimble_web_search/tests/conftest.py b/sources/nimble_web_search/tests/conftest.py new file mode 100644 index 000000000..a26e2b7c3 --- /dev/null +++ b/sources/nimble_web_search/tests/conftest.py @@ -0,0 +1,154 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Shared fixtures and output-contract helpers for nimble_web_search tests. + +The assertion helper exposed here defines the provider's structural output +contract once, so the recorded-response replay tests and the key-gated live +integration test certify exactly the same success criteria. Everything is +exposed as pytest fixtures (not module imports) so the test modules work under +any pytest import mode. +""" + +import json +import re +import sys +import types +import xml.etree.ElementTree as ET +from pathlib import Path +from unittest.mock import AsyncMock +from unittest.mock import MagicMock + +import pytest + +FIXTURES_DIR = Path(__file__).parent / "fixtures" + +# Whole-block extraction rather than splitting on the "\n\n---\n\n" joiner: +# deep-mode page_content is markdown that can itself contain that byte +# sequence (horizontal rules), which would make separator-splitting mis-slice +# a document into fragments. Extracting ... spans is +# unambiguous because the renderer html-escapes all interior fields, so no +# raw '<'/'>' can appear inside a block. +_DOCUMENT_BLOCK_RE = re.compile(r'.*?', re.DOTALL) +_DOCUMENT_HEAD_RE = re.compile(r'\n\n(?P<title>.*?)\n\n', re.DOTALL) + + +class FakeDocument: + """Mimic a langchain ``Document`` just enough for the provider's renderer.""" + + def __init__(self, url="", title="", page_content="", description="", position=1): + self.page_content = page_content + self.metadata = { + "url": url, + "title": title, + "description": description, + "position": position, + "entity_type": "OrganicResult", + } + + +def _extract_blocks(result: str) -> list[str]: + return _DOCUMENT_BLOCK_RE.findall(result) + + +def _assert_search_contract(result: str, max_results: int) -> list[str]: + """Assert the provider's structural output contract; return the blocks. + + These are the provider's CI success criteria: + + 1. ``result`` is a non-empty string that is neither a provider ``Error:`` + message nor the ``Search returned no results`` sentinel. + 2. It contains between 1 and ``max_results`` ```` blocks. + 3. Every block has a non-empty http(s) ``href`` and a non-empty title. + 4. Every block parses as XML (the renderer html-escapes all interior + fields, so a well-formed response must parse). + 5. At least one block has non-empty body content (individual results may + legitimately have an empty description, so "all bodies non-empty" + would be flaky). + """ + assert isinstance(result, str) and result.strip(), "empty result" + assert not result.startswith("Error:"), f"provider returned an error: {result[:200]}" + assert result != "Search returned no results" + + blocks = _extract_blocks(result) + assert 1 <= len(blocks) <= max_results, f"expected 1..{max_results} blocks, got {len(blocks)}" + + bodies = [] + for block in blocks: + head = _DOCUMENT_HEAD_RE.match(block) + assert head is not None, f"malformed head: {block[:120]!r}" + href = head.group("href") + assert href.startswith(("http://", "https://")), f"non-http(s) href: {href!r}" + assert head.group("title").strip(), "empty title" + element = ET.fromstring(block) + body = (element.text or "") + "".join(child.tail or "" for child in element) + bodies.append(body.strip()) + assert any(bodies), "every block had an empty body" + return blocks + + +@pytest.fixture +def fake_langchain_nimble(monkeypatch): + """Install a fake ``langchain_nimble`` module so tests never hit the network. + + Returns the shared ``NimbleSearchRetriever`` instance the registration + will create; set ``instance.ainvoke.return_value`` to control results. + """ + module = types.ModuleType("langchain_nimble") + instance = MagicMock() + instance.ainvoke = AsyncMock() + module.NimbleSearchRetriever = MagicMock(return_value=instance) + monkeypatch.setitem(sys.modules, "langchain_nimble", module) + return instance + + +@pytest.fixture +def assert_search_contract(): + """The structural output-contract assertion shared by replay + live tests.""" + return _assert_search_contract + + +@pytest.fixture +def extract_document_blocks(): + """Whole-block ```` extraction helper.""" + return _extract_blocks + + +@pytest.fixture +def make_fake_document(): + """Factory for renderer-compatible fake documents.""" + return FakeDocument + + +@pytest.fixture +def load_recorded_fixture(): + """Load a recorded SDK-boundary fixture: name -> (payload, documents).""" + + def _load(name: str): + with (FIXTURES_DIR / name).open(encoding="utf-8") as fh: + payload = json.load(fh) + documents = [ + FakeDocument( + url=doc["metadata"].get("url", ""), + title=doc["metadata"].get("title", ""), + description=doc["metadata"].get("description", ""), + page_content=doc.get("page_content", ""), + position=doc["metadata"].get("position", 1), + ) + for doc in payload["documents"] + ] + return payload, documents + + return _load From e40fd3013db361b30929dfe231bd0e864494a86e Mon Sep 17 00:00:00 2001 From: Kobi Kadosh Date: Thu, 9 Jul 2026 23:12:59 -0700 Subject: [PATCH 05/14] test(nimble_web_search): add recorded-response replay tests Replays real NimbleSearchRetriever responses, captured at the SDK boundary and redacted by construction (result fields only; no headers or auth material), through the registered function end-to-end. Covers the lite metadata-only response (description fallback), a deep full-content response, and a synthetic case where a document body contains the block-joiner sequence. Deterministic, credential-free test mode; fixtures/README.md documents provenance and the refresh procedure. Co-Authored-By: Claude Fable 5 Signed-off-by: Kobi Kadosh --- .../tests/fixtures/README.md | 52 ++++++++++ .../fixtures/recorded_deep_response.json | 45 +++++++++ .../fixtures/recorded_lite_response.json | 45 +++++++++ .../tests/test_nimble_recorded_replay.py | 96 +++++++++++++++++++ 4 files changed, 238 insertions(+) create mode 100644 sources/nimble_web_search/tests/fixtures/README.md create mode 100644 sources/nimble_web_search/tests/fixtures/recorded_deep_response.json create mode 100644 sources/nimble_web_search/tests/fixtures/recorded_lite_response.json create mode 100644 sources/nimble_web_search/tests/test_nimble_recorded_replay.py diff --git a/sources/nimble_web_search/tests/fixtures/README.md b/sources/nimble_web_search/tests/fixtures/README.md new file mode 100644 index 000000000..d3d4650c4 --- /dev/null +++ b/sources/nimble_web_search/tests/fixtures/README.md @@ -0,0 +1,52 @@ +# Recorded Nimble response fixtures + +Real `NimbleSearchRetriever.ainvoke()` responses, captured at the SDK boundary +(the exact seam the unit tests mock) and replayed by +`test_nimble_recorded_replay.py` to exercise the full provider pipeline with no +network and no credentials. + +## Provenance + +| File | Captured | Query | Config | +|---|---|---|---| +| `recorded_lite_response.json` | 2026-07-10 | `NVIDIA CUDA Toolkit documentation` | `search_depth=lite`, `max_results=5`, `focus=general`, `country=US`, `locale=en` | +| `recorded_deep_response.json` | 2026-07-10 | `NVIDIA CUDA Toolkit documentation` | `search_depth=deep`, `max_results=5`, `focus=general`, `country=US`, `locale=en` | + +The query matches the live integration test's `CANNED_QUERY`, so the recorded +and live layers certify the same scenario. + +## Redaction contract + +Fixtures are redacted **by construction**: capture happens at the retriever's +return value (a list of documents), never at the HTTP layer, so request/response +headers and auth material are never present. Only the fields the provider +consumes are kept — `page_content` (truncated to 2000 chars) and the +`url` / `title` / `description` / `position` / `entity_type` metadata keys. + +## Refreshing a fixture + +```python +import asyncio, json +from langchain_nimble import NimbleSearchRetriever # requires NIMBLE_API_KEY + +KEPT = ("url", "title", "description", "position", "entity_type") +retriever = NimbleSearchRetriever(max_results=5, search_depth="lite", focus="general", country="US", locale="en") +docs = asyncio.run(retriever.ainvoke("NVIDIA CUDA Toolkit documentation")) +payload = { + "_description": "Recorded NimbleSearchRetriever.ainvoke() response, search_depth=lite", + "_captured": "YYYY-MM-DD", + "_query": "NVIDIA CUDA Toolkit documentation", + "_config": {"max_results": 5, "search_depth": "lite", "focus": "general", "country": "US", "locale": "en"}, + "_redaction": "only url/title/description/position/entity_type/page_content kept; page_content truncated to 2000 chars; no headers/auth", + "documents": [ + {"page_content": (d.page_content or "")[:2000], "metadata": {k: (d.metadata or {}).get(k, "") for k in KEPT}} + for d in docs + ], +} +print(json.dumps(payload, indent=2, ensure_ascii=False)) +``` + +Update the `_captured` date and re-run the replay tests after refreshing. + +Licensed under the Apache License, Version 2.0 (SPDX-License-Identifier: Apache-2.0); +JSON cannot carry a license header, so this note covers the fixture files. diff --git a/sources/nimble_web_search/tests/fixtures/recorded_deep_response.json b/sources/nimble_web_search/tests/fixtures/recorded_deep_response.json new file mode 100644 index 000000000..174d48c44 --- /dev/null +++ b/sources/nimble_web_search/tests/fixtures/recorded_deep_response.json @@ -0,0 +1,45 @@ +{ + "_description": "Recorded NimbleSearchRetriever.ainvoke() response, search_depth=deep", + "_captured": "2026-07-10", + "_query": "NVIDIA CUDA Toolkit documentation", + "_config": { + "max_results": 5, + "search_depth": "deep", + "focus": "general", + "country": "US", + "locale": "en" + }, + "_redaction": "only url/title/description/position/entity_type/page_content kept; page_content truncated to 2000 chars; no headers/auth", + "documents": [ + { + "page_content": "", + "metadata": { + "url": "https://docs.nvidia.com/cuda/", + "title": "CUDA Toolkit Documentation", + "description": "Find installation instructions, launch highlights, programming guides, compiler documentation, API references, CUDA libraries, profiling tools, samples, and ...", + "position": 1, + "entity_type": "OrganicResult" + } + }, + { + "page_content": "Toggle Navigation\n\n* [Home](https://developer.nvidia.com/)\n* [Blog](https://developer.nvidia.com/blog/)\n* [Forums](https://forums.developer.nvidia.com/)\n* [Docs](https://docs.nvidia.com/)\n* [Downloads](https://developer.nvidia.com/downloads)\n* [Training](https://www.nvidia.com/en-us/deep-learning-ai/education/)\n\n* [Join](https://developer.nvidia.com/login)\n\n* Topics\n\n + Artificial Intelligence\n + [Overview](https://developer.nvidia.com/topics/ai/)\n + [AI Inference](https://developer.nvidia.com/topics/ai/ai-inference/)\n + [Conversational AI](https://developer.nvidia.com/topics/ai/conversational-ai/)\n + [Cybersecurity](https://developer.nvidia.com/topics/ai/cybersecurity-ai/)\n + [Data Science](https://developer.nvidia.com/topics/ai/data-science)\n + [Generative AI](https://developer.nvidia.com/generative-ai)\n + [Retrieval-Augmented Generation](https://developer.nvidia.com/topics/ai/retrieval-augmented-generation)\n + [Vision AI](https://developer.nvidia.com/computer-vision)\n + Cloud and Data Center\n + [Cloud-Native Technologies](https://developer.nvidia.com/cloud-native)\n + [IO Acceleration](https://developer.nvidia.com/magnum-io)\n + [Networking](https://developer.nvidia.com/networking)\n + Design and Simulation\n + [Computer Aided Engineering](https://developer.nvidia.com/topics/cae)\n + [Extended Reality (XR)](https://developer.nvidia.com/xr)\n + [Physics and Dynamics Simulation](https://developer.nvidia.com/physx-sdk)\n + [Robotics Simulation](https://developer.nvidia.com/isaac/sim)\n + [Self-Driving Vehicles Simulation](https://developer.nvidia.com/drive/simulation)\n + Graphics and Rendering\n + [Ray Tracing](https://developer.nvidia.com/rtx/ray-tracing)\n + [Video, Broadcast, and Display](https://developer.nvidia.com/video-and-audio-solutions)\n + [Game Engines](https://developer.nvidia.com/game-engines)\n + [Image Processing](https://developer.nvidia.com/image-processing)\n + High-Performance Computing\n + [Overview](https://developer.nvidia.com/hpc)", + "metadata": { + "url": "https://developer.nvidia.com/cuda-toolkit-archive", + "title": "CUDA Toolkit Archive", + "description": "Previous releases of the CUDA Toolkit, GPU Computing SDK, documentation and developer drivers can be found using the links below. Please select the release...", + "position": 2, + "entity_type": "OrganicResult" + } + }, + { + "page_content": "NVIDIACUDA Toolkit Documentation\n\nSearch In:\nEntire Site\nJust This Document\nclear search\nsearch\n\n▼CUDA Toolkit\n\n* [Release Notes](cuda-toolkit-release-notes/index.html \"The Release Notes for the CUDA Toolkit from v4.0 to today.\")\n* [EULA](eula/index.html \"The End User License Agreements for the NVIDIA CUDA Toolkit, the NVIDIA CUDA Samples, the NVIDIA Display Driver, and NVIDIA NSight (Visual Studio Edition).\")\n\n▼[Getting Started Guides](#getting-started-guides)\n\n* [Getting Started Linux](cuda-getting-started-guide-for-linux/index.html \"This guide discusses how to install and check for correct operation of the CUDA Development Tools on GNU/Linux systems.\")\n* [Getting Started Mac OS X](cuda-getting-started-guide-for-mac-os-x/index.html \"This guide discusses how to install and check for correct operation of the CUDA Development Tools on Mac OS X systems.\")\n* [Getting Started Windows](cuda-getting-started-guide-for-microsoft-windows/index.html \"This guide discusses how to install and check for correct operation of the CUDA Development Tools on Microsoft Windows systems.\")\n\n▼[Programming Guides](#programming-guides)\n\n* [Programming Guide](cuda-c-programming-guide/index.html \"This guide provides a detailed discussion of the CUDA programming model and programming interface. It then describes the hardware implementation, and provides guidance on how to achieve maximum performance. The Appendixes include a list of all CUDA-enabled devices, detailed description of all extensions to the C language, listings of supported mathematical functions, C++ features supported in host and device code, details on texture fetching, technical specifications of various devices, and concludes by introducing the low-level driver API.\")\n* [Best Practices Guide](cuda-c-best-practices-guide/index.html \"This guide presents established parallelization and optimization techniques and explains coding metaphors and idioms that can greatly simplify programming for CUDA-capable GPU architectures. The in", + "metadata": { + "url": "https://cseweb.ucsd.edu/classes/wi15/cse262-a/static/cuda-5.5-doc/html/index.html", + "title": "CUDA Toolkit Documentation", + "description": "This guide presents established parallelization and optimization techniques and explains coding metaphors and idioms that can greatly simplify programming for...", + "position": 3, + "entity_type": "OrganicResult" + } + } + ] +} diff --git a/sources/nimble_web_search/tests/fixtures/recorded_lite_response.json b/sources/nimble_web_search/tests/fixtures/recorded_lite_response.json new file mode 100644 index 000000000..3ee00ce45 --- /dev/null +++ b/sources/nimble_web_search/tests/fixtures/recorded_lite_response.json @@ -0,0 +1,45 @@ +{ + "_description": "Recorded NimbleSearchRetriever.ainvoke() response, search_depth=lite", + "_captured": "2026-07-10", + "_query": "NVIDIA CUDA Toolkit documentation", + "_config": { + "max_results": 5, + "search_depth": "lite", + "focus": "general", + "country": "US", + "locale": "en" + }, + "_redaction": "only url/title/description/position/entity_type/page_content kept; page_content truncated to 2000 chars; no headers/auth", + "documents": [ + { + "page_content": "", + "metadata": { + "url": "https://docs.nvidia.com/cuda/", + "title": "CUDA Toolkit Documentation", + "description": "Find installation instructions, launch highlights, programming guides, compiler documentation, API references, CUDA libraries, profiling tools, samples, and ...", + "position": 1, + "entity_type": "OrganicResult" + } + }, + { + "page_content": "", + "metadata": { + "url": "https://developer.nvidia.com/cuda-toolkit-archive", + "title": "CUDA Toolkit Archive", + "description": "Previous releases of the CUDA Toolkit, GPU Computing SDK, documentation and developer drivers can be found using the links below. Please select the release...", + "position": 2, + "entity_type": "OrganicResult" + } + }, + { + "page_content": "", + "metadata": { + "url": "https://cseweb.ucsd.edu/classes/wi15/cse262-a/static/cuda-5.5-doc/html/index.html", + "title": "CUDA Toolkit Documentation", + "description": "This guide presents established parallelization and optimization techniques and explains coding metaphors and idioms that can greatly simplify programming for...", + "position": 3, + "entity_type": "OrganicResult" + } + } + ] +} diff --git a/sources/nimble_web_search/tests/test_nimble_recorded_replay.py b/sources/nimble_web_search/tests/test_nimble_recorded_replay.py new file mode 100644 index 000000000..48dde685f --- /dev/null +++ b/sources/nimble_web_search/tests/test_nimble_recorded_replay.py @@ -0,0 +1,96 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Recorded-response replay tests — the credential-free test-mode path. + +Each file in ``fixtures/`` is a real ``NimbleSearchRetriever.ainvoke`` response +captured at the SDK boundary and redacted by construction (only the result +fields the provider consumes are kept; no headers, no auth material — see +``fixtures/README.md``). Replaying them through the registered function +exercises the full provider pipeline — config, retry wrapper, rendering, +truncation, escaping — deterministically, with no network and no key. + +These tests certify the same structural output contract as the key-gated live +integration test (``test_nimble_live_integration.py``), via the shared +``assert_search_contract`` fixture. +""" + +from unittest.mock import MagicMock + +import pytest +from nimble_web_search.register import NimbleWebSearchToolConfig +from nimble_web_search.register import nimble_web_search + + +@pytest.fixture(autouse=True) +def _replay_env(monkeypatch): + """Replay never talks to the network; a placeholder key satisfies registration.""" + monkeypatch.setenv("NIMBLE_API_KEY", "test-key-not-real") # pragma: allowlist secret + + +async def _run_provider(config: NimbleWebSearchToolConfig, question: str) -> str: + async with nimble_web_search(config, MagicMock()) as info: + return await info.single_fn(question) + + +class TestRecordedReplay: + @pytest.mark.parametrize("fixture_name", ["recorded_lite_response.json", "recorded_deep_response.json"]) + async def test_recorded_response_meets_output_contract( + self, fixture_name, fake_langchain_nimble, load_recorded_fixture, assert_search_contract + ): + payload, documents = load_recorded_fixture(fixture_name) + fake_langchain_nimble.ainvoke.return_value = documents + config = NimbleWebSearchToolConfig(**payload["_config"]) + + result = await _run_provider(config, payload["_query"]) + + blocks = assert_search_contract(result, config.max_results) + assert len(blocks) == len(documents) + + async def test_lite_fixture_renders_description_fallback( + self, fake_langchain_nimble, load_recorded_fixture, extract_document_blocks + ): + """lite results carry no page_content; the renderer must fall back to the description.""" + payload, documents = load_recorded_fixture("recorded_lite_response.json") + assert any(not doc.page_content for doc in documents), "fixture no longer covers the lite fallback" + fake_langchain_nimble.ainvoke.return_value = documents + + result = await _run_provider(NimbleWebSearchToolConfig(**payload["_config"]), payload["_query"]) + + for block, doc in zip(extract_document_blocks(result), documents, strict=True): + if not doc.page_content and doc.metadata["description"]: + assert doc.metadata["description"][:40] in block + + async def test_body_containing_separator_sequence_yields_exact_block_count( + self, fake_langchain_nimble, make_fake_document, assert_search_contract + ): + """Synthetic (not recorded): deep-mode markdown content can contain the + ``\\n\\n---\\n\\n`` joiner sequence itself; whole-block extraction must + still count documents exactly rather than mis-slicing on separators. + """ + documents = [ + make_fake_document( + url="https://example.com/a", + title="A", + page_content="intro\n\n---\n\n[Release Notes](notes/index.html)\n: description text", + ), + make_fake_document(url="https://example.com/b", title="B", description="plain snippet"), + ] + fake_langchain_nimble.ainvoke.return_value = documents + + result = await _run_provider(NimbleWebSearchToolConfig(), "q") + + blocks = assert_search_contract(result, max_results=5) + assert len(blocks) == 2 From f55be3b2ab7e6e054538dd42bae3f7e391d5df4b Mon Sep 17 00:00:00 2001 From: Kobi Kadosh Date: Thu, 9 Jul 2026 23:12:59 -0700 Subject: [PATCH 06/14] test(nimble_web_search): add key-gated live integration test One opt-in live API call (AIQ_NIMBLE_LIVE_TESTS=1 plus NIMBLE_API_KEY), bounded at 120 seconds, running the canned query "NVIDIA CUDA Toolkit documentation" with shipped defaults and asserting the structural output contract. Mirrors the opt-in gating idiom of the OpenSearch live tests. The test adds no retries of its own -- the provider's retry loop is the code under test, so a failure means three consecutive attempts failed. Co-Authored-By: Claude Fable 5 Signed-off-by: Kobi Kadosh --- .../tests/test_nimble_live_integration.py | 84 +++++++++++++++++++ 1 file changed, 84 insertions(+) create mode 100644 sources/nimble_web_search/tests/test_nimble_live_integration.py diff --git a/sources/nimble_web_search/tests/test_nimble_live_integration.py b/sources/nimble_web_search/tests/test_nimble_live_integration.py new file mode 100644 index 000000000..936e7014c --- /dev/null +++ b/sources/nimble_web_search/tests/test_nimble_live_integration.py @@ -0,0 +1,84 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Live Nimble integration test — the minimal repeatable CI check. + +Opt-in (mirrors ``tests/knowledge_layer_tests/test_opensearch_live.py``): set +``AIQ_NIMBLE_LIVE_TESTS=1`` and provide a real ``NIMBLE_API_KEY``. Cost per +run: exactly one API call, bounded at 120 seconds. + +The test performs no retries of its own — the provider's built-in retry loop +(``max_retries=3`` with backoff) is part of the code under test, so a failure +here means three consecutive provider attempts failed, which is precisely the +reliability signal CI should surface. + +Run: + + AIQ_NIMBLE_LIVE_TESTS=1 NIMBLE_API_KEY= \ + uv run pytest sources/nimble_web_search/tests -m integration -v +""" + +import asyncio +import os +from unittest.mock import MagicMock + +import pytest +from nimble_web_search.register import NimbleWebSearchToolConfig +from nimble_web_search.register import nimble_web_search + +# A concrete, time-invariant entity query that stays on-topic across time and +# regions — certified for repeated CI use. Keep in sync with +# fixtures/README.md (the recorded fixtures were captured with this query). +CANNED_QUERY = "NVIDIA CUDA Toolkit documentation" + + +def _env_bool(name: str, default: bool = False) -> bool: + """env bool.""" + value = os.environ.get(name) + if value is None: + return default + return value.lower() in {"1", "true", "yes", "on"} + + +pytestmark = [ + pytest.mark.integration, + pytest.mark.skipif( + not _env_bool("AIQ_NIMBLE_LIVE_TESTS"), + reason="Set AIQ_NIMBLE_LIVE_TESTS=1 to run live Nimble integration tests.", + ), + pytest.mark.skipif( + not os.environ.get("NIMBLE_API_KEY"), + reason="NIMBLE_API_KEY not set; live Nimble tests require a real key.", + ), +] + + +class TestLiveNimbleAPI: + async def test_live_lite_search_returns_parseable_documents(self, assert_search_contract): + """One live call with shipped defaults must satisfy the output contract. + + Success criteria (shared with the replay tests via + ``assert_search_contract``): a non-error, non-empty response containing + 1..max_results ```` blocks, every block with an http(s) href + and a title, every block XML-parseable, and at least one non-empty body. + Assertions are structural, never content-exact, so ordinary result + variation cannot flake the test. + """ + config = NimbleWebSearchToolConfig() # shipped defaults: lite / 5 / general / US / en + + async with nimble_web_search(config, MagicMock()) as info: + result = await asyncio.wait_for(info.single_fn(CANNED_QUERY), timeout=120) + + assert_search_contract(result, config.max_results) From b6cda6cee63e642a008bc57fe655e06e2ce15b39 Mon Sep 17 00:00:00 2001 From: Kobi Kadosh Date: Thu, 9 Jul 2026 23:12:59 -0700 Subject: [PATCH 07/14] docs(nimble_web_search): document test modes and CI success criteria README now describes the three test layers (mocked, recorded replay, key-gated live) with exact commands, the canned CI query, per-run cost (one API call, at most 120 seconds), and the structural success criteria. Refreshes stale test counts. Also corrects the country field example to an ISO 3166 code: the API accepts 'GB', not 'UK' (the latter returns a 400 client error). Co-Authored-By: Claude Fable 5 Signed-off-by: Kobi Kadosh --- sources/nimble_web_search/README.md | 17 +++++++++++++---- sources/nimble_web_search/src/register.py | 2 +- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/sources/nimble_web_search/README.md b/sources/nimble_web_search/README.md index be70c022d..6d5821a23 100644 --- a/sources/nimble_web_search/README.md +++ b/sources/nimble_web_search/README.md @@ -42,25 +42,34 @@ You can alternatively set `api_key` directly in the YAML (as a string). Both pat ## Test +The package ships three test layers; the first two are credential-free and run everywhere. + ```bash -# Unit tests (credential-free, mocked) +# 1. Mocked unit tests + recorded-response replay (credential-free; the default run) uv run pytest sources/nimble_web_search -v +# → 36 passed, 1 skipped (the live test, opt-in below) + +# 2. Live integration test (opt-in; exactly one API call, bounded at 120 s) +AIQ_NIMBLE_LIVE_TESTS=1 NIMBLE_API_KEY= \ + uv run pytest sources/nimble_web_search/tests -m integration -v # Lint uv run ruff check sources/nimble_web_search uv run ruff format --check sources/nimble_web_search ``` -The unit tests cover: config defaults / all fields / invalid `search_depth` rejection, missing-key stub + warn-once, key-from-config env hydration, successful render + description fallback, deep depth passthrough, query/content truncation, empty/error handling, retry-then-success, final-retry failure, 401, 403 enterprise-tier, non-default country/locale passthrough, and renderer behavior on titles containing special characters. +- **Mocked unit tests** (`test_nimble_register.py`) cover: config defaults / all fields / invalid `search_depth` rejection, missing-key stub + warn-once, key-from-config env hydration, successful render + description fallback, deep depth passthrough, query/content truncation, empty/error handling, retry-then-success, final-retry failure, 401, 403 enterprise-tier, non-default country/locale passthrough, and renderer behavior on titles containing special characters. +- **Recorded-response replay** (`test_nimble_recorded_replay.py`) replays real, redacted `NimbleSearchRetriever` responses ([`tests/fixtures/README.md`](tests/fixtures/README.md)) through the full provider pipeline — a deterministic test mode that needs no network and no key. +- **Live integration** (`test_nimble_live_integration.py`) runs the canned query `NVIDIA CUDA Toolkit documentation` once against the real API with the shipped defaults and asserts the structural output contract: a non-error response containing 1..`max_results` `` blocks, every block with an http(s) `href` and a title, every block XML-parseable, and at least one non-empty body. Assertions are structural — never content-exact — so ordinary result variation cannot flake the run. Suitable for CI: add `NIMBLE_API_KEY` as a repository secret and set `AIQ_NIMBLE_LIVE_TESTS=1` in the job. ## Verification Beyond the mocked unit tests above, verify the provider is fully integrated with AI-Q by walking the [adding-a-data-source checklist](../../docs/source/extending/adding-a-data-source.md): ```bash -# 1. Mocked unit tests pass (CI-safe, no credentials) +# 1. Mocked unit tests + recorded replay pass (CI-safe, no credentials) uv run pytest sources/nimble_web_search -q -# → 21 passed in <1s +# → 36 passed, 1 skipped in <1s # 2. NAT discovers the registered function nat info components --types function | grep nimble_web_search diff --git a/sources/nimble_web_search/src/register.py b/sources/nimble_web_search/src/register.py index 67c49f538..d0b3d693a 100644 --- a/sources/nimble_web_search/src/register.py +++ b/sources/nimble_web_search/src/register.py @@ -71,7 +71,7 @@ class NimbleWebSearchToolConfig(FunctionBaseConfig, name="nimble_web_search"): ) country: str = Field( default="US", - description="ISO country code passed to Nimble (e.g. 'US', 'UK', 'FR').", + description="ISO 3166 country code passed to Nimble (e.g. 'US', 'GB', 'FR').", ) locale: str = Field( default="en", From 29fba4aa16988ce36351015a319008fa2e10a211 Mon Sep 17 00:00:00 2001 From: Kobi Kadosh Date: Fri, 10 Jul 2026 04:01:07 -0700 Subject: [PATCH 08/14] fix(nimble_web_search): drop unresolvable-URL results and retry when none remain The live API intermittently returns results whose url is empty or a server-relative redirect token (e.g. "/goto?url=...") instead of a resolvable link -- observed in about 2% of calls during a 100-run soak of the shipped defaults. Rendering those results hands agents citations that cannot be followed. The provider now drops such results from a response and, when nothing usable remains, treats the response as transient so the retry loop re-queries. A truly empty result list keeps its existing non-retried behavior. Includes unit tests for the mixed, all-unresolvable (retry and exhaustion), and empty-response paths, and fixes the remaining non-ISO country example in the README. Co-Authored-By: Claude Fable 5 Signed-off-by: Kobi Kadosh --- sources/nimble_web_search/README.md | 6 +- sources/nimble_web_search/src/register.py | 16 +++ .../tests/test_nimble_result_filtering.py | 101 ++++++++++++++++++ 3 files changed, 120 insertions(+), 3 deletions(-) create mode 100644 sources/nimble_web_search/tests/test_nimble_result_filtering.py diff --git a/sources/nimble_web_search/README.md b/sources/nimble_web_search/README.md index 6d5821a23..e7cbc1dd3 100644 --- a/sources/nimble_web_search/README.md +++ b/sources/nimble_web_search/README.md @@ -47,7 +47,7 @@ The package ships three test layers; the first two are credential-free and run e ```bash # 1. Mocked unit tests + recorded-response replay (credential-free; the default run) uv run pytest sources/nimble_web_search -v -# → 36 passed, 1 skipped (the live test, opt-in below) +# → 40 passed, 1 skipped (the live test, opt-in below) # 2. Live integration test (opt-in; exactly one API call, bounded at 120 s) AIQ_NIMBLE_LIVE_TESTS=1 NIMBLE_API_KEY= \ @@ -69,7 +69,7 @@ Beyond the mocked unit tests above, verify the provider is fully integrated with ```bash # 1. Mocked unit tests + recorded replay pass (CI-safe, no credentials) uv run pytest sources/nimble_web_search -q -# → 36 passed, 1 skipped in <1s +# → 40 passed, 1 skipped in <1s # 2. NAT discovers the registered function nat info components --types function | grep nimble_web_search @@ -96,7 +96,7 @@ The provider exposes the following Nimble-specific surface. Defaults are tuned f | Result count | `max_results` | `5` | Range `1-100` (Nimble's documented cap). Soft cap (Nimble may return up to N+2; see Known limitations). | | Search depth | `search_depth` | `lite` | See the dedicated [Search depth](#search-depth) section below. | | Search focus | `focus` | `general` | Nimble focus mode: `general` (default, broad web/research), `news` (news-publisher sources ordered by recency — not a recency filter; older articles still appear), or domain-specific `location` / `shopping` / `geo` / `social`. Leave `general` for normal research; the LLM never selects focus, so general queries can't drift to `news`. | -| Localization — country | `country` | `US` | Two-letter country code (e.g. `FR`, `JP`, `UK`). Reaches the SDK constructor verbatim. | +| Localization — country | `country` | `US` | Two-letter ISO 3166 country code (e.g. `FR`, `JP`, `GB`). Reaches the SDK constructor verbatim. | | Localization — language | `locale` | `en` | ISO 639-1 language code (e.g. `fr`, `ja`). | | Per-result content size | `max_content_length` | `10000` chars | Truncates each result's body to N chars (3-char ellipsis included). Minimum `1`; set to `null` to disable truncation; omit to use default. | | Retries | `max_retries` | `3` | Exponential backoff on transient errors. Final failure surfaces a friendly per-status message (401, 403, generic). | diff --git a/sources/nimble_web_search/src/register.py b/sources/nimble_web_search/src/register.py index d0b3d693a..57bb98ae8 100644 --- a/sources/nimble_web_search/src/register.py +++ b/sources/nimble_web_search/src/register.py @@ -186,6 +186,22 @@ def _truncate_content(content: str) -> str: if not docs: raise ValueError("Search returned no results") + # The API intermittently returns results whose URL is empty or + # a server-relative redirect token (e.g. "/goto?url=...") + # instead of a resolvable link — observed in ~2% of calls + # during a 100-run soak. Such results break citation + # downstream, so drop them; if none remain, treat the response + # as transient so the retry loop re-queries instead of + # rendering unusable documents. + def _has_resolvable_url(doc) -> bool: + url = str((getattr(doc, "metadata", {}) or {}).get("url", "") or "") + return bool(url) and not url.startswith("/") + + usable_docs = [doc for doc in docs if _has_resolvable_url(doc)] + if not usable_docs: + raise RuntimeError("Search returned only results without resolvable URLs") + docs = usable_docs + def _render(doc) -> str: metadata = getattr(doc, "metadata", {}) or {} url = metadata.get("url", "") or "" diff --git a/sources/nimble_web_search/tests/test_nimble_result_filtering.py b/sources/nimble_web_search/tests/test_nimble_result_filtering.py new file mode 100644 index 000000000..e30f2481c --- /dev/null +++ b/sources/nimble_web_search/tests/test_nimble_result_filtering.py @@ -0,0 +1,101 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for unresolvable-URL result filtering. + +The live API intermittently returns results whose ``url`` is a redirect token +(e.g. ``/goto?url=...``) or empty instead of a resolvable link. The provider +drops such results and, when a response contains nothing usable, treats it as +transient so the retry loop re-queries rather than rendering documents that +break citation downstream. +""" + +from unittest.mock import MagicMock + +import pytest +from nimble_web_search.register import NimbleWebSearchToolConfig +from nimble_web_search.register import nimble_web_search + + +async def _no_sleep(_): + return None + + +@pytest.fixture(autouse=True) +def _env(monkeypatch): + monkeypatch.setenv("NIMBLE_API_KEY", "test-key-not-real") # pragma: allowlist secret + + +async def _run_provider(config: NimbleWebSearchToolConfig, question: str) -> str: + async with nimble_web_search(config, MagicMock()) as info: + return await info.single_fn(question) + + +class TestUnresolvableUrlFiltering: + async def test_mixed_response_renders_only_resolvable_results( + self, fake_langchain_nimble, make_fake_document, assert_search_contract + ): + fake_langchain_nimble.ainvoke.return_value = [ + make_fake_document(url="/goto?url=CAESSwHuR6pN", title="Token", description="redirect token"), + make_fake_document(url="https://example.com/a", title="A", description="fine"), + make_fake_document(url="", title="Empty", description="no url at all"), + make_fake_document(url="https://example.com/b", title="B", description="also fine"), + ] + + result = await _run_provider(NimbleWebSearchToolConfig(), "q") + + blocks = assert_search_contract(result, max_results=5) + assert len(blocks) == 2 + assert "example.com/a" in result and "example.com/b" in result + assert "/goto?url=" not in result + + async def test_all_unresolvable_response_is_retried( + self, fake_langchain_nimble, make_fake_document, assert_search_contract, monkeypatch + ): + monkeypatch.setattr("nimble_web_search.register.asyncio.sleep", _no_sleep) + fake_langchain_nimble.ainvoke.side_effect = [ + [make_fake_document(url="/goto?url=abc", title="T1", description="d")], + [make_fake_document(url="https://example.com/ok", title="OK", description="d")], + ] + + result = await _run_provider(NimbleWebSearchToolConfig(max_retries=3), "q") + + assert_search_contract(result, max_results=5) + assert fake_langchain_nimble.ainvoke.call_count == 2 + + async def test_all_unresolvable_on_every_attempt_returns_error( + self, fake_langchain_nimble, make_fake_document, monkeypatch + ): + monkeypatch.setattr("nimble_web_search.register.asyncio.sleep", _no_sleep) + fake_langchain_nimble.ainvoke.return_value = [ + make_fake_document(url="/goto?url=abc", title="T", description="d"), + ] + + config = NimbleWebSearchToolConfig(max_retries=2) + result = await _run_provider(config, "q") + + assert result.startswith("Error:") + assert "resolvable URLs" in result + assert fake_langchain_nimble.ainvoke.call_count == config.max_retries + + async def test_truly_empty_response_still_short_circuits(self, fake_langchain_nimble, monkeypatch): + """An empty result list stays a non-retried sentinel (unchanged behavior).""" + monkeypatch.setattr("nimble_web_search.register.asyncio.sleep", _no_sleep) + fake_langchain_nimble.ainvoke.return_value = [] + + result = await _run_provider(NimbleWebSearchToolConfig(max_retries=3), "q") + + assert result == "Search returned no results" + assert fake_langchain_nimble.ainvoke.call_count == 1 From 6ac63275806775806b1b4bbd214dc35844401bdf Mon Sep 17 00:00:00 2001 From: Kobi Kadosh Date: Tue, 14 Jul 2026 01:26:06 -0700 Subject: [PATCH 09/14] fix(nimble_web_search): pass API key to the retriever config, not os.environ Addresses review feedback. Resolve the configured key from tool_config and pass it directly to NimbleSearchRetriever (its api_key field) instead of writing it into os.environ; when no config key is set, the SDK reads NIMBLE_API_KEY from the environment itself. This keeps the secret out of process-global state, so it can't leak to child processes or race across concurrent workflows, and removes the un-cleaned env mutation the config test relied on. Also hoist the result helpers (_has_resolvable_url, _render) above the retry loop so they aren't re-allocated per attempt. Co-Authored-By: Claude Fable 5 Signed-off-by: Kobi Kadosh --- sources/nimble_web_search/src/register.py | 82 +++++++++++-------- .../tests/test_nimble_register.py | 8 +- 2 files changed, 53 insertions(+), 37 deletions(-) diff --git a/sources/nimble_web_search/src/register.py b/sources/nimble_web_search/src/register.py index 57bb98ae8..1461dabdb 100644 --- a/sources/nimble_web_search/src/register.py +++ b/sources/nimble_web_search/src/register.py @@ -110,10 +110,12 @@ async def nimble_web_search( """ from langchain_nimble import NimbleSearchRetriever - if not os.environ.get("NIMBLE_API_KEY") and tool_config.api_key: - os.environ["NIMBLE_API_KEY"] = tool_config.api_key.get_secret_value() + # Resolve the key from config or environment. A config-supplied key is passed + # straight to the retriever below (never written to os.environ), so the secret + # does not enter process-global state where it could leak to child processes. + api_key = tool_config.api_key - if not os.environ.get("NIMBLE_API_KEY"): + if api_key is None and not os.environ.get("NIMBLE_API_KEY"): global _missing_key_warned if not _missing_key_warned: logger.warning( @@ -145,13 +147,19 @@ async def _nimble_web_search_stub(question: str) -> str: # the retriever. `focus` is passed explicitly so the default is provably # "general" rather than relying on the SDK's (unvalidated) field default. # `include_answer` is intentionally not exposed in this initial integration. - retriever = NimbleSearchRetriever( - max_results=tool_config.max_results, - search_depth=tool_config.search_depth, - focus=tool_config.focus, - country=tool_config.country, - locale=tool_config.locale, - ) + retriever_kwargs = { + "max_results": tool_config.max_results, + "search_depth": tool_config.search_depth, + "focus": tool_config.focus, + "country": tool_config.country, + "locale": tool_config.locale, + } + # Pass the configured key through the SDK's `api_key` field instead of the + # environment; when no config key is set, the SDK resolves NIMBLE_API_KEY + # from the environment itself. + if api_key is not None: + retriever_kwargs["api_key"] = api_key + retriever = NimbleSearchRetriever(**retriever_kwargs) async def _nimble_web_search(question: str) -> str: """Search the web with Nimble and return relevant sources for a question. @@ -179,6 +187,33 @@ def _truncate_content(content: str) -> str: return content[: limit - 3] + "..." return content + def _has_resolvable_url(doc) -> bool: + """Return True when the result carries a resolvable absolute URL. + + The API intermittently returns results whose URL is empty or a + server-relative redirect token (e.g. "/goto?url=...") -- observed in + ~2% of calls during a 100-run soak. Those break citation downstream, + so they are dropped by the caller. + """ + url = str((getattr(doc, "metadata", {}) or {}).get("url", "") or "") + return bool(url) and not url.startswith("/") + + def _render(doc) -> str: + """Render one result as an escaped XML ```` block.""" + metadata = getattr(doc, "metadata", {}) or {} + url = metadata.get("url", "") or "" + title = metadata.get("title", "") or "" + page_content = getattr(doc, "page_content", "") or "" + description = metadata.get("description", "") or "" + body = _truncate_content(page_content if page_content else description) + # Escape untrusted API fields so they can't break the + # markup or inject into downstream renderers/parsers. + return ( + f'\n' + f"\n{html.escape(title)}\n\n" + f"{html.escape(body)}\n" + ) + for attempt in range(tool_config.max_retries): try: docs = await retriever.ainvoke(question) @@ -186,37 +221,14 @@ def _truncate_content(content: str) -> str: if not docs: raise ValueError("Search returned no results") - # The API intermittently returns results whose URL is empty or - # a server-relative redirect token (e.g. "/goto?url=...") - # instead of a resolvable link — observed in ~2% of calls - # during a 100-run soak. Such results break citation - # downstream, so drop them; if none remain, treat the response - # as transient so the retry loop re-queries instead of + # Drop results without a resolvable URL; if none remain, treat the + # response as transient so the retry loop re-queries instead of # rendering unusable documents. - def _has_resolvable_url(doc) -> bool: - url = str((getattr(doc, "metadata", {}) or {}).get("url", "") or "") - return bool(url) and not url.startswith("/") - usable_docs = [doc for doc in docs if _has_resolvable_url(doc)] if not usable_docs: raise RuntimeError("Search returned only results without resolvable URLs") docs = usable_docs - def _render(doc) -> str: - metadata = getattr(doc, "metadata", {}) or {} - url = metadata.get("url", "") or "" - title = metadata.get("title", "") or "" - page_content = getattr(doc, "page_content", "") or "" - description = metadata.get("description", "") or "" - body = _truncate_content(page_content if page_content else description) - # Escape untrusted API fields so they can't break the - # markup or inject into downstream renderers/parsers. - return ( - f'\n' - f"\n{html.escape(title)}\n\n" - f"{html.escape(body)}\n" - ) - web_search_results = "\n\n---\n\n".join(_render(doc) for doc in docs) return web_search_results if web_search_results else "Search returned no results" diff --git a/sources/nimble_web_search/tests/test_nimble_register.py b/sources/nimble_web_search/tests/test_nimble_register.py index 77ad75871..f1f867c33 100644 --- a/sources/nimble_web_search/tests/test_nimble_register.py +++ b/sources/nimble_web_search/tests/test_nimble_register.py @@ -177,7 +177,7 @@ async def test_warn_once_when_key_missing(self, caplog): class TestNimbleWebSearchLive: - async def test_api_key_from_config_sets_env(self, fake_langchain_nimble): + async def test_api_key_from_config_passed_to_retriever(self, fake_langchain_nimble): fake_langchain_nimble.ainvoke.return_value = [ _FakeDoc(url="https://a.example", title="A", page_content="body a") ] @@ -187,7 +187,11 @@ async def test_api_key_from_config_sets_env(self, fake_langchain_nimble): async with nimble_web_search(config, builder) as info: out = await info.single_fn("question") - assert os.environ.get("NIMBLE_API_KEY") == "sk-from-config" + # The configured key is passed straight to the retriever, never written + # to the process-global environment. + retriever_cls = sys.modules["langchain_nimble"].NimbleSearchRetriever + assert retriever_cls.call_args.kwargs["api_key"].get_secret_value() == "sk-from-config" + assert os.environ.get("NIMBLE_API_KEY") is None assert "https://a.example" in out assert "body a" in out From 28a538d433774e820d1928031d349096fef31e46 Mon Sep 17 00:00:00 2001 From: Kobi Kadosh Date: Tue, 14 Jul 2026 01:26:06 -0700 Subject: [PATCH 10/14] fix(nimble_web_search): install runtime deps from the lockfile; ISO country example The Dockerfile pinned langchain-nimble==3.0.0 and nimble-python==0.18.0 in a separate uv pip install after 'uv sync --frozen', which resolved them outside the lockfile and pinned nimble-python below the locked version. The workspace sync already installs both from the frozen lock (as it does for the Exa and Tavily langchain deps), so drop the redundant line. Also correct the configuration-reference country example to the ISO 3166 code GB (UK is not valid), matching the provider docs and implementation. Co-Authored-By: Claude Fable 5 Signed-off-by: Kobi Kadosh --- deploy/Dockerfile | 1 - docs/source/customization/configuration-reference.md | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/deploy/Dockerfile b/deploy/Dockerfile index 43a507115..a3b6810bf 100644 --- a/deploy/Dockerfile +++ b/deploy/Dockerfile @@ -84,7 +84,6 @@ RUN uv pip install --no-deps -e . \ && uv pip install --no-deps -e ./sources/tavily_web_search \ && uv pip install --no-deps -e ./sources/exa_web_search \ && uv pip install --no-deps -e ./sources/nimble_web_search \ - && uv pip install langchain-nimble==3.0.0 nimble-python==0.18.0 \ && uv pip install --no-deps -e "./sources/knowledge_layer[all]" \ && uv pip install --no-deps -e ./frontends/aiq_api \ && uv pip install "psycopg[binary]>=3.0.0" diff --git a/docs/source/customization/configuration-reference.md b/docs/source/customization/configuration-reference.md index 84df679b7..293981870 100644 --- a/docs/source/customization/configuration-reference.md +++ b/docs/source/customization/configuration-reference.md @@ -226,7 +226,7 @@ functions: | `max_retries` | `int` | `3` | Number of retry attempts on search failure. | | `search_depth` | `str` | `"lite"` | Nimble search depth. See options below. | | `focus` | `str` | `"general"` | Nimble focus mode. See options below. | -| `country` | `str` | `"US"` | ISO country code passed to Nimble (e.g. `US`, `UK`, `FR`). | +| `country` | `str` | `"US"` | ISO 3166 country code passed to Nimble (e.g. `US`, `GB`, `FR`). | | `locale` | `str` | `"en"` | Language/locale passed to Nimble (e.g. `en`, `fr`, `es`). | | `max_content_length` | `int` | `10000` | Max characters per result's page content. Set to `None` to disable truncation. | From bf210f0972bc894c70bc4c0d99edeae1274ca20c Mon Sep 17 00:00:00 2001 From: Kobi Kadosh Date: Tue, 14 Jul 2026 01:46:22 -0700 Subject: [PATCH 11/14] fix(nimble_web_search): bound max_retries and cap retry backoff Addresses review feedback on unbounded retry behavior. Add an upper bound of 10 to max_retries (ge=1, le=10) and cap the exponential backoff at 30s (min(2**attempt, 30)) so a misconfigured max_retries cannot produce an unbounded wait. Add a config-validation case for the new upper bound. Also mark max_content_length as int | None in the configuration reference, matching the field (None disables truncation). Co-Authored-By: Claude Fable 5 Signed-off-by: Kobi Kadosh --- docs/source/customization/configuration-reference.md | 2 +- sources/nimble_web_search/src/register.py | 8 ++++++-- sources/nimble_web_search/tests/test_nimble_register.py | 1 + 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/docs/source/customization/configuration-reference.md b/docs/source/customization/configuration-reference.md index 293981870..05ed5a27b 100644 --- a/docs/source/customization/configuration-reference.md +++ b/docs/source/customization/configuration-reference.md @@ -228,7 +228,7 @@ functions: | `focus` | `str` | `"general"` | Nimble focus mode. See options below. | | `country` | `str` | `"US"` | ISO 3166 country code passed to Nimble (e.g. `US`, `GB`, `FR`). | | `locale` | `str` | `"en"` | Language/locale passed to Nimble (e.g. `en`, `fr`, `es`). | -| `max_content_length` | `int` | `10000` | Max characters per result's page content. Set to `None` to disable truncation. | +| `max_content_length` | `int \| None` | `10000` | Max characters per result's page content. Set to `None` to disable truncation. | **`search_depth` options:** diff --git a/sources/nimble_web_search/src/register.py b/sources/nimble_web_search/src/register.py index 1461dabdb..3bbec3501 100644 --- a/sources/nimble_web_search/src/register.py +++ b/sources/nimble_web_search/src/register.py @@ -46,7 +46,9 @@ class NimbleWebSearchToolConfig(FunctionBaseConfig, name="nimble_web_search"): description="Maximum number of search results to return (Nimble accepts 1-100).", ) api_key: SecretStr | None = Field(default=None, description="The API key for the Nimble service") - max_retries: int = Field(default=3, ge=1, description="Maximum number of retries for the search request") + max_retries: int = Field( + default=3, ge=1, le=10, description="Maximum number of retries for the search request (1-10)." + ) search_depth: Literal["lite", "fast", "deep"] = Field( default="lite", description=( @@ -252,7 +254,9 @@ def _render(doc) -> str: # Transient error: retry with backoff, or give up on the last attempt. if attempt == tool_config.max_retries - 1: return f"Error: Web search failed - {error_msg}" - await asyncio.sleep(2**attempt) + # Cap the exponential delay so a large max_retries can't produce + # an unbounded wait. + await asyncio.sleep(min(2**attempt, 30)) return "Error: Search failed after all retries" diff --git a/sources/nimble_web_search/tests/test_nimble_register.py b/sources/nimble_web_search/tests/test_nimble_register.py index f1f867c33..8e8da1699 100644 --- a/sources/nimble_web_search/tests/test_nimble_register.py +++ b/sources/nimble_web_search/tests/test_nimble_register.py @@ -131,6 +131,7 @@ def test_no_include_answer_field(self): ("max_results", 0), # below ge=1 ("max_results", 101), # above le=100 (Nimble's documented cap) ("max_retries", 0), # below ge=1 + ("max_retries", 11), # above le=10 ("max_content_length", 0), # below ge=1 (use None to disable truncation) ], ) From f66c5911c0bcb8f13fa882cecd7fae7215102352 Mon Sep 17 00:00:00 2001 From: Kyle Zheng Date: Tue, 14 Jul 2026 13:48:27 -0700 Subject: [PATCH 12/14] fix(nimble_web_search): tighten setup guidance and URL handling Signed-off-by: Kyle Zheng --- .../customization/configuration-reference.md | 2 +- docs/source/get-started/installation.md | 2 +- sources/nimble_web_search/README.md | 12 ++++---- sources/nimble_web_search/src/register.py | 10 +++++-- .../tests/test_nimble_register.py | 28 ++++++++++++------- .../tests/test_nimble_result_filtering.py | 19 +++++++++++++ 6 files changed, 54 insertions(+), 19 deletions(-) diff --git a/docs/source/customization/configuration-reference.md b/docs/source/customization/configuration-reference.md index 05ed5a27b..cee0c5ffb 100644 --- a/docs/source/customization/configuration-reference.md +++ b/docs/source/customization/configuration-reference.md @@ -213,7 +213,7 @@ functions: max_results: 5 max_content_length: 10000 - deep_web_search_tool: + advanced_web_search_tool: _type: nimble_web_search max_results: 5 search_depth: deep diff --git a/docs/source/get-started/installation.md b/docs/source/get-started/installation.md index 3ae9f5ff8..04f5d4bd2 100644 --- a/docs/source/get-started/installation.md +++ b/docs/source/get-started/installation.md @@ -137,7 +137,7 @@ Then edit `deploy/.env` and fill in your keys. |----------|----------|---------| | `TAVILY_API_KEY` | [Tavily](https://tavily.com/) | Web search (Tavily provider) | | `EXA_API_KEY` | [Exa](https://exa.ai/) | Web search (Exa provider) | -| `NIMBLE_API_KEY` | [Nimble](https://nimbleway.com/) | Web search (Nimble provider) | +| `NIMBLE_API_KEY` | [Nimble API keys](https://docs.nimbleway.com/nimble-sdk/admin/account-management) | Web search (Nimble provider) | | `SERPER_API_KEY` | [Serper](https://serper.dev/) | Google Scholar paper search with `provider: serper` (the default) | | `SERPAPI_API_KEY` | [SerpAPI](https://serpapi.com/) | Google Scholar paper search with `provider: serpapi` | | `SEARCHAPI_API_KEY` | [SearchAPI](https://www.searchapi.io/) | Google Scholar paper search with `provider: searchapi` | diff --git a/sources/nimble_web_search/README.md b/sources/nimble_web_search/README.md index e7cbc1dd3..e005cdac3 100644 --- a/sources/nimble_web_search/README.md +++ b/sources/nimble_web_search/README.md @@ -38,6 +38,8 @@ See [`docs/source/customization/configuration-reference.md`](../../docs/source/c NIMBLE_API_KEY=... ``` +Create a key in the [Nimble dashboard](https://docs.nimbleway.com/nimble-sdk/admin/account-management): open **Account Settings > API Keys**, select **Create New API Key**, and copy the key when it is displayed. Nimble displays a new key only once, so store it securely. + You can alternatively set `api_key` directly in the YAML (as a string). Both paths use the standard `nat.data_models.function.FunctionBaseConfig` `SecretStr` handling — the key is not logged. ## Test @@ -47,7 +49,7 @@ The package ships three test layers; the first two are credential-free and run e ```bash # 1. Mocked unit tests + recorded-response replay (credential-free; the default run) uv run pytest sources/nimble_web_search -v -# → 40 passed, 1 skipped (the live test, opt-in below) +# → credential-free tests pass; the opt-in live test is skipped # 2. Live integration test (opt-in; exactly one API call, bounded at 120 s) AIQ_NIMBLE_LIVE_TESTS=1 NIMBLE_API_KEY= \ @@ -58,7 +60,7 @@ uv run ruff check sources/nimble_web_search uv run ruff format --check sources/nimble_web_search ``` -- **Mocked unit tests** (`test_nimble_register.py`) cover: config defaults / all fields / invalid `search_depth` rejection, missing-key stub + warn-once, key-from-config env hydration, successful render + description fallback, deep depth passthrough, query/content truncation, empty/error handling, retry-then-success, final-retry failure, 401, 403 enterprise-tier, non-default country/locale passthrough, and renderer behavior on titles containing special characters. +- **Mocked unit tests** (`test_nimble_register.py`) cover: config defaults / all fields / invalid `search_depth` rejection, missing-key stub + warn-once, direct config-key passthrough to the SDK, successful render + description fallback, deep depth passthrough, query/content truncation, empty/error handling, retry-then-success, final-retry failure, 401, 403 enterprise-tier, non-default country/locale passthrough, and renderer behavior on titles containing special characters. - **Recorded-response replay** (`test_nimble_recorded_replay.py`) replays real, redacted `NimbleSearchRetriever` responses ([`tests/fixtures/README.md`](tests/fixtures/README.md)) through the full provider pipeline — a deterministic test mode that needs no network and no key. - **Live integration** (`test_nimble_live_integration.py`) runs the canned query `NVIDIA CUDA Toolkit documentation` once against the real API with the shipped defaults and asserts the structural output contract: a non-error response containing 1..`max_results` `` blocks, every block with an http(s) `href` and a title, every block XML-parseable, and at least one non-empty body. Assertions are structural — never content-exact — so ordinary result variation cannot flake the run. Suitable for CI: add `NIMBLE_API_KEY` as a repository secret and set `AIQ_NIMBLE_LIVE_TESTS=1` in the job. @@ -69,7 +71,7 @@ Beyond the mocked unit tests above, verify the provider is fully integrated with ```bash # 1. Mocked unit tests + recorded replay pass (CI-safe, no credentials) uv run pytest sources/nimble_web_search -q -# → 40 passed, 1 skipped in <1s +# → credential-free tests pass; the opt-in live test is skipped # 2. NAT discovers the registered function nat info components --types function | grep nimble_web_search @@ -100,9 +102,9 @@ The provider exposes the following Nimble-specific surface. Defaults are tuned f | Localization — language | `locale` | `en` | ISO 639-1 language code (e.g. `fr`, `ja`). | | Per-result content size | `max_content_length` | `10000` chars | Truncates each result's body to N chars (3-char ellipsis included). Minimum `1`; set to `null` to disable truncation; omit to use default. | | Retries | `max_retries` | `3` | Exponential backoff on transient errors. Final failure surfaces a friendly per-status message (401, 403, generic). | -| Auth | `api_key` / `NIMBLE_API_KEY` | env or config | `pydantic.SecretStr`; never logged. Config-side `api_key` hydrates the env var so the underlying SDK can read it. | +| Auth | `api_key` / `NIMBLE_API_KEY` | env or config | `pydantic.SecretStr`; never logged. A config-side `api_key` is passed directly to the SDK without modifying the process environment. | -Each `` block in the rendered output carries an `entity_type` (`"OrganicResult"` for SERP results). The `include_answer=True` capability — which would produce an `entity_type="answer"` block first — is intentionally **not exposed** in this initial integration. See [Known limitations](#known-limitations). +Each rendered `` block contains the result URL, title, and body. Other provider metadata, including `entity_type`, is not included. The `include_answer=True` capability is intentionally **not exposed** in this initial integration. See [Known limitations](#known-limitations). ## Search depth diff --git a/sources/nimble_web_search/src/register.py b/sources/nimble_web_search/src/register.py index 3bbec3501..75096d89c 100644 --- a/sources/nimble_web_search/src/register.py +++ b/sources/nimble_web_search/src/register.py @@ -19,6 +19,7 @@ import os from collections.abc import AsyncGenerator from typing import Literal +from urllib.parse import urlparse from pydantic import Field from pydantic import SecretStr @@ -132,7 +133,8 @@ async def _nimble_web_search_stub(question: str) -> str: return ( "Error: Nimble web search is unavailable because NIMBLE_API_KEY is not set.\n" "To enable this tool:\n" - "1. Get an API key from https://nimbleway.com/\n" + "1. Get an API key from " + "https://docs.nimbleway.com/nimble-sdk/admin/account-management\n" "2. Set the API key in your environment or in your .env file\n" "3. Restart the application" ) @@ -198,7 +200,11 @@ def _has_resolvable_url(doc) -> bool: so they are dropped by the caller. """ url = str((getattr(doc, "metadata", {}) or {}).get("url", "") or "") - return bool(url) and not url.startswith("/") + try: + parsed = urlparse(url) + return parsed.scheme.lower() in {"http", "https"} and bool(parsed.hostname) + except ValueError: + return False def _render(doc) -> str: """Render one result as an escaped XML ```` block.""" diff --git a/sources/nimble_web_search/tests/test_nimble_register.py b/sources/nimble_web_search/tests/test_nimble_register.py index 8e8da1699..69a0c2b3f 100644 --- a/sources/nimble_web_search/tests/test_nimble_register.py +++ b/sources/nimble_web_search/tests/test_nimble_register.py @@ -230,7 +230,9 @@ async def test_description_used_when_page_content_empty(self, fake_langchain_nim async def test_search_depth_deep_passes_through(self, fake_langchain_nimble, monkeypatch): monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") - fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="u", title="t", page_content="body")] + fake_langchain_nimble.ainvoke.return_value = [ + _FakeDoc(url="https://example.com", title="t", page_content="body") + ] config = NimbleWebSearchToolConfig(search_depth="deep") builder = MagicMock() @@ -255,7 +257,7 @@ async def test_focus_defaults_to_general(self, fake_langchain_nimble, monkeypatc so general research queries never silently use a news/other focus. """ monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") - fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="u", title="t", page_content="b")] + fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="https://example.com", title="t", page_content="b")] config = NimbleWebSearchToolConfig() builder = MagicMock() @@ -271,7 +273,7 @@ async def test_focus_news_passes_through_when_configured(self, fake_langchain_ni drift to news on their own.) """ monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") - fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="u", title="t", page_content="b")] + fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="https://example.com", title="t", page_content="b")] config = NimbleWebSearchToolConfig(focus="news") builder = MagicMock() @@ -283,7 +285,9 @@ async def test_focus_news_passes_through_when_configured(self, fake_langchain_ni async def test_truncates_long_query(self, fake_langchain_nimble, monkeypatch): monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") - fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="u", title="t", page_content="body")] + fake_langchain_nimble.ainvoke.return_value = [ + _FakeDoc(url="https://example.com", title="t", page_content="body") + ] config = NimbleWebSearchToolConfig() builder = MagicMock() @@ -297,7 +301,9 @@ async def test_truncates_long_query(self, fake_langchain_nimble, monkeypatch): async def test_truncates_content(self, fake_langchain_nimble, monkeypatch): monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") - fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="u", title="t", page_content="abcdefghijklmnop")] + fake_langchain_nimble.ainvoke.return_value = [ + _FakeDoc(url="https://example.com", title="t", page_content="abcdefghijklmnop") + ] config = NimbleWebSearchToolConfig(max_content_length=8) builder = MagicMock() @@ -312,7 +318,9 @@ async def test_truncates_content_small_limit_no_negative_slice(self, fake_langch result never exceeds the configured budget (the ellipsis needs 3 chars of headroom). """ monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") - fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="u", title="t", page_content="abcdefghij")] + fake_langchain_nimble.ainvoke.return_value = [ + _FakeDoc(url="https://example.com", title="t", page_content="abcdefghij") + ] config = NimbleWebSearchToolConfig(max_content_length=2) builder = MagicMock() @@ -340,7 +348,7 @@ async def test_retries_then_succeeds(self, fake_langchain_nimble, monkeypatch): fake_langchain_nimble.ainvoke.side_effect = [ RuntimeError("transient"), - [_FakeDoc(url="u", title="t", page_content="ok")], + [_FakeDoc(url="https://example.com", title="t", page_content="ok")], ] config = NimbleWebSearchToolConfig(max_retries=3) @@ -422,7 +430,7 @@ async def test_non_transient_errors_short_circuit_without_retry(self, fake_langc async def test_non_default_country_passthrough(self, fake_langchain_nimble, monkeypatch): """VERIFIES: NimbleWebSearchToolConfig(country="FR") forwards country='FR' to the SDK.""" monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") - fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="u", title="t", page_content="b")] + fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="https://example.com", title="t", page_content="b")] config = NimbleWebSearchToolConfig(country="FR") builder = MagicMock() @@ -435,7 +443,7 @@ async def test_non_default_country_passthrough(self, fake_langchain_nimble, monk async def test_non_default_locale_passthrough(self, fake_langchain_nimble, monkeypatch): """VERIFIES: NimbleWebSearchToolConfig(locale="fr") forwards locale='fr' to the SDK.""" monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") - fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="u", title="t", page_content="b")] + fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="https://example.com", title="t", page_content="b")] config = NimbleWebSearchToolConfig(locale="fr") builder = MagicMock() @@ -448,7 +456,7 @@ async def test_non_default_locale_passthrough(self, fake_langchain_nimble, monke async def test_country_locale_combined_passthrough(self, fake_langchain_nimble, monkeypatch): """VERIFIES: Both country and locale non-defaults are forwarded together to the SDK.""" monkeypatch.setenv("NIMBLE_API_KEY", "sk-env") - fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="u", title="t", page_content="b")] + fake_langchain_nimble.ainvoke.return_value = [_FakeDoc(url="https://example.com", title="t", page_content="b")] config = NimbleWebSearchToolConfig(country="JP", locale="ja") builder = MagicMock() diff --git a/sources/nimble_web_search/tests/test_nimble_result_filtering.py b/sources/nimble_web_search/tests/test_nimble_result_filtering.py index e30f2481c..356fd421f 100644 --- a/sources/nimble_web_search/tests/test_nimble_result_filtering.py +++ b/sources/nimble_web_search/tests/test_nimble_result_filtering.py @@ -61,6 +61,25 @@ async def test_mixed_response_renders_only_resolvable_results( assert "example.com/a" in result and "example.com/b" in result assert "/goto?url=" not in result + async def test_non_http_or_hostless_urls_are_filtered( + self, fake_langchain_nimble, make_fake_document, assert_search_contract + ): + fake_langchain_nimble.ainvoke.return_value = [ + make_fake_document(url="javascript:alert(1)", title="Script", description="invalid scheme"), + make_fake_document(url="example.com/no-scheme", title="No scheme", description="relative URL"), + make_fake_document(url="ftp://example.com/file", title="FTP", description="invalid scheme"), + make_fake_document(url="https:///missing-host", title="No host", description="invalid absolute URL"), + make_fake_document(url="https://example.com/ok", title="OK", description="valid URL"), + ] + + result = await _run_provider(NimbleWebSearchToolConfig(), "q") + + blocks = assert_search_contract(result, max_results=5) + assert len(blocks) == 1 + assert "https://example.com/ok" in result + assert "javascript:" not in result + assert "ftp://" not in result + async def test_all_unresolvable_response_is_retried( self, fake_langchain_nimble, make_fake_document, assert_search_contract, monkeypatch ): From da967653942992898504d3834f7bc3b8f3a93029 Mon Sep 17 00:00:00 2001 From: Kyle Zheng Date: Tue, 14 Jul 2026 13:53:18 -0700 Subject: [PATCH 13/14] docs(nimble_web_search): use provider homepage for setup Signed-off-by: Kyle Zheng --- docs/source/get-started/installation.md | 2 +- sources/nimble_web_search/README.md | 2 +- sources/nimble_web_search/src/register.py | 3 +-- 3 files changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/source/get-started/installation.md b/docs/source/get-started/installation.md index 04f5d4bd2..3ae9f5ff8 100644 --- a/docs/source/get-started/installation.md +++ b/docs/source/get-started/installation.md @@ -137,7 +137,7 @@ Then edit `deploy/.env` and fill in your keys. |----------|----------|---------| | `TAVILY_API_KEY` | [Tavily](https://tavily.com/) | Web search (Tavily provider) | | `EXA_API_KEY` | [Exa](https://exa.ai/) | Web search (Exa provider) | -| `NIMBLE_API_KEY` | [Nimble API keys](https://docs.nimbleway.com/nimble-sdk/admin/account-management) | Web search (Nimble provider) | +| `NIMBLE_API_KEY` | [Nimble](https://nimbleway.com/) | Web search (Nimble provider) | | `SERPER_API_KEY` | [Serper](https://serper.dev/) | Google Scholar paper search with `provider: serper` (the default) | | `SERPAPI_API_KEY` | [SerpAPI](https://serpapi.com/) | Google Scholar paper search with `provider: serpapi` | | `SEARCHAPI_API_KEY` | [SearchAPI](https://www.searchapi.io/) | Google Scholar paper search with `provider: searchapi` | diff --git a/sources/nimble_web_search/README.md b/sources/nimble_web_search/README.md index e005cdac3..fbe915eda 100644 --- a/sources/nimble_web_search/README.md +++ b/sources/nimble_web_search/README.md @@ -38,7 +38,7 @@ See [`docs/source/customization/configuration-reference.md`](../../docs/source/c NIMBLE_API_KEY=... ``` -Create a key in the [Nimble dashboard](https://docs.nimbleway.com/nimble-sdk/admin/account-management): open **Account Settings > API Keys**, select **Create New API Key**, and copy the key when it is displayed. Nimble displays a new key only once, so store it securely. +Visit [Nimble](https://nimbleway.com/) to create an account, obtain an API key, and access the provider's setup guides. Store the key securely. You can alternatively set `api_key` directly in the YAML (as a string). Both paths use the standard `nat.data_models.function.FunctionBaseConfig` `SecretStr` handling — the key is not logged. diff --git a/sources/nimble_web_search/src/register.py b/sources/nimble_web_search/src/register.py index 75096d89c..6ca1cb706 100644 --- a/sources/nimble_web_search/src/register.py +++ b/sources/nimble_web_search/src/register.py @@ -133,8 +133,7 @@ async def _nimble_web_search_stub(question: str) -> str: return ( "Error: Nimble web search is unavailable because NIMBLE_API_KEY is not set.\n" "To enable this tool:\n" - "1. Get an API key from " - "https://docs.nimbleway.com/nimble-sdk/admin/account-management\n" + "1. Get an API key from https://nimbleway.com/\n" "2. Set the API key in your environment or in your .env file\n" "3. Restart the application" ) From c989f00baef6999e9739e2b8fa9d1bfab00e4205 Mon Sep 17 00:00:00 2001 From: Kyle Zheng Date: Tue, 14 Jul 2026 14:17:21 -0700 Subject: [PATCH 14/14] docs(nimble_web_search): clarify live CI testing Signed-off-by: Kyle Zheng --- sources/nimble_web_search/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sources/nimble_web_search/README.md b/sources/nimble_web_search/README.md index fbe915eda..05bdca060 100644 --- a/sources/nimble_web_search/README.md +++ b/sources/nimble_web_search/README.md @@ -62,7 +62,7 @@ uv run ruff format --check sources/nimble_web_search - **Mocked unit tests** (`test_nimble_register.py`) cover: config defaults / all fields / invalid `search_depth` rejection, missing-key stub + warn-once, direct config-key passthrough to the SDK, successful render + description fallback, deep depth passthrough, query/content truncation, empty/error handling, retry-then-success, final-retry failure, 401, 403 enterprise-tier, non-default country/locale passthrough, and renderer behavior on titles containing special characters. - **Recorded-response replay** (`test_nimble_recorded_replay.py`) replays real, redacted `NimbleSearchRetriever` responses ([`tests/fixtures/README.md`](tests/fixtures/README.md)) through the full provider pipeline — a deterministic test mode that needs no network and no key. -- **Live integration** (`test_nimble_live_integration.py`) runs the canned query `NVIDIA CUDA Toolkit documentation` once against the real API with the shipped defaults and asserts the structural output contract: a non-error response containing 1..`max_results` `` blocks, every block with an http(s) `href` and a title, every block XML-parseable, and at least one non-empty body. Assertions are structural — never content-exact — so ordinary result variation cannot flake the run. Suitable for CI: add `NIMBLE_API_KEY` as a repository secret and set `AIQ_NIMBLE_LIVE_TESTS=1` in the job. +- **Live integration** (`test_nimble_live_integration.py`) runs the canned query `NVIDIA CUDA Toolkit documentation` once against the real API with the shipped defaults and asserts the structural output contract: a non-error response containing 1..`max_results` `` blocks, every block with an http(s) `href` and a title, every block XML-parseable, and at least one non-empty body. Assertions are structural — never content-exact — so ordinary result variation cannot flake the run. To run it in a dedicated opt-in CI job, add `NIMBLE_API_KEY` as a repository secret and set `AIQ_NIMBLE_LIVE_TESTS=1` in the job. ## Verification @@ -126,5 +126,5 @@ If you do not know your tier, leave `search_depth: lite` and let the description - API key handling follows the existing `EXA_API_KEY` / `TAVILY_API_KEY` pattern: env var or `SecretStr` config; never logged. - Untrusted API fields (`url`, `title`, body) are HTML-escaped before being rendered into the `` markup, so a result can't break the block or inject into downstream parsers. -- Tests are mocked; no live network in CI. +- Default CI is credential-free and network-free. An explicitly configured live-test job may access Nimble when `NIMBLE_API_KEY` and `AIQ_NIMBLE_LIVE_TESTS=1` are set. - The optional live smoke is documented in the PR description and uses a redacted output pattern.