Add a blueprint for Haystack Deep Research Agent

examples/basic/frameworks/haystack_deep_research_agent/README.md

@@ -0,0 +1,215 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Haystack Deep Research Agent
+
+This example demonstrates how to build a deep research agent using Haystack framework  that combines web search and Retrieval Augmented Generation (RAG) capabilities using the NeMo-Agent-Toolkit.
+
+## Overview
+
+The Haystack Deep Research Agent is an intelligent research assistant that can:
+
+- **Web Search**: Search the internet for current information using SerperDev API
+- **Document Retrieval**: Query an internal document database using RAG with OpenSearch
+- **Comprehensive Research**: Combine both sources to provide thorough, well-cited research reports
+- **Intelligent Routing**: Automatically decide when to use web search vs. internal documents
+
+## Architecture
+
+The workflow consists of three main components:
+
+1. **Web Search Tool**: Uses Haystack's SerperDevWebSearch and LinkContentFetcher to search the web and extract content from web pages
+2. **RAG Tool**: Uses OpenSearchDocumentStore to index and query internal documents with semantic retrieval
+3. **Deep Research Agent** (`register.py`): Orchestrates the agent and imports modular pipelines from `src/aiq_haystack_deep_research_agent/pipelines/`:
+   - `search.py`: builds the web search tool
+   - `rag.py`: builds the RAG pipeline and tool
+   - `indexing.py`: startup indexing (PDF/TXT/MD) into OpenSearch
+
+## Prerequisites
+
+Before using this workflow, ensure you have:
+
+1. **NVIDIA API Key**: Required for the chat generator and RAG functionality
+   - Get your key from [NVIDIA API Catalog](https://build.nvidia.com/)
+   - Set as environment variable: `export NVIDIA_API_KEY=your_key_here`
+
+2. **SerperDev API Key**: Required for web search functionality
+   - Get your key from [SerperDev](https://serper.dev/api-key)
+   - Set as environment variable: `export SERPERDEV_API_KEY=your_key_here`
+
+3. **OpenSearch Instance**: Required for RAG functionality
+   - You can run OpenSearch locally using `docker`
+
+## Installation and Usage
+
+Follow the instructions in the [Install Guide](../../../../docs/source/quick-start/installing.md#install-from-source) to create the development environment and install AIQ toolkit.
+
+### Step 1: Set Your API Keys
+
+```bash
+export NVIDIA_API_KEY=<YOUR_NVIDIA_API_KEY>
+export SERPERDEV_API_KEY=<YOUR_SERPERDEV_API_KEY>
+```
+
+### Step 2: Start OpenSearch (if not already running)
+
+```bash
+docker run -d --name opensearch -p 9200:9200 -p 9600:9600 \
+  -e "discovery.type=single-node" \
+  -e "plugins.security.disabled=true" \
+  opensearchproject/opensearch:2.11.1
+```
+
+### Step 3: Install the Workflow
+
+```bash
+uv pip install -e examples/basic/frameworks/haystack_deep_research_agent
+```
+
+### Step 4: Add Sample Documents (Optional)
+
+Place documents in the example `data/` directory to enable RAG (PDF, TXT, or MD). On startup, the workflow indexes files from:
+
+- `workflow.data_dir` (default: `/data`)
+- If empty/missing, it falls back to this example's bundled `data/` directory
+
+```bash
+# Example: Download a sample PDF
+wget "https://docs.aws.amazon.com/pdfs/bedrock/latest/userguide/bedrock-ug.pdf" \
+  -O examples/basic/frameworks/haystack_deep_research_agent/data/bedrock-ug.pdf
+```
+
+### Step 5: Run the Workflow
+
+```bash
+aiq run --config_file=examples/basic/frameworks/haystack_deep_research_agent/src/aiq_haystack_deep_research_agent/configs/config.yml --input "What are the latest updates on the Artemis moon mission?"
+```
+
+## Example Queries
+
+Here are some example queries you can try:
+
+**Web Search Examples:**
+
+```bash
+# Current events
+aiq run --config_file=examples/basic/frameworks/haystack_deep_research_agent/src/aiq_haystack_deep_research_agent/configs/config.yml --input "What are the latest developments in AI research for 2024?"
+
+# Technology news
+aiq run --config_file=examples/basic/frameworks/haystack_deep_research_agent/src/aiq_haystack_deep_research_agent/configs/config.yml --input "What are the new features in the latest Python release?"
+```
+
+**RAG Examples (if you have documents indexed):**
+
+```bash
+# Document-specific queries
+aiq run --config_file=examples/basic/frameworks/haystack_deep_research_agent/src/aiq_haystack_deep_research_agent/configs/config.yml --input "What are the key features of AWS Bedrock?"
+
+# Mixed queries (will use both web search and RAG)
+aiq run --config_file=examples/basic/frameworks/haystack_deep_research_agent/src/aiq_haystack_deep_research_agent/configs/config.yml --input "How does AWS Bedrock compare to other AI platforms in 2024?"
+```
+
+**Web Search + RAG Examples:**
+
+```bash
+aiq run --config_file=examples/basic/frameworks/haystack_deep_research_agent/src/aiq_haystack_deep_research_agent/configs/config.yml --input "Is panna (heavy cream) needed on carbonara? Check online the recipe and compare it with the one from our internal dataset."
+```
+
+## Testing
+
+### Quick smoke test (no external services)
+
+- Validates the workflow config without hitting LLMs or OpenSearch.
+
+```bash
+# In your virtual environment
+pytest -q examples/basic/frameworks/haystack_deep_research_agent/tests -k config_yaml_loads_and_has_keys
+```
+
+### End-to-end test (requires keys + OpenSearch)
+
+- Prerequisites:
+  - Set keys: `NVIDIA_API_KEY` and `SERPERDEV_API_KEY`
+  - OpenSearch running on `http://localhost:9200` (start with Docker):
+
+```bash
+docker run -d --name opensearch -p 9200:9200 -p 9600:9600 \
+  -e "discovery.type=single-node" \
+  -e "plugins.security.disabled=true" \
+  opensearchproject/opensearch:2.11.1
+```
+
+- Run the e2e test (ensure `pytest-asyncio` is installed in your venv):
+
+```bash
+pip install pytest-asyncio  # if not already installed
+export NVIDIA_API_KEY=<YOUR_KEY>
+export SERPERDEV_API_KEY=<YOUR_KEY>
+pytest -q examples/basic/frameworks/haystack_deep_research_agent/tests -k full_workflow_e2e
+```
+
+## Configuration
+
+The workflow is configured via `config.yml`. Key configuration options include:
+
+- **Web Search Tool**:
+  - `top_k`: Number of search results to retrieve (default: 10)
+  - `timeout`: Timeout for fetching web content (default: 3 seconds)
+  - `retry_attempts`: Number of retry attempts for failed requests (default: 2)
+
+- **RAG Tool**:
+  - `opensearch_url`: OpenSearch host URL (default: `http://localhost:9200`)
+  - `index_name`: OpenSearch index name (fixed: `deep_research_docs`)
+  - `top_k`: Number of documents to retrieve (default: 15)
+  - `index_on_startup`: If true, run indexing pipeline on start
+  - `data_dir`: Directory to scan for documents; if empty/missing, falls back to example `data/`
+
+- **Agent**:
+  - `max_agent_steps`: Maximum number of agent steps (default: 20)
+  - `system_prompt`: Customizable system prompt for the agent
+
+## Customization
+
+You can customize the workflow by:
+
+1. **Modifying the system prompt** in `config.yml` to change the agent's behavior
+2. **Adding more document types** by extending the RAG tool to support other file formats
+3. **Changing the LLM model** by updating the top-level `llms` section in `config.yml`. This example defines `agent_llm` and `rag_llm` using the `nim` provider so they can leverage common parameters like `temperature`, `top_p`, and `max_tokens`. The workflow references them via the builder. See Haystack's NvidiaChatGenerator docs: [NvidiaChatGenerator](https://docs.haystack.deepset.ai/docs/nvidiachatgenerator)
+4. **Adjusting search parameters** to optimize for your use case
+
+## Troubleshooting
+
+**Common Issues:**
+
+1. **OpenSearch Connection Error**: Ensure OpenSearch is running and accessible at the configured host
+2. **Missing API Keys**: Verify that both NVIDIA_API_KEY and SERPERDEV_API_KEY are set
+3. **No Documents Found**: Check that PDF files are placed in the data directory and the path is correct
+4. **Web Search Fails**: Verify your SerperDev API key is valid and has remaining quota
+
+**Logs**: Check the NeMo-Agent-Toolkit logs for detailed error information and debugging.
+
+## Architecture Details
+
+The workflow demonstrates several key NeMo-Agent-Toolkit patterns:
+
+- **Workflow Registration**: The agent is exposed as a workflow function with a pydantic config
+- **Builder LLM Integration**: LLMs are defined under top-level `llms:` and accessed via `builder.get_llm_config(...)`
+- **Component Integration**: Haystack components are composed into tools within the workflow
+- **Error Handling**: Robust error handling with fallback behaviors
+- **Async Operations**: All operations are asynchronous for better performance
+
+This example showcases how the Haystack AI framework can be seamlessly integrated into NeMo-Agent-Toolkit workflows while maintaining the flexibility and power of the underlying architecture.

examples/basic/frameworks/haystack_deep_research_agent/configs

@@ -0,0 +1 @@
+src/aiq_haystack_deep_research_agent/configs

examples/basic/frameworks/haystack_deep_research_agent/data

@@ -0,0 +1 @@
+src/aiq_haystack_deep_research_agent/data

examples/basic/frameworks/haystack_deep_research_agent/pyproject.toml

@@ -0,0 +1,28 @@
+[build-system]
+build-backend = "setuptools.build_meta"
+requires = ["setuptools >= 64", "setuptools-scm>=8"]
+
+[tool.setuptools_scm]
+root = "../../../.."
+
+[project]
+name = "aiq_haystack_deep_research_agent"
+dynamic = ["version"]
+dependencies = [
+  "aiqtoolkit~=1.2",
+  "haystack-ai>=2.17.0,<2.19",
+  "opensearch-haystack~=4.2.0",
+  "nvidia-haystack~=0.3.0",
+  "trafilatura~=2.0.0",
+  "pypdf~=5.8.0",
+  "docstring-parser~=0.16",
+]
+requires-python = ">=3.11,<3.13"
+description = "Haystack Deep Research Agent workflow for AIQ toolkit"
+classifiers = ["Programming Language :: Python"]
+
+[tool.uv.sources]
+aiqtoolkit = { path = "../../../..", editable = true }
+
+[project.entry-points.'aiq.components']
+aiq_haystack_deep_research_agent = "aiq_haystack_deep_research_agent.register"

examples/basic/frameworks/haystack_deep_research_agent/src/aiq_haystack_deep_research_agent/__init__.py

@@ -0,0 +1,10 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+# Re-export pipelines helpers for convenience
+try:
+	from .pipelines.search import create_search_tool  # noqa: F401
+	from .pipelines.rag import create_rag_tool  # noqa: F401
+	from .pipelines.indexing import run_startup_indexing  # noqa: F401
+except Exception:  # pragma: no cover - optional during install time
+	pass

examples/basic/frameworks/haystack_deep_research_agent/src/aiq_haystack_deep_research_agent/configs/config.yml

@@ -0,0 +1,34 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+general:
+  use_uvloop: true
+
+llms:
+  rag_llm:
+    _type: nim
+    model: nvidia/llama-3.3-nemotron-super-49b-v1
+  agent_llm:
+    _type: nim
+    model: nvidia/llama-3.3-nemotron-super-49b-v1
+
+workflow:
+  _type: haystack_deep_research_agent
+  max_agent_steps: 20
+  search_top_k: 10
+  rag_top_k: 15
+  opensearch_url: http://localhost:9200
+  index_on_startup: true
+  data_dir: /data

examples/basic/frameworks/haystack_deep_research_agent/src/aiq_haystack_deep_research_agent/data/carbonara.md

@@ -0,0 +1,58 @@
+# Roman Carbonara (Original Recipe)
+
+This is a faithful Roman-style carbonara: no cream, no onion, no garlic. Just eggs, Pecorino Romano, guanciale, and black pepper.
+
+## Ingredients (serves 2–3)
+- 200 g spaghetti (or rigatoni)
+- 100 g guanciale, cut into lardons (about 0.5–1 cm thick)
+- 2 large egg yolks + 1 whole large egg (room temperature)
+- 60 g Pecorino Romano, very finely grated (plus extra to serve)
+- 1 to 1.5 tsp freshly ground black pepper (to taste)
+- Kosher salt (for pasta water only)
+
+Notes:
+- Guanciale is traditional. Pancetta is a second-best substitute; bacon is a distant third.
+- Pecorino Romano is traditional. If too sharp, you may use a 50/50 mix with Parmigiano Reggiano, but pure Pecorino is canonical.
+
+## Equipment
+- Large pot for boiling pasta
+- Large sauté pan (cold start) for guanciale
+- Mixing bowl (heatproof)
+- Tongs or spider
+- Fine grater and pepper mill
+
+## Method
+1) Prep the egg-and-cheese emulsion
+- In a bowl, whisk together egg yolks, whole egg, grated Pecorino, and black pepper until thick like a loose paste. Set aside.
+
+2) Render the guanciale
+- Place guanciale in a cold pan. Turn heat to medium-low and render slowly until fat has melted and the meat is lightly crisp at the edges (about 6–10 minutes). Do not brown deeply. Turn off heat.
+- Reserve the pan with all rendered fat. If there is too much fat, spoon off a little, but keep enough to coat the pasta generously.
+
+3) Cook the pasta
+- Boil pasta in well-salted water until just shy of al dente. Reserve about 250 ml (1 cup) of starchy pasta water.
+
+4) Marry pasta, guanciale, and fat
+- Transfer pasta directly into the pan with guanciale and fat over low heat. Toss to coat. Add a splash of hot pasta water to loosen and create a glossy base.
+
+5) Create the sauce off-heat
+- Remove the pan from heat. Wait 15–30 seconds so it is hot but not scorching.
+- Add the egg-cheese mixture to the pasta, tossing vigorously and adding small splashes of hot pasta water as needed to form a silky, glossy sauce that clings to the noodles. The residual heat cooks the eggs gently—avoid scrambling.
+
+6) Adjust and serve
+- Taste and adjust with more Pecorino and black pepper if desired. Plate immediately.
+- Top with a final dusting of Pecorino and a twist of pepper.
+
+## Keys to success
+- Temperature control: combine eggs off-heat and use pasta water to reach a creamy emulsion.
+- Very fine cheese grating: ensures quick melting and smooth texture.
+- Freshly cracked pepper: be generous.
+- Work fast: carbonara should be glossy and served immediately.
+
+## Canonical checklist (what not to add)
+- No cream
+- No garlic or onion
+- No butter or oil (beyond guanciale fat)
+- No wine
+
+Buon appetito!

examples/basic/frameworks/haystack_deep_research_agent/src/aiq_haystack_deep_research_agent/data/data

@@ -0,0 +1 @@
+data