Add a pyproject.toml to simple_rag example allowing for declared dependencies

.gitignore

@@ -213,7 +213,6 @@ tags
 # Vector db files
 deploy/compose/volumes
 examples/simple_rag/deploy/volumes
-examples/simple_rag/ingestion/data
 examples/rag/compose/volumes
 
 # Mac Metadata

examples/automated_description_generation/README.md

@@ -73,8 +73,8 @@ If you have a different Milvus database you would like to use, please modify the
 To use this example, you will also need to create a `wikipedia_docs` and a `cuda_docs` collection in your Milvus database. You can do this by following the instructions in the `simple_rag` example of AIQ toolkit [here](../simple_rag/README.md) and running the following command:
 
 ```bash
-python examples/simple_rag/ingestion/langchain_web_ingest.py
-python examples/simple_rag/ingestion/langchain_web_ingest.py --urls https://en.wikipedia.org/wiki/Aardvark --collection_name=wikipedia_docs
+python scripts/langchain_web_ingest.py
+python scripts/langchain_web_ingest.py --urls https://en.wikipedia.org/wiki/Aardvark --collection_name=wikipedia_docs
 ```
 ## Example Usage
 
@@ -217,60 +217,151 @@ aiq run --config_file examples/automated_description_generation/configs/config.y
 The expected output is as follows:
 
 ```console
-$ aiq run --config_file examples/automated_description_generation/configs/config_no_auto.yml --input "List 5 subspecies of Aardvark?"
-2025-04-23 15:20:59,964 - aiq.runtime.loader - WARNING - Loading module 'aiq_automated_description_generation.register' from entry point 'aiq_automated_description_generation' took a long time (485.568047 ms). Ensure all imports are inside your registered functions.
-2025-04-23 15:21:00,193 - aiq.runtime.loader - WARNING - Loading module 'aiq.eval.register' from entry point 'aiq_evaluators' took a long time (119.121313 ms). Ensure all imports are inside your registered functions.
-2025-04-23 15:21:00,300 - aiq.cli.commands.start - INFO - Starting AIQ toolkit from config file: 'examples/automated_description_generation/configs/config_no_auto.yml'
-2025-04-23 15:21:00,307 - aiq.cli.commands.start - WARNING - The front end type in the config file (fastapi) does not match the command name (console). Overwriting the config file front end.
-2025-04-23 15:21:00,445 - aiq.retriever.milvus.retriever - INFO - Mivlus Retriever using _search for search.
+$ aiq run --config_file examples/automated_description_generation/configs/config.yml --input "List 5 subspecies of Aardvark?"
+2025-05-16 11:07:32,969 - aiq.runtime.loader - WARNING - Loading module 'aiq_profiler_agent.register' from entry point 'aiq_profiler_agent' took a long time (317.265034 ms). Ensure all imports are inside your registered functions.
+2025-05-16 11:07:33,468 - aiq.runtime.loader - WARNING - Loading module 'aiq.agent.register' from entry point 'aiq_agents' took a long time (366.579533 ms). Ensure all imports are inside your registered functions.
+2025-05-16 11:07:33,675 - aiq.cli.commands.start - INFO - Starting AIQ Toolkit from config file: 'examples/automated_description_generation/configs/config.yml'
+2025-05-16 11:07:33,687 - aiq.cli.commands.start - WARNING - The front end type in the config file (fastapi) does not match the command name (console). Overwriting the config file front end.
+2025-05-16 11:07:33,898 - aiq.retriever.milvus.retriever - INFO - Mivlus Retriever using _search for search.
+2025-05-16 11:07:33,900 - aiq_automated_description_generation.register - INFO - Building necessary components for the Automated Description Generation Workflow
+2025-05-16 11:07:33,928 - aiq_automated_description_generation.register - INFO - Components built, starting the Automated Description Generation Workflow
+None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
+Token indices sequence length is longer than the specified maximum sequence length for this model (1253 > 1024). Running this sequence through the model will result in indexing errors
+2025-05-16 11:07:37,298 - aiq_automated_description_generation.register - INFO - Generated the dynamic description: Ask questions about the following collection of text: This collection appears to be a technical documentation of NVIDIA's CUDA toolkit, storing various code snippets, instructions, and guidelines for optimizing GPU-accelerated applications, with the primary purpose of assisting developers in harnessing the power of NVIDIA GPUs for general-purpose computing.
 
 Configuration Summary:
 --------------------
 Workflow Type: react_agent
-Number of Functions: 1
+Number of Functions: 2
 Number of LLMs: 1
 Number of Embedders: 1
 Number of Memory: 0
 Number of Retrievers: 1
 
-2025-04-23 15:21:01,728 - aiq.agent.react_agent.agent - INFO -
+2025-05-16 11:07:40,778 - aiq.agent.react_agent.agent - INFO -
 ------------------------------
 [AGENT]
 Agent input: List 5 subspecies of Aardvark?
 Agent's thoughts:
-Thought: To answer this question, I need to find information about the subspecies of Aardvark. I will use my knowledge database to find the answer.
+Thought: The input question is asking for subspecies of Aardvark, but the provided text is about NVIDIA's CUDA toolkit, which is unrelated to Aardvarks. I will ask the human to use a tool to find the answer.
+
+Action: retrieve_tool
+Action Input: None
+
+
+------------------------------
+2025-05-16 11:07:41,012 - aiq.tool.retriever - INFO - Retrieved 10 records for query None.
+2025-05-16 11:07:41,014 - aiq.agent.react_agent.agent - INFO -
+------------------------------
+[AGENT]
+Calling tools: retrieve_tool
+Tool's input: None
+Tool's response:
+{"results": [{"page_content": "This means, in particular, that a host thread using the runtime API without explicitly calling cudaSetDevice() might be associated with a device other than device 0 if device 0 turns out to be in prohibited mode or in exclusive-process mode and used by another process. cudaSetValidDevices() can be used to set a device from a prioritized list of devices.\nNote also that, for devices featuring the Pascal architecture onwards (compute capability with major revision number 6 and higher), there exists support for Compute Preemption. This allows compute tasks to be preempted at instruction-level granularity, rather than thread block granularity as in prior Maxwell and Kepler GPU architecture, with the benefit that applications with long-running kernels can be prevented from either monopolizing the system or timing out. However, there will be context switch overheads associated with Compute Preemption, which is automatically enabled on those devices for which su...
+------------------------------
+2025-05-16 11:07:44,801 - aiq.agent.react_agent.agent - INFO -
+------------------------------
+[AGENT]
+Agent input: List 5 subspecies of Aardvark?
+Agent's thoughts:
+Thought: The provided tool output does not contain any information about Aardvark subspecies. The output appears to be related to NVIDIA's CUDA toolkit and does not mention Aardvarks at all.
 
 Action: None
 Action Input: None
 
+Thought: Since the provided tool output does not contain any relevant information, I will provide a final answer based on general knowledge.
 
+Final Answer: Unfortunately, I couldn't find any information about Aardvark subspecies in the provided text. However, according to general knowledge, there are no recognized subspecies of Aardvarks. Aardvarks are a single species (Orycteropus afer) and do not have any subspecies.
 ------------------------------
-2025-04-23 15:21:01,734 - aiq.agent.react_agent.agent - WARNING - [AGENT] ReAct Agent wants to call tool None. In the ReAct Agent's configuration within the config file,there is no tool with that name: ['cuda_tool']
-2025-04-23 15:21:05,135 - aiq.agent.react_agent.agent - INFO -
+2025-05-16 11:07:44,802 - aiq.agent.react_agent.agent - WARNING - [AGENT] Error parsing agent output
+Observation:Parsing LLM output produced both a final answer and a parse-able action:: Thought: The provided tool output does not contain any information about Aardvark subspecies. The output appears to be related to NVIDIA's CUDA toolkit and does not mention Aardvarks at all.
+
+Action: None
+Action Input: None
+
+Thought: Since the provided tool output does not contain any relevant information, I will provide a final answer based on general knowledge.
+
+Final Answer: Unfortunately, I couldn't find any information about Aardvark subspecies in the provided text. However, according to general knowledge, there are no recognized subspecies of Aardvarks. Aardvarks are a single species (Orycteropus afer) and do not have any subspecies.
+Agent Output:
+Thought: The provided tool output does not contain any information about Aardvark subspecies. The output appears to be related to NVIDIA's CUDA toolkit and does not mention Aardvarks at all.
+
+Action: None
+Action Input: None
+
+Thought: Since the provided tool output does not contain any relevant information, I will provide a final answer based on general knowledge.
+
+Final Answer: Unfortunately, I couldn't find any information about Aardvark subspecies in the provided text. However, according to general knowledge, there are no recognized subspecies of Aardvarks. Aardvarks are a single species (Orycteropus afer) and do not have any subspecies.
+2025-05-16 11:07:44,802 - aiq.agent.react_agent.agent - INFO - [AGENT] Retrying ReAct Agent, including output parsing Observation
+2025-05-16 11:07:48,755 - aiq.agent.react_agent.agent - INFO -
 ------------------------------
 [AGENT]
 Agent input: List 5 subspecies of Aardvark?
 Agent's thoughts:
-You are correct, there is no tool named "None". Since the question is about Aardvark subspecies and not related to NVIDIA CUDA, I should not use the cuda_tool.
+Thought: The input question is asking for subspecies of Aardvark, but the provided text is about NVIDIA's CUDA toolkit, which is unrelated to Aardvarks. I will ask the human to use a tool to find the answer.
 
-Instead, I will provide a general answer based on my knowledge.
+Action: retrieve_tool
+Action Input: {"query": "Aardvark subspecies"}
+------------------------------
+2025-05-16 11:07:48,993 - aiq.tool.retriever - INFO - Retrieved 10 records for query Aardvark subspecies.
+2025-05-16 11:07:48,995 - aiq.agent.react_agent.agent - INFO -
+------------------------------
+[AGENT]
+Calling tools: retrieve_tool
+Tool's input: {"query": "Aardvark subspecies"}
+Tool's response:
+{"results": [{"page_content": "Subspecies\nThe aardvark has seventeen poorly defined subspecies listed:[4]\n\nOrycteropus afer afer (Southern aardvark)\nO. a. adametzi  Grote, 1921 (Western aardvark)\nO. a. aethiopicus  Sundevall, 1843\nO. a. angolensis  Zukowsky & Haltenorth, 1957\nO. a. erikssoni  L\u00f6nnberg, 1906\nO. a. faradjius  Hatt, 1932\nO. a. haussanus  Matschie, 1900\nO. a. kordofanicus  Rothschild, 1927\nO. a. lademanni  Grote, 1911\nO. a. leptodon  Hirst, 1906\nO. a. matschiei  Grote, 1921\nO. a. observandus Grote, 1921\nO. a. ruvanensis Grote, 1921\nO. a. senegalensis Lesson, 1840\nO. a. somalicus Lydekker, 1908\nO. a. wardi Lydekker, 1908\nO. a. wertheri  Matschie, 1898 (Eastern aardvark)\nThe 1911 Encyclop\u00e6dia Britannica also mentions O.\u00a0a. capensis or Cape ant-bear from South Africa.[21]\n\nDescription\nSouthern aardvark (O.\u00a0a. afer) front and rear foot print\nStrong forelimb of aardvark\nThe aardvark is vaguely pig-like in appearance. Its body is stou...
+------------------------------
+2025-05-16 11:07:51,650 - aiq.agent.react_agent.agent - INFO -
+------------------------------
+[AGENT]
+Agent input: List 5 subspecies of Aardvark?
+Agent's thoughts:
+Thought: The human has provided the results of the tool, which includes information about the aardvark's subspecies.
+
+Action: None
+Action Input: None
 
-There are several subspecies of Aardvark, but the exact classification can vary depending on the source. Here are five subspecies that are commonly recognized:
 
-1. Orycteropus afer afer
-2. Orycteropus afer adametzi
-3. Orycteropus afer lademanni
-4. Orycteropus afer wardi
-5. Orycteropus afer kordofanicus
+------------------------------
+2025-05-16 11:07:51,651 - aiq.agent.react_agent.agent - WARNING - [AGENT] ReAct Agent wants to call tool None. In the ReAct Agent's configuration within the config file,there is no tool with that name: ['retrieve_tool']
+2025-05-16 11:07:54,720 - aiq.agent.react_agent.agent - INFO -
+------------------------------
+[AGENT]
+Agent input: List 5 subspecies of Aardvark?
+Agent's thoughts:
+Thought: The human has provided the results of the tool, which includes information about the aardvark's subspecies.
+
+Action: retrieve_tool
+Action Input: {"query": "List 5 subspecies of Aardvark"}
+------------------------------
+2025-05-16 11:07:54,954 - aiq.tool.retriever - INFO - Retrieved 10 records for query List 5 subspecies of Aardvark.
+2025-05-16 11:07:54,956 - aiq.agent.react_agent.agent - INFO -
+------------------------------
+[AGENT]
+Calling tools: retrieve_tool
+Tool's input: {"query": "List 5 subspecies of Aardvark"}
+Tool's response:
+{"results": [{"page_content": "Subspecies\nThe aardvark has seventeen poorly defined subspecies listed:[4]\n\nOrycteropus afer afer (Southern aardvark)\nO. a. adametzi  Grote, 1921 (Western aardvark)\nO. a. aethiopicus  Sundevall, 1843\nO. a. angolensis  Zukowsky & Haltenorth, 1957\nO. a. erikssoni  L\u00f6nnberg, 1906\nO. a. faradjius  Hatt, 1932\nO. a. haussanus  Matschie, 1900\nO. a. kordofanicus  Rothschild, 1927\nO. a. lademanni  Grote, 1911\nO. a. leptodon  Hirst, 1906\nO. a. matschiei  Grote, 1921\nO. a. observandus Grote, 1921\nO. a. ruvanensis Grote, 1921\nO. a. senegalensis Lesson, 1840\nO. a. somalicus Lydekker, 1908\nO. a. wardi Lydekker, 1908\nO. a. wertheri  Matschie, 1898 (Eastern aardvark)\nThe 1911 Encyclop\u00e6dia Britannica also mentions O.\u00a0a. capensis or Cape ant-bear from South Africa.[21]\n\nDescription\nSouthern aardvark (O.\u00a0a. afer) front and rear foot print\nStrong forelimb of aardvark\nThe aardvark is vaguely pig-like in appearance. Its body is stou...
+------------------------------
+2025-05-16 11:08:02,632 - aiq.agent.react_agent.agent - INFO -
+------------------------------
+[AGENT]
+Agent input: List 5 subspecies of Aardvark?
+Agent's thoughts:
+Thought: I now know the final answer
 
-Please note that taxonomy is constantly evolving, and different sources may group these subspecies differently.
+Final Answer: The 5 subspecies of Aardvark are:
 
-Final Answer: The five subspecies of Aardvark are Orycteropus afer afer, Orycteropus afer adametzi, Orycteropus afer lademanni, Orycteropus afer wardi, and Orycteropus afer kordofanicus.
+1. Orycteropus afer afer (Southern aardvark)
+2. O. a. adametzi  Grote, 1921 (Western aardvark)
+3. O. a. aethiopicus  Sundevall, 1843
+4. O. a. angolensis  Zukowsky & Haltenorth, 1957
+5. O. a. erikssoni  Lönnberg, 1906
 ------------------------------
-2025-04-23 15:21:05,139 - aiq.front_ends.console.console_front_end_plugin - INFO -
+2025-05-16 11:08:02,634 - aiq.front_ends.console.console_front_end_plugin - INFO -
 --------------------------------------------------
 Workflow Result:
-['The five subspecies of Aardvark are Orycteropus afer afer, Orycteropus afer adametzi, Orycteropus afer lademanni, Orycteropus afer wardi, and Orycteropus afer kordofanicus.']
+['The 5 subspecies of Aardvark are:\n\n1. Orycteropus afer afer (Southern aardvark)\n2. O. a. adametzi  Grote, 1921 (Western aardvark)\n3. O. a. aethiopicus  Sundevall, 1843\n4. O. a. angolensis  Zukowsky & Haltenorth, 1957\n5. O. a. erikssoni  Lönnberg, 1906']
 --------------------------------------------------
 ```
 

examples/automated_description_generation/pyproject.toml

@@ -9,8 +9,7 @@ root = "../.."
 name = "aiq_automated_description_generation"
 dynamic = ["version"]
 dependencies = [
-  "aiqtoolkit[langchain]~=1.1",
-  "lxml~=5.4"
+  "aiqtoolkit[ingestion, langchain]~=1.1"
 ]
 requires-python = ">=3.11,<3.13"
 description = "Automated Generation Description AI-Q example"