add RAG under topics (microsoft#1990)

* add RAG

* demo

* correct notebook

* Update quarto installation

* Update gitignore

* Update format

* RAG doc

---------

Co-authored-by: Li Jiang <[email protected]>

.gitignore

@@ -178,5 +178,7 @@ test/agentchat/test_agent_scripts/*
 
 # test cache
 .cache_test
+.db
+
 
 notebook/result.png

notebook/agentchat_RetrieveChat.ipynb

@@ -92,7 +92,7 @@
     "\n",
     "## Construct agents for RetrieveChat\n",
     "\n",
-    "We start by initializing the `RetrieveAssistantAgent` and `RetrieveUserProxyAgent`. The system message needs to be set to \"You are a helpful assistant.\" for RetrieveAssistantAgent. The detailed instructions are given in the user message. Later we will use the `RetrieveUserProxyAgent.generate_init_prompt` to combine the instructions and a retrieval augmented generation task for an initial prompt to be sent to the LLM assistant."
+    "We start by initializing the `RetrieveAssistantAgent` and `RetrieveUserProxyAgent`. The system message needs to be set to \"You are a helpful assistant.\" for RetrieveAssistantAgent. The detailed instructions are given in the user message. Later we will use the `RetrieveUserProxyAgent.message_generator` to combine the instructions and a retrieval augmented generation task for an initial prompt to be sent to the LLM assistant."
    ]
   },
   {
@@ -3037,7 +3037,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.12"
+   "version": "3.10.13"
   },
   "skip_test": "Requires interactive usage"
  },

website/README.md

@@ -23,9 +23,9 @@ yarn install
 
 `quarto` is used to render notebooks.
 
-Install it [here](https://quarto.org/docs/get-started/).
+Install it [here](https://github.com/quarto-dev/quarto-cli/releases).
 
-> Note: Support for Docusaurus 3.0 in Quarto is from version `1.4`. Ensure that your `quarto` version is `1.4` or higher.
+> Note: Ensure that your `quarto` version is `1.5.23` or higher.
 
 ## Local Development
 

website/docs/topics/retrieval_augmentation.md

@@ -0,0 +1,68 @@
+# Retrieval Augmentation
+
+Retrieval Augmented Generation (RAG) is a powerful technique that combines language models with external knowledge retrieval to improve the quality and relevance of generated responses.
+
+One way to realize RAG in AutoGen is to construct agent chats with `RetrieveAssistantAgent` and `RetrieveUserProxyAgent` classes.
+
+## Example Setup: RAG with Retrieval Augmented Agents
+The following is an example setup demonstrating how to create retrieval augmented agents in AutoGen:
+
+### Step 1. Create an instance of `RetrieveAssistantAgent` and `RetrieveUserProxyAgent`.
+
+Here `RetrieveUserProxyAgent` instance acts as a proxy agent that retrieves relevant information based on the user's input.
+```python
+assistant = RetrieveAssistantAgent(
+    name="assistant",
+    system_message="You are a helpful assistant.",
+    llm_config={
+        "timeout": 600,
+        "cache_seed": 42,
+        "config_list": config_list,
+    },
+)
+ragproxyagent = RetrieveUserProxyAgent(
+    name="ragproxyagent",
+    human_input_mode="NEVER",
+    max_consecutive_auto_reply=3,
+    retrieve_config={
+        "task": "code",
+        "docs_path": [
+            "https://raw.githubusercontent.com/microsoft/FLAML/main/website/docs/Examples/Integrate%20-%20Spark.md",
+            "https://raw.githubusercontent.com/microsoft/FLAML/main/website/docs/Research.md",
+            os.path.join(os.path.abspath(""), "..", "website", "docs"),
+        ],
+        "custom_text_types": ["mdx"],
+        "chunk_token_size": 2000,
+        "model": config_list[0]["model"],
+        "client": chromadb.PersistentClient(path="/tmp/chromadb"),
+        "embedding_model": "all-mpnet-base-v2",
+        "get_or_create": True,  # set to False if you don't want to reuse an existing collection, but you'll need to remove the collection manually
+    },
+    code_execution_config=False,  # set to False if you don't want to execute the code
+)
+```
+
+### Step 2. Initiating Agent Chat with Retrieval Augmentation
+
+Once the retrieval augmented agents are set up, you can initiate a chat with retrieval augmentation using the following code:
+
+```python
+code_problem = "How can I use FLAML to perform a classification task and use spark to do parallel training. Train 30 seconds and force cancel jobs if time limit is reached."
+ragproxyagent.initiate_chat(
+    assistant, message=ragproxyagent.message_generator, problem=code_problem, search_string="spark"
+)  # search_string is used as an extra filter for the embeddings search, in this case, we only want to search documents that contain "spark".
+```
+
+## Online Demo
+[Retrival-Augmented Chat Demo on Huggingface](https://huggingface.co/spaces/thinkall/autogen-demos)
+
+## More Examples and Notebooks
+For more detailed examples and notebooks showcasing the usage of retrieval augmented agents in AutoGen, refer to the following:
+- Automated Code Generation and Question Answering with Retrieval Augmented Agents - [View Notebook](/docs/notebooks/agentchat_RetrieveChat)
+- Automated Code Generation and Question Answering with [Qdrant](https://qdrant.tech/) based Retrieval Augmented Agents - [View Notebook](https://github.com/microsoft/autogen/blob/main/notebook/agentchat_qdrant_RetrieveChat.ipynb)
+- Chat with OpenAI Assistant with Retrieval Augmentation - [View Notebook](https://github.com/microsoft/autogen/blob/main/notebook/agentchat_oai_assistant_retrieval.ipynb)
+- **RAG**: Group Chat with Retrieval Augmented Generation (with 5 group member agents and 1 manager agent) - [View Notebook](/docs/notebooks/agentchat_groupchat_RAG)
+
+## Roadmap
+
+Explore our detailed roadmap [here](https://github.com/microsoft/autogen/issues/1657) for further advancements plan around RAG. Your contributions, feedback, and use cases are highly appreciated! We invite you to engage with us and play a pivotal role in the development of this impactful feature.