docs rework

README.md

@@ -1,5 +1,7 @@
 # 🦫 OpenRag — The Open RAG Experimentation Playground
 
+![RAG Architecture](./RAG_architecture.png)
+
 OpenRag is a lightweight, modular and extensible Retrieval-Augmented Generation (RAG) framework designed to explore and test advanced RAG techniques — 100% open source and focused on experimentation, not lock-in.
 
 > Built by the Linagora, OpenRag offers a sovereign-by-design alternative to mainstream RAG stacks.
@@ -9,17 +11,14 @@ OpenRag is a lightweight, modular and extensible Retrieval-Augmented Generation
 - [Table of Contents](#table-of-contents)
 - [🎯 Goals](#-goals)
 - [✨ Key Features](#-key-features)
-  - [Supported File Formats](#supported-file-formats)
-  - [Chunking](#chunking)
-  - [Indexing](#indexing)
-  - [Document Retrieval & Reranking](#document-retrieval--reranking)
-  - [🔌 OpenAI API Compatibility](#-openai-api-compatibility)
-- [🚀 Getting Started](#-getting-started)
+- [🚧 Coming Soon](#-coming-soon)
+- [🚀 Installation](#-installation)
   - [Prerequisites](#prerequisites)
   - [Installation and Configuration](#installation-and-configuration)
   - [Configuration](#configuration)
 - [🔧 Troubleshooting](#-troubleshooting)
 - [🤝 Contributing](#-contributing)
+- [📜 License](#-license)
 
 
 ## 🎯 Goals
@@ -28,75 +27,112 @@ OpenRag is a lightweight, modular and extensible Retrieval-Augmented Generation
 - Collaborate with the community to innovate and push the boundaries of RAG applications
 
 ## ✨ Key Features
-This section provides a detailed explanation of the currently supported features.
+### 📁 Rich File Format Support
+OpenRag supports a comprehensive range of file formats for seamless document ingestion:
 
-The **`.hydra_config`** directory contains all the configuration files for the application. These configurations are structured using the [Hydra configuration framework](https://hydra.cc/docs/intro/). This directory will be referenced for setting up the RAG (Retrieval-Augmented Generation) pipeline.
+* **Text Files**: `txt`, `md`
+* **Document Files**: `pdf`, `docx`, `doc`, `pptx` - Advanced PDF parsing with OCR support and Office document processing
+* **Audio Files**: `wav`, `mp3`, `mp4`, `ogg`, `flv`, `wma`, `aac` - Audio transcription and content extraction
+* **Images**: `png`, `jpeg`, `jpg`, `svg` - Vision Language Model (VLM) powered image captioning and analysis
 
-### Supported File Formats
-This branch currently supports the following file types:
+All files are intelligently converted to **Markdown format** with images replaced by AI-generated captions, ensuring consistent processing across all document types.
 
-* **TextFiles**: `txt`, `md`
-* **Document Files**: `pdf`, `docx`, `doc`, `pptx`
-* **Audio Files**: `wav`, `mp3`, `mp4`, `ogg`, `flv`, `wma`, `aac`
-* **Images**: `png:, jpeg, jpg, svg`
+### 🎛️ Native Web-Based Indexer UI
+Experience intuitive document management through our built-in web interface.
 
-Files are converted tp **Markdown**, with images replaced by captions generated by a **Vision Language Model (VLM)**. (Refer to the **Configuration** section for additional details.) The final Markdown output is then split into chunks and indexed in the [Milvus vector database](https://milvus.io/).
+<details>
 
->[!NOTE]
-> **Upcoming Support**: Future releases will expand compatibility to include additional formats such as `csv`, `odt`, `html`, and other widely used open-source document types.
+<summary>Indexer UI Features</summary>
 
-### Chunking
-Multiple [chunking strategies](./.hydra_config/chunker) are supported: **`semantic`, `markdown`, and `recursive`** chunking. Files are converted to markdown and the **same chunker** is used for all types. Format-specific chunkers (e.g., for CSV, HTML) will be added later. 
+* **Drag-and-drop file upload** with batch processing capabilities
+* **Real-time indexing progress** monitoring and status updates
+* **Partition management** - organize documents into logical collections
+* **Visual document preview** and metadata inspection
+* **Search and filtering** capabilities for indexed content
 
-```yml
-# .hydra_config/chunker/markdown_splitter.yaml
-defaults:
-  - base
-name: markdown_splitter
-chunk_size: 512
-chunk_overlap: 100
-```
+</details>
 
-The **`chunk_size`** and **`chunk_overlap`** values are expressed in **tokens**, not characters. For enhanced retrieval, enable the **contextual retrieval** — a technique introduced by Anthropic to improve retrieval performance ([Contextual Retrieval](https://www.anthropic.com/news/contextual-retrieval)).
+### 🗂️ Partition-Based Architecture
+Organize your knowledge base with flexible partition management:
+* **Multi-tenant support** - isolate different document collections
 
-### Indexing
-Chunks are stored in the **Milvus** vector database using the `Qwen/Qwen3-Embedding-0.6B` embedder via VLLM. To explore alternatives, check the [MTEB benchmark](https://huggingface.co/spaces/mteb/leaderboard).
+### 💬 Interactive Chat UI with Source Attribution
+Engage with your documents through our sophisticated chat interface:
 
-> \[!IMPORTANT]
-> Use an embedding model suited to your document languages and context window needs. The default model supports English and French.
+<details>
 
+<summary>Chat UI Features</summary>
 
-### Document Retrieval & Reranking
-* Search Pipeline: We use a **hybrid search** combining **semantic search** and **BM25** keyword matching for broader coverage. Results are merged and ranked with [Reciprocal Rank Fusion (RRF)](https://milvus.io/docs/reranking.md) for optimal relevance.
+* **Chainlit-powered UI** - modern, responsive chat experience
+* **Source transparency** - every response includes relevant document references
+</details>
 
-> \[!IMPORTANT]
-> Semantic similarity doesn't always mean relevance. Rerankers help refine results and reduce hallucinations by prioritizing the most relevant documents.
-
-* *Reranker: Documents are then reranked using the multilingual reranker **`Alibaba-NLP/gte-multilingual-reranker-base`** model from Hugging Face.
 
 ### 🔌 OpenAI API Compatibility
+OpenRAG API is tailored to be compatible with the OpenAI format (see the [openai-compatibility section](docs/api_documentation.md#-openai-compatible-chat) for more details), enabling seamless integration of your deployed RAG into popular frontends and workflows such as OpenWebUI, LangChain, N8N, and more. This ensures flexibility and ease of adoption without requiring custom adapters.
+
+<details>
+
+<summary>Summary of features</summary>
+
+* **Drop-in replacement** for OpenAI API endpoints
+* **Compatible with popular frontends** like OpenWebUI, LangChain, N8N, and more
+* **Authentication support** - secure your API with token-based auth
+
+</details>
+
+
+### ⚡ Distributed Ray Deployment
+Scale your RAG pipeline across multiple machines and GPUs.
+<details>
+
+<summary>Distributed Ray Deployment</summary>
+
+* **Horizontal scaling** - distribute processing across worker nodes
+* **GPU acceleration** - optimize inference across available hardware
+* **Resource management** - intelligent allocation of compute resources
+* **Monitoring dashboard** - real-time cluster health and performance metrics
+
+See the section on [distributed deployment in a ray cluster](#5-distributed-deployment-in-a-ray-cluster) for more details
 
-Our RAG pipeline natively supports the **OpenAI API**, ensuring seamless integration with any tool or workflow built around the OpenAI interface. This means you can plug it directly into popular frontends like **`OpenWebUI`**—no custom adapters needed.
+</details>
 
-Looking to get started? Check out the [API documentation](docs/api_documentation.md#-openai-compatible-chat) for full details.
+### 🔍 Advanced Retrieval & Reranking
+OpenRAG Leverages state-of-the-art retrieval techniques for superior accuracy.
 
+<details>
+s
+<summary>Implemented advanced retrieval techniques</summary>
 
+* **Hybrid search** - combines semantic similarity with BM25 keyword matching
+* **Contextual retrieval** - Anthropic's technique for enhanced chunk relevance
+* **Multilingual reranking** - using `Alibaba-NLP/gte-multilingual-reranker-base`
 
-## 🚀 Getting Started
+For more details, [see this file](docs/features_in_details.md)
+
+</details>
+
+
+## 🚧 Coming Soon
+* **📂 Expanded Format Support**: Future updates will introduce compatibility with additional formats such as `csv`, `odt`, `html`, and other widely used open-source document types.
+* **🔄 Unified Markdown Conversion**: All files will continue to be converted to markdown using a consistent chunker. Format-specific chunkers (e.g., for CSV, HTML) are planned for enhanced processing.
+* **🤖 Advanced Features**: Upcoming releases will include Tool Calling, Agentic RAG, and MCP to elevate your RAG workflows.
+* **Enhanced Security**: Ensures data encryption both during transit and at rest.
+
+## 🚀 Installation
 
 ### Prerequisites
 - **Python 3.12** or higher recommended
 - **Docker** and **Docker Compose**
 - For GPU capable machines, ensure you have the NVIDIA Container Toolkit installed. Refer to the [NVIDIA documentation](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) for installation instructions.
 
-OpenRag is designed to run in a containerized environment under Linux on x86_64 architecture. ARM processors are not supported, this is subject to change in the future.
-
 ### Installation and Configuration
 #### 1. Clone the repository:
 ```bash
 git clone git@github.com:linagora/openrag.git
 
-# git clone --recurse-submodules git@github.com:linagora/openrag.git # to clone the repo with the associated submodules
+# # to clone the repo with the associated submodules
+# git clone --recurse-submodules git@github.com:linagora/openrag.git
 
 cd openrag
 git checkout main # or a given release
@@ -121,10 +157,9 @@ uv sync
 ```
 
 #### 3. Create a `.env` File
-
 Create a `.env` file at the root of the project, mirroring the structure of `.env.example`, to configure your environment.
 
-###### File Parser configuration 
+##### File Parser configuration 
 > For PDF indexing, multiple loader options are available. Set your choice using the **`PDFLoader`** env variable:
   * **`MarkerLoader`** and **`DoclingLoader`** are recommended for optimal performance, especially on OCR-processed PDFs. They support both GPU and CPU execution.
   * For lightweight testing on CPU, use **`PyMuPDF4LLMLoader`** or **`PyMuPDFLoader`**.
@@ -139,7 +174,7 @@ Other file formats (`txt`, `docx`, `doc`, `pptx`, audio type files, etc) are pre
 BASE_URL=
 API_KEY=
 MODEL=
-LLM_SEMAPHORE=10
+LLM_SEMAPHORE=10 # change with respect to your llm's capabilities
 
 # VLM for image captioning. You can put your LLM here if it's multimodal 
 VLM_BASE_URL=
@@ -156,17 +191,17 @@ RETRIEVER_TOP_K=20 # Number of documents to return before reranking
 
 # EMBEDDER
 EMBEDDER_MODEL_NAME=Qwen/Qwen3-Embedding-0.6B
+EMBEDDER_API_KEY=EMPTY
 # VLLM_PORT=8000 forwarded port
 # EMBEDDER_BASE_URL=http://vllm:8000/v1
-EMBEDDER_API_KEY=EMPTY
 
 # RERANKER
 RERANKER_ENABLED=true
 RERANKER_MODEL=Alibaba-NLP/gte-multilingual-reranker-base
 RERANKER_TOP_K=5 # Number of documents to return after reranking. increment it for better results if your llm has a wider context window
 
 # Prompts
-PROMPTS_DIR=../prompts/example3_en # you can the fr version of the prompts ../prompts/example3
+PROMPTS_DIR=../prompts/example3_en # you can use the fr version of the prompts '../prompts/example3'
 
 # Loaders
 PDFLoader=MarkerLoader
@@ -176,15 +211,15 @@ MARKER_MAX_PROCESSES=2 # increment if you've enough gpu capacity
 RAY_DEDUP_LOGS=0
 RAY_NUM_GPUS=0.1
 RAY_POOL_SIZE=1 # increment if you a cluster of machines
-RAY_MAX_TASKS_PER_WORKER=5 # Number of tasks per serializer instance
+RAY_MAX_TASKS_PER_WORKER=6 # Number of tasks per serializer instance
 RAY_DASHBOARD_PORT=8265
 RAY_RUNTIME_ENV_HOOK=ray._private.runtime_env.uv_runtime_env_hook.hook
 
 # To enable HTTP authentication via HTTPBearer for the api endpoints
 AUTH_TOKEN=super-secret-token
 ```
 
-###### Indexer UI
+##### Indexer UI
 >[!IMPORTANT]
 > Before launching the app, You might want to configure **`Indexer UI` (A Web interface for intuitive document ingestion, indexing, and management.)** following the dedicated guide:
 ➡ [Deploy with Indexer UI](docs/setup_indexerui.md)
@@ -199,7 +234,7 @@ docker compose up --build -d  # Use 'down' to stop
 # Start with CPU
 docker compose --profile cpu up --build -d # Use '--profile cpu down' to stop it properly
 ```
-> \[!TIPS]
+> \[!TIP]
 > For quick testing on CPU, you can reduce computational load by adjusting the following settings in the **`.env`** file:
 
 ```bash
@@ -217,7 +252,7 @@ Once the app is up and running, visit `http://localhost:APP_PORT` or `http:X.X.X
 1. **`/docs`** – FastAPI’s full API documentation. See [this guide](docs/api_documentation.md) for more details on the endpoints.
 2. **`/chainlit`** – [Chainlit chat UI](https://docs.chainlit.io/get-started/overview) to chat with your partitions. To disable it (e.g., for backend-only use), set `WITH_CHAINLIT_UI=False`.
 
-> \[!NOTE]
+> [!NOTE]
 > Chainlit UI has no authentication by default. To enable it, set the following in your `.env`:
 
 ```bash
@@ -245,7 +280,10 @@ To scale **OpenRag** in a distributed environment using **Ray**, follow the dedi
 This FastAPI-powered backend offers capabilities for document-based question answering (RAG), semantic search, and document indexing across multiple partitions. It exposes endpoints for interacting with a vector database and managing document ingestion, processing, and querying. See this document for [detailed overview of our api](docs/api_documentation.md).
 
 
+
 ## 🔧 Troubleshooting
+<details>
+<summary>Troubleshooting</summary>
 
 ### Error on dependencies installation
 
@@ -271,12 +309,22 @@ While executing OpenRag, if you encounter a problem that prevents you from downl
 sudo mkdir /app/model_weights
 sudo chmod 775 /app/model_weights
 ```
+</details>
+
+
+## 🤝 Support and Contributions
+We ❤️ your contributions!
+
+We encourage you to contribute to OpenRag! Here's how you can get involved:
+1. Fork this repository.
+2. Create a new branch for your feature or fix.
+3. Submit a pull request for review.
+
+Feel free to ask **`questions`, `suggest features`, or `report bugs` via the GitHub Issues page**. Your feedback helps us improve!
+
 
-## 🤝 Contributing
+## 📜 License
 
-We ❤️ contributions!
+OpenRag is licensed under the [Apache License 2.0](LICENSE). You are free to use, modify, and distribute this software in compliance with the terms of the license.
 
-Contributions are welcome! Please follow standard GitHub workflow:
-1. Fork the repository
-2. Create a feature branch
-3. Submit a pull request
+For more details, refer to the [LICENSE](LICENSE) file in the repository.

docs/features_in_details.md

@@ -0,0 +1,46 @@
+## ✨ Key Features
+This section provides a detailed explanation of the currently supported features.
+
+The **`.hydra_config`** directory contains all the configuration files for the application. These configurations are structured using the [Hydra configuration framework](https://hydra.cc/docs/intro/). This directory will be referenced for setting up the RAG (Retrieval-Augmented Generation) pipeline.
+
+### Supported File formats
+This branch currently supports the following file types:
+
+* **TextFiles**: `txt`, `md`
+* **Document Files**: `pdf`, `docx`, `doc`, `pptx`
+* **Audio Files**: `wav`, `mp3`, `mp4`, `ogg`, `flv`, `wma`, `aac`
+* **Images**: `png:, jpeg, jpg, svg`
+
+Files are converted tp **Markdown**, with images replaced by captions generated by a **Vision Language Model (VLM)**. (Refer to the **Configuration** section for additional details.) The final Markdown output is then split into chunks and indexed in the [Milvus vector database](https://milvus.io/).
+
+> [!NOTE]
+> **Upcoming Support**: Future releases will expand compatibility to include additional formats such as `csv`, `odt`, `html`, and other widely used open-source document types.
+
+### Chunking
+Multiple [chunking strategies](./.hydra_config/chunker) are supported: **`semantic`, `markdown`, and `recursive`** chunking. Files are converted to markdown and the **same chunker** is used for all types. Format-specific chunkers (e.g., for CSV, HTML) will be added later. 
+
+```yml
+# .hydra_config/chunker/markdown_splitter.yaml
+defaults:
+  - base
+name: markdown_splitter
+chunk_size: 512
+chunk_overlap: 100
+```
+
+The **`chunk_size`** and **`chunk_overlap`** values are expressed in **tokens**, not characters. For enhanced retrieval, enable the **contextual retrieval** — a technique introduced by Anthropic to improve retrieval performance ([Contextual Retrieval](https://www.anthropic.com/news/contextual-retrieval)).
+
+### Indexing
+Chunks are stored in the **Milvus** vector database using the `Qwen/Qwen3-Embedding-0.6B` embedder via VLLM. To explore alternatives, check the [MTEB benchmark](https://huggingface.co/spaces/mteb/leaderboard).
+
+> \[!IMPORTANT]
+> Use an embedding model suited to your document languages and context window needs. The default model supports English and French.
+
+
+### Document Retrieval & Reranking
+* Search Pipeline: We use a **hybrid search** combining **semantic search** and **BM25** keyword matching for broader coverage. Results are merged and ranked with [Reciprocal Rank Fusion (RRF)](https://milvus.io/docs/reranking.md) for optimal relevance.
+
+> \[!IMPORTANT]
+> Semantic similarity doesn't always mean relevance. Rerankers help refine results and reduce hallucinations by prioritizing the most relevant documents.
+
+* *Reranker: Documents are then reranked using the multilingual reranker **`Alibaba-NLP/gte-multilingual-reranker-base`** model from Hugging Face.