⚕️ Multi-Agent-Medical-Assistant :

AI-powered multi-agentic system for medical diagnosis and assistance

---

📚 Table of Contents

• Overview
• Demo
• Technical Flow Chart
• Key Features
• Tech Stack
• Installation and Setup
• Usage
• Contributions
• License
• Citing
• Contact

---

📌 Overview

The Multi-Agent Medical Assistant is an AI-powered chatbot designed to assist with medical diagnosis, research, and patient interactions.

🚀 Powered by Multi-Agent Intelligence, this system integrates:

• 🤖 Large Language Models (LLMs)
• 🖼️ Computer Vision Models for medical imaging analysis
• 📚 Retrieval-Augmented Generation (RAG) leveraging vector databases
• 🌐 Real-time Web Search for up-to-date medical insights
• 👨‍⚕️ Human-in-the-Loop Validation to verify AI-based medical image diagnoses

What You’ll Learn from This Project 👨‍💻📖

🔹 Multi-Agent Orchestration with structured graph workflows
🔹 Advanced RAG Techniques – hybrid retrieval, semantic chunking, and vector search
🔹 ⚡ Confidence-Based Routing & Agent-to-Agent Handoff
🔹 🔒 Scalable, Production-Ready AI with Modularized Code & Robust Exception Handling

📂 For learners: Check out agents/README.md for a detailed breakdown of the agentic workflow! 🎯

---

💫 Demo

Multi-Agent-Medical-Assistant-compressed.mp4

If you like what you see and would want to support the project's developer, you can ! :)

---

🛡️ Technical Flow Chart

---

✨ Key Features

• 🤖 Multi-Agent Architecture : Specialized agents working in harmony to handle diagnosis, information retrieval, reasoning, and more

• 🔍 Advanced RAG Retrieval System : Leveraging Qdrant for precise vector search and sophisticated hybrid retrieval techniques, supported file types: .txt, .csv, .json, .pdf

• 🏥 Medical Imaging Analysis

  • Brain Tumor Detection
  • Chest X-ray Disease Classification
  • Skin Lesion Segmentation

• 🌐 Real-time Research Integration : Web search agent that retrieves the latest medical research papers and findings

• 📊 Confidence-Based Verification : Log probability analysis ensures high accuracy in medical recommendations

• 🎙️ Voice Interaction Capabilities : Seamless speech-to-text and text-to-speech powered by Eleven Labs API

• 👩‍⚕️ Expert Oversight System : Human-in-the-loop verification by medical professionals before finalizing outputs

• ⚔️ Input & Output Guardrails : Ensures safe, unbiased, and reliable medical responses while filtering out harmful or misleading content

• 💻 Intuitive User Interface : Designed for healthcare professionals with minimal technical expertise

---

🛠️ Technology Stack

Component	Technologies
🔹 Backend Framework	FastAPI, Flask
🔹 Agent Orchestration	LangGraph
🔹 Knowledge Storage	Qdrant Vector Database
🔹 Medical Imaging	Computer Vision Models
	• Brain Tumor: Object Detection
	• Chest X-ray: Image Classification
	• Skin Lesion: Semantic Segmentation
🔹 Guardrails	LangChain
🔹 Speech Processing	Eleven Labs API
🔹 Frontend	HTML, CSS, JavaScript
🔹 Deployment	Docker, CI/CD Pipeline

---

🚀 Installation & Setup

1️⃣ Clone the Repository

git clone https://github.com/souvikmajumder26/Multi-Agent-Medical-Assistant.git  
cd Multi-Agent-Medical-Assistant  

2️⃣ Create & Activate Virtual Environment

• If using conda:

conda create --name <environment-name> python=3.11
conda activate <environment-name>

• If using python venv:

python -m venv <environment-name>
source <environment-name>/bin/activate  # For Mac/Linux
<environment-name>\Scripts\activate     # For Windows  

3️⃣ Install Dependencies

Important

ffmpeg is required for speech service to work.

• If using conda:

conda install -c conda-forge ffmpeg
pip install -r requirements.txt  

• If using python venv:

winget install ffmpeg
pip install -r requirements.txt  

4️⃣ Set Up API Keys

• Create a .env file and add the following API keys:

Note

You may use any llm and embedding model of your choice...

1. If using Azure OpenAI, no modification required.
2. If using direct OpenAI, modify the llm and embedding model definitions in the 'config.py' na provide appropriate env variables.
3. If using local models, appropriate code changes will be required throughout the codebase especially in 'agents'.

Warning

If all necessary env variables are not provided, errors will be thrown in console.

# LLM Configuration (Azure Open AI - gpt-4o used in development)
# If using any other LLM API key or local LLM, appropriate code modification is required
deployment_name = 
model_name = gpt-4o
azure_endpoint = 
openai_api_key = 
openai_api_version = 

# Embedding Model Configuration (Azure Open AI - text-embedding-ada-002 used in development)
# If using any other embedding model, appropriate code modification is required
embedding_deployment_name =
embedding_model_name = text-embedding-ada-002
embedding_azure_endpoint = 
embedding_openai_api_key = 
embedding_openai_api_version = 

# Speech API Key (Free credits available with new Eleven Labs Account)
ELEVEN_LABS_API_KEY = 

# Web Search API Key (Free credits available with new Tavily Account)
TAVILY_API_KEY = 

# Hugging Face Token - using reranker model "ms-marco-TinyBERT-L-6"
HUGGINGFACE_TOKEN = 

# (OPTIONAL) If using Qdrant server version, local does not require API key
QDRANT_URL = 
QDRANT_API_KEY = 

5️⃣ Run the Application

• Run the following commands one after another in separate windows with same directorty and virtual environment. Keep both running simultanesouly.

uvicorn api.fastapi_backend:app --reload

python app.py

6️⃣ Ingest additional data into the Vector DB

• Run any one of the following commands as required. First one to ingest one document at a time, second one to ingest multiple documents from a directory.

python ingest_rag_data.py --file ./data/raw/brain_tumors_ucni.pdf

python ingest_rag_data.py --dir ./data/raw

---

🧠 Usage

• Upload medical images for AI-based diagnosis. Task specific Computer Vision model powered agents - upload images from 'sample_images' folder to try out.
• Ask medical queries to leverage retrieval-augmented generation (RAG) if information in memory or web-search to retrieve latest information.
• Use voice-based interaction (speech-to-text and text-to-speech).
• Review AI-generated insights with human-in-the-loop verification.

---

🤝 Contributions

Contributions are welcome! Please check the issues tab for feature requests and improvements.

---

⚖️ License

This project is licensed under the Apache-2.0 License. See the LICENSE file for details.

---

📝 Citing

@misc{Souvik2025,
  Author = {Souvik Majumder},
  Title = {Multi Agent Medical Assistant},
  Year = {2025},
  Publisher = {GitHub},
  Journal = {GitHub repository},
  Howpublished = {\url{https://github.com/souvikmajumder26/Multi-Agent-Medical-Assistant}}
}

---

📬 Contact

For any questions or collaboration inquiries, reach out to Souvik Majumder on:

🔗 LinkedIn: https://www.linkedin.com/in/souvikmajumder26

🔗 GitHub: https://github.com/souvikmajumder26

🔝 Return

---