![](https://private-user-images.githubusercontent.com/13723743/293135587-5ee0314a-d983-444a-8671-88cc0b52b752.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU1OTE0OTIsIm5iZiI6MTcyNTU5MTE5MiwicGF0aCI6Ii8xMzcyMzc0My8yOTMxMzU1ODctNWVlMDMxNGEtZDk4My00NDRhLTg2NzEtODhjYzBiNTJiNzUyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTA2VDAyNTMxMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWZkYjQ0YWZiYWRkMWE4M2NlOTg3Y2QwZGEwNTFiMWFkMTRlOTFlNzMyYzg5NDdkM2RhZTg5NWVmYzc5OWMxOTkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.kd3UdiWEetj-NHODTxaRo4hGZ7xzAktw2vzDtC-YkX0)
ControlLLM: Augmenting Large Language Models with Tools by Searching on Graphs
[Paper] [Project Page] [Demo] [🤗 Space]
We present ControlLLM, a novel framework that enables large language models (LLMs) to utilize multi-modal tools for solving complex real-world tasks. Despite the remarkable performance of LLMs, they still struggle with tool invocation due to ambiguous user prompts, inaccurate tool selection and parameterization, and inefficient tool scheduling. To overcome these challenges, our framework comprises three key components: (1) a
cllm_demo.mp4
- Image Perception
- Image Editing
- Image Generation
- Video Perception
- Video Editing
- Video Generation
- Audio Perception
- Audio Generation
- Multi-Solution
- Pointing Inputs
- Resource Type Awareness
- ✅ (🔥 New) Rlease online demo and 🤗Hugging Face space.
- ✅ (🔥 New) Support PixArt-alpha, a state-of-the-art method for Text-to-Image synthesis.
- Linux
- Python 3.10+
- PyTorch 2.0+
- CUDA 11.8+
Execute the following command in the root directory:
git clone https://github.com/OpenGVLab/ControlLLM.git
cd controlllm
Setup environment:
conda create -n cllm python=3.10
conda activate cllm
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia
Install LLaVA:
pip install git+https://github.com/haotian-liu/LLaVA.git
Then install other dependencies:
pip install -r requirements.txt
pip install -e .
Please put your personal OpenAI Key and Weather Key into the corresponding environment variables.
😬 Launch all in one endpoint:
# openai key
export OPENAI_API_KEY="..."
# openai base
export OPENAI_BASE_URL="..."
# weather api key
export WEATHER_API_KEY="..."
# resource dir
export SERVER_ROOT="./server_resources"
python -m cllm.services.launch --port 10056 --host 0.0.0.0
Take image generation as an example, we first launch the service.
python -m cllm.services.image_generation.launch --port 10011 --host 0.0.0.0
Then, we can call the services via python api:
from cllm.services.image_generation.api import *
setup(port=10011)
text2image('A horse')
export OPENAI_BASE_URL="..."
export OPENAI_API_KEY="..."
python -m cllm.services.tog.launch --port 10052 --host 0.0.0.0
Use openssl
to generate the certificate:
mkdir certificate
openssl req -x509 -newkey rsa:4096 -keyout certificate/key.pem -out certificate/cert.pem -sha256 -days 365 -nodes
Last, you can launch gradio demo in your server:
export TOG_PORT=10052
export CLLM_SERVICES_PORT=10056
export CLIENT_ROOT="./client_resources"
export GRADIO_TEMP_DIR="$HOME/.tmp"
export OPENAI_BASE_URL="..."
export OPENAI_API_KEY="..."
python -m cllm.app.gradio --controller "cllm.agents.tog.Controller" --server-port 10003 --https
Alternatively, you can set above variables in run.sh
and launch all services by running:
bash ./run.sh
This project is released under the Apache 2.0 license.
If you find this project useful in your research, please cite our paper:
@article{2023controlllm,
title={ControlLLM: Augment Language Models with Tools by Searching on Graphs},
author={Liu, Zhaoyang and Lai, Zeqiang and Gao, Zhangwei and Cui, Erfei and Li, Zhiheng and Zhu, Xizhou and Lu, Lewei and Chen, Qifeng and Qiao, Yu and Dai, Jifeng and Wang, Wenhai},
journal={arXiv preprint arXiv:2305.10601},
year={2023}
}
- Thanks to the open source of the following projects: Hugging Face LangChain SAM Stable Diffusion ControlNet InstructPix2Pix EasyOCR ImageBind PixArt-alpha LLaVA Modelscope AudioCraft Whisper Llama 2 LLaMA
If you want to join our WeChat group, please scan the following QR Code to add our assistant as a Wechat friend: