A streamlit ChatBot running StableLM Zephyr 3B with Openvino and Optimum Intel
- StableLM-3B Chatbot - A streamlit CHATBOT interface with stablelm-zephyr-3b quantized in 4bit and optimum-intel. The Interface has a kind text streaming effect, and the number of turns are handled to not exceed the context window. The Model used is published on Hugging Face Hub and was created with the free HF Space hosting the NCCF-quantization tool.
python312 -m venv venv
.\venv\Scripts\activate
python -m pip install --upgrade pip
pip install openvino-genai==2024.4.0
pip install optimum-intel[openvino] tiktoken streamlit==1.36.0
from HuggingFace
https://huggingface.co/FM-1976/stablelm-zephyr-3b-openvino-4bit
streamlit run .\stappStableLM.py