2. Services

Various services that are integrated with Harbor. The link in the service name will lead you to a dedicated page in Harbor's wiki with details on getting started with the service.

Frontends

This section covers services that can provide you with an interface for interacting with the language models.

• Open WebUI
  widely adopted and feature rich web interface for interacting with LLMs. Supports OpenAI-compatible and Ollama backends, multi-users, multi-model chats, custom prompts, TTS, Web RAG, RAG, and much much more.

• ComfyUI
  The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface.

• LibreChat
  Open-source ChatGPT UI alternative supporting multiple AI providers (Anthropic, AWS, OpenAI, Azure, Groq, Mistral, Google) with features like model switching, message search, and multi-user support. Includes integration with DALL-E-3 and various APIs.

• HuggingFace ChatUI
  A chat interface using open source models, eg OpenAssistant or Llama. It is a SvelteKit app and it powers the HuggingChat app on hf.co/chat.

• Lobe Chat
  An open-source, modern-design AI chat framework. Supports Multi AI Providers( OpenAI / Claude 3 / Gemini / Ollama / Azure / DeepSeek), Knowledge Base (file upload / knowledge management / RAG ), Multi-Modals (Vision/TTS) and plugin system.

• hollama
  A minimal web-UI for talking to Ollama servers.

• parllama
  TUI for Ollama

• BionicGPT
  on-premise LLM web UI with support for OpenAI-compatible backends

• AnythingLLM
  The all-in-one Desktop & Docker AI application with built-in RAG, AI agents, and more.

• Chat Nio
  Comprehensive LLM web interface with built-in marketplace

Backends

This section covers services that provide the LLM inference capabilities.

• Ollama
  Get up and running with Llama 3.2, Mistral, Gemma 2, and other large language models.

• llama.cpp
  LLM inference in C/C++

• vLLM
  A high-throughput and memory-efficient inference and serving engine for LLMs

• TabbyAPI
  An OAI compatible exllamav2 API that's both lightweight and fast

• Aphrodite Engine
  Large-scale LLM inference engine

• mistral.rs
  Blazingly fast LLM inference.

• openedai-speech
  An OpenAI API compatible text to speech server using Coqui AI's xtts_v2 and/or piper tts as the backend.

• Parler
  Inference and training library for high-quality TTS models.

• text-generation-inference
  Inference engine from HuggingFace.

• lmdeploy

• AirLLM
  70B inference with single 4GB GPU (very slow, though)

• SGLang
  SGLang is a fast serving framework for large language models and vision language models.

• ktransformers
  A Flexible Framework for Experiencing Cutting-edge LLM Inference Optimizations

• Speaches, aka Faster Whisper Server
  an OpenAI API-compatible speech server (formerly faster-whisper-server), both TTS and STT

• Nexa SDK
  Nexa SDK is a comprehensive toolkit for supporting ONNX and GGML models.

• KoboldCpp
  KoboldCpp is an easy-to-use AI text-generation software for GGML and GGUF models.

Satellite services

Additional services that can be integrated with various Frontends and Backends to enable more features.

• SearXNG
  A privacy-respecting, hackable metasearch engine. Highly configurable and can be used for Web RAG use-cases.

• Perplexica
  An AI-powered search engine. It is an Open source alternative to Perplexity AI.

• Dify
  An open-source LLM app development platform.

• Plandex
  AI driven development in your terminal.

• 🚅 LiteLLM
  LLM proxy that can aggregate multiple inference APIs together into a single endpoint.

• langfuse
  Open source LLM engineering platform: LLM Observability, metrics, evals, prompt management, playground, datasets.

• ● Open Interpreter
  A natural language interface for computers.

• cloudflared
  A helper service allowing to expose Harbor services over the internet.

• cmdh
  Create Linux commands from natural language, in the shell.

• fabric
  LLM-driven processing of the text data in the terminal.

• txtai RAG
  RAG WebUI built with txtai.

• TextGrad
  Automatic "Differentiation" via Text - using large language models to backpropagate textual gradients.

• aider
  Aider is AI pair programming in your terminal.

• aichat
  All-in-one LLM CLI tool featuring Shell Assistant, Chat-REPL, RAG, AI tools & agents.

• autogpt
  Create, deploy, and manage continuous AI agents that automate complex workflows.

• omnichain
  Visual programming for AI language models

• Harbor Bench
  Harbor's own tool to evaluate LLMs and inference backends against custom tasks.

• lm-evaluation-harness
  A de-facto standard framework for the few-shot evaluation of language models.

• JupyterLab
  Helper service to author/run Jupyter notebooks in Python with access to Harbor services.

• ol1
  A simple Gradio app implementing an o1-like chain of reasoning with Ollama.

• Harbor Boost
  Connects to downstream LLM API and serves a wrapper with custom workflow. For example, it can be used to add a CoT (Chain of Thought) to an existing LLM API, and much more. Scriptable with Python.

• OpenHands
  A platform for software development agents powered by AI.

• LitLytics
  Simple analytics platform that leverages LLMs to automate data analysis.

• Repopack
  A powerful tool that packs your entire repository into a single, AI-friendly file.

• n8n
  Fair-code workflow automation platform with native AI capabilities.

• Bolt.new
  Prompt, run, edit, and deploy full-stack web applications.

• Open WebUI Pipelines
  UI-Agnostic OpenAI API Plugin Framework.

• Qdrant
  Qdrant - High-performance, massive-scale Vector Database and Vector Search Engine.

• K6
  A modern load testing tool, using Go and JavaScript - https://k6.io

• Promptfoo
  Test your prompts, agents, and RAGs. A developer-friendly local tool for testing LLM applications.

• Webtop
  Linux in a web browser supporting popular desktop environments.

• OmniParser
  A simple screen parsing tool towards pure vision based GUI agent.

• Flowise
  Drag & drop UI to build your customized LLM flow.

• LangFlow
  A low-code app builder for RAG and multi-agent AI applications.

• OptiLLM
  Optimising LLM proxy that implements many advanced workflows to boost the performance of the LLMs.