Skip to content

docs(docker): add section on connecting to local inference servers (vLLM, Ollama) - #12335

Closed
Magicray1217 wants to merge 1 commit into
NousResearch:mainfrom
Magicray1217:docs/docker-vllm-setup
Closed

docs(docker): add section on connecting to local inference servers (vLLM, Ollama)#12335
Magicray1217 wants to merge 1 commit into
NousResearch:mainfrom
Magicray1217:docs/docker-vllm-setup

Conversation

@Magicray1217

Copy link
Copy Markdown

Summary

Adds a comprehensive guide for connecting Dockerized Hermes to local inference servers like vLLM and Ollama.

What's added

A new section Connecting to local inference servers (vLLM, Ollama, etc.) in the Docker docs page, covering:

  • Docker Compose networking (recommended) — both services on the same network, using container names as hostnames
  • Standalone Docker run — \host.docker.internal\ for macOS/Windows, --network host\ for Linux
  • Connectivity verification — how to test from inside the container
  • Ollama example — specific config for Ollama

Why

Users frequently hit networking issues when connecting Docker Hermes to local inference servers (#12308). The main gotchas:

  1. Using \localhost\ instead of the container name or \host.docker.internal\
  2. Mismatched --served-model-name\ vs \model:\ config value
  3. Trailing slash in \�ase_url\
  4. Server bound to \127.0.0.1\ instead of \

…LLM, Ollama)

Adds a comprehensive guide for connecting Dockerized Hermes to local
inference servers like vLLM and Ollama, covering:
- Docker Compose networking (recommended)
- Standalone Docker run with host.docker.internal / --network host
- Connectivity verification steps
- Ollama-specific example

Closes NousResearch#12308
@mxnstrexgl

Copy link
Copy Markdown

✓ Automated scan: docs only. api_key: "none" is example config, not real credentials.

@alt-glitch alt-glitch added type/docs Documentation improvements P3 Low — cosmetic, nice to have area/docker Docker image, Compose, packaging labels Apr 23, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Related to #7965 (open PR for same Docker-to-host LLM guide) and #7963 (the docs issue). May be competing PRs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/docker Docker image, Compose, packaging P3 Low — cosmetic, nice to have type/docs Documentation improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants