Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion docs/inference/inference-options.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,11 @@ $$nemoclaw onboard
```

When the wizard asks you to choose an inference provider, select **Other OpenAI-compatible endpoint**.
Enter the base URL of your local server, for example `http://localhost:8000/v1`.
Enter the base URL of your local server, for example `http://localhost:8000/v1` on the default Docker-driver topology.
If you are on the legacy containerized OpenShell gateway path and the gateway cannot reach host `localhost`, use the host-gateway URL for your setup, commonly `http://host.openshell.internal:8000/v1`.

For GGUF models, start a compatible server such as `llama-server` yourself and enter the model id that server reports from `/v1/models`.
The Ollama local path selects Ollama model tags; it does not accept a raw `.gguf` file path directly.

The wizard prompts for an API key.
If your server does not require authentication, enter any non-empty string (for example, `dummy`).
Expand Down
32 changes: 32 additions & 0 deletions docs/inference/use-local-inference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,38 @@ Use **Other OpenAI-compatible endpoint** for vLLM, TensorRT-LLM, llama.cpp, Loca
For compatible endpoints, NemoClaw uses `/v1/chat/completions` by default because some local backends accept `/v1/responses` but drop system prompts or tool definitions.
Set `NEMOCLAW_PREFERRED_API=openai-responses` only after you have verified that the backend streams the events OpenClaw requires.

Use this path when you already have a local server for a raw model file, such as a GGUF model served by `llama-server`.
The Ollama picker accepts Ollama model tags, not a filesystem path to a `.gguf` file.
To use a downloaded GGUF file directly, start an OpenAI-compatible server first, then point NemoClaw at that server.

For example, start llama.cpp on the host:

```bash
llama-server \
-m /models/NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf \
--host 0.0.0.0 \
--port 8000 \
-c 16384 \
-ngl 999 \
--parallel 1 \
--chat-template chatml
```

Then select **Other OpenAI-compatible endpoint** during onboarding and enter the server base URL, for example `http://localhost:8000/v1` on the default Docker-driver topology.
For non-interactive setup:

```bash
NEMOCLAW_PROVIDER=custom \
NEMOCLAW_ENDPOINT_URL=http://localhost:8000/v1 \
NEMOCLAW_MODEL=NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf \
COMPATIBLE_API_KEY=dummy \
$$nemoclaw onboard --non-interactive
```

Use the model id reported by the server's `/v1/models` response.
If your setup uses the legacy containerized OpenShell gateway path instead of the default host-process gateway, the reachable host-service URL can differ; `http://host.openshell.internal:8000/v1` is the usual legacy host-gateway route and port `8000` is included in NemoClaw's `local-inference` policy preset.
After onboarding, run `$$nemoclaw <name> status` and check the `Inference` row before starting long agent work.

For the full compatible-endpoint prompt flow, non-interactive variables, API-path controls, managed vLLM profiles, NIM setup, and timeout settings, refer to [Inference Options](inference-options#setup-details-for-local-and-compatible-providers).

## Managed vLLM and NIM
Expand Down
Loading