Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 23 additions & 21 deletions docs/inference/set-up-vllm.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -92,40 +92,42 @@ The first run can take 10 to 30 minutes.
Later runs reuse the cached image and model weights.
The managed container uses Docker's `--restart unless-stopped` policy, so Docker restarts it unless an operator explicitly stops it.

### Check Storage Before Downloads
### Check Storage Before the Image Pull

For an uncached image, NemoClaw checks Docker image storage before the watched image pull starts.
It checks the Hugging Face model cache before that pull and checks it again afterward before `hf download` starts.
The image estimate includes compressed content, unpacked layers, and pull-staging headroom.
The model estimate includes the model's full published size plus temporary-write headroom because a local cache cannot prove which files a mutable remote revision will reuse.
For an uncached pinned image, NemoClaw checks Docker image storage before the watched image pull starts.
The estimate reserves three times the image's compressed registry size plus `3 GiB` for compressed content, unpacked layers, and pull staging.

For Docker's classic image store, NemoClaw checks the absolute `DockerRootDir` reported by `docker info`.
For Docker's containerd image store, it checks the top-level `root` from `/etc/containerd/config.toml`, or `/var/lib/containerd` when that configuration file is absent, and also checks `DockerRootDir` for pull staging.
The check uses the location with the least available capacity.
The model-cache check uses `~/.cache/huggingface`, or its nearest existing parent filesystem when the cache directory does not exist yet.

NemoClaw recognizes an image cache hit only when Docker contains the exact platform digest.
A cache hit skips the cold-pull capacity check, but the explicit watched `docker pull` still runs against that immutable digest.
Before trusting the host model-cache capacity, NemoClaw also verifies that the selected Docker daemon maps bind mounts to the local host; a remote or unverifiable Docker host makes the model-cache check inconclusive.
An explicit Unix socket is trusted only when it is Docker's default `/var/run/docker.sock`; any other socket can forward to a daemon with a different filesystem and is treated as unverifiable.
Known container markers are rejected immediately.
Before trusting client-side image-store capacity, NemoClaw requires the selected default Docker socket peer to have a visible PID and share the client's mount namespace; a nested PID namespace or unverifiable peer fails closed before the image pull.
Before trusting model-cache capacity, NemoClaw also asks the pinned cached vLLM image to read a random mode-`0600` sentinel through a bounded, read-only Docker bind mount.
The pull is idempotent for the cached digest and cannot select different layers from a moved release tag.
The model-download helper and the long-lived vLLM container use `--pull=never`, so an implicit pull cannot bypass the storage gate.
The second model-cache check detects capacity consumed by the image pull before the model download command can start.

When capacity is insufficient or NemoClaw cannot verify it, the warning identifies the image or model, available capacity, approximate requirement, checked path and source, remediation, and diagnostic commands.
A cache hit skips the capacity check, but the explicit watched `docker pull` still runs against that immutable digest.
The model-download helper and the long-lived vLLM container use `--pull=never`, so an implicit pull cannot select another image.

NemoClaw treats `/var/run/docker.sock` and `/run/docker.sock` as the local default Docker socket paths.
It cannot inspect daemon storage through a remote endpoint, a named non-default Docker context, or another Unix socket.
When both `DOCKER_CONTEXT` and `DOCKER_HOST` are set, NemoClaw follows Docker CLI precedence and uses `DOCKER_CONTEXT`.
When the check is inconclusive, NemoClaw prints the reason, approximate image requirement, and diagnostic commands, then continues with the pull without claiming that capacity is sufficient.

When verified capacity is insufficient, the warning identifies the image, available capacity, approximate requirement, checked path and source, remediation, and diagnostic commands.
Interactive setup continues only when you enter `y` or `yes` explicitly.
Pressing **Enter**, entering `n` or `no`, or entering any other response stops before the guarded pull or model download starts.
Pressing **Enter**, entering `n` or `no`, or entering any other response stops before the guarded image pull starts.

<Warning>
`--yes` and `NEMOCLAW_YES=1` do not accept managed-vLLM storage warnings.
`--yes` and `NEMOCLAW_YES=1` do not accept a verified insufficient-storage warning.
Non-interactive setup stops unless you set `NEMOCLAW_IGNORE_VLLM_DISK_SPACE=1` exactly.
This override accepts both insufficient-capacity and inconclusive checks for Docker image storage and the model cache.
Use it only after you independently verify that both storage locations can complete the downloads.
The override applies only when NemoClaw verifies that Docker image storage is insufficient.
Use it only after you independently verify that Docker storage can complete the image pull.
Inconclusive checks continue automatically and do not require the override.
</Warning>

<Note>
This preflight checks only the pinned vLLM container image.
It does not estimate or gate the Hugging Face model-weight download into `~/.cache/huggingface`.
Before the first managed-vLLM run, verify that the filesystem backing your model cache has enough free space for the selected model.
</Note>

If image pull output stops making progress, a watchdog stops the stalled pull without imposing a fixed wall-clock limit on slow but active downloads.
If vLLM does not become ready, NemoClaw prints a short tail of the container logs before exiting.

Expand Down
2 changes: 1 addition & 1 deletion docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3250,7 +3250,7 @@ Set them before running `$$nemoclaw onboard`.
| `NEMOCLAW_INSTALL_TAG` | release tag | For internal installer commands: the release tag to install. Defaults to the admin-promoted `lkg` tag when unset. Overridden by the `--install-tag` flag. |
| `NEMOCLAW_VLLM_MODEL` | registry slug or Hugging Face model id | Selects the model the managed-vLLM install path serves. Recognised slugs: `qwen3.6-27b`, `qwen3.6-35b-a3b-nvfp4`, `nemotron-3-nano-4b`, `deepseek-v4-flash`, `deepseek-r1-distill-70b`. Unset uses the per-platform profile default. Gated models (e.g. `deepseek-r1-distill-70b`) require `HF_TOKEN` or `HUGGING_FACE_HUB_TOKEN`. |
| `NEMOCLAW_VLLM_EXTRA_ARGS_JSON` | JSON array of non-blank strings | Appends advanced operator-owned tokens to the managed `vllm serve` command after NemoClaw's registry defaults. Example: `["--max-num-seqs","2"]`. Malformed JSON, non-string tokens, or blank tokens fail before Docker work starts. |
| `NEMOCLAW_IGNORE_VLLM_DISK_SPACE` | exactly `1` | Allows managed vLLM to continue when Docker image-storage or model-cache capacity is insufficient or cannot be verified; other values do not bypass the checks. |
| `NEMOCLAW_IGNORE_VLLM_DISK_SPACE` | exactly `1` | Allows managed vLLM to continue when verified Docker image-storage capacity is insufficient; other values do not bypass the check. Inconclusive image-storage checks report diagnostics and continue automatically. The check does not cover Hugging Face model-cache capacity. |
<AgentOnly variant="openclaw">
| `NEMOCLAW_MINIMAL_BOOTSTRAP` | `1` to enable | Skips default OpenClaw workspace-template seeding for new pristine workspaces. Existing files are not deleted; refer to [Understand Runtime Changes](../manage-sandboxes/configure-sandboxes/understand-runtime-changes). |
</AgentOnly>
Expand Down
2 changes: 1 addition & 1 deletion scripts/install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2655,7 +2655,7 @@ detect_express_platform() {
fi
case "$model" in
*DGX*Spark*) printf "DGX Spark" ;;
*DGX*Station*) printf "DGX Station" ;;
*DGX*Station* | *Station*GB300*) printf "DGX Station" ;;
*) ;;
esac
}
Expand Down
19 changes: 0 additions & 19 deletions src/lib/inference/vllm-models.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,25 +16,6 @@ import {
} from "./vllm-models";

describe("vllm model registry", () => {
it("records a finite positive Hugging Face download size for every model", () => {
for (const model of VLLM_MODELS) {
expect(Number.isFinite(model.downloadSizeBytes)).toBe(true);
expect(model.downloadSizeBytes).toBeGreaterThan(0);
}
});

it("pins the official Hugging Face repository totals", () => {
expect(
Object.fromEntries(VLLM_MODELS.map((model) => [model.envValue, model.downloadSizeBytes])),
).toEqual({
"qwen3.6-27b": 30_900_000_000,
"deepseek-r1-distill-70b": 141_000_000_000,
"nemotron-3-nano-4b": 5_280_000_000,
"deepseek-v4-flash": 160_000_000_000,
"qwen3.6-35b-a3b-nvfp4": 23_500_000_000,
});
});

it("returns null when NEMOCLAW_VLLM_MODEL is unset so the caller can fall back to the profile default", () => {
expect(selectVllmModelFromEnv({} as NodeJS.ProcessEnv)).toBeNull();
});
Expand Down
7 changes: 0 additions & 7 deletions src/lib/inference/vllm-models.ts
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,6 @@ export interface VllmModelDef {
label: string;
/** Stable identifier accepted via `NEMOCLAW_VLLM_MODEL`. */
envValue: string;
/** Approximate full Hugging Face repository download size in bytes. */
downloadSizeBytes: number;
/** `--max-model-len` flag value. */
maxModelLen: number;
/** Model-specific flags appended after the shared serving flags. */
Expand Down Expand Up @@ -65,7 +63,6 @@ export const VLLM_MODELS: readonly VllmModelDef[] = [
id: "Qwen/Qwen3.6-27B-FP8",
label: "Qwen3.6 27B FP8",
envValue: "qwen3.6-27b",
downloadSizeBytes: 30_900_000_000,
maxModelLen: 262144,
modelArgs: [
"--gpu-memory-utilization",
Expand All @@ -88,7 +85,6 @@ export const VLLM_MODELS: readonly VllmModelDef[] = [
id: "deepseek-ai/DeepSeek-R1-Distill-Llama-70B",
label: "DeepSeek-R1 Distill Llama 70B",
envValue: "deepseek-r1-distill-70b",
downloadSizeBytes: 141_000_000_000,
maxModelLen: 32768,
modelArgs: [
"--gpu-memory-utilization",
Expand All @@ -108,7 +104,6 @@ export const VLLM_MODELS: readonly VllmModelDef[] = [
id: "nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8",
label: "NVIDIA Nemotron-3 Nano 4B FP8",
envValue: "nemotron-3-nano-4b",
downloadSizeBytes: 5_280_000_000,
// Matches the model card's `max_position_embeddings` and the vLLM
// example NVIDIA publishes for this checkpoint. The previous value
// (262000) was an undocumented round-down with no headroom rationale.
Expand Down Expand Up @@ -138,7 +133,6 @@ export const VLLM_MODELS: readonly VllmModelDef[] = [
id: "deepseek-ai/DeepSeek-V4-Flash",
label: "DeepSeek V4 Flash",
envValue: "deepseek-v4-flash",
downloadSizeBytes: 160_000_000_000,
maxModelLen: 1048576,
modelArgs: [
"--kv-cache-dtype",
Expand Down Expand Up @@ -179,7 +173,6 @@ export const VLLM_MODELS: readonly VllmModelDef[] = [
id: "nvidia/Qwen3.6-35B-A3B-NVFP4",
label: "Qwen3.6 35B-A3B NVFP4",
envValue: "qwen3.6-35b-a3b-nvfp4",
downloadSizeBytes: 23_500_000_000,
maxModelLen: 262144,
// Additive flags on top of the shared serving defaults. The shared flags
// already cover --tensor-parallel-size/--pipeline-parallel-size/
Expand Down
Loading
Loading