diff --git a/.agents/skills/nemoclaw-user-configure-inference/SKILL.md b/.agents/skills/nemoclaw-user-configure-inference/SKILL.md index 3644caced47..749dbc79c63 100644 --- a/.agents/skills/nemoclaw-user-configure-inference/SKILL.md +++ b/.agents/skills/nemoclaw-user-configure-inference/SKILL.md @@ -372,7 +372,9 @@ For an already-running vLLM server, run `nemohermes onboard` and select **Local If vLLM is already running, NemoClaw detects the running model and validates the endpoint. If vLLM is not running and your host matches a DGX Spark or DGX Station managed profile, NemoClaw shows the **Install vLLM** or **Start vLLM** entry by default. Generic Linux NVIDIA GPU hosts still require `NEMOCLAW_EXPERIMENTAL=1` or `NEMOCLAW_PROVIDER=install-vllm` before the managed entry appears. -NemoClaw pulls the vLLM image, downloads model weights into `~/.cache/huggingface`, starts the `nemoclaw-vllm` container on `localhost:8000`, and prints progress markers while the model loads. +NemoClaw pulls the vLLM image, downloads model weights into `~/.cache/huggingface`, starts the `nemoclaw-vllm` container on `localhost:8000`, streams Hugging Face download progress, and polls `/v1/models` until the model is ready. +If Docker pull output stops making progress, a watchdog stops the stalled pull instead of failing slow but active downloads on a fixed wall-clock timeout. +If vLLM never becomes ready, NemoClaw prints a short tail of the vLLM container logs before exiting. The first run can take 10 to 30 minutes. Later runs reuse the cached image and model weights. diff --git a/.agents/skills/nemoclaw-user-configure-inference/references/switch-inference-providers.md b/.agents/skills/nemoclaw-user-configure-inference/references/switch-inference-providers.md index c9640d3af80..384aa069e51 100644 --- a/.agents/skills/nemoclaw-user-configure-inference/references/switch-inference-providers.md +++ b/.agents/skills/nemoclaw-user-configure-inference/references/switch-inference-providers.md @@ -172,6 +172,8 @@ session and the baked image. Switching to a different provider family (for example, from NVIDIA Endpoints to Anthropic) also uses `nemoclaw inference set`. The command updates both the gateway route and the OpenClaw provider namespace in the running sandbox config. +If the in-sandbox config sync fails after the gateway route is updated, NemoClaw keeps the host registry aligned with the gateway and prints a rebuild hint. +Run the rebuild before relying on the running agent if the warning says the image config could not be patched. ```bash nemoclaw inference set --provider anthropic-prod --model claude-sonnet-4-6 --no-verify @@ -181,6 +183,8 @@ nemoclaw inference set --provider anthropic-prod --model claude-sonnet-4-6 --no- Switching to a different provider family (for example, from NVIDIA Endpoints to Anthropic) also uses `nemohermes inference set`. The command updates both the gateway route and `/sandbox/.hermes/config.yaml`. +If the Hermes config sync fails after the gateway route is updated, NemoClaw keeps the host registry aligned with the gateway and prints a rebuild hint. +Run the rebuild before relying on the running agent if the warning says the image config could not be patched. ```bash nemohermes inference set --provider anthropic-prod --model claude-sonnet-4-6 --no-verify diff --git a/.agents/skills/nemoclaw-user-manage-sandboxes/SKILL.md b/.agents/skills/nemoclaw-user-manage-sandboxes/SKILL.md index a924fcb1a20..ce9f7cf4f6c 100644 --- a/.agents/skills/nemoclaw-user-manage-sandboxes/SKILL.md +++ b/.agents/skills/nemoclaw-user-manage-sandboxes/SKILL.md @@ -170,6 +170,8 @@ When the default port is already held by another sandbox, `nemoclaw onboard` sca When the default API port is already held by another sandbox, `nemohermes onboard` scans for the next free port and records it for the sandbox. +If you intentionally run separate OpenShell gateways on the same host, set a different `NEMOCLAW_GATEWAY_PORT` before each onboarding run. +NemoClaw isolates the gateway name and local state by port so one port-specific gateway does not replace another. ```bash diff --git a/.agents/skills/nemoclaw-user-manage-sandboxes/references/messaging-channels.md b/.agents/skills/nemoclaw-user-manage-sandboxes/references/messaging-channels.md index 32ee5b10068..48966fc6d8c 100644 --- a/.agents/skills/nemoclaw-user-manage-sandboxes/references/messaging-channels.md +++ b/.agents/skills/nemoclaw-user-manage-sandboxes/references/messaging-channels.md @@ -94,10 +94,12 @@ Set `DISCORD_USER_ID` to restrict access to one user; otherwise, any member of t Slack uses Socket Mode and requires two tokens. Use `SLACK_BOT_TOKEN` for the bot user OAuth token (`xoxb-...`) and `SLACK_APP_TOKEN` for the app-level Socket Mode token (`xapp-...`). +NemoClaw validates both tokens before it saves Slack credentials or enables the channel. Set `SLACK_ALLOWED_USERS` to comma-separated Slack member IDs to authorize those users for DMs and for channel `@mention` events in channels where the Slack app is present. Set `SLACK_ALLOWED_CHANNELS` to comma-separated Slack channel IDs to restrict channel `@mention` handling to those channels. When both Slack allowlists are set, NemoClaw requires the mention to come from one of the allowed channels and one of the allowed members. Channel messages still require an explicit bot mention. +During sandbox startup, NemoClaw normalizes OpenShell credential placeholders into the environment shape expected by the Slack runtime, so post-rebuild Slack starts use the gateway-managed tokens instead of literal placeholder strings. WeChat (experimental) delivers messages over Tencent's iLink gateway through the upstream `@tencent-weixin/openclaw-weixin` plugin baked into the sandbox base image and the built-in Hermes iLink WeChat adapter. The supported mode in this release is **personal WeChat** (`bot_type=3`). diff --git a/.agents/skills/nemoclaw-user-reference/references/commands-nemohermes.md b/.agents/skills/nemoclaw-user-reference/references/commands-nemohermes.md index 13d10bd7aec..51db2edb9e4 100644 --- a/.agents/skills/nemoclaw-user-reference/references/commands-nemohermes.md +++ b/.agents/skills/nemoclaw-user-reference/references/commands-nemohermes.md @@ -122,6 +122,8 @@ Three tiers are available: After selecting a tier, the wizard shows a combined preset and access-mode screen where you can include or exclude individual presets and toggle each between read and read-write access. For details on tiers and the presets each includes, see [Network Policies](network-policies.md#policy-tiers). +When you finish the policy step, NemoClaw records the finalized built-in preset selection for that sandbox. +Later re-onboard runs seed from that finalized selection, so presets you intentionally removed stay removed unless you select them again or override the policy mode. In non-interactive mode, set the tier with `NEMOCLAW_POLICY_TIER` (default: `balanced`): @@ -199,6 +201,9 @@ If you enable Telegram during onboarding, the wizard can also prompt for whether Set `TELEGRAM_REQUIRE_MENTION=1` for non-interactive onboarding when you want mention-only group replies. Pairing and `TELEGRAM_ALLOWED_IDS` still govern direct messages. +If you cancel a brand-new onboarding run at the policy preset step, NemoClaw rolls back the sandbox, registry entry, and onboarding session instead of leaving a default sandbox with unfinished policy state. +Existing live sandboxes are not deleted by this cancel rollback path. + If you run onboarding again with the same sandbox name and choose a different inference provider or model, NemoClaw detects the drift and recreates the sandbox so the running agent config matches your selection. In interactive mode, the wizard asks for confirmation before delete and recreate. In non-interactive mode, NemoClaw recreates automatically when the stored selection is readable and differs; if NemoClaw cannot read the stored selection, NemoClaw reuses by default. @@ -362,6 +367,10 @@ If another terminal is already connected to the sandbox, `connect` prints a note An unknown slug or a gated model (for example `deepseek-r1-distill-70b`) with no `HF_TOKEN` or `HUGGING_FACE_HUB_TOKEN` exits non-zero with the same error the installer would emit, before any sandbox readiness probe or SSH attach. Unset the variable, or supply the missing token, before retrying. +When the live OpenShell gateway inference route differs from the route recorded in the NemoClaw registry, `connect` prints an explicit warning and realigns the shared gateway to the recorded route. +Use `nemohermes inference set --provider --model ` to make an intentional route change. +If the sandbox is registered locally but missing from a healthy gateway, `connect` preserves the registry entry and points you to `rebuild --yes`, `onboard`, or `destroy` instead of deleting the metadata needed for recovery. + After a host reboot, the OpenShell gateway rotates its SSH host keys. `connect` detects the resulting identity drift, prunes stale `openshell-*` entries from `~/.ssh/known_hosts`, and retries automatically. You no longer need to re-run `nemohermes onboard` after a reboot in this case. @@ -416,6 +425,8 @@ Pass `--json` to emit a structured per-sandbox report instead of the text render The JSON output includes at least `schemaVersion`, `name`, `found`, `model`, `provider`, `phase`, `gatewayState`, `inferenceHealth`, `rpcIssue`, `hostGpuDetected`, `sandboxGpuEnabled`, `sandboxGpuMode`, `sandboxGpuDevice`, `openshellDriver`, `openshellVersion`, `policies`, `failureLayer`, and `dockerPaused`. `openshellDriver` and `openshellVersion` are always strings (falling back to `"unknown"` when the registry has no value), so consumers can rely on `typeof` checks. `failureLayer` is `null` when no preflight failure was detected and otherwise one of `docker_unreachable`, `sandbox_container_stopped`, or `sandbox_dashboard_port_conflict`; when set, `inferenceHealth` is suppressed to `null` so automation does not see a stale remote-provider healthy status during a local outage. +`dockerPaused` is `true` when NemoClaw detects that the Docker-driver sandbox container is paused. +In that case, text output keeps OpenShell's authoritative phase but prints a `docker unpause ` recovery hint instead of sending you directly to rebuild. The command exits non-zero when the sandbox is missing locally, the gateway state is not `present`, the gateway reports a schema/protobuf mismatch (mirrored as `rpcIssue`), or `failureLayer` is non-null. The alias form `nemohermes status --json` requires the sandbox to be registered locally; the canonical form `nemohermes sandbox status --json` is the one to use from automation that may run against an unknown sandbox name, since it still emits a JSON document with `found: false` instead of a text error. @@ -449,6 +460,7 @@ When the host Docker daemon is reachable but the per-sandbox container is stoppe If the sandbox's recorded dashboard port is also held by a foreign listener, the header escalates to `Failure layer: sandbox_dashboard_port_conflict — sandbox container is stopped and the dashboard port is held by a foreign listener.` so the operator can recover the port before restarting the sandbox. If the sandbox or gateway cannot be verified, the command exits non-zero instead of reporting healthy inference from stale registry state. +When a locally registered sandbox is missing from the live gateway, status preserves the registry entry so the suggested `rebuild --yes` recovery can still find the sandbox metadata. Gateway and dashboard health checks treat HTTP `401` from device auth as a live service, not as an offline gateway. A `Connected` line reports whether the sandbox has any active SSH sessions and, if so, how many. @@ -634,6 +646,8 @@ Unchecking a preset in the onboard TUI checkbox also removes it from the sandbox Add a host alias to the sandbox pod template. Use this when a sandbox needs a stable LAN-only name, such as a local SearXNG or internal model endpoint, without dropping to `docker exec` and `kubectl patch`. +Host alias commands use the legacy Kubernetes gateway `Sandbox` resource path. +They are not supported on Docker-driver or VM-driver sandboxes because those drivers do not run the gateway cluster container that owns this resource. ```bash nemohermes my-assistant hosts-add searxng.local 192.168.1.105 @@ -1224,6 +1238,8 @@ nemohermes inference set --provider --model [--sandbox Pass both `--provider` and `--model` when you want NemoClaw to update the OpenShell inference route and sync the selected sandbox's agent config. If you only want the lower-level OpenShell route operation, run `openshell inference set -g nemoclaw --model --provider ` directly. When either flag is missing, `nemohermes inference set` prints that OpenShell command instead of an oclif flag-validation error. +The command updates the host registry immediately after the gateway route changes. +If the in-sandbox config sync fails, NemoClaw keeps the gateway and registry aligned, warns that the running image may still need a rebuild, and points you to `nemohermes rebuild`. Supported provider names are `nvidia-prod`, `nvidia-nim`, `nvidia-router`, `openai-api`, `anthropic-prod`, `compatible-anthropic-endpoint`, `gemini-api`, `compatible-endpoint`, `hermes-provider`, `ollama-local`, and `vllm-local`. Use `--no-verify` only when OpenShell cannot verify the provider at switch time but you have already confirmed the provider and credential. @@ -1429,6 +1445,7 @@ All ports must be non-privileged integers between 1024 and 65535. If a port value is not a valid integer or falls outside the allowed range, the CLI exits with an error. `NEMOCLAW_GATEWAY_PORT` also cannot overlap the configured dashboard, vLLM, Ollama, or Ollama proxy ports, and cannot use the dashboard auto-allocation range `18789` through `18799` or the default inference/proxy ports `8000`, `11434`, and `11435`. +When you run multiple NemoClaw gateways with different `NEMOCLAW_GATEWAY_PORT` values, NemoClaw derives a separate gateway name, state directory, and compatibility container name from the port so one gateway does not tear down another. On non-WSL hosts, `NEMOCLAW_OLLAMA_PORT` and `NEMOCLAW_OLLAMA_PROXY_PORT` must be different. If you run Ollama on port 11435, set `NEMOCLAW_OLLAMA_PROXY_PORT` to another free port before onboarding. diff --git a/.agents/skills/nemoclaw-user-reference/references/commands.md b/.agents/skills/nemoclaw-user-reference/references/commands.md index 4a72dc6b811..efba872fa24 100644 --- a/.agents/skills/nemoclaw-user-reference/references/commands.md +++ b/.agents/skills/nemoclaw-user-reference/references/commands.md @@ -166,6 +166,8 @@ Three tiers are available: After selecting a tier, the wizard shows a combined preset and access-mode screen where you can include or exclude individual presets and toggle each between read and read-write access. For details on tiers and the presets each includes, see [Network Policies](network-policies.md#policy-tiers). +When you finish the policy step, NemoClaw records the finalized built-in preset selection for that sandbox. +Later re-onboard runs seed from that finalized selection, so presets you intentionally removed stay removed unless you select them again or override the policy mode. In non-interactive mode, set the tier with `NEMOCLAW_POLICY_TIER` (default: `balanced`): @@ -250,6 +252,9 @@ If you enable Telegram during onboarding, the wizard can also prompt for whether Set `TELEGRAM_REQUIRE_MENTION=1` for non-interactive onboarding when you want mention-only group replies. Pairing and `TELEGRAM_ALLOWED_IDS` still govern direct messages. +If you cancel a brand-new onboarding run at the policy preset step, NemoClaw rolls back the sandbox, registry entry, and onboarding session instead of leaving a default sandbox with unfinished policy state. +Existing live sandboxes are not deleted by this cancel rollback path. + If you run onboarding again with the same sandbox name and choose a different inference provider or model, NemoClaw detects the drift and recreates the sandbox so the running agent config matches your selection. In interactive mode, the wizard asks for confirmation before delete and recreate. In non-interactive mode, NemoClaw recreates automatically when the stored selection is readable and differs; if NemoClaw cannot read the stored selection, NemoClaw reuses by default. @@ -422,6 +427,10 @@ If another terminal is already connected to the sandbox, `connect` prints a note An unknown slug or a gated model (for example `deepseek-r1-distill-70b`) with no `HF_TOKEN` or `HUGGING_FACE_HUB_TOKEN` exits non-zero with the same error the installer would emit, before any sandbox readiness probe or SSH attach. Unset the variable, or supply the missing token, before retrying. +When the live OpenShell gateway inference route differs from the route recorded in the NemoClaw registry, `connect` prints an explicit warning and realigns the shared gateway to the recorded route. +Use `nemoclaw inference set --provider --model ` to make an intentional route change. +If the sandbox is registered locally but missing from a healthy gateway, `connect` preserves the registry entry and points you to `rebuild --yes`, `onboard`, or `destroy` instead of deleting the metadata needed for recovery. + After a host reboot, the OpenShell gateway rotates its SSH host keys. `connect` detects the resulting identity drift, prunes stale `openshell-*` entries from `~/.ssh/known_hosts`, and retries automatically. You no longer need to re-run `nemoclaw onboard` after a reboot in this case. @@ -490,6 +499,8 @@ Pass `--json` to emit a structured per-sandbox report instead of the text render The JSON output includes at least `schemaVersion`, `name`, `found`, `model`, `provider`, `phase`, `gatewayState`, `inferenceHealth`, `rpcIssue`, `hostGpuDetected`, `sandboxGpuEnabled`, `sandboxGpuMode`, `sandboxGpuDevice`, `openshellDriver`, `openshellVersion`, `policies`, `failureLayer`, and `dockerPaused`. `openshellDriver` and `openshellVersion` are always strings (falling back to `"unknown"` when the registry has no value), so consumers can rely on `typeof` checks. `failureLayer` is `null` when no preflight failure was detected and otherwise one of `docker_unreachable`, `sandbox_container_stopped`, or `sandbox_dashboard_port_conflict`; when set, `inferenceHealth` is suppressed to `null` so automation does not see a stale remote-provider healthy status during a local outage. +`dockerPaused` is `true` when NemoClaw detects that the Docker-driver sandbox container is paused. +In that case, text output keeps OpenShell's authoritative phase but prints a `docker unpause ` recovery hint instead of sending you directly to rebuild. The command exits non-zero when the sandbox is missing locally, the gateway state is not `present`, the gateway reports a schema/protobuf mismatch (mirrored as `rpcIssue`), or `failureLayer` is non-null. The alias form `nemoclaw status --json` requires the sandbox to be registered locally; the canonical form `nemoclaw sandbox status --json` is the one to use from automation that may run against an unknown sandbox name, since it still emits a JSON document with `found: false` instead of a text error. @@ -523,6 +534,7 @@ When the host Docker daemon is reachable but the per-sandbox container is stoppe If the sandbox's recorded dashboard port is also held by a foreign listener, the header escalates to `Failure layer: sandbox_dashboard_port_conflict — sandbox container is stopped and the dashboard port is held by a foreign listener.` so the operator can recover the port before restarting the sandbox. If the sandbox or gateway cannot be verified, the command exits non-zero instead of reporting healthy inference from stale registry state. +When a locally registered sandbox is missing from the live gateway, status preserves the registry entry so the suggested `rebuild --yes` recovery can still find the sandbox metadata. Gateway and dashboard health checks treat HTTP `401` from device auth as a live service, not as an offline gateway. A `Connected` line reports whether the sandbox has any active SSH sessions and, if so, how many. @@ -787,6 +799,8 @@ Unchecking a preset in the onboard TUI checkbox also removes it from the sandbox Add a host alias to the sandbox pod template. Use this when a sandbox needs a stable LAN-only name, such as a local SearXNG or internal model endpoint, without dropping to `docker exec` and `kubectl patch`. +Host alias commands use the legacy Kubernetes gateway `Sandbox` resource path. +They are not supported on Docker-driver or VM-driver sandboxes because those drivers do not run the gateway cluster container that owns this resource. ```bash nemoclaw my-assistant hosts-add searxng.local 192.168.1.105 @@ -1422,6 +1436,8 @@ nemoclaw inference set --provider --model [--sandbox ] Pass both `--provider` and `--model` when you want NemoClaw to update the OpenShell inference route and sync the selected sandbox's agent config. If you only want the lower-level OpenShell route operation, run `openshell inference set -g nemoclaw --model --provider ` directly. When either flag is missing, `nemoclaw inference set` prints that OpenShell command instead of an oclif flag-validation error. +The command updates the host registry immediately after the gateway route changes. +If the in-sandbox config sync fails, NemoClaw keeps the gateway and registry aligned, warns that the running image may still need a rebuild, and points you to `nemoclaw rebuild`. Supported provider names are `nvidia-prod`, `nvidia-nim`, `nvidia-router`, `openai-api`, `anthropic-prod`, `compatible-anthropic-endpoint`, `gemini-api`, `compatible-endpoint`, `hermes-provider`, `ollama-local`, and `vllm-local`. Use `--no-verify` only when OpenShell cannot verify the provider at switch time but you have already confirmed the provider and credential. @@ -1627,6 +1643,7 @@ All ports must be non-privileged integers between 1024 and 65535. If a port value is not a valid integer or falls outside the allowed range, the CLI exits with an error. `NEMOCLAW_GATEWAY_PORT` also cannot overlap the configured dashboard, vLLM, Ollama, or Ollama proxy ports, and cannot use the dashboard auto-allocation range `18789` through `18799` or the default inference/proxy ports `8000`, `11434`, and `11435`. +When you run multiple NemoClaw gateways with different `NEMOCLAW_GATEWAY_PORT` values, NemoClaw derives a separate gateway name, state directory, and compatibility container name from the port so one gateway does not tear down another. On non-WSL hosts, `NEMOCLAW_OLLAMA_PORT` and `NEMOCLAW_OLLAMA_PROXY_PORT` must be different. If you run Ollama on port 11435, set `NEMOCLAW_OLLAMA_PROXY_PORT` to another free port before onboarding. diff --git a/.agents/skills/nemoclaw-user-reference/references/troubleshooting.md b/.agents/skills/nemoclaw-user-reference/references/troubleshooting.md index 19b914b2f71..57ee07d784d 100644 --- a/.agents/skills/nemoclaw-user-reference/references/troubleshooting.md +++ b/.agents/skills/nemoclaw-user-reference/references/troubleshooting.md @@ -584,6 +584,12 @@ nemoclaw rebuild The sandbox may have been stopped or deleted. Run `nemoclaw onboard` to recreate the sandbox from the same blueprint and policy definitions. +### Sandbox is registered locally but missing from the gateway + +After a gateway restart, host reboot, or manual OpenShell cleanup, NemoClaw may still have a local registry entry for a sandbox that the live gateway no longer lists. +`nemoclaw status` and `nemoclaw connect` preserve that local registry entry and print recovery guidance instead of deleting it automatically. +Run `nemoclaw rebuild --yes` when you want NemoClaw to recreate the sandbox from the recorded metadata, or run `nemoclaw destroy` when you intentionally want to remove the stale entry. + ### Status shows "not running" inside the sandbox This is expected behavior. diff --git a/docs/inference/switch-inference-providers.mdx b/docs/inference/switch-inference-providers.mdx index 2f1defd029c..d37bc28d10d 100644 --- a/docs/inference/switch-inference-providers.mdx +++ b/docs/inference/switch-inference-providers.mdx @@ -181,6 +181,8 @@ session and the baked image. Switching to a different provider family (for example, from NVIDIA Endpoints to Anthropic) also uses `nemoclaw inference set`. The command updates both the gateway route and the OpenClaw provider namespace in the running sandbox config. +If the in-sandbox config sync fails after the gateway route is updated, NemoClaw keeps the host registry aligned with the gateway and prints a rebuild hint. +Run the rebuild before relying on the running agent if the warning says the image config could not be patched. ```bash nemoclaw inference set --provider anthropic-prod --model claude-sonnet-4-6 --no-verify @@ -190,6 +192,8 @@ nemoclaw inference set --provider anthropic-prod --model claude-sonnet-4-6 --no- Switching to a different provider family (for example, from NVIDIA Endpoints to Anthropic) also uses `nemohermes inference set`. The command updates both the gateway route and `/sandbox/.hermes/config.yaml`. +If the Hermes config sync fails after the gateway route is updated, NemoClaw keeps the host registry aligned with the gateway and prints a rebuild hint. +Run the rebuild before relying on the running agent if the warning says the image config could not be patched. ```bash nemohermes inference set --provider anthropic-prod --model claude-sonnet-4-6 --no-verify diff --git a/docs/inference/use-local-inference.mdx b/docs/inference/use-local-inference.mdx index fa46c694c60..4e875cb5b18 100644 --- a/docs/inference/use-local-inference.mdx +++ b/docs/inference/use-local-inference.mdx @@ -372,7 +372,9 @@ For an already-running vLLM server, run `nemohermes onboard` and select **Local If vLLM is already running, NemoClaw detects the running model and validates the endpoint. If vLLM is not running and your host matches a DGX Spark or DGX Station managed profile, NemoClaw shows the **Install vLLM** or **Start vLLM** entry by default. Generic Linux NVIDIA GPU hosts still require `NEMOCLAW_EXPERIMENTAL=1` or `NEMOCLAW_PROVIDER=install-vllm` before the managed entry appears. -NemoClaw pulls the vLLM image, downloads model weights into `~/.cache/huggingface`, starts the `nemoclaw-vllm` container on `localhost:8000`, and prints progress markers while the model loads. +NemoClaw pulls the vLLM image, downloads model weights into `~/.cache/huggingface`, starts the `nemoclaw-vllm` container on `localhost:8000`, streams Hugging Face download progress, and polls `/v1/models` until the model is ready. +If Docker pull output stops making progress, a watchdog stops the stalled pull instead of failing slow but active downloads on a fixed wall-clock timeout. +If vLLM never becomes ready, NemoClaw prints a short tail of the vLLM container logs before exiting. The first run can take 10 to 30 minutes. Later runs reuse the cached image and model weights. diff --git a/docs/manage-sandboxes/lifecycle.mdx b/docs/manage-sandboxes/lifecycle.mdx index e91d0a5d588..17ba9fd9cc9 100644 --- a/docs/manage-sandboxes/lifecycle.mdx +++ b/docs/manage-sandboxes/lifecycle.mdx @@ -172,6 +172,8 @@ When the default port is already held by another sandbox, `nemoclaw onboard` sca When the default API port is already held by another sandbox, `nemohermes onboard` scans for the next free port and records it for the sandbox. +If you intentionally run separate OpenShell gateways on the same host, set a different `NEMOCLAW_GATEWAY_PORT` before each onboarding run. +NemoClaw isolates the gateway name and local state by port so one port-specific gateway does not replace another. ```bash diff --git a/docs/manage-sandboxes/messaging-channels.mdx b/docs/manage-sandboxes/messaging-channels.mdx index cee2035378d..3684938fa6c 100644 --- a/docs/manage-sandboxes/messaging-channels.mdx +++ b/docs/manage-sandboxes/messaging-channels.mdx @@ -103,10 +103,12 @@ Set `DISCORD_USER_ID` to restrict access to one user; otherwise, any member of t Slack uses Socket Mode and requires two tokens. Use `SLACK_BOT_TOKEN` for the bot user OAuth token (`xoxb-...`) and `SLACK_APP_TOKEN` for the app-level Socket Mode token (`xapp-...`). +NemoClaw validates both tokens before it saves Slack credentials or enables the channel. Set `SLACK_ALLOWED_USERS` to comma-separated Slack member IDs to authorize those users for DMs and for channel `@mention` events in channels where the Slack app is present. Set `SLACK_ALLOWED_CHANNELS` to comma-separated Slack channel IDs to restrict channel `@mention` handling to those channels. When both Slack allowlists are set, NemoClaw requires the mention to come from one of the allowed channels and one of the allowed members. Channel messages still require an explicit bot mention. +During sandbox startup, NemoClaw normalizes OpenShell credential placeholders into the environment shape expected by the Slack runtime, so post-rebuild Slack starts use the gateway-managed tokens instead of literal placeholder strings. WeChat (experimental) delivers messages over Tencent's iLink gateway through the upstream `@tencent-weixin/openclaw-weixin` plugin baked into the sandbox base image and the built-in Hermes iLink WeChat adapter. The supported mode in this release is **personal WeChat** (`bot_type=3`). diff --git a/docs/reference/commands-nemohermes.mdx b/docs/reference/commands-nemohermes.mdx index 6a9d356ef23..659576bdd7c 100644 --- a/docs/reference/commands-nemohermes.mdx +++ b/docs/reference/commands-nemohermes.mdx @@ -131,6 +131,8 @@ Three tiers are available: After selecting a tier, the wizard shows a combined preset and access-mode screen where you can include or exclude individual presets and toggle each between read and read-write access. For details on tiers and the presets each includes, see [Network Policies](network-policies#policy-tiers). +When you finish the policy step, NemoClaw records the finalized built-in preset selection for that sandbox. +Later re-onboard runs seed from that finalized selection, so presets you intentionally removed stay removed unless you select them again or override the policy mode. In non-interactive mode, set the tier with `NEMOCLAW_POLICY_TIER` (default: `balanced`): @@ -208,6 +210,9 @@ If you enable Telegram during onboarding, the wizard can also prompt for whether Set `TELEGRAM_REQUIRE_MENTION=1` for non-interactive onboarding when you want mention-only group replies. Pairing and `TELEGRAM_ALLOWED_IDS` still govern direct messages. +If you cancel a brand-new onboarding run at the policy preset step, NemoClaw rolls back the sandbox, registry entry, and onboarding session instead of leaving a default sandbox with unfinished policy state. +Existing live sandboxes are not deleted by this cancel rollback path. + If you run onboarding again with the same sandbox name and choose a different inference provider or model, NemoClaw detects the drift and recreates the sandbox so the running agent config matches your selection. In interactive mode, the wizard asks for confirmation before delete and recreate. In non-interactive mode, NemoClaw recreates automatically when the stored selection is readable and differs; if NemoClaw cannot read the stored selection, NemoClaw reuses by default. @@ -371,6 +376,10 @@ If another terminal is already connected to the sandbox, `connect` prints a note An unknown slug or a gated model (for example `deepseek-r1-distill-70b`) with no `HF_TOKEN` or `HUGGING_FACE_HUB_TOKEN` exits non-zero with the same error the installer would emit, before any sandbox readiness probe or SSH attach. Unset the variable, or supply the missing token, before retrying. +When the live OpenShell gateway inference route differs from the route recorded in the NemoClaw registry, `connect` prints an explicit warning and realigns the shared gateway to the recorded route. +Use `nemohermes inference set --provider --model ` to make an intentional route change. +If the sandbox is registered locally but missing from a healthy gateway, `connect` preserves the registry entry and points you to `rebuild --yes`, `onboard`, or `destroy` instead of deleting the metadata needed for recovery. + After a host reboot, the OpenShell gateway rotates its SSH host keys. `connect` detects the resulting identity drift, prunes stale `openshell-*` entries from `~/.ssh/known_hosts`, and retries automatically. You no longer need to re-run `nemohermes onboard` after a reboot in this case. @@ -425,6 +434,8 @@ Pass `--json` to emit a structured per-sandbox report instead of the text render The JSON output includes at least `schemaVersion`, `name`, `found`, `model`, `provider`, `phase`, `gatewayState`, `inferenceHealth`, `rpcIssue`, `hostGpuDetected`, `sandboxGpuEnabled`, `sandboxGpuMode`, `sandboxGpuDevice`, `openshellDriver`, `openshellVersion`, `policies`, `failureLayer`, and `dockerPaused`. `openshellDriver` and `openshellVersion` are always strings (falling back to `"unknown"` when the registry has no value), so consumers can rely on `typeof` checks. `failureLayer` is `null` when no preflight failure was detected and otherwise one of `docker_unreachable`, `sandbox_container_stopped`, or `sandbox_dashboard_port_conflict`; when set, `inferenceHealth` is suppressed to `null` so automation does not see a stale remote-provider healthy status during a local outage. +`dockerPaused` is `true` when NemoClaw detects that the Docker-driver sandbox container is paused. +In that case, text output keeps OpenShell's authoritative phase but prints a `docker unpause ` recovery hint instead of sending you directly to rebuild. The command exits non-zero when the sandbox is missing locally, the gateway state is not `present`, the gateway reports a schema/protobuf mismatch (mirrored as `rpcIssue`), or `failureLayer` is non-null. The alias form `nemohermes status --json` requires the sandbox to be registered locally; the canonical form `nemohermes sandbox status --json` is the one to use from automation that may run against an unknown sandbox name, since it still emits a JSON document with `found: false` instead of a text error. @@ -458,6 +469,7 @@ When the host Docker daemon is reachable but the per-sandbox container is stoppe If the sandbox's recorded dashboard port is also held by a foreign listener, the header escalates to `Failure layer: sandbox_dashboard_port_conflict — sandbox container is stopped and the dashboard port is held by a foreign listener.` so the operator can recover the port before restarting the sandbox. If the sandbox or gateway cannot be verified, the command exits non-zero instead of reporting healthy inference from stale registry state. +When a locally registered sandbox is missing from the live gateway, status preserves the registry entry so the suggested `rebuild --yes` recovery can still find the sandbox metadata. Gateway and dashboard health checks treat HTTP `401` from device auth as a live service, not as an offline gateway. A `Connected` line reports whether the sandbox has any active SSH sessions and, if so, how many. @@ -659,6 +671,8 @@ Unchecking a preset in the onboard TUI checkbox also removes it from the sandbox Add a host alias to the sandbox pod template. Use this when a sandbox needs a stable LAN-only name, such as a local SearXNG or internal model endpoint, without dropping to `docker exec` and `kubectl patch`. +Host alias commands use the legacy Kubernetes gateway `Sandbox` resource path. +They are not supported on Docker-driver or VM-driver sandboxes because those drivers do not run the gateway cluster container that owns this resource. ```bash nemohermes my-assistant hosts-add searxng.local 192.168.1.105 @@ -1249,6 +1263,8 @@ nemohermes inference set --provider --model [--sandbox Pass both `--provider` and `--model` when you want NemoClaw to update the OpenShell inference route and sync the selected sandbox's agent config. If you only want the lower-level OpenShell route operation, run `openshell inference set -g nemoclaw --model --provider ` directly. When either flag is missing, `nemohermes inference set` prints that OpenShell command instead of an oclif flag-validation error. +The command updates the host registry immediately after the gateway route changes. +If the in-sandbox config sync fails, NemoClaw keeps the gateway and registry aligned, warns that the running image may still need a rebuild, and points you to `nemohermes rebuild`. Supported provider names are `nvidia-prod`, `nvidia-nim`, `nvidia-router`, `openai-api`, `anthropic-prod`, `compatible-anthropic-endpoint`, `gemini-api`, `compatible-endpoint`, `hermes-provider`, `ollama-local`, and `vllm-local`. Use `--no-verify` only when OpenShell cannot verify the provider at switch time but you have already confirmed the provider and credential. @@ -1454,6 +1470,7 @@ All ports must be non-privileged integers between 1024 and 65535. If a port value is not a valid integer or falls outside the allowed range, the CLI exits with an error. `NEMOCLAW_GATEWAY_PORT` also cannot overlap the configured dashboard, vLLM, Ollama, or Ollama proxy ports, and cannot use the dashboard auto-allocation range `18789` through `18799` or the default inference/proxy ports `8000`, `11434`, and `11435`. +When you run multiple NemoClaw gateways with different `NEMOCLAW_GATEWAY_PORT` values, NemoClaw derives a separate gateway name, state directory, and compatibility container name from the port so one gateway does not tear down another. On non-WSL hosts, `NEMOCLAW_OLLAMA_PORT` and `NEMOCLAW_OLLAMA_PROXY_PORT` must be different. If you run Ollama on port 11435, set `NEMOCLAW_OLLAMA_PROXY_PORT` to another free port before onboarding. diff --git a/docs/reference/commands.mdx b/docs/reference/commands.mdx index bfad7ecc334..dade43cefa2 100644 --- a/docs/reference/commands.mdx +++ b/docs/reference/commands.mdx @@ -173,6 +173,8 @@ Three tiers are available: After selecting a tier, the wizard shows a combined preset and access-mode screen where you can include or exclude individual presets and toggle each between read and read-write access. For details on tiers and the presets each includes, see [Network Policies](network-policies#policy-tiers). +When you finish the policy step, NemoClaw records the finalized built-in preset selection for that sandbox. +Later re-onboard runs seed from that finalized selection, so presets you intentionally removed stay removed unless you select them again or override the policy mode. In non-interactive mode, set the tier with `NEMOCLAW_POLICY_TIER` (default: `balanced`): @@ -257,6 +259,9 @@ If you enable Telegram during onboarding, the wizard can also prompt for whether Set `TELEGRAM_REQUIRE_MENTION=1` for non-interactive onboarding when you want mention-only group replies. Pairing and `TELEGRAM_ALLOWED_IDS` still govern direct messages. +If you cancel a brand-new onboarding run at the policy preset step, NemoClaw rolls back the sandbox, registry entry, and onboarding session instead of leaving a default sandbox with unfinished policy state. +Existing live sandboxes are not deleted by this cancel rollback path. + If you run onboarding again with the same sandbox name and choose a different inference provider or model, NemoClaw detects the drift and recreates the sandbox so the running agent config matches your selection. In interactive mode, the wizard asks for confirmation before delete and recreate. In non-interactive mode, NemoClaw recreates automatically when the stored selection is readable and differs; if NemoClaw cannot read the stored selection, NemoClaw reuses by default. @@ -429,6 +434,10 @@ If another terminal is already connected to the sandbox, `connect` prints a note An unknown slug or a gated model (for example `deepseek-r1-distill-70b`) with no `HF_TOKEN` or `HUGGING_FACE_HUB_TOKEN` exits non-zero with the same error the installer would emit, before any sandbox readiness probe or SSH attach. Unset the variable, or supply the missing token, before retrying. +When the live OpenShell gateway inference route differs from the route recorded in the NemoClaw registry, `connect` prints an explicit warning and realigns the shared gateway to the recorded route. +Use `nemoclaw inference set --provider --model ` to make an intentional route change. +If the sandbox is registered locally but missing from a healthy gateway, `connect` preserves the registry entry and points you to `rebuild --yes`, `onboard`, or `destroy` instead of deleting the metadata needed for recovery. + After a host reboot, the OpenShell gateway rotates its SSH host keys. `connect` detects the resulting identity drift, prunes stale `openshell-*` entries from `~/.ssh/known_hosts`, and retries automatically. You no longer need to re-run `nemoclaw onboard` after a reboot in this case. @@ -497,6 +506,8 @@ Pass `--json` to emit a structured per-sandbox report instead of the text render The JSON output includes at least `schemaVersion`, `name`, `found`, `model`, `provider`, `phase`, `gatewayState`, `inferenceHealth`, `rpcIssue`, `hostGpuDetected`, `sandboxGpuEnabled`, `sandboxGpuMode`, `sandboxGpuDevice`, `openshellDriver`, `openshellVersion`, `policies`, `failureLayer`, and `dockerPaused`. `openshellDriver` and `openshellVersion` are always strings (falling back to `"unknown"` when the registry has no value), so consumers can rely on `typeof` checks. `failureLayer` is `null` when no preflight failure was detected and otherwise one of `docker_unreachable`, `sandbox_container_stopped`, or `sandbox_dashboard_port_conflict`; when set, `inferenceHealth` is suppressed to `null` so automation does not see a stale remote-provider healthy status during a local outage. +`dockerPaused` is `true` when NemoClaw detects that the Docker-driver sandbox container is paused. +In that case, text output keeps OpenShell's authoritative phase but prints a `docker unpause ` recovery hint instead of sending you directly to rebuild. The command exits non-zero when the sandbox is missing locally, the gateway state is not `present`, the gateway reports a schema/protobuf mismatch (mirrored as `rpcIssue`), or `failureLayer` is non-null. The alias form `nemoclaw status --json` requires the sandbox to be registered locally; the canonical form `nemoclaw sandbox status --json` is the one to use from automation that may run against an unknown sandbox name, since it still emits a JSON document with `found: false` instead of a text error. @@ -530,6 +541,7 @@ When the host Docker daemon is reachable but the per-sandbox container is stoppe If the sandbox's recorded dashboard port is also held by a foreign listener, the header escalates to `Failure layer: sandbox_dashboard_port_conflict — sandbox container is stopped and the dashboard port is held by a foreign listener.` so the operator can recover the port before restarting the sandbox. If the sandbox or gateway cannot be verified, the command exits non-zero instead of reporting healthy inference from stale registry state. +When a locally registered sandbox is missing from the live gateway, status preserves the registry entry so the suggested `rebuild --yes` recovery can still find the sandbox metadata. Gateway and dashboard health checks treat HTTP `401` from device auth as a live service, not as an offline gateway. A `Connected` line reports whether the sandbox has any active SSH sessions and, if so, how many. @@ -810,6 +822,8 @@ Unchecking a preset in the onboard TUI checkbox also removes it from the sandbox Add a host alias to the sandbox pod template. Use this when a sandbox needs a stable LAN-only name, such as a local SearXNG or internal model endpoint, without dropping to `docker exec` and `kubectl patch`. +Host alias commands use the legacy Kubernetes gateway `Sandbox` resource path. +They are not supported on Docker-driver or VM-driver sandboxes because those drivers do not run the gateway cluster container that owns this resource. ```bash nemoclaw my-assistant hosts-add searxng.local 192.168.1.105 @@ -1445,6 +1459,8 @@ nemoclaw inference set --provider --model [--sandbox ] Pass both `--provider` and `--model` when you want NemoClaw to update the OpenShell inference route and sync the selected sandbox's agent config. If you only want the lower-level OpenShell route operation, run `openshell inference set -g nemoclaw --model --provider ` directly. When either flag is missing, `nemoclaw inference set` prints that OpenShell command instead of an oclif flag-validation error. +The command updates the host registry immediately after the gateway route changes. +If the in-sandbox config sync fails, NemoClaw keeps the gateway and registry aligned, warns that the running image may still need a rebuild, and points you to `nemoclaw rebuild`. Supported provider names are `nvidia-prod`, `nvidia-nim`, `nvidia-router`, `openai-api`, `anthropic-prod`, `compatible-anthropic-endpoint`, `gemini-api`, `compatible-endpoint`, `hermes-provider`, `ollama-local`, and `vllm-local`. Use `--no-verify` only when OpenShell cannot verify the provider at switch time but you have already confirmed the provider and credential. @@ -1650,6 +1666,7 @@ All ports must be non-privileged integers between 1024 and 65535. If a port value is not a valid integer or falls outside the allowed range, the CLI exits with an error. `NEMOCLAW_GATEWAY_PORT` also cannot overlap the configured dashboard, vLLM, Ollama, or Ollama proxy ports, and cannot use the dashboard auto-allocation range `18789` through `18799` or the default inference/proxy ports `8000`, `11434`, and `11435`. +When you run multiple NemoClaw gateways with different `NEMOCLAW_GATEWAY_PORT` values, NemoClaw derives a separate gateway name, state directory, and compatibility container name from the port so one gateway does not tear down another. On non-WSL hosts, `NEMOCLAW_OLLAMA_PORT` and `NEMOCLAW_OLLAMA_PROXY_PORT` must be different. If you run Ollama on port 11435, set `NEMOCLAW_OLLAMA_PROXY_PORT` to another free port before onboarding. diff --git a/docs/reference/troubleshooting.mdx b/docs/reference/troubleshooting.mdx index ed3c74df8e9..9b63c34c3a8 100644 --- a/docs/reference/troubleshooting.mdx +++ b/docs/reference/troubleshooting.mdx @@ -597,6 +597,12 @@ nemoclaw rebuild The sandbox may have been stopped or deleted. Run `nemoclaw onboard` to recreate the sandbox from the same blueprint and policy definitions. +### Sandbox is registered locally but missing from the gateway + +After a gateway restart, host reboot, or manual OpenShell cleanup, NemoClaw may still have a local registry entry for a sandbox that the live gateway no longer lists. +`nemoclaw status` and `nemoclaw connect` preserve that local registry entry and print recovery guidance instead of deleting it automatically. +Run `nemoclaw rebuild --yes` when you want NemoClaw to recreate the sandbox from the recorded metadata, or run `nemoclaw destroy` when you intentionally want to remove the stale entry. + ### Status shows "not running" inside the sandbox This is expected behavior.