fix(onboard): diagnose incomplete custom plugin images

.github/workflows/regression-e2e.yaml

@@ -268,8 +268,8 @@ jobs:
             /tmp/nemoclaw-e2e-model-router-response.log
           if-no-files-found: ignore
 
-  # ── OpenClaw plugin runtime-deps EXDEV E2E ─────────────────────
-  # Coverage guard for #3513 / #3127. On Ubuntu/OpenShell sandbox layouts
+  # ── OpenClaw custom-plugin lifecycle and runtime-deps EXDEV E2E ─
+  # Coverage guard for #6108 / #3513 / #3127. On Ubuntu/OpenShell sandbox layouts
   # where /tmp and /sandbox can live on different filesystems, OpenClaw's
   # first CLI bootstrap must not fail plugin runtime dependency installation
   # with EXDEV cross-device rename errors.
@@ -281,7 +281,7 @@ jobs:
     runs-on: ubuntu-latest
     permissions:
       contents: read
-    timeout-minutes: 45
+    timeout-minutes: 75
     steps:
       - name: Checkout
         uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
@@ -300,7 +300,7 @@ jobs:
       - name: Build CLI
         run: npm run build:cli
 
-      - name: Run OpenClaw plugin runtime-deps EXDEV Vitest test
+      - name: Run OpenClaw custom-plugin lifecycle and runtime-deps EXDEV Vitest test
         env:
           E2E_ARTIFACT_DIR: ${{ github.workspace }}/e2e-artifacts/live/openclaw-plugin-runtime-exdev
           NEMOCLAW_RUN_LIVE_E2E: "1"

docs/deployment/install-openclaw-plugins.mdx

@@ -3,8 +3,8 @@
 # SPDX-License-Identifier: Apache-2.0
 title: "Install OpenClaw Plugins"
 sidebar-title: "Install OpenClaw Plugins"
-description: "How to install OpenClaw plugins in a NemoClaw-managed sandbox today."
-description-agent: "Explains the difference between OpenClaw plugins and agent skills, and shows the current Dockerfile-based workflow for baking a plugin into a NemoClaw sandbox, including `.dockerignore` handling for custom build contexts. Use when users ask how to install, build, or configure OpenClaw plugins under NemoClaw."
+description: "How to install an OpenClaw plugin from a version-matched full NemoClaw runtime source context."
+description-agent: "Explains the difference between OpenClaw plugins and agent skills, the complete-image contract of `nemoclaw onboard --from`, and the source-based workflow for adding a plugin without removing the managed runtime. Use when users ask how to install, build, or configure OpenClaw plugins under NemoClaw."
 keywords: ["nemoclaw plugins", "openclaw plugins", "install openclaw plugin", "nemoclaw onboard from dockerfile", "nemoclaw dockerignore"]
 content:
   type: "how_to"
@@ -18,83 +18,160 @@ They are different from NemoClaw-managed agent skills:
 - **Skills** are `SKILL.md` directories that teach an agent how to perform a task.
 - **Policy presets** are network-egress rules that control what sandboxed code can reach.
 
-To install supported OpenClaw plugins under NemoClaw, bake the plugin into a custom sandbox image and onboard from that Dockerfile.
+Until NemoClaw provides a managed plugin lifecycle, bake the plugin into a version-matched full NemoClaw runtime image.
 
-## Prepare a Build Directory
+## Understand the Custom Image Contract
 
-Place the Dockerfile and everything it needs to `COPY` in one directory.
-`nemoclaw onboard --from <Dockerfile>` uses the Dockerfile's parent directory as the Docker build context.
-Add a `.dockerignore` next to the Dockerfile to exclude local caches, generated artifacts, model files, or other paths that are not needed by the image build.
-NemoClaw still applies its own secret-safety exclusions for credential-like paths such as `.env*`, `.ssh/`, `.aws/`, `.npmrc`, `secrets/`, `*.pem`, and `*.key`, even if `.dockerignore` negates them.
+The `--from` option supplies the complete sandbox image definition.
+It does not add your Dockerfile as a layer on top of NemoClaw's normal managed runtime.
+
+<Warning>
+Do not start a custom OpenClaw image from `ghcr.io/nvidia/nemoclaw/sandbox-base` alone.
+That intermediate image contains Node.js, OpenClaw, and other runtime dependencies, but it does not contain `nemoclaw-start`, the generated `openclaw.json`, or the managed gateway health check.
+A sandbox created from the base image alone can report a successful create while the gateway and dashboard remain unavailable.
+</Warning>
+
+`nemoclaw onboard --from <Dockerfile>` uses the Dockerfile's parent directory as its Docker build context.
+The workaround therefore starts with the complete source context for the installed NemoClaw release.
+
+For a first-class `add`, `list`, `status`, and `remove` lifecycle that does not require a source checkout, follow [issue #5998](https://github.com/NVIDIA/NemoClaw/issues/5998).
+
+## Prepare a Version-Matched Build Context
+
+Use the same NemoClaw release for the installed CLI, source checkout, and `sandbox-base` image.
+The following commands reproduce the workflow for NemoClaw `v0.0.71`, which includes OpenClaw `2026.5.27`.
+
+```bash
+nemoclaw --version
+git clone --depth 1 --branch v0.0.71 https://github.com/NVIDIA/NemoClaw.git my-plugin-sandbox
+cp -R /path/to/my-plugin ./my-plugin-sandbox/my-plugin
+cd my-plugin-sandbox
+```
+
+For a different release, replace `v0.0.71` everywhere in this workflow and start from that release's stock Dockerfile.
+Do not reuse this patch across releases because the managed image contract can change.
+
+The plugin directory must contain the inputs used by its build, including its manifest and dependency lockfile.
+This example expects the following files:
 
 ```text
-my-plugin-sandbox/
-├── Dockerfile
-└── my-plugin/
-    ├── package-lock.json
-    ├── package.json
-    └── src/
+my-plugin/
+├── openclaw.plugin.json
+├── package-lock.json
+├── package.json
+├── src/
+└── tsconfig.json
 ```
 
 The example Dockerfile uses `npm ci`, so `my-plugin/` must include `package-lock.json`.
 If your plugin does not have a lockfile yet, create one in the plugin project with `npm install --package-lock-only`.
+OpenClaw `2026.5.27` also requires the plugin manifest to declare each registered tool in `contracts.tools`; for example, a weather plugin that registers `get_weather` must declare `"tools": ["get_weather"]`.
 
-## Example Dockerfile
+## Extend the Full Managed Dockerfile
 
-Use the custom image to copy the plugin into the OpenClaw extensions directory.
-Then let OpenClaw refresh its config before NemoClaw starts the sandbox.
+Make two changes to the stock `Dockerfile` from the `v0.0.71` checkout.
+First, pin the base image to the same release.
 
-```dockerfile
-ARG SANDBOX_BASE=ghcr.io/nvidia/nemoclaw/sandbox-base:latest
-FROM ${SANDBOX_BASE}
+```diff
+-ARG BASE_IMAGE=ghcr.io/nvidia/nemoclaw/sandbox-base:latest
++ARG BASE_IMAGE=ghcr.io/nvidia/nemoclaw/sandbox-base:v0.0.71
+```
 
-COPY my-plugin/ /opt/my-plugin/
-WORKDIR /opt/my-plugin
-RUN npm ci --no-audit --no-fund && npm run build
+Next, name the completed runtime stage so the plugin layer can extend it.
+
+```diff
+-FROM ${BASE_IMAGE}
++FROM ${BASE_IMAGE} AS nemoclaw-runtime
+```
 
-RUN mkdir -p /sandbox/.openclaw/extensions \
- && cp -a /opt/my-plugin /sandbox/.openclaw/extensions/my-plugin \
- && openclaw doctor --fix
+Append the following stages to the end of the stock Dockerfile.
+Replace `weather` with the plugin ID from `openclaw.plugin.json` in the stage name, staging directory, and OpenClaw plugin commands.
 
-WORKDIR /opt/nemoclaw
+```dockerfile
+# Build the plugin from its lockfile.
+FROM builder AS weather-plugin-builder
+WORKDIR /opt/my-plugin
+COPY my-plugin/package.json my-plugin/package-lock.json my-plugin/tsconfig.json ./
+RUN npm ci --no-audit --no-fund
+COPY my-plugin/openclaw.plugin.json ./
+COPY my-plugin/src/ ./src/
+RUN npm run build && npm prune --omit=dev
+
+# Extend the completed managed runtime.
+FROM nemoclaw-runtime AS weather-runtime
+COPY --from=weather-plugin-builder --chown=sandbox:sandbox \
+    /opt/my-plugin/package.json \
+    /opt/my-plugin/package-lock.json \
+    /opt/my-plugin/openclaw.plugin.json \
+    /opt/weather-plugin/
+COPY --from=weather-plugin-builder --chown=sandbox:sandbox \
+    /opt/my-plugin/dist/ /opt/weather-plugin/dist/
+COPY --from=weather-plugin-builder --chown=sandbox:sandbox \
+    /opt/my-plugin/node_modules/ /opt/weather-plugin/node_modules/
+
+USER sandbox
+RUN HOME=/sandbox openclaw plugins install /opt/weather-plugin \
+    && HOME=/sandbox openclaw plugins enable weather \
+    && HOME=/sandbox openclaw plugins inspect weather --json > /dev/null
+
+# Enabling the plugin changes openclaw.json after the managed runtime hashes it.
+# hadolint ignore=DL3002
+USER root
+RUN chown sandbox:sandbox /sandbox/.openclaw/openclaw.json \
+    && chmod 660 /sandbox/.openclaw/openclaw.json \
+    && sha256sum /sandbox/.openclaw/openclaw.json > /sandbox/.openclaw/.config-hash \
+    && chown sandbox:sandbox /sandbox/.openclaw/.config-hash \
+    && chmod 660 /sandbox/.openclaw/.config-hash
 ```
 
-If the plugin needs configuration in `openclaw.json`, apply it after `openclaw doctor --fix` so the base config exists first.
+The final stage inherits the stock runtime entrypoint, command, gateway health check, generated configuration, and file permissions.
+The local install copies the staged plugin into OpenClaw's extensions tree, records the install, and leaves existing managed plugin load paths intact before the explicit enable and inspect steps.
+The last `RUN` refreshes the managed config hash after `openclaw plugins enable` updates `openclaw.json`.
 
-## Create the Sandbox
+## Create and Verify the Sandbox
 
-Point `nemoclaw onboard --from` at the Dockerfile in the build directory.
+Pin NemoClaw's base-image resolver to the same release when you onboard.
 
 ```bash
-nemoclaw onboard --from ./my-plugin-sandbox/Dockerfile
+NEMOCLAW_SANDBOX_BASE_IMAGE_REF=ghcr.io/nvidia/nemoclaw/sandbox-base:v0.0.71 \
+  nemoclaw onboard \
+    --fresh \
+    --no-gpu \
+    --name weather-agent \
+    --from "$PWD/Dockerfile"
 ```
 
-To run a second sandbox alongside an existing one, use a dedicated build directory and rerun onboarding with the sandbox name and ports you intend to use.
+Verify the managed runtime and plugin after onboarding completes.
 
-## Build Performance
+```bash
+nemoclaw weather-agent status
+nemoclaw weather-agent exec -- test -s /tmp/gateway.log
+nemoclaw weather-agent exec -- env HOME=/sandbox openclaw plugins inspect weather --runtime --json
+nemoclaw weather-agent exec -- bash -lc \
+  ". /tmp/nemoclaw-proxy-env.sh && HOME=/sandbox openclaw gateway call tools.catalog --params '{\"includePlugins\":true}' --json"
+nemoclaw weather-agent exec -- bash -lc \
+  ". /tmp/nemoclaw-proxy-env.sh && HOME=/sandbox openclaw gateway call tools.invoke --params '{\"name\":\"get_weather\",\"args\":{\"location\":\"Santa Clara\"}}' --json"
+```
 
-Custom plugin images are normal Docker builds, so build time depends on the build context size and the Docker layer cache rather than on NemoClaw.
+The plugin inspection must report the `weather` plugin, the gateway tool catalog must contain `get_weather`, and the invocation must return the deterministic `Santa Clara` weather result.
 
-Keep the build context small and dedicated.
-The Dockerfile's parent directory is staged as the build context before the Docker build starts, so a broad directory can make onboarding look stuck while Docker is only preparing context.
-A small build directory stages quickly:
+Keep the source checkout and Dockerfile at the recorded path if you plan to run `nemoclaw weather-agent rebuild --yes`.
+This source-based workflow can reproduce the plugin during that rebuild, but it is not the durable managed-plugin lifecycle proposed in issue #5998.
+Rerun the plugin inspection and tool-catalog checks after a gateway restart or sandbox rebuild to verify that the plugin remains available.
 
-```text
-my-plugin-sandbox/        # fast: only what the image needs
-├── Dockerfile
-├── .dockerignore
-└── my-plugin/
-```
+If the plugin imports runtime packages, keep those packages in `dependencies` rather than `devDependencies` so `npm prune --omit=dev` preserves them.
+If the plugin needs configuration in `openclaw.json`, apply it before refreshing `.config-hash`.
 
-A Dockerfile placed in a large tree stages slowly:
+Never bake plugin credentials into the Dockerfile or `openclaw.json`.
+Use OpenShell credential providers for secrets.
 
-```text
-~/                        # slow: stages the whole home directory
-├── Dockerfile
-├── Downloads/
-├── datasets/
-└── models/
-```
+## Build Performance
+
+Custom plugin images are normal Docker builds, so build time depends on the build context size and the Docker layer cache rather than on NemoClaw.
+
+Keep the Dockerfile at the release checkout root because the full managed image copies repository scripts, blueprint files, and the built-in NemoClaw plugin.
+Use the checkout's `.dockerignore`, and do not place unrelated datasets, model files, caches, or credentials under that directory.
+NemoClaw also applies secret-safety exclusions for credential-like paths such as `.env*`, `.ssh/`, `.aws/`, `.npmrc`, `secrets/`, `*.pem`, and `*.key`.
 
 Distinguish cold builds from warm rebuilds.
 The first build on a fresh host is a cold build that downloads the base image and package indexes, so it is the slowest run.
@@ -126,7 +203,10 @@ For custom preset workflows, refer to [Customize Network Policy](../network-poli
 The following mistakes commonly mix plugin installation with other NemoClaw extension paths.
 
 - Do not use `nemoclaw <sandbox> skill install` for OpenClaw plugins. That command only installs `SKILL.md` agent skills.
-- Do not put a Dockerfile in a broad directory such as `/tmp` unless you intend to send that whole directory as the Docker build context.
+- Do not use `sandbox-base` as the final custom image. It is an intermediate dependency image.
+- Do not combine one NemoClaw release's Dockerfile with another release's base image.
+- Do not copy a plugin into `/sandbox/.openclaw/extensions/<id>` and then run `plugins install --link` on that same path. Stage it outside the managed extensions directory and use the local install shown above.
+- Do not move or delete the recorded source checkout before rebuilding the sandbox.
 - Do not rely on `.dockerignore` to include credential-like paths; NemoClaw excludes those from staged custom build contexts for safety.
 - Keep plugin dependencies in the build stage or plugin directory, and avoid copying unrelated host files into the sandbox image.
 
@@ -135,3 +215,4 @@ The following mistakes commonly mix plugin installation with other NemoClaw exte
 - Review [Sandbox Hardening](sandbox-hardening) before adding plugin code to a shared or long-lived sandbox.
 - Review [Network Policies](../reference/network-policies) to plan plugin egress rules.
 - Follow [Customize Network Policy](../network-policy/customize-network-policy) if the plugin needs a custom preset.
+- Follow [issue #5998](https://github.com/NVIDIA/NemoClaw/issues/5998) for the no-source-checkout managed plugin lifecycle.

docs/reference/commands-nemohermes.mdx

@@ -279,6 +279,7 @@ The poll count is clamped to a minimum of `1` so the probe always runs at least
 #### `--from <Dockerfile>`
 
 Build the sandbox image from a custom Dockerfile instead of the stock NemoClaw image.
+The supplied Dockerfile defines the complete sandbox image, and NemoClaw does not layer it on top of the stock managed runtime.
 The entire parent directory of the specified file is used as the Docker build context, so any files your Dockerfile references (scripts, config, etc.) must live alongside it.
 If that directory contains a `.dockerignore`, onboarding applies those rules while calculating the context size and staging files for Docker.
 NemoClaw also applies additional secret-safety exclusions that override `.dockerignore` negation rules: credential-style files and directories such as `.env*`, `.ssh/`, `.aws/`, `.netrc`, `.npmrc`, `secrets/`, `*.pem`, and `*.key` are still skipped even if `.dockerignore` tries to include them.

docs/reference/commands.mdx

@@ -373,6 +373,7 @@ The poll count is clamped to a minimum of `1` so the probe always runs at least
 #### `--from <Dockerfile>`
 
 Build the sandbox image from a custom Dockerfile instead of the stock NemoClaw image.
+The supplied Dockerfile defines the complete sandbox image, and NemoClaw does not layer it on top of the stock managed runtime.
 The entire parent directory of the specified file is used as the Docker build context, so any files your Dockerfile references (scripts, config, etc.) must live alongside it.
 If that directory contains a `.dockerignore`, onboarding applies those rules while calculating the context size and staging files for Docker.
 NemoClaw also applies additional secret-safety exclusions that override `.dockerignore` negation rules: credential-style files and directories such as `.env*`, `.ssh/`, `.aws/`, `.netrc`, `.npmrc`, `secrets/`, `*.pem`, and `*.key` are still skipped even if `.dockerignore` tries to include them.
@@ -389,6 +390,14 @@ $$nemoclaw onboard --from path/to/Dockerfile
 The Dockerfile path must exist.
 Missing paths fail during command parsing before preflight, gateway setup, inference setup, or sandbox creation starts.
 
+<AgentOnly variant="openclaw">
+
+If deployment verification cannot reach the gateway for a custom OpenClaw image, NemoClaw checks for `/tmp/gateway.log`, `/usr/local/bin/nemoclaw-start`, and `/sandbox/.openclaw/openclaw.json`.
+When all three paths are absent, onboarding reports that the custom image lacks the managed runtime instead of treating repeated port-forward retries as the recovery path.
+For the version-matched full-runtime plugin workflow, refer to [Install OpenClaw Plugins](../manage-sandboxes/install-openclaw-plugins).
+
+</AgentOnly>
+
 The file can have any name; if it is not already named `Dockerfile`, onboard copies it to `Dockerfile` inside the staged build context automatically.
 To create an isolated build context, create a dedicated directory that contains only the Dockerfile and the files it needs:
 
@@ -2573,6 +2582,13 @@ Set them before running `$$nemoclaw onboard`.
 | `NEMOCLAW_AUTO_FIX_FIREWALL` | `1` to enable | Opts in to automatic UFW remediation when Linux Docker-driver sandbox containers cannot reach the host gateway after a proven TCP failure. NemoClaw runs `sudo -n` only, validates the narrow Docker bridge subnet → gateway IP:port rule before invoking UFW, re-probes after applying it, and otherwise falls back to the printed manual command. |
 | `NEMOCLAW_WECHAT_QUIET` | `1` to enable | Silences the `[wechat]` diagnostic lines printed during the host-side WeChat QR login (poll status, IDC redirects, swallowed gateway errors), which are visible by default while the experimental WeChat path stabilizes; set `1` once the flow is reliable in your environment. |
 
+<AgentOnly variant="openclaw">
+
+Set `NEMOCLAW_SANDBOX_BASE_IMAGE_REF` to an OpenClaw sandbox-base tag or digest to override base-image resolution during onboarding.
+Use a release-matched tag or immutable digest for a source-based custom image; NemoClaw resolves the reference and pins a repository digest into a stock-style `ARG BASE_IMAGE` declaration when possible.
+
+</AgentOnly>
+
 ### Onboard Profiling Traces
 
 Set `NEMOCLAW_TRACE=1` before `$$nemoclaw onboard` to write an OpenTelemetry-style JSON trace for the run.