Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions docs/reference/troubleshooting.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -893,6 +893,43 @@ To silence the warning when the host is intentionally small, set `NEMOCLAW_IGNOR

<AgentOnly variant="openclaw,hermes">

### Managed Sandbox Image Build Requires Local BuildKit

On a local Docker-driver gateway, NemoClaw builds each generated OpenClaw or Hermes sandbox image with host-side BuildKit.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Document the portable profile exemption.

Line 898 states that NemoClaw builds each generated OpenClaw or Hermes sandbox image with host-side BuildKit. prebuildSandboxImageIfEligible clears requiresLocalBuildKit when isPortableExperimentalProfile(env) is true, and it builds with rootless Podman instead. The exception list at Line 930 and Line 931 names user-supplied --from contexts and the generated Deep Agents Code image, but not the portable profile.

An operator who runs the portable profile reads a requirement that does not apply to their setup.

📝 Proposed addition
 This requirement does not change user-supplied `--from` contexts, which continue to use the OpenShell gateway builder.
 It also preserves the gateway fallback when a generated LangChain Deep Agents Code image does not complete its local prebuild.
+The portable experimental profile builds these images with rootless Podman instead of host-side BuildKit, so this requirement does not apply to it.

As per coding guidelines: "Verify commands, flags, API names, defaults, behavior, and technical claims against checked-in source code, tests, scripts, or another accepted source of truth".

Also applies to: 930-931

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/reference/troubleshooting.mdx` at line 898, Update the troubleshooting
documentation around the host-side BuildKit requirement and its exception list
to explicitly exempt the portable experimental profile. State that
prebuildSandboxImageIfEligible uses rootless Podman when
isPortableExperimentalProfile(env) is true, and include this profile alongside
the existing --from and generated Deep Agents Code exceptions.

Source: Coding guidelines

The generated Dockerfiles include BuildKit-only file-mode, per-step network, and mount controls.
NemoClaw stops before sandbox creation in these cases:

- The local build is disabled.
- The staged build context fails trust validation.
- Docker cannot start the build.
- The build exits with an error.

It does not send that generated Dockerfile to the OpenShell gateway's classic Docker API builder because that builder cannot enforce the same instructions.

Keep the local prebuild enabled, and verify that the host Docker installation provides BuildKit:

```bash
unset NEMOCLAW_SANDBOX_PREBUILD
docker info
docker buildx version
```

Repair Docker access or the Docker Buildx plugin when either Docker command fails.
Then rerun the original onboarding or rebuild command.
For a resumable onboarding session, run:

```bash
$$nemoclaw onboard --resume
```

A passing recovery completes the local BuildKit build before sandbox creation starts.
If NemoClaw rejects the staged build context trust boundary, do not change its permissions or move its Dockerfile.
Rerun the command so NemoClaw creates a new private staged context.
If the new context is also rejected, preserve the complete error and stop instead of forcing the gateway builder.

This requirement does not change user-supplied `--from` contexts, which continue to use the OpenShell gateway builder.
It also preserves the gateway fallback when a generated LangChain Deep Agents Code image does not complete its local prebuild.

### Re-onboard fails because port 18789 is held by SSH

After destroying a sandbox and gateway, the SSH port-forward process for the dashboard can be left running.
Expand Down
87 changes: 86 additions & 1 deletion src/lib/adapters/fs/regular-file.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import fs from "node:fs";
import os from "node:os";
import path from "node:path";

import { describe, expect, it } from "vitest";
import { describe, expect, it, vi } from "vitest";

import { openRegularFileNoFollow } from "./regular-file";

Expand Down Expand Up @@ -71,6 +71,91 @@ describe("regular file adapter", () => {
}
});

it("reads bounded bytes without changing their representation", () => {
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-regular-file-"));
const filePath = path.join(tmp, "runtime.bundle");
const expected = Buffer.from([0x00, 0xff, 0x7f, 0x0a]);
fs.writeFileSync(filePath, expected);

try {
const file = openRegularFileNoFollow(filePath);
try {
expect(file.readBytes(expected.length)).toEqual(expected);
expect(() => file.readBytes(expected.length - 1)).toThrow(RangeError);
} finally {
file.close();
}
expect(fs.readFileSync(filePath)).toEqual(expected);
} finally {
fs.rmSync(tmp, { recursive: true, force: true });
}
});

it("rejects a hard-linked regular file without changing either link", () => {
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-regular-file-"));
const filePath = path.join(tmp, "runtime.bundle");
const aliasPath = path.join(tmp, "runtime-alias.bundle");
const expected = Buffer.from("reviewed runtime\n");
fs.writeFileSync(filePath, expected);
fs.linkSync(filePath, aliasPath);

try {
expect(() => openRegularFileNoFollow(filePath)).toThrow(/changed during validation/);
expect(fs.readFileSync(filePath)).toEqual(expected);
expect(fs.readFileSync(aliasPath)).toEqual(expected);
} finally {
fs.rmSync(tmp, { recursive: true, force: true });
}
});

it("rejects a replaced path when reading bytes from its original descriptor", () => {
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-regular-file-"));
const filePath = path.join(tmp, "runtime.bundle");
const openedPath = path.join(tmp, "opened-runtime.bundle");
fs.writeFileSync(filePath, "reviewed\n");

try {
const file = openRegularFileNoFollow(filePath);
try {
fs.renameSync(filePath, openedPath);
fs.writeFileSync(filePath, "replacement\n");
expect(() => file.readBytes(64)).toThrow(/changed during validation/);
} finally {
file.close();
}
expect(fs.readFileSync(openedPath, "utf8")).toBe("reviewed\n");
expect(fs.readFileSync(filePath, "utf8")).toBe("replacement\n");
} finally {
fs.rmSync(tmp, { recursive: true, force: true });
}
});

it("rejects descriptor metadata changes during a byte read", () => {
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-regular-file-"));
const filePath = path.join(tmp, "runtime.bundle");
const expected = Buffer.from("reviewed runtime\n");
fs.writeFileSync(filePath, expected, { mode: 0o644 });
fs.chmodSync(filePath, 0o644);

try {
const file = openRegularFileNoFollow(filePath);
const originalReadSync = fs.readSync.bind(fs);
const readSpy = vi.spyOn(fs, "readSync").mockImplementation(((...args: unknown[]) => {
fs.chmodSync(filePath, 0o600);
return Reflect.apply(originalReadSync, fs, args);
}) as typeof fs.readSync);
try {
expect(() => file.readBytes(expected.length)).toThrow(/changed while reading/);
} finally {
readSpy.mockRestore();
file.close();
}
expect(fs.readFileSync(filePath)).toEqual(expected);
} finally {
fs.rmSync(tmp, { recursive: true, force: true });
}
});

it("refuses to follow a symbolic link", () => {
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-regular-file-"));
const targetPath = path.join(tmp, "target");
Expand Down
34 changes: 33 additions & 1 deletion src/lib/adapters/fs/regular-file.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import fs from "node:fs";

export interface OpenRegularFile {
close(): void;
readBytes(maxBytes: number): Buffer;
readUtf8(maxBytes?: number): string;
replaceUtf8(contents: string, mode: number): void;
}
Expand Down Expand Up @@ -33,7 +34,7 @@ export function openRegularFileNoFollow(
closed = true;
fs.closeSync(descriptor);
};
const assertPathIdentity = () => {
const assertPathIdentity = (): fs.Stats => {
const descriptorStats = fs.fstatSync(descriptor);
const pathStats = fs.lstatSync(target);
if (
Expand All @@ -47,6 +48,7 @@ export function openRegularFileNoFollow(
) {
throw new Error(`regular file changed during validation: ${target}`);
}
return descriptorStats;
};
try {
const descriptorStats = fs.fstatSync(descriptor);
Expand All @@ -61,8 +63,38 @@ export function openRegularFileNoFollow(
close();
throw error;
}
const readBytes = (maxBytes: number): Buffer => {
if (!Number.isSafeInteger(maxBytes) || maxBytes < 0) {
throw new RangeError(`regular file read limit must be a non-negative integer: ${target}`);
}
const beforeRead = assertPathIdentity();
if (beforeRead.size > maxBytes) {
throw new RangeError(`regular file exceeds the ${maxBytes}-byte read limit: ${target}`);
}
const bytes = Buffer.alloc(beforeRead.size);
let offset = 0;
while (offset < beforeRead.size) {
const read = fs.readSync(descriptor, bytes, offset, beforeRead.size - offset, offset);
if (read === 0) throw new Error(`short read from regular file: ${target}`);
offset += read;
}
const afterRead = assertPathIdentity();
if (
beforeRead.dev !== afterRead.dev ||
beforeRead.ino !== afterRead.ino ||
beforeRead.nlink !== afterRead.nlink ||
beforeRead.mode !== afterRead.mode ||
beforeRead.size !== afterRead.size ||
beforeRead.mtimeMs !== afterRead.mtimeMs ||
beforeRead.ctimeMs !== afterRead.ctimeMs
) {
throw new Error(`regular file changed while reading: ${target}`);
}
return bytes;
};
return {
close,
readBytes,
readUtf8: (maxBytes) => {
const size = fs.fstatSync(descriptor).size;
if (maxBytes !== undefined && size > maxBytes) {
Expand Down
73 changes: 71 additions & 2 deletions src/lib/onboard/sandbox-create-launch.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -627,11 +627,44 @@ describe("prepareSandboxCreateLaunchWithPrebuild", () => {
expect(buildImage).toHaveBeenCalledOnce();
});

it("renders the original Dockerfile for Hermes after a local build failure", async () => {
it.each([
["OpenClaw", null],
["Hermes", { name: "hermes" }],
])("fails closed for a generated %s image after a local BuildKit failure", async (_agentName, agent) => {
const buildCtx = createTrustedBuildContext();
const dockerfile = path.join(buildCtx, "Dockerfile");

await expect(
prepareSandboxCreateLaunchWithPrebuild({
agent: agent as any,
chatUiUrl: "",
createArgs: ["--from", dockerfile, "--name", "demo"],
env: {},
extraPlaceholderKeys: [],
getDashboardForwardPort: () => "0",
hermesDashboardState: disabledHermesDashboardState,
manageDashboard: false,
openshellShellCommand: (args) => args.join(" "),
sandboxName: "demo",
buildEnv: () => ({}),
prebuild: {
buildCtx,
buildId: "build-123",
dockerDriverGateway: true,
env: { NEMOCLAW_SANDBOX_PREBUILD: "1" },
buildImage: async () => 1,
log: vi.fn(),
origin: "generated",
},
}),
).rejects.toThrow("Local BuildKit build failed (exit 1)");
});

it("preserves the gateway builder for generated Deep Agents Code images", async () => {
const buildCtx = createTrustedBuildContext();
const dockerfile = path.join(buildCtx, "Dockerfile");
const result = await prepareSandboxCreateLaunchWithPrebuild({
agent: { name: "hermes" } as any,
agent: { name: "langchain-deepagents-code" } as any,
chatUiUrl: "",
createArgs: ["--from", dockerfile, "--name", "demo"],
env: {},
Expand Down Expand Up @@ -661,4 +694,40 @@ describe("prepareSandboxCreateLaunchWithPrebuild", () => {
expect(result.createCommand).toContain(`sandbox create --from ${dockerfile} --name demo`);
expect(result.createCommand).not.toContain("nemoclaw-sandbox-local");
});

it("preserves the rootless gateway path for a generated portable Hermes image", async () => {
const buildCtx = createTrustedBuildContext();
const dockerfile = path.join(buildCtx, "Dockerfile");
const result = await prepareSandboxCreateLaunchWithPrebuild({
agent: { name: "hermes" } as any,
chatUiUrl: "",
createArgs: ["--from", dockerfile, "--name", "demo"],
env: {},
extraPlaceholderKeys: [],
getDashboardForwardPort: () => "0",
hermesDashboardState: disabledHermesDashboardState,
manageDashboard: false,
openshellShellCommand: (args) => args.join(" "),
sandboxName: "demo",
buildEnv: () => ({}),
prebuild: {
buildCtx,
buildId: "build-123",
dockerDriverGateway: true,
env: {
NEMOCLAW_EXPERIMENTAL_PROFILE: "portable",
NEMOCLAW_SANDBOX_PREBUILD: "1",
},
buildImage: async () => 1,
log: vi.fn(),
origin: "generated",
},
});

expect(result.prebuild).toEqual({
createArgs: ["--from", dockerfile, "--name", "demo"],
imageRef: null,
imageId: null,
});
});
});
4 changes: 4 additions & 0 deletions src/lib/onboard/sandbox-create-launch.ts
Original file line number Diff line number Diff line change
Expand Up @@ -338,9 +338,13 @@ export async function prepareSandboxCreateLaunchWithPrebuild(
input: SandboxCreateLaunchWithPrebuildInput,
): Promise<SandboxCreateLaunchWithPrebuild> {
const { prebuild: prebuildInput, ...launchInput } = input;
const requiresLocalBuildKit =
prebuildInput.origin === "generated" &&
(input.agent == null || input.agent.name === "openclaw" || input.agent.name === "hermes");
const prebuild = await prebuildSandboxImageIfEligible({
...prebuildInput,
createArgs: input.createArgs,
requiresLocalBuildKit,
sandboxName: input.sandboxName,
});
return {
Expand Down
Loading
Loading