diff --git a/docs/reference/architecture.mdx b/docs/reference/architecture.mdx index 70cd6736046..fc34d1dadb9 100644 --- a/docs/reference/architecture.mdx +++ b/docs/reference/architecture.mdx @@ -87,12 +87,14 @@ Onboarding validates and reuses a healthy selected service. It enables or restarts the service when startup or verified runtime drift requires it, then checks gateway health. The marked service generates the local OpenShell mTLS bundle and reads `$XDG_CONFIG_HOME/openshell/gateway.env`, or `~/.config/openshell/gateway.env` when `XDG_CONFIG_HOME` is not absolute. NemoClaw preserves unrelated environment entries, accepts `DOCKER_HOST` only for an absolute local `unix://` socket, and refuses foreign or symlinked managed files. -The standalone Linux process is used only when the systemd user manager is unavailable; after a service is selected, startup or health failure stops onboarding. +If the selected systemd user service fails inspection, startup, or its health check, NemoClaw prints its `journalctl` log command and attempts the standalone fallback. On Apple Silicon macOS, Homebrew makes the official OpenShell formula authoritative. The installer stages the formula and onboarding starts its `openshell` service. -When Homebrew is present, a missing formula, a formula from another tap, a service-start failure, or a health failure stops onboarding. -Only a host without Homebrew uses the standalone macOS gateway fallback. +If the service fails inspection, startup, or its health check, NemoClaw prints the formula log command and attempts the standalone fallback. +A host without Homebrew, or with no installed OpenShell formula, also uses the standalone macOS gateway. +On both platforms, standalone startup requires exclusive ownership of the gateway port. +Managed-service trust failures and unsafe environment configuration remain hard failures. NemoClaw-managed gateways on custom ports remain detached and separate from the default service. An externally supervised gateway can use any matching configured port; its declared supervisor retains lifecycle authority. diff --git a/docs/reference/troubleshooting.mdx b/docs/reference/troubleshooting.mdx index e35128ca0fa..514f5ea50e4 100644 --- a/docs/reference/troubleshooting.mdx +++ b/docs/reference/troubleshooting.mdx @@ -137,6 +137,7 @@ It rejects remote endpoints and relative socket paths before service startup. ### Onboarding reports `invalid_docker_host` The `invalid_docker_host` advisory means that `DOCKER_HOST` is not an absolute local `unix://` socket path that NemoClaw can write to the managed OpenShell gateway service environment. +NemoClaw does not use the standalone gateway fallback when this validation fails. Remove the override to use Docker's default local socket: ```bash @@ -931,8 +932,14 @@ Follow these steps to reconnect. brew services restart openshell ``` - If Homebrew is present but `openshell` is missing or comes from another tap, rerun the NemoClaw installer. - Onboarding does not use the standalone fallback while Homebrew is present. + During onboarding, NemoClaw uses the standalone fallback when the `openshell` formula is missing. + If the installed service fails inspection, startup, or its health check, NemoClaw prints this log command: + + ```bash + tail -n 200 "$(brew --prefix)/var/log/openshell/openshell-gateway.out.log" "$(brew --prefix)/var/log/openshell/openshell-gateway.err.log" + ``` + + Rerun the NemoClaw installer to restore the managed service for later onboarding runs. On Linux package installs, inspect and restart the upstream service. @@ -941,6 +948,12 @@ Follow these steps to reconnect. systemctl --user restart openshell-gateway ``` + If the service fails inspection, startup, or its health check, NemoClaw prints this log command: + + ```bash + journalctl --user --unit openshell-gateway --no-pager --lines=200 + ``` + On Linux tarball installs, inspect and restart the marked NemoClaw service. ```bash @@ -948,8 +961,23 @@ Follow these steps to reconnect. systemctl --user restart nemoclaw-openshell-gateway ``` + If the service fails inspection, startup, or its health check, NemoClaw prints this log command: + + ```bash + journalctl --user --unit nemoclaw-openshell-gateway --no-pager --lines=200 + ``` + The tarball unit is under `$XDG_CONFIG_HOME/systemd/user`, or `~/.config/systemd/user` when `XDG_CONFIG_HOME` is not absolute. It starts with your user session; NemoClaw does not enable lingering. + If a managed service fails inspection, startup, or its health check, NemoClaw attempts the standalone fallback on either platform. + The standalone gateway starts only after NemoClaw verifies exclusive ownership of the gateway port. + The fallback does not bypass managed-service trust validation or unsafe environment configuration. + These conditions remain hard failures: + + - Homebrew formula identity query, metadata, or official-tap validation errors + - Foreign or symlinked systemd units, or an untrusted systemd executable identity + - An invalid `DOCKER_HOST` or a symlinked service environment file + Without Homebrew on macOS, or without a reachable systemd user manager on Linux, continue with the standalone recovery step below. 1. Check sandbox state. diff --git a/src/lib/onboard.ts b/src/lib/onboard.ts index f37f41615f2..f07ded4ddee 100644 --- a/src/lib/onboard.ts +++ b/src/lib/onboard.ts @@ -650,6 +650,7 @@ const { getDockerDriverGatewayEnv, getDockerDriverGatewayPid, getDockerDriverGatewayPortListenerPid, + getDockerDriverGatewayPortListenerScan, getDockerDriverGatewayReuseDrift: getGatewayReuseDrift, getDockerDriverGatewayRuntimeDrift, getDockerDriverGatewayRuntimeDriftFromSnapshot, @@ -1898,15 +1899,7 @@ async function startGatewayWithOptions( process.env.OPENSHELL_GATEWAY = GATEWAY_NAME; } -/** - * Reconcile or create the host Docker-driver gateway. The public onboard() - * entrypoint holds acquireOnboardLock()'s atomic cross-process filesystem lock - * (created with openSync("wx")) across this whole call, so separate concurrent - * `nemoclaw onboard` CLI processes cannot race creation. - * The strict post-reap bind check below remains a second boundary against - * recovery commands or external processes that do not participate in that - * lock; the OS then permits only one child to bind the port. - */ +/** Reconcile the host Docker-driver gateway under the onboard lock and strict port checks. */ async function startDockerDriverGateway({ exitOnFailure = true, skipSandboxBridgeReachability = false, @@ -1950,84 +1943,88 @@ async function startDockerDriverGateway({ exitOnFailure, }), }); - if ( - await dockerDriverGatewayEnv.startPackageManagedDockerDriverGatewayWithEnvOverride({ - clearDockerDriverGatewayRuntimeFiles, - exitOnFailure, - gatewayEnv: driftGatewayEnv, - gatewayName: GATEWAY_NAME, - isDockerDriverGatewayReady: () => - isDockerDriverGatewayHttpReady(undefined, undefined, driftGatewayEnv), - registerDockerDriverGatewayEndpoint, - preparePortForOpenShellGatewayUserServiceStart: servicePortOwnership.preparePort, - runCaptureOpenshell, - skipSandboxBridgeReachability, - validatePortOwnerForOpenShellGatewayUserServiceStart: servicePortOwnership.validatePortOwner, - verifySandboxBridgeGatewayReachableOrExit: (fail, options) => - verifySandboxBridgeGatewayReachableOrExit(fail, { - ...options, - port: GATEWAY_PORT, - }), - }) - ) - return; - const initialHealth = dockerDriverGatewayCutover.readDockerDriverGatewayHealth( - runCaptureOpenshell, - GATEWAY_NAME, - ); - const cutover = await dockerDriverGatewayCutover.runDockerDriverGatewayCutover( - { - gatewayBin, - identityGatewayBin, - driftGatewayBin, - driftGatewayEnv, - exitOnFailure, - skipSandboxBridgeReachability, - stateDir, - portListenerScan: servicePortOwnership.portListenerScan, - pidFileGatewayPid: getDockerDriverGatewayPid(), - initialHealth, - }, - { - isDockerDriverGatewayProcessAlive, - isGatewayHealthy, - getDockerDriverGatewayRuntimeDrift, - logDockerDriverGatewayRestart, - registerDockerDriverGatewayEndpoint, - isDockerDriverGatewayHttpReady: () => - isDockerDriverGatewayHttpReady(undefined, undefined, driftGatewayEnv), - verifySandboxBridgeGatewayReachableOrExit: (fail, options) => - verifySandboxBridgeGatewayReachableOrExit(fail, { - ...options, - port: GATEWAY_PORT, - }), - readGatewayHealth: () => ({ - status: runCaptureOpenshell(["status"], { ignoreError: true }), - namedInfo: runCaptureOpenshell(["gateway", "info", "-g", GATEWAY_NAME], { - ignoreError: true, - }), - activeInfo: runCaptureOpenshell(["gateway", "info"], { ignoreError: true }), + const cutover = await dockerDriverGatewayCutover.runDockerDriverGatewayManagedFallback( + () => + dockerDriverGatewayEnv.startPackageManagedDockerDriverGatewayWithEnvOverride({ + clearDockerDriverGatewayRuntimeFiles, + exitOnFailure, + gatewayEnv: driftGatewayEnv, + gatewayName: GATEWAY_NAME, + isDockerDriverGatewayReady: () => + isDockerDriverGatewayHttpReady(undefined, undefined, driftGatewayEnv), + registerDockerDriverGatewayEndpoint, + preparePortForOpenShellGatewayUserServiceStart: servicePortOwnership.preparePort, + runCaptureOpenshell, + skipSandboxBridgeReachability, + validatePortOwnerForOpenShellGatewayUserServiceStart: + servicePortOwnership.validatePortOwner, + verifySandboxBridgeGatewayReachableOrExit: (fail, options) => + verifySandboxBridgeGatewayReachableOrExit(fail, { + ...options, + port: GATEWAY_PORT, + }), }), - rememberDockerDriverGatewayPid, - reapDuplicateHostGatewaysExceptOrFail, - reapHostGatewayBeforeLaunchOrFail, - isGatewayPortAvailable: async () => { - const probe = await checkGatewayPortAvailable(); - return probe.ok && !probe.warning; - }, - reportUntrustedGatewayPort: servicePortOwnership.reportUntrustedGatewayPort, - reportMissingGatewayBinary: () => { - console.error(" OpenShell Docker-driver gateway binary not found."); - console.error( - ` Install OpenShell v${SUPPORTED_OPENSHELL_FALLBACK_VERSION}, or set NEMOCLAW_OPENSHELL_GATEWAY_BIN.`, - ); - if (exitOnFailure) process.exit(1); - throw new Error("OpenShell gateway binary not found"); - }, - log: (message) => console.log(message), - }, + async () => + dockerDriverGatewayCutover.runDockerDriverGatewayCutover( + { + gatewayBin, + identityGatewayBin, + driftGatewayBin, + driftGatewayEnv, + exitOnFailure, + skipSandboxBridgeReachability, + stateDir, + portListenerScan: getDockerDriverGatewayPortListenerScan( + await checkGatewayPortAvailable(), + { gatewayBin: identityGatewayBin }, + ), + pidFileGatewayPid: getDockerDriverGatewayPid(), + initialHealth: dockerDriverGatewayCutover.readDockerDriverGatewayHealth( + runCaptureOpenshell, + GATEWAY_NAME, + ), + }, + { + isDockerDriverGatewayProcessAlive, + isGatewayHealthy, + getDockerDriverGatewayRuntimeDrift, + logDockerDriverGatewayRestart, + registerDockerDriverGatewayEndpoint, + isDockerDriverGatewayHttpReady: () => + isDockerDriverGatewayHttpReady(undefined, undefined, driftGatewayEnv), + verifySandboxBridgeGatewayReachableOrExit: (fail, options) => + verifySandboxBridgeGatewayReachableOrExit(fail, { + ...options, + port: GATEWAY_PORT, + }), + readGatewayHealth: () => ({ + status: runCaptureOpenshell(["status"], { ignoreError: true }), + namedInfo: runCaptureOpenshell(["gateway", "info", "-g", GATEWAY_NAME], { + ignoreError: true, + }), + activeInfo: runCaptureOpenshell(["gateway", "info"], { ignoreError: true }), + }), + rememberDockerDriverGatewayPid, + reapDuplicateHostGatewaysExceptOrFail, + reapHostGatewayBeforeLaunchOrFail, + isGatewayPortAvailable: async () => { + const probe = await checkGatewayPortAvailable(); + return probe.ok && !probe.warning; + }, + reportUntrustedGatewayPort: servicePortOwnership.reportUntrustedGatewayPort, + reportMissingGatewayBinary: () => { + console.error(" OpenShell Docker-driver gateway binary not found."); + console.error( + ` Install OpenShell v${SUPPORTED_OPENSHELL_FALLBACK_VERSION}, or set NEMOCLAW_OPENSHELL_GATEWAY_BIN.`, + ); + if (exitOnFailure) process.exit(1); + throw new Error("OpenShell gateway binary not found"); + }, + log: (message) => console.log(message), + }, + ), ); - if (cutover === "reused") return; + if (cutover !== "launch") return; if (!gatewayBin || !gatewayLaunch) { throw new Error("OpenShell gateway launch missing after cutover"); } @@ -2037,9 +2034,8 @@ async function startDockerDriverGateway({ const logFd = dockerDriverGatewayLaunch.openDockerDriverGatewayLog(logPath, { exitOnFailure }); console.log(" Starting OpenShell Docker-driver gateway..."); console.log(` Gateway log: ${logPath}`); - const launch = gatewayLaunch; - dockerDriverGatewayLaunch.prepareAndLogDockerDriverGatewayLaunch(launch); - const child = dockerDriverGatewayLaunch.spawnDockerDriverGateway(launch, logFd); + dockerDriverGatewayLaunch.prepareAndLogDockerDriverGatewayLaunch(gatewayLaunch); + const child = dockerDriverGatewayLaunch.spawnDockerDriverGateway(gatewayLaunch, logFd); const childExit = trackChildExit(child); // #3111 zombie-safe liveness child.unref(); const childPid = child.pid ?? 0; @@ -4706,6 +4702,7 @@ module.exports = { buildControlUiUrls, startGateway, + startDockerDriverGateway, findAvailableDashboardPort, startGatewayForRecovery, openshellArgv, diff --git a/src/lib/onboard/docker-driver-gateway-cutover.ts b/src/lib/onboard/docker-driver-gateway-cutover.ts index bec2b8f7cf2..9fe5f0aadd5 100644 --- a/src/lib/onboard/docker-driver-gateway-cutover.ts +++ b/src/lib/onboard/docker-driver-gateway-cutover.ts @@ -70,6 +70,14 @@ export interface DockerDriverGatewayCutoverDeps { log(message: string): void; } +export async function runDockerDriverGatewayManagedFallback( + startManagedGateway: () => Promise, + runStandaloneCutover: () => Promise<"reused" | "launch">, +): Promise<"managed" | "reused" | "launch"> { + if (await startManagedGateway()) return "managed"; + return runStandaloneCutover(); +} + /** * Resolve reuse, adoption, or replacement for the host Docker-driver gateway. * Every reuse path requires a complete listener scan; replacement reaps only diff --git a/src/lib/onboard/docker-driver-gateway-env-service.test.ts b/src/lib/onboard/docker-driver-gateway-env-service.test.ts index 5cf5c839536..3989bceb86b 100644 --- a/src/lib/onboard/docker-driver-gateway-env-service.test.ts +++ b/src/lib/onboard/docker-driver-gateway-env-service.test.ts @@ -43,7 +43,7 @@ describe("package-managed Docker-driver gateway env service", () => { skipSandboxBridgeReachability: false, startOpenShellGatewayUserService: (opts) => { opts?.prepareServiceEnv?.(); - return { attempted: true, fallbackAllowed: false, started: true }; + return { attempted: true, started: true }; }, verifySandboxBridgeGatewayReachableOrExit: async () => undefined, }), @@ -69,7 +69,7 @@ describe("package-managed Docker-driver gateway env service", () => { const envFile = path.join(tempHome, ".config", "openshell", "gateway.env"); const startService = vi.fn((opts?: { prepareServiceEnv?: () => void }) => { opts?.prepareServiceEnv?.(); - return { attempted: true, fallbackAllowed: false, started: true }; + return { attempted: true, started: true }; }); try { @@ -126,11 +126,18 @@ describe("package-managed Docker-driver gateway env service", () => { skipSandboxBridgeReachability: false, startOpenShellGatewayUserService: (opts) => { opts?.prepareServiceEnv?.(); - return { attempted: true, fallbackAllowed: false, started: true }; + return { attempted: true, started: true }; }, verifySandboxBridgeGatewayReachableOrExit: async () => undefined, }), - ).rejects.toThrow("Refusing to write symlinked OpenShell gateway env file"); + ).rejects.toMatchObject({ + name: "OpenShellGatewayServiceEnvironmentError", + cause: expect.objectContaining({ + message: expect.stringContaining( + "Refusing to write symlinked OpenShell gateway env file", + ), + }), + }); expect(fs.readFileSync(targetFile, "utf-8")).toBe("KEEP_ME=1\n"); } finally { diff --git a/src/lib/onboard/docker-driver-gateway-env.test.ts b/src/lib/onboard/docker-driver-gateway-env.test.ts index 210c8343d28..d24f7aee4ac 100644 --- a/src/lib/onboard/docker-driver-gateway-env.test.ts +++ b/src/lib/onboard/docker-driver-gateway-env.test.ts @@ -292,7 +292,7 @@ describe("writeDockerGatewayDebEnvOverride", () => { skipSandboxBridgeReachability: false, startOpenShellGatewayUserService: (opts) => { opts?.prepareServiceEnv?.(); - return { attempted: true, fallbackAllowed: false, started: true }; + return { attempted: true, started: true }; }, verifySandboxBridgeGatewayReachableOrExit: async () => undefined, }), diff --git a/src/lib/onboard/docker-driver-gateway-env.ts b/src/lib/onboard/docker-driver-gateway-env.ts index a44ac85c4ab..d090eb129c0 100644 --- a/src/lib/onboard/docker-driver-gateway-env.ts +++ b/src/lib/onboard/docker-driver-gateway-env.ts @@ -20,10 +20,13 @@ import { } from "./docker-driver-gateway-config"; import { buildDockerDriverGatewayLocalTlsEnv } from "./docker-driver-gateway-local-tls"; import { + getOpenShellGatewayManagedServiceLogCommand, getOpenShellUserConfigHome, hasOpenShellGatewayUserService, + OpenShellGatewayServiceEnvironmentError, type PackageManagedDockerDriverGatewayOptions, startPackageManagedDockerDriverGateway, + stopOpenShellGatewayUserService, } from "./docker-driver-gateway-service"; export { getGatewayHttpsEndpoint, startPackageManagedDockerDriverGateway }; @@ -348,15 +351,24 @@ export function startPackageManagedDockerDriverGatewayWithEnvOverride( hasOpenShellGatewayUserService: options.hasOpenShellGatewayUserService ?? (() => hasOpenShellGatewayUserService({ env, home: effectiveHome })), + managedServiceLogCommand: + options.managedServiceLogCommand ?? getOpenShellGatewayManagedServiceLogCommand(), prepareOpenShellGatewayUserServiceEnv: () => { - const serviceGatewayEnv = { ...gatewayEnv }; - delete serviceGatewayEnv.DOCKER_HOST; - const dockerHost = normalizePackageServiceDockerHost(env.DOCKER_HOST); - if (dockerHost) serviceGatewayEnv.DOCKER_HOST = dockerHost; - writeDockerGatewayDebEnvOverrideFile(() => serviceGatewayEnv, { - env, - home: effectiveHome, - }); + try { + const serviceGatewayEnv = { ...gatewayEnv }; + delete serviceGatewayEnv.DOCKER_HOST; + const dockerHost = normalizePackageServiceDockerHost(env.DOCKER_HOST); + if (dockerHost) serviceGatewayEnv.DOCKER_HOST = dockerHost; + writeDockerGatewayDebEnvOverrideFile(() => serviceGatewayEnv, { + env, + home: effectiveHome, + }); + } catch (error) { + throw new OpenShellGatewayServiceEnvironmentError(error); + } }, + stopOpenShellGatewayUserService: + options.stopOpenShellGatewayUserService ?? + (() => stopOpenShellGatewayUserService({ env, home: effectiveHome })), }); } diff --git a/src/lib/onboard/docker-driver-gateway-service.test.ts b/src/lib/onboard/docker-driver-gateway-service.test.ts index 505a2398e33..0d22c5b67ac 100644 --- a/src/lib/onboard/docker-driver-gateway-service.test.ts +++ b/src/lib/onboard/docker-driver-gateway-service.test.ts @@ -6,12 +6,14 @@ import { describe, expect, it, vi } from "vitest"; import { createVirtualClock } from "./__test-helpers__/virtual-clock"; import { getNemoclawOpenShellGatewayUserServicePath, + getOpenShellGatewayManagedServiceLogCommand, getOpenShellGatewayUserServiceBinaryPaths, getOpenShellGatewayUserServicePaths, getOpenShellUserConfigHome, getTrustedActiveOpenShellGatewayUserServicePid, hasOpenShellGatewayUserService, NEMOCLAW_OPENSHELL_GATEWAY_USER_SERVICE_MARKER, + OpenShellGatewayServiceTrustError, type SpawnSyncLikeResult, startOpenShellGatewayUserService, startPackageManagedDockerDriverGateway, @@ -147,14 +149,14 @@ describe("docker-driver-gateway-service", () => { ).toThrow("must come from nvidia/openshell"); }); - it("rejects a missing Homebrew formula when Homebrew is available (#6903)", () => { - expect(() => + it("reports no managed service when the Homebrew formula is missing (#8104)", () => { + expect( hasOpenShellGatewayUserService({ commandExists: () => true, platform: "darwin", spawnSyncImpl: () => spawnResult(1, "formula not installed"), }), - ).toThrow("official OpenShell Homebrew formula is not installed"); + ).toBe(false); }); it("uses the effective XDG config home and accepts only a marked regular unit (#6903)", () => { @@ -215,7 +217,7 @@ describe("docker-driver-gateway-service", () => { }); expect(result).toMatchObject({ - fallbackAllowed: false, + logCommand: "journalctl --user --unit nemoclaw-openshell-gateway --no-pager --lines=200", manager: "systemd", serviceName: "nemoclaw-openshell-gateway", started: true, @@ -395,7 +397,10 @@ describe("docker-driver-gateway-service", () => { }, }); - expect(result).toMatchObject({ fallbackAllowed: false, started: false }); + expect(result).toMatchObject({ + logCommand: "journalctl --user --unit openshell-gateway --no-pager --lines=200", + started: false, + }); expect(result.reason).toContain("unknown listener"); expect(events.some((event) => /^(stop|enable|restart)/.test(event))).toBe(false); }); @@ -415,7 +420,13 @@ describe("docker-driver-gateway-service", () => { validatePortOwnerForServiceStart: () => events.push("validate-port"), }); - expect(result).toMatchObject({ manager: "homebrew", serviceName: "openshell", started: true }); + expect(result).toMatchObject({ + logCommand: + 'tail -n 200 "$(brew --prefix)/var/log/openshell/openshell-gateway.out.log" "$(brew --prefix)/var/log/openshell/openshell-gateway.err.log"', + manager: "homebrew", + serviceName: "openshell", + started: true, + }); expect(events).toEqual([ "list --formula openshell", "info --json=v2 openshell", @@ -428,10 +439,10 @@ describe("docker-driver-gateway-service", () => { }); it.each([ - ["manager unavailable", "daemon-reload", "Failed to connect to bus", true], - ["foreign executable", "show", "", false], - ["inactive service", "is-active", "inactive", false], - ])("returns the expected fallback decision for a selected systemd service failure: %s (#6903)", (_case, failedCommand, detail, fallbackAllowed) => { + ["the manager is unavailable", "daemon-reload", "Failed to connect to bus", false], + ["the executable is foreign", "show", "", true], + ["the service is inactive", "is-active", "inactive", false], + ])("reports the selected systemd log command when %s (#8104)", (_case, failedCommand, detail, standaloneFallbackBlocked) => { const result = startOpenShellGatewayUserService({ commandExists: () => true, env: {}, @@ -455,7 +466,23 @@ describe("docker-driver-gateway-service", () => { }), }); - expect(result).toMatchObject({ fallbackAllowed, started: false }); + expect(result).toMatchObject({ + logCommand: "journalctl --user --unit openshell-gateway --no-pager --lines=200", + standaloneFallbackBlocked, + started: false, + }); + }); + + it("selects the managed service log command without service validation (#8104)", () => { + expect(getOpenShellGatewayManagedServiceLogCommand({ platform: "darwin" })).toContain( + "openshell-gateway.err.log", + ); + expect( + getOpenShellGatewayManagedServiceLogCommand({ + existsSync: (candidate) => candidate === "/lib/systemd/user/openshell-gateway.service", + platform: "linux", + }), + ).toBe("journalctl --user --unit openshell-gateway --no-pager --lines=200"); }); it("uses managed service only after metadata and direct gRPC health are ready (#6903)", async () => { @@ -489,7 +516,6 @@ describe("docker-driver-gateway-service", () => { }, startOpenShellGatewayUserService: () => ({ attempted: true, - fallbackAllowed: false, started: true, statusCommand: "systemctl --user status nemoclaw-openshell-gateway", }), @@ -502,8 +528,28 @@ describe("docker-driver-gateway-service", () => { expect(events).toEqual(["register", "sleep", "register", "ready", "clear", "verify"]); }); - it("allows detached fallback only before a service authority starts (#6903)", async () => { + it.each([ + [ + "systemd", + "journalctl --user --unit nemoclaw-openshell-gateway --no-pager --lines=200", + "systemd" as const, + "nemoclaw-openshell-gateway", + ], + [ + "Homebrew", + 'tail -n 200 "$(brew --prefix)/var/log/openshell/openshell-gateway.out.log" "$(brew --prefix)/var/log/openshell/openshell-gateway.err.log"', + "homebrew" as const, + "openshell", + ], + ])("prints the %s log command before standalone fallback (#8104)", async (_case, logCommand, manager, serviceName) => { const register = vi.fn(() => true); + const stopService = vi.fn(() => ({ + attempted: true, + standaloneFallbackAllowed: false, + stopped: true, + })); + const warn = vi.spyOn(console, "warn").mockImplementation(() => undefined); + await expect( startPackageManagedDockerDriverGateway({ clearDockerDriverGatewayRuntimeFiles: vi.fn(), @@ -515,19 +561,30 @@ describe("docker-driver-gateway-service", () => { skipSandboxBridgeReachability: false, startOpenShellGatewayUserService: () => ({ attempted: true, - fallbackAllowed: true, - reason: "user manager unavailable", + logCommand, + manager, + reason: "managed service failed", + serviceName, started: false, }), + stopOpenShellGatewayUserService: stopService, verifySandboxBridgeGatewayReachableOrExit: vi.fn(), }), ).resolves.toBe(false); expect(register).not.toHaveBeenCalled(); + expect(stopService).toHaveBeenCalledOnce(); + expect(warn.mock.calls.flat().join("\n")).toContain(`Logs: ${logCommand}`); }); - it("keeps runtime breadcrumbs when managed service health fails (#6903)", async () => { + it("stops an unhealthy managed service before standalone fallback (#8104)", async () => { const clear = vi.fn(); const clock = createVirtualClock(); + const stopService = vi.fn(() => ({ + attempted: true, + standaloneFallbackAllowed: false, + stopped: true, + })); + const warn = vi.spyOn(console, "warn").mockImplementation(() => undefined); await expect( startPackageManagedDockerDriverGateway({ clearDockerDriverGatewayRuntimeFiles: clear, @@ -544,14 +601,247 @@ describe("docker-driver-gateway-service", () => { sleepSeconds: clock.advance, startOpenShellGatewayUserService: () => ({ attempted: true, - fallbackAllowed: false, + logCommand: "journalctl --user --unit nemoclaw-openshell-gateway --no-pager --lines=200", started: true, statusCommand: "systemctl --user status nemoclaw-openshell-gateway", }), + stopOpenShellGatewayUserService: stopService, verifySandboxBridgeGatewayReachableOrExit: vi.fn(), }), - ).rejects.toThrow("configured 1s health deadline"); + ).resolves.toBe(false); expect(clear).not.toHaveBeenCalled(); + expect(stopService).toHaveBeenCalledOnce(); + expect(warn.mock.calls.flat().join("\n")).toContain( + "journalctl --user --unit nemoclaw-openshell-gateway --no-pager --lines=200", + ); + }); + + it("continues to standalone fallback when managed service cleanup fails (#8104)", async () => { + const warn = vi.spyOn(console, "warn").mockImplementation(() => undefined); + + await expect( + startPackageManagedDockerDriverGateway({ + clearDockerDriverGatewayRuntimeFiles: vi.fn(), + exitOnFailure: false, + gatewayName: "nemoclaw", + hasOpenShellGatewayUserService: () => true, + registerDockerDriverGatewayEndpoint: vi.fn(), + runCaptureOpenshell: vi.fn(), + skipSandboxBridgeReachability: false, + startOpenShellGatewayUserService: () => ({ + attempted: true, + reason: "restart failed", + started: false, + }), + stopOpenShellGatewayUserService: () => { + throw new Error("service manager unavailable"); + }, + verifySandboxBridgeGatewayReachableOrExit: vi.fn(), + }), + ).resolves.toBe(false); + expect(warn.mock.calls.flat().join("\n")).toContain( + "standalone startup will verify gateway port ownership", + ); + }); + + it("blocks standalone fallback after unsafe managed service preparation (#8104)", async () => { + const stopService = vi.fn(); + + await expect( + startPackageManagedDockerDriverGateway({ + clearDockerDriverGatewayRuntimeFiles: vi.fn(), + exitOnFailure: false, + gatewayName: "nemoclaw", + hasOpenShellGatewayUserService: () => true, + registerDockerDriverGatewayEndpoint: vi.fn(), + runCaptureOpenshell: vi.fn(), + skipSandboxBridgeReachability: false, + startOpenShellGatewayUserService: () => ({ + attempted: true, + reason: "unsafe service environment", + standaloneFallbackBlocked: true, + started: false, + }), + stopOpenShellGatewayUserService: stopService, + verifySandboxBridgeGatewayReachableOrExit: vi.fn(), + }), + ).rejects.toThrow("unsafe service environment"); + expect(stopService).not.toHaveBeenCalled(); + }); + + it("uses standalone fallback when managed service inspection fails (#8104)", async () => { + const warn = vi.spyOn(console, "warn").mockImplementation(() => undefined); + const stopService = vi.fn(() => ({ + attempted: true, + standaloneFallbackAllowed: false, + stopped: true, + })); + + await expect( + startPackageManagedDockerDriverGateway({ + clearDockerDriverGatewayRuntimeFiles: vi.fn(), + exitOnFailure: true, + gatewayName: "nemoclaw", + hasOpenShellGatewayUserService: () => { + throw new Error("inspect failed"); + }, + managedServiceLogCommand: + 'tail -n 200 "$(brew --prefix)/var/log/openshell/openshell-gateway.out.log" "$(brew --prefix)/var/log/openshell/openshell-gateway.err.log"', + registerDockerDriverGatewayEndpoint: vi.fn(), + runCaptureOpenshell: vi.fn(), + skipSandboxBridgeReachability: false, + startOpenShellGatewayUserService: vi.fn(), + stopOpenShellGatewayUserService: stopService, + verifySandboxBridgeGatewayReachableOrExit: vi.fn(), + }), + ).resolves.toBe(false); + expect(stopService).toHaveBeenCalledOnce(); + expect(warn.mock.calls.flat().join("\n")).toContain("openshell-gateway.err.log"); + }); + + it("uses standalone fallback when Homebrew has no OpenShell formula (#8104)", async () => { + const startService = vi.fn(); + + await expect( + startPackageManagedDockerDriverGateway({ + clearDockerDriverGatewayRuntimeFiles: vi.fn(), + exitOnFailure: true, + gatewayName: "nemoclaw", + hasOpenShellGatewayUserService: () => + hasOpenShellGatewayUserService({ + commandExists: () => true, + platform: "darwin", + spawnSyncImpl: () => spawnResult(1, "formula not installed"), + }), + managedServiceLogCommand: getOpenShellGatewayManagedServiceLogCommand({ + platform: "darwin", + }), + registerDockerDriverGatewayEndpoint: vi.fn(), + runCaptureOpenshell: vi.fn(), + skipSandboxBridgeReachability: false, + startOpenShellGatewayUserService: startService, + stopOpenShellGatewayUserService: vi.fn(), + verifySandboxBridgeGatewayReachableOrExit: vi.fn(), + }), + ).resolves.toBe(false); + expect(startService).not.toHaveBeenCalled(); + }); + + it("blocks standalone fallback when managed service inspection fails trust validation (#8104)", async () => { + const startService = vi.fn(); + + await expect( + startPackageManagedDockerDriverGateway({ + clearDockerDriverGatewayRuntimeFiles: vi.fn(), + exitOnFailure: false, + gatewayName: "nemoclaw", + hasOpenShellGatewayUserService: () => { + throw new OpenShellGatewayServiceTrustError("foreign managed service"); + }, + managedServiceLogCommand: + "journalctl --user --unit nemoclaw-openshell-gateway --no-pager --lines=200", + registerDockerDriverGatewayEndpoint: vi.fn(), + runCaptureOpenshell: vi.fn(), + skipSandboxBridgeReachability: false, + startOpenShellGatewayUserService: startService, + stopOpenShellGatewayUserService: vi.fn(), + verifySandboxBridgeGatewayReachableOrExit: vi.fn(), + }), + ).rejects.toThrow("foreign managed service"); + expect(startService).not.toHaveBeenCalled(); + }); + + it("blocks standalone fallback when managed service cleanup fails trust validation (#8104)", async () => { + await expect( + startPackageManagedDockerDriverGateway({ + clearDockerDriverGatewayRuntimeFiles: vi.fn(), + exitOnFailure: false, + gatewayName: "nemoclaw", + hasOpenShellGatewayUserService: () => true, + registerDockerDriverGatewayEndpoint: vi.fn(), + runCaptureOpenshell: vi.fn(), + skipSandboxBridgeReachability: false, + startOpenShellGatewayUserService: () => ({ + attempted: true, + reason: "restart failed", + started: false, + }), + stopOpenShellGatewayUserService: () => ({ + attempted: true, + reason: "foreign managed service", + standaloneFallbackAllowed: false, + standaloneFallbackBlocked: true, + stopped: false, + }), + verifySandboxBridgeGatewayReachableOrExit: vi.fn(), + }), + ).rejects.toThrow("foreign managed service"); + }); + + it("exits when unhealthy managed service cleanup fails trust validation (#8104)", async () => { + const clock = createVirtualClock(); + const exitSpy = vi.spyOn(process, "exit").mockImplementation((() => { + throw new Error("process.exit(1)"); + }) as typeof process.exit); + vi.spyOn(console, "error").mockImplementation(() => undefined); + vi.spyOn(console, "warn").mockImplementation(() => undefined); + + await expect( + startPackageManagedDockerDriverGateway({ + clearDockerDriverGatewayRuntimeFiles: vi.fn(), + exitOnFailure: true, + gatewayName: "nemoclaw", + hasOpenShellGatewayUserService: () => true, + healthPollCount: 1, + healthPollInterval: 1, + isDockerDriverGatewayReady: async () => false, + now: clock.now, + registerDockerDriverGatewayEndpoint: () => true, + runCaptureOpenshell: (args) => (args[0] === "status" ? STATUS_CONNECTED : GATEWAY_INFO), + skipSandboxBridgeReachability: false, + sleepSeconds: clock.advance, + startOpenShellGatewayUserService: () => ({ attempted: true, started: true }), + stopOpenShellGatewayUserService: () => ({ + attempted: true, + reason: "foreign managed service", + standaloneFallbackAllowed: false, + standaloneFallbackBlocked: true, + stopped: false, + }), + verifySandboxBridgeGatewayReachableOrExit: vi.fn(), + }), + ).rejects.toThrow("process.exit(1)"); + expect(exitSpy).toHaveBeenCalledWith(1); + }); + + it("uses standalone fallback when managed service startup fails unexpectedly (#8104)", async () => { + const stopService = vi.fn(() => ({ + attempted: true, + standaloneFallbackAllowed: false, + stopped: true, + })); + const warn = vi.spyOn(console, "warn").mockImplementation(() => undefined); + + await expect( + startPackageManagedDockerDriverGateway({ + clearDockerDriverGatewayRuntimeFiles: vi.fn(), + exitOnFailure: true, + gatewayName: "nemoclaw", + hasOpenShellGatewayUserService: () => true, + managedServiceLogCommand: + "journalctl --user --unit nemoclaw-openshell-gateway --no-pager --lines=200", + registerDockerDriverGatewayEndpoint: vi.fn(), + runCaptureOpenshell: vi.fn(), + skipSandboxBridgeReachability: false, + startOpenShellGatewayUserService: () => { + throw new Error("systemctl invocation failed"); + }, + stopOpenShellGatewayUserService: stopService, + verifySandboxBridgeGatewayReachableOrExit: vi.fn(), + }), + ).resolves.toBe(false); + expect(stopService).toHaveBeenCalledOnce(); + expect(warn.mock.calls.flat().join("\n")).toContain("journalctl --user --unit"); }); it("stops the trusted systemd gateway unit without disabling it (#7904)", () => { @@ -605,7 +895,11 @@ describe("docker-driver-gateway-service", () => { ), }); - expect(result).toMatchObject({ attempted: true, stopped: false }); + expect(result).toMatchObject({ + attempted: true, + standaloneFallbackBlocked: true, + stopped: false, + }); expect(result.reason).toContain("service identity is not a trusted OpenShell gateway"); expect(events).toEqual([ "show nemoclaw-openshell-gateway --property=FragmentPath --property=ExecStart", diff --git a/src/lib/onboard/docker-driver-gateway-service.ts b/src/lib/onboard/docker-driver-gateway-service.ts index 96026ca3bdf..ecd446660ac 100644 --- a/src/lib/onboard/docker-driver-gateway-service.ts +++ b/src/lib/onboard/docker-driver-gateway-service.ts @@ -40,10 +40,11 @@ export interface OpenShellGatewayUserServiceOptions { export interface OpenShellGatewayUserServiceStartResult { attempted: boolean; - fallbackAllowed: boolean; + logCommand?: string; manager?: "homebrew" | "systemd"; reason?: string; serviceName?: string; + standaloneFallbackBlocked?: boolean; statusCommand?: string; started: boolean; } @@ -54,10 +55,25 @@ export interface OpenShellGatewayUserServiceStopResult { manager?: "homebrew" | "systemd"; reason?: string; serviceName?: string; + standaloneFallbackBlocked?: boolean; statusCommand?: string; stopped: boolean; } +export class OpenShellGatewayServiceEnvironmentError extends Error { + constructor(error: unknown) { + super(formatError(error), { cause: error }); + this.name = "OpenShellGatewayServiceEnvironmentError"; + } +} + +export class OpenShellGatewayServiceTrustError extends Error { + constructor(message: string) { + super(message); + this.name = "OpenShellGatewayServiceTrustError"; + } +} + export interface SpawnSyncLikeResult { error?: Error; status: number | null; @@ -79,6 +95,7 @@ export interface PackageManagedDockerDriverGatewayOptions { healthPollCount?: number; healthPollInterval?: number; isDockerDriverGatewayReady?: () => Promise; + managedServiceLogCommand?: string; now?: () => number; prepareOpenShellGatewayUserServiceEnv?: () => void; preparePortForOpenShellGatewayUserServiceStart?: () => void; @@ -92,6 +109,7 @@ export interface PackageManagedDockerDriverGatewayOptions { "preparePortForServiceStart" | "prepareServiceEnv" | "validatePortOwnerForServiceStart" >, ) => OpenShellGatewayUserServiceStartResult; + stopOpenShellGatewayUserService?: () => OpenShellGatewayUserServiceStopResult; validatePortOwnerForOpenShellGatewayUserServiceStart?: () => void; verifySandboxBridgeGatewayReachableOrExit: ( exitOnFailure: boolean, @@ -100,6 +118,7 @@ export interface PackageManagedDockerDriverGatewayOptions { } interface OpenShellGatewayUserServiceTarget { + logCommand: string; manager: "homebrew" | "systemd"; serviceName: string; statusCommand: string; @@ -107,6 +126,27 @@ interface OpenShellGatewayUserServiceTarget { trustedUnitPaths: string[]; } +function getSystemdGatewayLogCommand(serviceName: string): string { + return `journalctl --user --unit ${serviceName} --no-pager --lines=200`; +} + +function getHomebrewGatewayLogCommand(): string { + return 'tail -n 200 "$(brew --prefix)/var/log/openshell/openshell-gateway.out.log" "$(brew --prefix)/var/log/openshell/openshell-gateway.err.log"'; +} + +export function getOpenShellGatewayManagedServiceLogCommand( + opts: Pick = {}, +): string | undefined { + const platform = opts.platform ?? process.platform; + if (platform === "darwin") return getHomebrewGatewayLogCommand(); + if (platform !== "linux") return undefined; + return getSystemdGatewayLogCommand( + hasUpstreamOpenShellGatewayUserService(opts) + ? OPENSHELL_GATEWAY_USER_SERVICE + : NEMOCLAW_OPENSHELL_GATEWAY_USER_SERVICE, + ); +} + export function getOpenShellGatewayUserServicePaths(): string[] { return [ "/usr/local/lib/systemd/user/openshell-gateway.service", @@ -275,14 +315,17 @@ function hasOfficialHomebrewFormula( const spawnSyncImpl = opts.spawnSyncImpl ?? spawnSync; if ( !runBrew(["list", "--formula", OPENSHELL_GATEWAY_HOMEBREW_SERVICE], { env, spawnSyncImpl }).ok - ) { - throw new Error("The official OpenShell Homebrew formula is not installed"); - } + ) + return false; const info = runBrew(["info", "--json=v2", OPENSHELL_GATEWAY_HOMEBREW_SERVICE], { env, spawnSyncImpl, }); - if (!info.ok) throw new Error(`OpenShell Homebrew formula identity check failed: ${info.reason}`); + if (!info.ok) { + throw new OpenShellGatewayServiceTrustError( + `OpenShell Homebrew formula identity check failed: ${info.reason}`, + ); + } try { const parsed = JSON.parse(info.stdout ?? "") as { formulae?: Array<{ name?: string; tap?: string }>; @@ -291,13 +334,15 @@ function hasOfficialHomebrewFormula( (candidate) => candidate.name === OPENSHELL_GATEWAY_HOMEBREW_SERVICE, ); if (formula?.tap !== OPENSHELL_GATEWAY_HOMEBREW_TAP) { - throw new Error( + throw new OpenShellGatewayServiceTrustError( `OpenShell Homebrew formula must come from ${OPENSHELL_GATEWAY_HOMEBREW_TAP}`, ); } } catch (error) { if (error instanceof SyntaxError) { - throw new Error("OpenShell Homebrew formula identity check returned invalid JSON"); + throw new OpenShellGatewayServiceTrustError( + "OpenShell Homebrew formula identity check returned invalid JSON", + ); } throw error; } @@ -311,6 +356,7 @@ function resolveOpenShellGatewayUserService( if (platform === "darwin") { return hasOfficialHomebrewFormula(opts) ? { + logCommand: getHomebrewGatewayLogCommand(), manager: "homebrew", serviceName: OPENSHELL_GATEWAY_HOMEBREW_SERVICE, statusCommand: `brew services info ${OPENSHELL_GATEWAY_HOMEBREW_SERVICE}`, @@ -322,6 +368,7 @@ function resolveOpenShellGatewayUserService( if (platform !== "linux") return null; if (hasUpstreamOpenShellGatewayUserService(opts)) { return { + logCommand: getSystemdGatewayLogCommand(OPENSHELL_GATEWAY_USER_SERVICE), manager: "systemd", serviceName: OPENSHELL_GATEWAY_USER_SERVICE, statusCommand: `systemctl --user status ${OPENSHELL_GATEWAY_USER_SERVICE}`, @@ -335,12 +382,17 @@ function resolveOpenShellGatewayUserService( const servicePath = getNemoclawOpenShellGatewayUserServicePath(home, env); if (!(opts.existsSync ?? fs.existsSync)(servicePath)) return null; if (isSymbolicLink(servicePath, opts)) { - throw new Error(`Refusing symlinked NemoClaw gateway user service: ${servicePath}`); + throw new OpenShellGatewayServiceTrustError( + `Refusing symlinked NemoClaw gateway user service: ${servicePath}`, + ); } if (!isNemoclawManagedUnit(servicePath, opts)) { - throw new Error(`Refusing foreign NemoClaw gateway user service: ${servicePath}`); + throw new OpenShellGatewayServiceTrustError( + `Refusing foreign NemoClaw gateway user service: ${servicePath}`, + ); } return { + logCommand: getSystemdGatewayLogCommand(NEMOCLAW_OPENSHELL_GATEWAY_USER_SERVICE), manager: "systemd", serviceName: NEMOCLAW_OPENSHELL_GATEWAY_USER_SERVICE, statusCommand: `systemctl --user status ${NEMOCLAW_OPENSHELL_GATEWAY_USER_SERVICE}`, @@ -399,7 +451,7 @@ function extractSystemdExecStartPath(execStart: string): string | null { function validateSystemdServiceIdentity( service: OpenShellGatewayUserServiceTarget, opts: Required>, -): { ok: boolean; reason?: string } { +): { ok: boolean; reason?: string; trustFailure?: boolean } { const result = runSystemctlUser( ["show", service.serviceName, "--property=FragmentPath", "--property=ExecStart"], opts, @@ -412,7 +464,7 @@ function validateSystemdServiceIdentity( function validateSystemdServiceIdentityFromProperties( service: OpenShellGatewayUserServiceTarget, properties: Record, -): { ok: boolean; reason?: string } { +): { ok: boolean; reason?: string; trustFailure?: boolean } { const fragmentPath = path.normalize(properties.FragmentPath ?? ""); const execStartPath = extractSystemdExecStartPath(properties.ExecStart ?? ""); const trustedUnit = service.trustedUnitPaths.some( @@ -426,6 +478,7 @@ function validateSystemdServiceIdentityFromProperties( : { ok: false, reason: `service identity is not a trusted OpenShell gateway (${fragmentPath})`, + trustFailure: true, }; } @@ -505,12 +558,16 @@ function removeCompetingNemoclawUnit( Pick > & Pick, -): { ok: boolean; reason?: string } { +): { ok: boolean; reason?: string; trustFailure?: boolean } { if (service.serviceName !== OPENSHELL_GATEWAY_USER_SERVICE) return { ok: true }; const servicePath = getNemoclawOpenShellGatewayUserServicePath(opts.home, opts.env); if (!opts.existsSync(servicePath)) return { ok: true }; if (isSymbolicLink(servicePath, opts) || !isNemoclawManagedUnit(servicePath, opts)) { - return { ok: false, reason: `refusing to reconcile foreign unit ${servicePath}` }; + return { + ok: false, + reason: `refusing to reconcile foreign unit ${servicePath}`, + trustFailure: true, + }; } const disabled = runSystemctlUser( ["disable", "--now", NEMOCLAW_OPENSHELL_GATEWAY_USER_SERVICE], @@ -528,14 +585,15 @@ function removeCompetingNemoclawUnit( function serviceFailure( service: OpenShellGatewayUserServiceTarget, reason: string, - fallbackAllowed = false, + standaloneFallbackBlocked = false, ): OpenShellGatewayUserServiceStartResult { return { attempted: true, - fallbackAllowed, + logCommand: service.logCommand, manager: service.manager, reason, serviceName: service.serviceName, + standaloneFallbackBlocked, started: false, statusCommand: service.statusCommand, }; @@ -545,12 +603,17 @@ function runHook( hook: (() => void) | undefined, service: OpenShellGatewayUserServiceTarget, description: string, + standaloneFallbackBlocked = false, ): OpenShellGatewayUserServiceStartResult | null { try { hook?.(); return null; } catch (error) { - return serviceFailure(service, `${description}: ${formatError(error)}`); + return serviceFailure( + service, + `${description}: ${formatError(error)}`, + standaloneFallbackBlocked, + ); } } @@ -561,7 +624,6 @@ export function startOpenShellGatewayUserService( if (platform !== "linux" && platform !== "darwin") { return { attempted: false, - fallbackAllowed: true, started: false, reason: "unsupported platform", }; @@ -575,28 +637,27 @@ export function startOpenShellGatewayUserService( if (!service) { return { attempted: false, - fallbackAllowed: true, started: false, reason: "service not installed", }; } const command = stopServiceCommandName(service); if (!commandExists(command)) { - return serviceFailure(service, `${command} is not available`, true); + return serviceFailure(service, `${command} is not available`); } if (service.manager === "systemd") { const reloaded = runSystemctlUser(["daemon-reload"], { env, spawnSyncImpl }); if (!reloaded.ok) { - return serviceFailure( - service, - `systemctl --user daemon-reload failed: ${reloaded.reason}`, - userManagerLooksUnavailable(reloaded.reason ?? ""), - ); + return serviceFailure(service, `systemctl --user daemon-reload failed: ${reloaded.reason}`); } const identity = validateSystemdServiceIdentity(service, { env, spawnSyncImpl }); if (!identity.ok) - return serviceFailure(service, identity.reason ?? "service identity is invalid"); + return serviceFailure( + service, + identity.reason ?? "service identity is invalid", + identity.trustFailure, + ); } const ownershipFailure = runHook( @@ -620,6 +681,7 @@ export function startOpenShellGatewayUserService( return serviceFailure( service, `failed to reconcile gateway user services: ${reconciled.reason}`, + reconciled.trustFailure, ); } } @@ -628,17 +690,14 @@ export function startOpenShellGatewayUserService( opts.prepareServiceEnv, service, "failed to prepare OpenShell gateway service environment", + true, ); if (envFailure) return envFailure; const stop = runStopService(service, { env, spawnSyncImpl }); if (!stop.ok) { const prefix = service.manager === "homebrew" ? "brew services stop" : "systemctl --user stop"; - return serviceFailure( - service, - `${prefix} ${service.serviceName} failed: ${stop.reason}`, - service.manager === "systemd" && userManagerLooksUnavailable(stop.reason ?? ""), - ); + return serviceFailure(service, `${prefix} ${service.serviceName} failed: ${stop.reason}`); } const portFailure = runHook( @@ -663,16 +722,12 @@ export function startOpenShellGatewayUserService( : runSystemctlUser(args, { env, spawnSyncImpl }); if (!result.ok) { const prefix = service.manager === "homebrew" ? "brew" : "systemctl --user"; - return serviceFailure( - service, - `${prefix} ${args.join(" ")} failed: ${result.reason}`, - service.manager === "systemd" && userManagerLooksUnavailable(result.reason ?? ""), - ); + return serviceFailure(service, `${prefix} ${args.join(" ")} failed: ${result.reason}`); } } return { attempted: true, - fallbackAllowed: false, + logCommand: service.logCommand, manager: service.manager, serviceName: service.serviceName, started: true, @@ -707,15 +762,21 @@ export function stopOpenShellGatewayUserService( }; } - const describe = (stopped: boolean, reason?: string): OpenShellGatewayUserServiceStopResult => ({ + const describe = ( + stopped: boolean, + reason?: string, + standaloneFallbackBlocked = false, + ): OpenShellGatewayUserServiceStopResult => ({ attempted: true, standaloneFallbackAllowed: !stopped && + !standaloneFallbackBlocked && service.manager === "systemd" && userManagerLooksUnavailable(reason ?? "") && !hasSystemdUserServiceActivationLink(service, home, env, existsSync), manager: service.manager, serviceName: service.serviceName, + ...(standaloneFallbackBlocked ? { standaloneFallbackBlocked: true } : {}), statusCommand: service.statusCommand, stopped, ...(reason === undefined ? {} : { reason }), @@ -724,7 +785,13 @@ export function stopOpenShellGatewayUserService( if (!commandExists(command)) return describe(false, `${command} is not available`); if (service.manager === "systemd") { const identity = validateSystemdServiceIdentity(service, { env, spawnSyncImpl }); - if (!identity.ok) return describe(false, identity.reason ?? "service identity is invalid"); + if (!identity.ok) { + return describe( + false, + identity.reason ?? "service identity is invalid", + identity.trustFailure, + ); + } } const stop = runStopService(service, { env, spawnSyncImpl }); if (stop.ok) return describe(true); @@ -740,6 +807,7 @@ export async function startPackageManagedDockerDriverGateway({ healthPollCount, healthPollInterval, isDockerDriverGatewayReady = isDockerDriverGatewayHttpReady, + managedServiceLogCommand, now = Date.now, prepareOpenShellGatewayUserServiceEnv, preparePortForOpenShellGatewayUserServiceStart, @@ -748,35 +816,84 @@ export async function startPackageManagedDockerDriverGateway({ skipSandboxBridgeReachability, sleepSeconds: sleepSecondsImpl = sleepSeconds, startOpenShellGatewayUserService: startService = startOpenShellGatewayUserService, + stopOpenShellGatewayUserService: stopService = stopOpenShellGatewayUserService, validatePortOwnerForOpenShellGatewayUserServiceStart, verifySandboxBridgeGatewayReachableOrExit, }: PackageManagedDockerDriverGatewayOptions): Promise { - if (!hasService()) return false; + const stopBeforeStandaloneFallback = () => { + try { + const stopped = stopService(); + if (stopped.standaloneFallbackBlocked) { + throw new OpenShellGatewayServiceTrustError( + stopped.reason ?? "managed service identity is not trusted", + ); + } + if (stopped.attempted && !stopped.stopped) { + const detail = stopped.reason ? ` (${stopped.reason})` : ""; + console.warn( + ` OpenShell gateway managed service could not be stopped${detail}; standalone startup will verify gateway port ownership.`, + ); + } + } catch (error) { + if (error instanceof OpenShellGatewayServiceTrustError && exitOnFailure) process.exit(1); + if (error instanceof OpenShellGatewayServiceTrustError) throw error; + console.warn( + ` OpenShell gateway managed service cleanup failed (${formatError(error)}); standalone startup will verify gateway port ownership.`, + ); + } + }; + try { + if (!hasService()) return false; + } catch (error) { + if (error instanceof OpenShellGatewayServiceTrustError) throw error; + console.warn( + ` OpenShell gateway managed service could not be inspected (${formatError(error)}); using standalone fallback.`, + ); + if (managedServiceLogCommand) console.warn(` Logs: ${managedServiceLogCommand}`); + stopBeforeStandaloneFallback(); + return false; + } console.log(" Starting OpenShell Docker-driver gateway via managed service..."); - const serviceStart = startService({ - preparePortForServiceStart: preparePortForOpenShellGatewayUserServiceStart, - prepareServiceEnv: prepareOpenShellGatewayUserServiceEnv, - validatePortOwnerForServiceStart: validatePortOwnerForOpenShellGatewayUserServiceStart, - }); + let serviceStart: OpenShellGatewayUserServiceStartResult; + try { + serviceStart = startService({ + preparePortForServiceStart: preparePortForOpenShellGatewayUserServiceStart, + prepareServiceEnv: prepareOpenShellGatewayUserServiceEnv, + validatePortOwnerForServiceStart: validatePortOwnerForOpenShellGatewayUserServiceStart, + }); + } catch (error) { + if ( + error instanceof OpenShellGatewayServiceEnvironmentError || + error instanceof OpenShellGatewayServiceTrustError + ) { + throw error; + } + console.warn( + ` OpenShell gateway managed service startup failed (${formatError(error)}); using standalone fallback.`, + ); + if (managedServiceLogCommand) console.warn(` Logs: ${managedServiceLogCommand}`); + stopBeforeStandaloneFallback(); + return false; + } + const reportLogs = () => { + const logCommand = serviceStart.logCommand ?? managedServiceLogCommand; + if (logCommand) console.warn(` Logs: ${logCommand}`); + }; if (!serviceStart.started) { const detail = serviceStart.reason ? ` (${serviceStart.reason})` : ""; - if (serviceStart.fallbackAllowed) { - console.warn( - ` OpenShell gateway service is unavailable${detail}; using standalone fallback.`, - ); - return false; + if (serviceStart.standaloneFallbackBlocked) { + const message = `OpenShell gateway managed service failed to start${detail}.`; + console.error(` ${message}`); + if (exitOnFailure) process.exit(1); + throw new Error(message); } - const message = `OpenShell gateway service failed to start${detail}.`; - console.error(` ${message}`); - console.error( - ` Check: ${ - serviceStart.statusCommand ?? - `systemctl --user status ${serviceStart.serviceName ?? OPENSHELL_GATEWAY_USER_SERVICE}` - }`, + console.warn( + ` OpenShell gateway managed service failed to start${detail}; using standalone fallback.`, ); - if (exitOnFailure) process.exit(1); - throw new Error(message); + reportLogs(); + if (serviceStart.attempted) stopBeforeStandaloneFallback(); + return false; } const pollCount = healthPollCount ?? envInt("NEMOCLAW_HEALTH_POLL_COUNT", 30); @@ -812,15 +929,15 @@ export async function startPackageManagedDockerDriverGateway({ return true; } - const message = `OpenShell gateway service started but did not become healthy within the configured ${formatGatewayHealthWaitLimit( + const message = `OpenShell gateway managed service did not become healthy within the configured ${formatGatewayHealthWaitLimit( pollCount, pollInterval, - )}.`; - console.error(` ${message}`); - console.error( + )}; using standalone fallback.`; + console.warn(` ${message}`); + console.warn( ` Last readiness check: endpoint registered=${lastReadiness.registered ? "yes" : "no"}, OpenShell CLI health=${lastReadiness.cliHealthy ? "yes" : "no"}, direct gRPC health=${lastReadiness.grpcHealthy ? "yes" : "no"}.`, ); - console.error(` Check: ${serviceStart.statusCommand}`); - if (exitOnFailure) process.exit(1); - throw new Error(message); + reportLogs(); + if (serviceStart.attempted) stopBeforeStandaloneFallback(); + return false; } diff --git a/test/onboard-gateway-prelaunch-cutover.test.ts b/test/onboard-gateway-prelaunch-cutover.test.ts index c51f4415c63..e44ee075250 100644 --- a/test/onboard-gateway-prelaunch-cutover.test.ts +++ b/test/onboard-gateway-prelaunch-cutover.test.ts @@ -1,6 +1,9 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { spawnSync } from "node:child_process"; +import path from "node:path"; + import { describe, expect, it } from "vitest"; import { @@ -8,6 +11,7 @@ import { type DockerDriverGatewayCutoverInput, readDockerDriverGatewayHealth, runDockerDriverGatewayCutover, + runDockerDriverGatewayManagedFallback, } from "../src/lib/onboard/docker-driver-gateway-cutover"; type Event = { @@ -119,6 +123,119 @@ describe("Docker-driver gateway prelaunch cutover (#5968)", () => { expect(calls).toEqual([["status"], ["gateway", "info", "-g", "nemoclaw"], ["gateway", "info"]]); }); + it("skips standalone cutover when managed startup succeeds (#8104)", async () => { + let standaloneCalls = 0; + + await expect( + runDockerDriverGatewayManagedFallback( + async () => true, + async () => { + standaloneCalls += 1; + return "launch"; + }, + ), + ).resolves.toBe("managed"); + expect(standaloneCalls).toBe(0); + }); + + it("refreshes listener evidence through the onboard gateway caller (#8104)", () => { + const onboardPath = JSON.stringify(path.join(import.meta.dirname, "../src/lib/onboard.ts")); + const script = ` +const Module = require("node:module"); +const originalLoad = Module._load; +let managedResult = true; +let probe = 0; +let standaloneCalls = 0; +const observedListenerPids = []; + +Module._load = function(request, parent, isMain) { + const actual = () => originalLoad.call(this, request, parent, isMain); + if (request.endsWith("/preflight")) { + return { ...actual(), checkPortAvailable: async () => ({ ok: true, pid: ++probe }) }; + } + if (request.endsWith("/docker-driver-gateway-runtime")) { + const runtime = actual(); + return { + ...runtime, + createDockerDriverGatewayRuntimeHelpers: (deps) => ({ + ...runtime.createDockerDriverGatewayRuntimeHelpers(deps), + createGatewayServicePortOwnership: () => ({ + preparePort: () => {}, + reportUntrustedGatewayPort: () => {}, + validatePortOwner: () => {}, + }), + getDockerDriverGatewayEnv: () => ({}), + getDockerDriverGatewayPid: () => null, + getDockerDriverGatewayPortListenerScan: (portCheck) => ({ + complete: true, + pids: [portCheck.pid], + unverifiedPids: [], + }), + getDockerDriverGatewayStateDir: () => "/test/state", + resolveOpenShellGatewayBinary: () => null, + resolveOpenShellSandboxBinary: () => null, + }), + }; + } + if (request.endsWith("/docker-driver-gateway-env")) { + return { + ...actual(), + getGatewayPortCheckOptions: () => ({}), + startPackageManagedDockerDriverGatewayWithEnvOverride: async () => managedResult, + }; + } + if (request.endsWith("/docker-driver-gateway-cutover")) { + const cutover = actual(); + return { + ...cutover, + readDockerDriverGatewayHealth: () => ({ activeInfo: "", namedInfo: "", status: "" }), + runDockerDriverGatewayCutover: async (input) => { + standaloneCalls += 1; + observedListenerPids.push(input.portListenerScan.pids); + return "reused"; + }, + }; + } + if (request.endsWith("/openshell-cli")) { + return { + createOpenshellCliHelpers: () => ({ + getDockerDriverGatewayEndpointArg: () => "https://127.0.0.1:8080", + getGatewayPortArg: () => "8080", + getOpenshellBinary: () => "/test/bin/openshell", + openshellArgv: (args) => args, + openshellShellCommand: (args) => args.join(" "), + runCaptureOpenshell: () => "openshell 0.0.85", + runOpenshell: () => ({ status: 0 }), + }), + }; + } + return actual(); +}; + +const { startDockerDriverGateway } = require(${onboardPath}); +(async () => { + await startDockerDriverGateway({ exitOnFailure: false }); + const managedSuccessStandaloneCalls = standaloneCalls; + managedResult = false; + probe = 0; + await startDockerDriverGateway({ exitOnFailure: false }); + console.log(JSON.stringify({ managedSuccessStandaloneCalls, observedListenerPids })); +})().catch((error) => { + console.error(error && error.stack ? error.stack : error); + process.exit(1); +}); +`; + const result = spawnSync(process.execPath, ["-e", script], { + cwd: path.join(import.meta.dirname, ".."), + encoding: "utf8", + timeout: 30_000, + }); + + expect(result.status, result.stderr).toBe(0); + const payload = JSON.parse(result.stdout.trim().split("\n").at(-1) ?? "{}"); + expect(payload).toEqual({ managedSuccessStandaloneCalls: 0, observedListenerPids: [[2]] }); + }); + it("reaps stale port listeners before allowing a fresh launch", async () => { const harness = makeHarness({ listenerPids: [4242, 4343], @@ -167,6 +284,31 @@ describe("Docker-driver gateway prelaunch cutover (#5968)", () => { expect(harness.events.some((event) => event.type === "spawn-fresh")).toBe(false); }); + it("preserves the occupied-port gate after managed startup falls back (#8104)", async () => { + let listenerPids = [4242]; + let harness: ReturnType | undefined; + + await expect( + runDockerDriverGatewayManagedFallback( + async () => { + listenerPids = []; + return false; + }, + () => { + harness = makeHarness({ + listenerPids, + scanComplete: true, + pidFileGatewayPid: null, + postReapPortAvailable: false, + }); + return harness.run(); + }, + ), + ).rejects.toThrow("gateway port remains occupied"); + expect(harness?.events).toContainEqual({ type: "prelaunch-reap", extraPids: [] }); + expect(harness?.events.some((event) => event.type === "spawn-fresh")).toBe(false); + }); + it("never includes an unobserved pid-file process in port-scoped cleanup", async () => { const harness = makeHarness({ listenerPids: [4343], pidFileGatewayPid: 4242 });