fix(recovery): capture final handoff lifecycle state

docs/reference/commands.mdx

@@ -1134,8 +1134,9 @@ On Jetson/Tegra hosts, the compatibility path uses the NVIDIA runtime and adds e
 These include selected `/dev/nvmap`, `/dev/nvhost-*`, and `/dev/nvgpu/igpu0/*` nodes plus real `/dev/dri/renderD*` character devices.
 After compatibility recreation starts, onboarding keeps the pre-patch container as a rollback backup until the replacement passes the Ready, GPU, and applicable local-inference checks.
 If one of those checks fails before backup removal, onboarding prints failure diagnostics and attempts to restore the pre-patch container.
-To commit the replacement, NemoClaw stops it, removes the rollback backup, and waits until a successful OpenShell sandbox list has no row with that sandbox name.
-NemoClaw then starts the replacement as the final container lifecycle event and verifies OpenShell supervisor readiness again within the same handoff deadline.
+To commit the replacement, NemoClaw first asks OpenShell to stop the sandbox so its durable lifecycle row reaches `Stopped` before any irreversible Docker mutation.
+It then stops the exact transaction-owned replacement, removes the rollback backup, and asks OpenShell to start the sandbox so OpenShell owns the `Starting` lifecycle fence.
+NemoClaw verifies a `Ready` row, a working sandbox exec, and that the exact replacement is the sole running labeled container within the final handoff deadline.
 If that final handoff cannot be confirmed, onboarding exits with the container diagnostics and cleanup guidance instead of reporting success.
 If rollback fails, onboarding reports that the pre-patch container was not restored and prints container-cleanup guidance.
 GPU-proof diagnostics are captured before rollback and can print that guidance before the final container state is known, so inspect the sandbox and its labeled Docker containers before running a deletion command.
@@ -1844,6 +1845,10 @@ NemoClaw waits for the exact replacement to pass managed gateway health and Open
 After state restoration, it restarts the gateway in the exact replacement container and requires an authenticated `ok` result.
 It then runs the managed settle check.
 It commits only after the replacement identity, state restoration, gateway restart, and settle check pass.
+At the final commit handoff, NemoClaw asks OpenShell to stop the sandbox before it mutates either exact container.
+After OpenShell acknowledges that stop, NemoClaw stops the exact replacement, removes the rollback container, and asks OpenShell to start the sandbox through its authoritative lifecycle path.
+This preserves OpenShell's stopped/starting event fence while stale Docker removal snapshots settle; raw Docker stop/start events cannot strand the lifecycle row in `Error` or `Deleting`.
+If the authoritative stop fails, NemoClaw leaves both containers intact. If the start or final `Ready`/exec/exact-container proof fails after rollback-container removal, NemoClaw reports that automatic rollback is unavailable.
 If OpenShell re-registration, state restoration, or a later gateway check fails, NemoClaw attempts to roll back the replacement.
 The primary dashboard or API host forward stays stopped.
 NemoClaw removes the temporary state backup after a successful restore or rollback.
@@ -1852,20 +1857,24 @@ Mounted state remains available, but a committed swap does not retain other writ
 It is idempotent.
 When `recover` repairs a stopped built-in OpenClaw or Hermes gateway, it repeats the recovery action only for these exact transient results:
 
-- Status `1` with blank stdout and exactly one stderr line, `SUPERVISOR_BUSY`.
+- Status `1` with blank stdout and exactly one stderr line: `SUPERVISOR_NOT_RUNNING`, `SUPERVISOR_DISCOVERY_PENDING`, `PRIVILEGED_CONTROL_UNAVAILABLE`, `GATEWAY_HEALTH_TIMEOUT`, or `SUPERVISOR_BUSY`.
 - Status `137` with blank stdout and stderr.
 - Status `1` with blank stdout and exactly one stderr line, `Error response from daemon: Container <id> is restarting, wait until the container is running`.
 
 For the Docker result, `<id>` must be a 64-character lowercase hexadecimal ID that matches the selected registry-owned container.
 Recovery makes at most 11 controller attempts in total.
 It stops after 3 of those attempts return `SUPERVISOR_BUSY`.
-Managed settle confirmation treats exact `SUPERVISOR_BUSY` as inconclusive within its configured window.
+The managed controller emits `SUPERVISOR_DISCOVERY_PENDING` only when an incomplete process-table scan during startup cannot yet prove either one exact supervisor or clean supervisor absence.
+That result delays recovery but cannot authorize container recreation or accept a supervisor identity; a later request must perform the complete identity proof again.
+Managed settle confirmation treats exact `SUPERVISOR_BUSY` and `SUPERVISOR_DISCOVERY_PENDING` results as inconclusive within its configured window.
 Status `137` and the Docker restart result remain terminal during that confirmation.
-The managed supervisor startup waiter accepts the two container-transition results within its separate 11-attempt bound.
+The managed supervisor startup waiter accepts the four exact startup results, `SUPERVISOR_BUSY`, and the two container-transition results within its separate 11-attempt bound.
 Unbound container IDs, reformatted Docker errors, status `137` with nonblank output, and other diagnostic results are terminal.
 NemoClaw treats `SUPERVISOR_UNAVAILABLE` as terminal because the managed controller uses it for integrity refusals, ambiguous discovery, and process-identity changes.
 It does not repeat the recovery action or treat the settle probe as inconclusive, and instead prints host-side restart and rebuild guidance.
 Other controller failures also stop immediately.
+Only an exact `SUPERVISOR_NOT_RUNNING` result that remains after the bounded startup retries can enter transactional legacy keepalive recreation.
+The pinned controller probe must then confirm the missing supervisor before recreation proceeds.
 If the gateway is already running, the command exits zero without force-restarting it; it can still re-evaluate supported safety checks and check or recover host-side forwards.
 Use [`$$nemoclaw <name> gateway restart`](#$$nemoclaw-name-gateway-restart) when you deliberately need a running gateway to reload runtime configuration or plugins.
 

docs/reference/troubleshooting.mdx

@@ -1737,20 +1737,28 @@ $$nemoclaw <name> status
 
 When `recover` repairs a stopped built-in gateway, NemoClaw repeats the recovery action only for these exact transient results:
 
+- Status `1` with blank stdout and exactly one stderr line: `SUPERVISOR_NOT_RUNNING`, `SUPERVISOR_DISCOVERY_PENDING`, `PRIVILEGED_CONTROL_UNAVAILABLE`, or `GATEWAY_HEALTH_TIMEOUT`.
 - Status `1` with blank stdout and exactly one stderr line, `SUPERVISOR_BUSY`.
 - Status `137` with blank stdout and stderr.
 - Status `1` with blank stdout and exactly one stderr line, `Error response from daemon: Container <id> is restarting, wait until the container is running`.
 
 For the Docker result, `<id>` must be a 64-character lowercase hexadecimal ID that matches the selected registry-owned container.
 Recovery makes at most 11 controller attempts in total.
 It stops after 3 of those attempts return `SUPERVISOR_BUSY`.
-Managed settle confirmation treats exact `SUPERVISOR_BUSY` as inconclusive within its configured window.
+The managed controller emits `SUPERVISOR_DISCOVERY_PENDING` only when an incomplete process-table scan during startup cannot yet prove either one exact supervisor or clean supervisor absence.
+That result does not authorize container recreation or accept a supervisor identity; a later controller request must perform the full identity proof again.
+Managed settle confirmation treats exact `SUPERVISOR_BUSY` and `SUPERVISOR_DISCOVERY_PENDING` results as inconclusive within its configured window.
 Status `137` and the Docker restart result remain terminal during that confirmation.
-The managed supervisor startup waiter accepts the two container-transition results within its separate 11-attempt bound.
+The managed supervisor startup waiter accepts the four exact startup results, `SUPERVISOR_BUSY`, and the two container-transition results within its separate 11-attempt bound.
 Unbound container IDs, reformatted Docker errors, status `137` with nonblank output, and other diagnostic results stop immediately.
 NemoClaw treats `SUPERVISOR_UNAVAILABLE` as terminal because it can report unreadable or untrusted supervisor state, ambiguous discovery, or a process-identity change.
-`SUPERVISOR_NOT_RUNNING` is a separate result that requires two zero-supervisor scans with a stable PID 1 and does not enter that retry loop.
-On a supported local Docker-driver sandbox with the legacy keepalive startup, it can authorize a container-identity-pinned recreation that commits only after managed health and settle checks pass.
+`SUPERVISOR_NOT_RUNNING` is a separate result that requires two zero-supervisor scans with a stable PID 1.
+It enters the bounded startup retry first; only an exact missing-supervisor result that remains after the bound can authorize a container-identity-pinned recreation on a supported local Docker-driver sandbox with the legacy keepalive startup.
+That recreation commits only after managed health and settle checks pass.
+At the final commit handoff, NemoClaw asks OpenShell to stop the sandbox before it mutates either exact container.
+After OpenShell acknowledges that stop, NemoClaw stops the exact replacement, removes the rollback container, and asks OpenShell to start the sandbox through its authoritative lifecycle path.
+This preserves OpenShell's stopped/starting event fence while stale Docker removal snapshots settle; raw Docker stop/start events cannot strand the lifecycle row in `Error` or `Deleting`.
+If the authoritative stop fails, NemoClaw leaves both containers intact. If the start or final `Ready`/exec/exact-container proof fails after rollback-container removal, NemoClaw reports that automatic rollback is unavailable.
 To bypass that trusted recreation while troubleshooting, run `NEMOCLAW_DISABLE_SUPERVISOR_RELAUNCH=1 $$nemoclaw <name> recover`; NemoClaw leaves the container unchanged and returns rebuild or re-onboard guidance.
 If recovery stops after 3 `SUPERVISOR_BUSY` results, or if `gateway restart` reports `SUPERVISOR_BUSY`, wait for the active request to finish and retry the command.
 If recovery exhausts the transition bound after status `137` or the Docker restart result, wait for the container to finish restarting and retry the command.

scripts/managed-gateway-control.py

@@ -1010,21 +1010,31 @@ def _discover_supervisor(reader: ProcReader) -> ProcessIdentity:
     matches, inconclusive = _supervisor_candidates(reader, pid1, sandbox_uid)
     if inconclusive:
         # Busy agents can create or reap an unrelated short-lived process while
-        # /proc is being read. Retry only when one exact supervisor was already
-        # proven, and require that same pinned identity on every scan. A missing,
-        # changing, or duplicate supervisor still fails closed immediately.
-        if len(matches) != 1:
+        # /proc is being read. A restart can expose this churn before the
+        # supervisor's argv is observable, so zero matches plus an incomplete
+        # scan is not the clean two-scan absence proof below. Keep one exact
+        # supervisor pinned when it is already visible. If no supervisor was
+        # visible, require a fresh controller request after one appears rather
+        # than accepting an identity born during this ambiguous scan. The host
+        # retries only the exact SUPERVISOR_DISCOVERY_PENDING marker within its
+        # existing bound; duplicate or changing identities remain terminal.
+        if len(matches) > 1:
             raise ControlError("SUPERVISOR_UNAVAILABLE")
-        expected = matches[0].stable_key()
+        expected = matches[0].stable_key() if matches else None
         deadline = time.monotonic() + PROCESS_PROOF_GRACE_SECONDS
         while inconclusive:
             remaining = deadline - time.monotonic()
             if remaining <= 0:
-                raise ControlError("SUPERVISOR_UNAVAILABLE")
+                raise ControlError("SUPERVISOR_DISCOVERY_PENDING")
             time.sleep(min(PROCESS_PROOF_RETRY_SECONDS, remaining))
             _recapture_exact_identity(reader, pid1, deadline=deadline)
             matches, inconclusive = _supervisor_candidates(reader, pid1, sandbox_uid)
-            if len(matches) != 1 or matches[0].stable_key() != expected:
+            if len(matches) > 1:
+                raise ControlError("SUPERVISOR_UNAVAILABLE")
+            if expected is None:
+                if matches:
+                    raise ControlError("SUPERVISOR_DISCOVERY_PENDING")
+            elif len(matches) != 1 or matches[0].stable_key() != expected:
                 raise ControlError("SUPERVISOR_UNAVAILABLE")
     if len(matches) == 0:
         # A zero-match scan is the only absence signal that may authorize the

src/lib/actions/sandbox/gateway-restart.test.ts

@@ -21,6 +21,7 @@ const supervisorFailureMarkers: Array<
   ["SUPERVISOR_UNAVAILABLE", "privileged control unavailable"],
   ["SUPERVISOR_UNAVAILABLE\nNEMOCLAW_CONTROL_STAGE=await-replacement", "supervisor unavailable"],
   ["SUPERVISOR_NOT_RUNNING", "supervisor not running"],
+  ["SUPERVISOR_DISCOVERY_PENDING", "supervisor unavailable"],
   ["SUPERVISOR_REBUILD_REQUIRED", "privileged control unavailable"],
   ["SUPERVISOR_BUSY", "privileged control unavailable"],
   [MARKERS.SECRET_BOUNDARY_REFUSED, "secret-boundary refusal"],

src/lib/actions/sandbox/gateway-restart.ts

@@ -206,6 +206,12 @@ export function classifyGatewayRestartFailure(result: GatewayRestartCommandResul
       detail: detail || "the in-sandbox gateway supervisor is not running",
     };
   }
+  if (output.includes("SUPERVISOR_DISCOVERY_PENDING")) {
+    return {
+      layer: "supervisor unavailable",
+      detail: detail || "the managed gateway supervisor is still starting",
+    };
+  }
   if (output.includes("SUPERVISOR_UNAVAILABLE") && output.includes("NEMOCLAW_CONTROL_STAGE=")) {
     return {
       layer: "supervisor unavailable",

src/lib/actions/sandbox/process-recovery-managed-startup.test.ts

@@ -48,7 +48,12 @@ afterEach(() => {
 });
 
 describe("checkAndRecoverSandboxProcesses managed startup", () => {
-  it.each(["SUPERVISOR_NOT_RUNNING", "PRIVILEGED_CONTROL_UNAVAILABLE", "GATEWAY_HEALTH_TIMEOUT"])(
+  it.each([
+    "SUPERVISOR_NOT_RUNNING",
+    "SUPERVISOR_DISCOVERY_PENDING",
+    "PRIVILEGED_CONTROL_UNAVAILABLE",
+    "GATEWAY_HEALTH_TIMEOUT",
+  ])(
     "waits through the exact %s startup transition (#9466)",
     (startupMarker) => {
       const sandboxName = "startup-box";
@@ -81,6 +86,34 @@ describe("checkAndRecoverSandboxProcesses managed startup", () => {
     },
   );
 
+  it("does not retry a diagnostic-bearing supervisor-discovery result", () => {
+    const sandboxName = "diagnostic-start";
+    mockOpenClawSandbox(sandboxName);
+    vi.stubEnv("NEMOCLAW_GATEWAY_RECOVERY_POLL_INTERVAL_SECONDS", "0");
+    const requestGatewaySupervisorAction = vi.fn(() => ({
+      status: 1,
+      stdout: "",
+      stderr: "SUPERVISOR_DISCOVERY_PENDING\nunexpected diagnostic",
+    }));
+    const relaunchManagedSupervisorSessionImpl = vi.fn(() => null);
+
+    const result = checkAndRecoverSandboxProcesses(sandboxName, {
+      quiet: true,
+      isSandboxGatewayRunningImpl: () => false,
+      requestGatewaySupervisorAction,
+      relaunchManagedSupervisorSessionImpl,
+    });
+
+    expect(result).toMatchObject({
+      checked: true,
+      wasRunning: false,
+      recovered: false,
+      forwardRecovered: false,
+    });
+    expect(requestGatewaySupervisorAction).toHaveBeenCalledOnce();
+    expect(relaunchManagedSupervisorSessionImpl).not.toHaveBeenCalled();
+  });
+
   it("does not retry a managed-container identity mismatch (#9466)", () => {
     const sandboxName = "identity-box";
     mockOpenClawSandbox(sandboxName);

src/lib/actions/sandbox/process-recovery.test.ts

@@ -650,7 +650,7 @@ describe("confirmRecoveredSandboxGatewayManaged scope", () => {
     ).toBe(false);
   });
 
-  it("keeps unavailable supervisor results terminal while lease contention stays transient", () => {
+  it("keeps unavailable results terminal while exact transient results stay inconclusive", () => {
     const confirm = (stderr: string) =>
       confirmRecoveredSandboxGatewayManaged("my-sandbox", {
         getSandboxImpl: () => openClawEntry,
@@ -660,6 +660,8 @@ describe("confirmRecoveredSandboxGatewayManaged scope", () => {
 
     expect(confirm("SUPERVISOR_UNAVAILABLE")).toBe(false);
     expect(confirm("SUPERVISOR_BUSY")).toBeNull();
+    expect(confirm("SUPERVISOR_DISCOVERY_PENDING")).toBeNull();
+    expect(confirm("SUPERVISOR_DISCOVERY_PENDING\nunexpected diagnostic")).toBe(false);
   });
 });