fix(e2e): stabilize release candidate validation

ci/full-e2e-cold-path-calibration.json

@@ -1,5 +1,5 @@
 {
-  "$comment": "SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\nSPDX-License-Identifier: Apache-2.0\n\nFive independent workflow_dispatch full-e2e samples of current-main baseline 1a74b8348c2182fbf806726341186f008444e28e with the phase-measurement changes at 4544d07c8bfd500c3b64a74380ef5cd0e62089f5. All runs completed install, BuildKit prebuild without fallback, the silence assertion, and the expected first turn. Each budget is derived independently so phase caps diagnose regressions; they are not portions that must sum to the root-start budget.",
+  "$comment": "SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\nSPDX-License-Identifier: Apache-2.0\n\nFive independent workflow_dispatch full-e2e samples of current-main baseline 1a74b8348c2182fbf806726341186f008444e28e with the phase-measurement changes at 4544d07c8bfd500c3b64a74380ef5cd0e62089f5. All runs completed install, BuildKit prebuild without fallback, the silence assertion, and the expected first turn. Each baseline budget is derived independently so phase caps diagnose regressions; they are not portions that must sum to the root-start budget. The separate validation adjustment records four existing full-e2e job observations from three descendant heads after f62c278bd737f4f47be2e85436f65b270d5b4280 added the reviewed WeChat runtime graph to the supported sandbox image. The relevant image-building inputs listed in imageInputPaths were unchanged through 2adc8481ff3053a5a7be37d130cb183e222934ff; repository tests enforce that ancestry and unchanged-input boundary. All four jobs completed install, BuildKit prebuild without fallback, the silence assertion, and the expected first turn; two exceeded the prior root-start and/or sandbox-phase caps. The conclusion fields record the full-e2e job conclusions, not aggregate workflow conclusions. The adjustment raises only those two caps from the maximum observed value plus the existing headroom, rounded up to one second. This is a bounded post-change validation adjustment, not a replacement five-run exact-head calibration. Retire it by replacing the baseline and removing validationAdjustment after five successful full-e2e samples from one exact head that contains the image change.",
   "schemaVersion": 1,
   "calibratedAt": "2026-07-13",
   "baselineMainSha": "1a74b8348c2182fbf806726341186f008444e28e",
@@ -150,6 +150,183 @@
       }
     }
   ],
+  "validationAdjustment": {
+    "validatedAt": "2026-07-14",
+    "imageChangeSha": "f62c278bd737f4f47be2e85436f65b270d5b4280",
+    "imageInputsVerifiedThroughSha": "2adc8481ff3053a5a7be37d130cb183e222934ff",
+    "imageInputPaths": [
+      "Dockerfile",
+      "tsconfig.runtime-preloads.json",
+      "agents/openclaw/mcporter-runtime",
+      "agents/openclaw/wechat-runtime",
+      "nemoclaw/package.json",
+      "nemoclaw/package-lock.json",
+      "nemoclaw/tsconfig.json",
+      "nemoclaw/openclaw.plugin.json",
+      "nemoclaw/src",
+      "nemoclaw-blueprint/blueprint.yaml",
+      "nemoclaw-blueprint/policies",
+      "nemoclaw-blueprint/scripts",
+      "nemoclaw-blueprint/openclaw-plugins",
+      "nemoclaw-blueprint/model-specific-setup",
+      "scripts/checks/verify-openshell-policy-boundary-dependencies.mts",
+      "scripts/nemoclaw-start.sh",
+      "scripts/gateway-control.sh",
+      "scripts/managed-gateway-control.py",
+      "scripts/state-dir-guard.py",
+      "scripts/openclaw-config-guard.py",
+      "scripts/codex-acp-wrapper.sh",
+      "scripts/generate-openclaw-config.mts",
+      "scripts/validate-openclaw-tool-search.mts",
+      "scripts/lib/sandbox-init.sh",
+      "scripts/lib/gateway-supervisor.sh",
+      "scripts/lib/sandbox-rlimits.sh",
+      "scripts/lib/openclaw_device_approval_policy.py",
+      "scripts/lib/clean_runtime_shell_env_shim.py",
+      "scripts/lib/normalize_mutable_config_perms.py",
+      "src/lib/messaging",
+      "src/lib/tool-disclosure.ts",
+      "scripts/patch-openclaw-tool-catalog.js",
+      "scripts/patch-openclaw-chat-send.js",
+      "scripts/patch-openclaw-mcp-npx.mts",
+      "scripts/patch-openclaw-issue-4434-diagnostics.ts",
+      "scripts/patch-openclaw-device-self-approval.ts",
+      "scripts/verify-wechat-runtime-lock.mts",
+      "scripts/lib/reviewed-npm-archive.mts",
+      "src/lib/sandbox/build-context.ts"
+    ],
+    "adjustedMetrics": [
+      "rootStartToFirstTurnCompletion",
+      "nemoclaw.onboard.phase.sandbox"
+    ],
+    "derivation": {
+      "statistic": "maximum",
+      "minimumHeadroomMs": 5000,
+      "relativeHeadroomPercent": 10,
+      "roundUpMs": 1000
+    },
+    "retirement": {
+      "trigger": "successful-exact-head-calibration",
+      "minimumSampleCount": 5,
+      "allSamplesSameHead": true,
+      "imageChangeMustBeAncestor": true,
+      "action": "replace-baseline-and-remove-adjustment"
+    },
+    "runs": [
+      {
+        "runId": 29296660267,
+        "runUrl": "https://github.com/NVIDIA/NemoClaw/actions/runs/29296660267",
+        "headSha": "b634f011e78aa6c31537a70ff76d57bfa8ad7ee6",
+        "conclusion": "failure",
+        "installExitCode": 0,
+        "firstTurnExitCode": 0,
+        "performancePassed": false,
+        "usedBuildKitPrebuild": true,
+        "buildKitFallback": false,
+        "maxSilenceSecs": 32,
+        "responseChars": 23,
+        "measurementsMs": {
+          "onboardRoot": 191509,
+          "rootStartToFirstTurnCompletion": 199164,
+          "rootEndToInstallCompletion": 181,
+          "firstTurnCommand": 7470,
+          "rootEndToFirstTurnCompletion": 7655,
+          "phases": {
+            "nemoclaw.onboard.phase.preflight": 3526,
+            "nemoclaw.onboard.phase.gateway": 66,
+            "nemoclaw.onboard.phase.provider_selection": 1257,
+            "nemoclaw.onboard.phase.inference": 1309,
+            "nemoclaw.onboard.phase.sandbox": 149492
+          }
+        }
+      },
+      {
+        "runId": 29307095519,
+        "runUrl": "https://github.com/NVIDIA/NemoClaw/actions/runs/29307095519",
+        "headSha": "4f64b0ac19d321c6a85c88799c5675e33c3c1a7e",
+        "conclusion": "success",
+        "installExitCode": 0,
+        "firstTurnExitCode": 0,
+        "performancePassed": true,
+        "usedBuildKitPrebuild": true,
+        "buildKitFallback": false,
+        "maxSilenceSecs": 33,
+        "responseChars": 23,
+        "measurementsMs": {
+          "onboardRoot": 187160,
+          "rootStartToFirstTurnCompletion": 195237,
+          "rootEndToInstallCompletion": 168,
+          "firstTurnCommand": 7905,
+          "rootEndToFirstTurnCompletion": 8077,
+          "phases": {
+            "nemoclaw.onboard.phase.preflight": 1204,
+            "nemoclaw.onboard.phase.gateway": 63,
+            "nemoclaw.onboard.phase.provider_selection": 2187,
+            "nemoclaw.onboard.phase.inference": 1467,
+            "nemoclaw.onboard.phase.sandbox": 139921
+          }
+        }
+      },
+      {
+        "runId": 29313003676,
+        "runUrl": "https://github.com/NVIDIA/NemoClaw/actions/runs/29313003676",
+        "headSha": "2adc8481ff3053a5a7be37d130cb183e222934ff",
+        "conclusion": "failure",
+        "installExitCode": 0,
+        "firstTurnExitCode": 0,
+        "performancePassed": false,
+        "usedBuildKitPrebuild": true,
+        "buildKitFallback": false,
+        "maxSilenceSecs": 39,
+        "responseChars": 23,
+        "measurementsMs": {
+          "onboardRoot": 199724,
+          "rootStartToFirstTurnCompletion": 206895,
+          "rootEndToInstallCompletion": 139,
+          "firstTurnCommand": 7028,
+          "rootEndToFirstTurnCompletion": 7171,
+          "phases": {
+            "nemoclaw.onboard.phase.preflight": 1222,
+            "nemoclaw.onboard.phase.gateway": 64,
+            "nemoclaw.onboard.phase.provider_selection": 2634,
+            "nemoclaw.onboard.phase.inference": 1072,
+            "nemoclaw.onboard.phase.sandbox": 155080
+          }
+        }
+      },
+      {
+        "runId": 29313408425,
+        "runUrl": "https://github.com/NVIDIA/NemoClaw/actions/runs/29313408425",
+        "headSha": "2adc8481ff3053a5a7be37d130cb183e222934ff",
+        "conclusion": "success",
+        "installExitCode": 0,
+        "firstTurnExitCode": 0,
+        "performancePassed": true,
+        "usedBuildKitPrebuild": true,
+        "buildKitFallback": false,
+        "maxSilenceSecs": 31,
+        "responseChars": 23,
+        "measurementsMs": {
+          "onboardRoot": 185646,
+          "rootStartToFirstTurnCompletion": 193112,
+          "rootEndToInstallCompletion": 180,
+          "firstTurnCommand": 7282,
+          "rootEndToFirstTurnCompletion": 7466,
+          "phases": {
+            "nemoclaw.onboard.phase.preflight": 1262,
+            "nemoclaw.onboard.phase.gateway": 51,
+            "nemoclaw.onboard.phase.provider_selection": 1367,
+            "nemoclaw.onboard.phase.inference": 708,
+            "nemoclaw.onboard.phase.sandbox": 141977
+          }
+        }
+      }
+    ],
+    "derivedCapsMs": {
+      "rootStartToFirstTurnCompletionBudgetMs": 228000,
+      "sandboxPhaseBudgetMs": 171000
+    }
+  },
   "derivedBudgetsMs": {
     "rootStartToFirstTurnCompletionBudgetMs": 205000,
     "rootEndToFirstTurnCompletionBudgetMs": 14000,

ci/onboard-performance-budget.json

@@ -1,5 +1,5 @@
 {
-  "$comment": "SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\nSPDX-License-Identifier: Apache-2.0\n\nInitial advisory budget for the cloud-onboard-e2e warm-system trace signal. Profiling traces from #3769 were not available in durable CI artifacts when #3776 was implemented. The latest three release tags only exposed one tag-matching trace artifact, from a failed v0.0.66 nightly run, so this cap is calibrated from the latest ten distinct successful main full-trace samples available on 2026-06-23. Those samples had total durations of 298250 ms, 296926 ms, 304190 ms, 294859 ms, 305013 ms, 316147 ms, 300843 ms, 292702 ms, 201332 ms, and 206250 ms; the cap uses p95 via linear interpolation (index 8.55 between samples 8 and 9) plus 25 percent, rounded up to the nearest 30 seconds.\n\nThe full-e2e cold-path budgets are derived from the five current-main samples recorded in ci/full-e2e-cold-path-calibration.json. For each interval and phase independently, the budget is nearest-rank p95 plus the larger of 5 seconds or 10 percent, rounded up to the nearest second. Independent phase caps are diagnostic regression gates, not additive portions of the root-start interval.",
+  "$comment": "SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\nSPDX-License-Identifier: Apache-2.0\n\nInitial advisory budget for the cloud-onboard-e2e warm-system trace signal. Profiling traces from #3769 were not available in durable CI artifacts when #3776 was implemented. The latest three release tags only exposed one tag-matching trace artifact, from a failed v0.0.66 nightly run, so this cap is calibrated from the latest ten distinct successful main full-trace samples available on 2026-06-23. Those samples had total durations of 298250 ms, 296926 ms, 304190 ms, 294859 ms, 305013 ms, 316147 ms, 300843 ms, 292702 ms, 201332 ms, and 206250 ms; the cap uses p95 via linear interpolation (index 8.55 between samples 8 and 9) plus 25 percent, rounded up to the nearest 30 seconds.\n\nThe full-e2e cold-path baseline budgets are derived from the five current-main samples recorded in ci/full-e2e-cold-path-calibration.json. For each interval and phase independently, the baseline is nearest-rank p95 plus the larger of 5 seconds or 10 percent, rounded up to the nearest second. Four later functional full-e2e jobs completed install, BuildKit prebuild without fallback, and the expected first turn on their listed heads after the supported sandbox image changed; their separately recorded bounded validation adjustment raises only the root-start and sandbox-phase caps to the observed maximum plus 10 percent, rounded up to one second. The adjustment is retired after five successful samples from one head containing the image change, when that calibration replaces the baseline and the adjustment is removed. Independent phase caps are diagnostic regression gates, not additive portions of the root-start interval.",
   "schemaVersion": 1,
   "mode": "advisory",
   "scope": "cloud-onboard-e2e warm-system",
@@ -13,14 +13,14 @@
     "minPercent": 30
   },
   "fullE2eColdPath": {
-    "rootStartToFirstTurnCompletionBudgetMs": 205000,
+    "rootStartToFirstTurnCompletionBudgetMs": 228000,
     "rootEndToFirstTurnCompletionBudgetMs": 14000,
     "phaseBudgetsMs": {
       "nemoclaw.onboard.phase.preflight": 11000,
       "nemoclaw.onboard.phase.gateway": 6000,
       "nemoclaw.onboard.phase.provider_selection": 8000,
       "nemoclaw.onboard.phase.inference": 6000,
-      "nemoclaw.onboard.phase.sandbox": 146000
+      "nemoclaw.onboard.phase.sandbox": 171000
     }
   }
 }

ci/source-shape-test-budget.json

@@ -283,7 +283,12 @@
     },
     {
       "file": "test/onboard-performance-config-schema.test.ts",
-      "test": "keeps configured budgets derived from the checked-in samples",
+      "test": "keeps baseline budgets derived from the checked-in samples",
+      "category": "compatibility"
+    },
+    {
+      "file": "test/onboard-performance-config-schema.test.ts",
+      "test": "keeps interim cap adjustments tied to functional post-change evidence",
       "category": "compatibility"
     },
     {

docs/reference/commands.mdx

@@ -1059,8 +1059,11 @@ When the root-owned managed controller attests two unchanged zero-supervisor pro
 The recreation uses a credential-free managed startup command, pins the registered container identity, retains the previous container for rollback, and commits only after managed gateway health and the settle check pass.
 The recreation preserves mounted sandbox state, but a committed swap does not retain changes stored only in the previous container's writable layer.
 It is idempotent.
-When `recover` repairs a stopped built-in OpenClaw or Hermes gateway, it retries only when stdout is empty and stderr is exactly one `SUPERVISOR_BUSY` or `SUPERVISOR_UNAVAILABLE` line, with at most three controller attempts.
-Other controller failures stop immediately.
+When `recover` repairs a stopped built-in OpenClaw or Hermes gateway, it repeats the recovery action only for an exit status of `1` with blank stdout and a sole nonblank stderr line equal to `SUPERVISOR_BUSY`, with at most three controller attempts.
+The same result is inconclusive during managed settle confirmation and can be probed again only within the configured settle window.
+NemoClaw treats `SUPERVISOR_UNAVAILABLE` as terminal because the managed controller uses it for integrity refusals, ambiguous discovery, and process-identity changes.
+It does not repeat the recovery action or treat the settle probe as inconclusive, and instead prints host-side restart and rebuild guidance.
+Other controller failures also stop immediately.
 If the gateway is already running, the command exits zero without force-restarting it; it can still re-evaluate supported safety checks and check or recover host-side forwards.
 Use [`$$nemoclaw <name> gateway restart`](#$$nemoclaw-name-gateway-restart) when you deliberately need a running gateway to reload runtime configuration or plugins.
 

docs/reference/troubleshooting.mdx

@@ -973,15 +973,19 @@ On a direct-container deployment, first confirm that the sandbox is running:
 $$nemoclaw <name> status
 ```
 
-When `recover` repairs a stopped built-in gateway, NemoClaw automatically retries only when stdout is empty and stderr is exactly one `SUPERVISOR_BUSY` or `SUPERVISOR_UNAVAILABLE` line, with at most three controller attempts.
+When `recover` repairs a stopped built-in gateway, NemoClaw repeats the recovery action only for an exit status of `1` with blank stdout and a sole nonblank stderr line equal to `SUPERVISOR_BUSY`, with at most three controller attempts.
+The same result is inconclusive during managed settle confirmation and can be probed again only within the configured settle window.
+NemoClaw treats `SUPERVISOR_UNAVAILABLE` as terminal because it can report unreadable or untrusted supervisor state, ambiguous discovery, or a process-identity change.
 It does not retry other status or output combinations.
 `SUPERVISOR_NOT_RUNNING` is a separate result that requires two zero-supervisor scans with a stable PID 1 and does not enter that retry loop.
 On a supported local Docker-driver sandbox with the legacy keepalive startup, it can authorize a container-identity-pinned recreation that commits only after managed health and settle checks pass.
 To bypass that trusted recreation while troubleshooting, run `NEMOCLAW_DISABLE_SUPERVISOR_RELAUNCH=1 $$nemoclaw <name> recover`; NemoClaw leaves the container unchanged and returns rebuild or re-onboard guidance.
 If that bounded retry is exhausted, or if `gateway restart` reports `SUPERVISOR_BUSY`, wait for the active request to finish and retry the command.
-If the error mentions `SUPERVISOR_NOT_RUNNING` and trusted recreation could not proceed, `SUPERVISOR_REBUILD_REQUIRED`, `SUPERVISOR_UNAVAILABLE`, a missing `nemoclaw-gateway-control` helper, or a missing managed controller, the sandbox image may predate the current lifecycle contract.
-The managed controller also refuses an ambiguous or changing process tree instead of guessing which same-UID process is the gateway.
-Rebuild the image before retrying:
+If the error mentions `SUPERVISOR_NOT_RUNNING` and trusted recreation could not proceed, `SUPERVISOR_REBUILD_REQUIRED`, a missing `nemoclaw-gateway-control` helper, or a missing managed controller, the sandbox image may predate the current lifecycle contract.
+An exact `SUPERVISOR_UNAVAILABLE` result instead means the managed controller refused the current supervisor state rather than guessing which same-UID process is the gateway.
+The current recovery action and any managed settle confirmation stop immediately.
+If `recover` reports this result, follow its host-side `gateway restart` guidance.
+If restart also reports `SUPERVISOR_UNAVAILABLE`, or the image is incompatible, rebuild the image:
 
 ```bash
 $$nemoclaw <name> rebuild --yes

src/lib/actions/sandbox/gateway-restart.test.ts

@@ -121,7 +121,7 @@ describe("restartSandboxGateway — host-mediated gateway restart", () => {
     }
   });
 
-  it("uses the injected supervisor action for the managed settle probe", () => {
+  it("uses the injected supervisor action for managed settle probes", () => {
     const restore = silenceConsole();
     const previousSettleSeconds = process.env.NEMOCLAW_GATEWAY_RECOVERY_SETTLE_SECONDS;
     process.env.NEMOCLAW_GATEWAY_RECOVERY_SETTLE_SECONDS = "0.001";
@@ -147,6 +147,7 @@ describe("restartSandboxGateway — host-mediated gateway restart", () => {
       expect(requestGatewaySupervisorAction.mock.calls).toEqual([
         ["alpha", "restart", 210000],
         ["alpha", "probe"],
+        ["alpha", "probe"],
       ]);
     } finally {
       previousSettleSeconds === undefined