fix(sandbox): recover a gateway wedged in any not-serving state

docs/reference/troubleshooting.mdx

@@ -639,7 +639,18 @@ Docker then marks the container `(unhealthy)` even though `pgrep` still finds th
 NemoClaw prevents and self-heals this:
 
 - The generated sandbox config pins `gateway.reload.mode` to `hot`, so configuration changes never make the gateway restart itself out from under the sandbox supervisor.
-- A serving watchdog inside the sandbox kills a gateway that stops listening after it has served, and the supervisor relaunches it (look for `[gateway-watchdog]` lines in `$$nemoclaw <name> logs`).
+- A serving watchdog inside the sandbox kills a gateway that stops serving after it has served at least once, and the supervisor relaunches it (look for `[gateway-watchdog]` lines in `$$nemoclaw <name> logs`).
+
+The watchdog treats the gateway as serving only while the local health endpoint answers `200` or `401`, the same response requirement the sandbox applies when it waits for the gateway at startup.
+A refused port, a probe that times out, a connection that is accepted and then dropped, and an HTTP error response all count as not serving, so a gateway wedged in any of those states is recovered rather than left running.
+The watchdog logs the cause it observed on each failed probe, and acts after four consecutive failures (roughly two minutes).
+
+A gateway wedged this way shows up as `openclaw health` reporting `gateway_transport_error` — often `1006 abnormal closure (no close frame)` — while the container stays `running` with no restarts.
+Agent replies keep arriving during the wedge because the OpenClaw CLI falls back to running the agent in-process (its embedded fallback) instead of through the gateway, so the sandbox looks healthy from the outside while the gateway is unusable.
+If the watchdog reports `health probe inconclusive`, the probe itself could not run inside the sandbox; the gateway is left untouched, and the sandbox image needs repair with `$$nemoclaw <name> rebuild --yes`.
+
+The watchdog is recovery, not a fix for the underlying wedge: it converts a gateway that cannot serve back into a process exit that the sandbox supervisor already knows how to relaunch.
+It stays necessary until an OpenClaw gateway that cannot serve exits on its own.
 
 Because of the `hot` pin, restart-class configuration changes made inside the sandbox log `config reload requires gateway restart; hot mode ignoring` and do not take effect until the gateway restarts.
 For example, `openclaw plugins install` logs that message until the gateway restarts.

scripts/nemoclaw-start.sh

@@ -4715,13 +4715,86 @@ start_plugin_registry_refresh() {
 # startup failed: ... Process will stay alive"). The #2757 respawn loop only
 # observes process exit, so an alive-but-deaf gateway would stay wedged until
 # a human runs `nemoclaw <sandbox> recover`. This watchdog probes the local
-# health endpoint and — once it has seen a listener at least once — kills the
-# gateway after sustained connection-refused so the respawn loop relaunches
-# it. Only curl exit 7 counts as "listener gone": 200/401 mean serving, and
-# timeout / HTTP-error outcomes (curl 28/22) mean a listener exists and remain
-# the Docker HEALTHCHECK's responsibility. Arming only after the first
-# non-refused probe means a slow first boot is never killed; failed first
-# boots stay the respawn loop's and HEALTHCHECK's job.
+# health endpoint and — once it has seen the gateway serve at least once —
+# kills it after sustained not-serving probes so the respawn loop relaunches
+# it. Arming only after the first serving probe means a slow first boot is
+# never killed; failed first boots stay the respawn loop's and HEALTHCHECK's
+# job.
+#
+# "Serving" uses the same definition as the boot-time readiness gate
+# (openclaw_gateway_healthy): /health must answer 200 or 401. Before #7377 the
+# watchdog instead armed on "curl did not exit 7", so only an unbroken streak
+# of pure connection-refused probes could ever trigger recovery. Every other
+# wedge signature read as "serving" and silently reset the streak:
+#
+#   * curl 28 — the socket accepts but nothing answers within the timeout
+#   * curl 52/56 — accepted then dropped without a reply, which is what the
+#     `gateway closed (1006 abnormal closure (no close frame))` transport
+#     error on the WebSocket side looks like from an HTTP probe
+#   * curl 0 with an HTTP error status — the process answers /health but is
+#     not serving sessions
+#
+# Those outcomes were delegated to the Docker HEALTHCHECK, but an unhealthy
+# OpenShell sandbox container is never restarted by anything, so the gateway
+# stayed wedged indefinitely. Worse, a wedge that alternated between refused
+# and any other failure reset the consecutive-refusal counter on every other
+# probe and never reached the threshold at all, leaving an alive watchdog that
+# logged nothing and recovered nothing (#7377).
+#
+# Probes that fail for local reasons (curl missing or unable to run at all)
+# stay inconclusive: they neither arm the watchdog nor count toward the
+# threshold, so a broken probe can never turn into a kill loop.
+#
+# Source boundary: the wedge itself originates inside OpenClaw's gateway
+# lifecycle, which can leave the process running after its listener is gone
+# instead of exiting. NemoClaw cannot repair that from outside the process, so
+# it does two things it can do: the generated config pins `gateway.reload.mode`
+# to `hot` to remove the most common trigger, and this watchdog turns the
+# surviving cases back into process exits that the #2757 respawn loop already
+# handles. Remove this watchdog once an OpenClaw gateway that cannot serve
+# exits on its own — at that point the respawn loop observes the exit directly
+# and no external prober is needed.
+
+# Human-readable cause for a non-serving probe, used in the watchdog's log
+# lines so an operator can tell a refused port from a wedged-but-listening one.
+gateway_watchdog_curl_reason() {
+  case "$1" in
+    7) printf 'connection refused' ;;
+    28) printf 'probe timeout' ;;
+    52) printf 'empty reply from gateway' ;;
+    55) printf 'send error' ;;
+    56) printf 'connection reset' ;;
+    *) printf 'curl exit %s' "$1" ;;
+  esac
+}
+
+# Classify one health probe of the local gateway port.
+#   0 — serving: /health answered 200 or 401
+#   1 — not serving: refused, timed out, reset, or answered an HTTP error
+#   2 — inconclusive: the probe itself could not run; change no watchdog state
+# GATEWAY_WATCHDOG_PROBE_REASON carries the cause for the two failure returns.
+gateway_watchdog_probe_gateway() {
+  local port="$1"
+  local rc=0 code
+  GATEWAY_WATCHDOG_PROBE_REASON=""
+  code="$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 \
+    "http://127.0.0.1:${port}/health" 2>/dev/null)" || rc=$?
+  if [ "$rc" -ne 0 ]; then
+    GATEWAY_WATCHDOG_PROBE_REASON="$(gateway_watchdog_curl_reason "$rc")"
+    # Only transport outcomes prove the gateway is not serving. Anything else
+    # (curl absent, bad invocation, an unexpected local failure) is a broken
+    # probe, not a broken gateway, and must never escalate to a kill.
+    case "$rc" in
+      7 | 28 | 52 | 55 | 56) return 1 ;;
+      *) return 2 ;;
+    esac
+  fi
+  case "$code" in
+    200 | 401) return 0 ;;
+  esac
+  GATEWAY_WATCHDOG_PROBE_REASON="HTTP ${code:-000}"
+  return 1
+}
 
 # PID-reuse / tamper defense: only kill a process whose cmdline still looks
 # like the OpenClaw gateway. Match the PID 1 launch argv
@@ -4747,10 +4820,13 @@ gateway_watchdog_positive_int_ok() {
 
 start_gateway_serving_watchdog() {
   (
-    local interval refused_threshold armed=0 refused_streak=0
-    local pid start_identity extra tracked_identity last_identity="" rc msg
+    local interval unhealthy_threshold armed=0 unhealthy_streak=0
+    local pid start_identity extra tracked_identity last_identity="" msg
+    local probe_rc inconclusive_logged=0
     interval="${NEMOCLAW_GATEWAY_WATCHDOG_INTERVAL_SECONDS:-30}"
-    refused_threshold="${NEMOCLAW_GATEWAY_WATCHDOG_REFUSED_THRESHOLD:-4}"
+    # Env name kept from #4710 for compatibility; it now bounds consecutive
+    # not-serving probes rather than consecutive connection-refused ones.
+    unhealthy_threshold="${NEMOCLAW_GATEWAY_WATCHDOG_REFUSED_THRESHOLD:-4}"
     # Both knobs must be positive integers: a zero/garbage interval would
     # busy-loop the probe, and a zero threshold would kill on the first
     # refusal. Fall back to the defaults rather than trusting bad input.
@@ -4760,11 +4836,15 @@ start_gateway_serving_watchdog() {
       echo "[gateway-watchdog] invalid NEMOCLAW_GATEWAY_WATCHDOG_INTERVAL_SECONDS='${interval}'; defaulting to 30" >&2
       interval=30
     fi
-    if ! gateway_watchdog_positive_int_ok "$refused_threshold"; then
-      echo "[gateway-watchdog] invalid NEMOCLAW_GATEWAY_WATCHDOG_REFUSED_THRESHOLD='${refused_threshold}'; defaulting to 4" >&2
-      refused_threshold=4
+    if ! gateway_watchdog_positive_int_ok "$unhealthy_threshold"; then
+      echo "[gateway-watchdog] invalid NEMOCLAW_GATEWAY_WATCHDOG_REFUSED_THRESHOLD='${unhealthy_threshold}'; defaulting to 4" >&2
+      unhealthy_threshold=4
     fi
     [ -n "${_DASHBOARD_PORT:-}" ] || exit 0
+    if ! command -v curl >/dev/null 2>&1; then
+      echo "[gateway-watchdog] curl is unavailable; serving watchdog disabled (#7377)" >&2
+      exit 0
+    fi
     while :; do
       sleep "$interval"
       pid=""
@@ -4775,22 +4855,22 @@ start_gateway_serving_watchdog() {
         '' | *[!0-9]*)
           last_identity=""
           armed=0
-          refused_streak=0
+          unhealthy_streak=0
           continue
           ;;
       esac
       case "$start_identity" in
         '' | *[!0-9]*)
           last_identity=""
           armed=0
-          refused_streak=0
+          unhealthy_streak=0
           continue
           ;;
       esac
       if [ -n "$extra" ]; then
         last_identity=""
         armed=0
-        refused_streak=0
+        unhealthy_streak=0
         continue
       fi
       tracked_identity="${pid}:${start_identity}"
@@ -4800,42 +4880,53 @@ start_gateway_serving_watchdog() {
       if [ "$tracked_identity" != "$last_identity" ]; then
         last_identity="$tracked_identity"
         armed=0
-        refused_streak=0
+        unhealthy_streak=0
       fi
       if ! openclaw_supervised_pid_is_live "$pid" "$start_identity"; then
         # Process exit is the respawn loop's signal, not ours.
         last_identity=""
         armed=0
-        refused_streak=0
+        unhealthy_streak=0
+        continue
+      fi
+      probe_rc=0
+      gateway_watchdog_probe_gateway "$_DASHBOARD_PORT" || probe_rc=$?
+      if [ "$probe_rc" -eq 2 ]; then
+        # A probe that could not run tells us nothing about the gateway. Hold
+        # the current armed state and streak, and say so once per episode so
+        # the silence is not mistaken for a healthy gateway.
+        if [ "$inconclusive_logged" -eq 0 ]; then
+          echo "[gateway-watchdog] health probe inconclusive (${GATEWAY_WATCHDOG_PROBE_REASON}); leaving gateway pid $pid untouched (#7377)" >&2
+          inconclusive_logged=1
+        fi
         continue
       fi
-      rc=0
-      curl -s -o /dev/null --max-time 5 "http://127.0.0.1:${_DASHBOARD_PORT}/health" 2>/dev/null || rc=$?
-      if [ "$rc" -ne 7 ]; then
+      inconclusive_logged=0
+      if [ "$probe_rc" -eq 0 ]; then
         armed=1
-        refused_streak=0
+        unhealthy_streak=0
         continue
       fi
       [ "$armed" -eq 1 ] || continue
-      refused_streak=$((refused_streak + 1))
-      if [ "$refused_streak" -lt "$refused_threshold" ]; then
-        echo "[gateway-watchdog] gateway pid $pid alive but port ${_DASHBOARD_PORT} refused connection ($refused_streak/$refused_threshold) (#4710)" >&2
+      unhealthy_streak=$((unhealthy_streak + 1))
+      if [ "$unhealthy_streak" -lt "$unhealthy_threshold" ]; then
+        echo "[gateway-watchdog] gateway pid $pid alive but not serving port ${_DASHBOARD_PORT}: ${GATEWAY_WATCHDOG_PROBE_REASON} ($unhealthy_streak/$unhealthy_threshold) (#7377)" >&2
         continue
       fi
       if ! gateway_pid_is_openclaw_gateway "$pid"; then
         echo "[gateway-watchdog] pid $pid no longer looks like the openclaw gateway; not killing (#4710)" >&2
         armed=0
-        refused_streak=0
+        unhealthy_streak=0
         continue
       fi
       if ! openclaw_supervised_pid_is_live "$pid" "$start_identity"; then
         echo "[gateway-watchdog] pid $pid start identity changed; not killing (#4710)" >&2
         last_identity=""
         armed=0
-        refused_streak=0
+        unhealthy_streak=0
         continue
       fi
-      msg="[gateway-watchdog] CRITICAL: gateway pid $pid is alive but dropped its HTTP listener on port ${_DASHBOARD_PORT} ($refused_streak consecutive refused probes); killing it so the respawn loop can relaunch (#4710)"
+      msg="[gateway-watchdog] CRITICAL: gateway pid $pid is alive but stopped serving port ${_DASHBOARD_PORT} — ${GATEWAY_WATCHDOG_PROBE_REASON} ($unhealthy_streak consecutive not-serving probes); killing it so the respawn loop can relaunch (#7377)"
       echo "$msg" >&2
       append_openclaw_gateway_log_line "$msg" || true
       record_gateway_watchdog_kill "$tracked_identity"
@@ -4848,7 +4939,7 @@ start_gateway_serving_watchdog() {
         kill -KILL "$pid" 2>/dev/null || true
       fi
       armed=0
-      refused_streak=0
+      unhealthy_streak=0
     done
   ) &
   GATEWAY_WATCHDOG_PID=$!