fix: surface idle quota stops and Pi trust prompts promptly

.agents/skills/stuck-crewmate-recovery/SKILL.md

@@ -67,6 +67,21 @@ Never restart, stop, or update the shared daemon on a crewmate's claim.
 It is one instance serving every lane and home, so a restart kills other lanes' in-flight runs.
 Only positive socket refusal or absence is a daemon-down finding; escalate that finding, or a failed run record that names a daemon error, to the captain.
 
+## Quota-exhausted worker
+
+A `quota-exhausted` stale wake identifies a conservative rendered usage-limit stop, not a generic wedge.
+Confirm the targeted current state and pane still show that stop and that no active validation run owns the work before replacing the worker.
+Load `harness-adapters` and use the current dispatch resolver when available, then apply the ordinary dispatch eligibility and quota-array selection procedure to choose the next eligible candidate rather than retrying the exhausted model.
+Relaunch the same task in place through `bin/fm-control.sh <task-id> relaunch`, passing the selected harness, model, effort, and a progress note using its current help.
+Preserve existing work and report the recovery choice; silent automatic model switching is forbidden, and the watcher only reports evidence, never relaunches.
+If no eligible candidate can proceed, report the blocker and, when present, the raw delay, UTC observation time, and reset upper bound ("no later than"), not an exact reset time; do not repeatedly relaunch.
+`bin/fm-pane-stop-lib.sh` owns the supported rendered stops; `bin/fm-watch.sh`'s header owns wake timing, reset upper bounds, and deduplication.
+An unknown reset must not be invented.
+
+A `blocked-at-prompt` stale wake instead calls for trust handling, including workers that have not yet written a status event.
+Load `harness-adapters` and follow that harness's documented trust procedure; do not blindly send Enter or manufacture consent, because some dialogs require an operator decision and some default to exit.
+The watcher never accepts a prompt or changes trust settings.
+
 ## Live-endpoint escalation
 
 Escalate in order:

AGENTS.md

@@ -159,7 +159,7 @@ state/               runtime records and signals; gitignored
   .watch.lock .wake-queue.lock watcher singleton and queue serialization locks
   .claude-autoarm.lock .claude-autoarm-epoch .claude-autoarm-failure-notified .claude-autoarm-failure-alarmed .turnend-claude-blocks .turnend-claude-blocks.lock   Claude Stop auto-arm single-flight, epoch, failure-episode, attended-alarm, guard-budget, and budget-lock records; never touch
   .cursor-park-owner .cursor-park-owner.lock .turnend-cursor-blocks   Cursor stop-hook owner record, publication and commit lock, and bounded repair-nag budget; never touch
-  .hash-* .count-* .stale-* .stale-since-* .churn-since-* .paused-* .wedge-escalations-* .dead-reported-* .writing-* .waiting-* .seen-* .hb-surfaced-* .last-* .heartbeat-streak   watcher internals; never touch
+  .hash-* .count-* .stale-* .stale-since-* .pane-stop-* .churn-since-* .paused-* .wedge-escalations-* .dead-reported-* .writing-* .waiting-* .seen-* .hb-surfaced-* .last-* .heartbeat-streak   watcher internals; never touch
   .watch-triage.log  watcher's absorbed-wake debug log (size-capped); never relied on, safe to delete
   .last-watcher-beat watcher liveness beacon, touched every poll (including while absorbing benign wakes); guard scripts read it
   .subsuper-* .supervise-daemon.*   sub-supervisor internals; never touch

bin/fm-pane-stop-lib.sh

@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+# Conservative rendered stop recognition for fm-watch.sh; no network calls.
+# fm_pane_stop <harness> <pane> prints kind<TAB>provider<TAB>reset-delay<TAB>display.
+# Unknown reset delays use '-' (never infer a weekly reset date).
+# Callers establish a live, idle worker first. Only standalone lines qualify;
+# quota errors must be in the last 12 nonblank lines, dialogs in the last 40.
+# An unknown error or changed vendor wording stays on ordinary stale triage.
+fm_pane_stop() {
+  local harness=$1 pane=$2 allowed kind provider pattern line normalized recent
+  local hours minutes seconds duration
+  normalized=$(printf '%s\n' "$pane" | sed -E $'s/\033\\[[0-9;]*[mK]//g; s/^[[:space:]│┃]+//; s/[[:space:]│┃]+$//; /^[[:space:]]*$/d' | tail -n 40)
+  while IFS='|' read -r allowed kind provider pattern; do
+    case ",$allowed," in *",$harness,"*) ;; *) continue ;; esac
+    recent=$normalized
+    [ "$kind" != quota-exhausted ] || recent=$(printf '%s\n' "$normalized" | tail -n 12)
+    while IFS= read -r line; do
+      [[ $line =~ $pattern ]] || continue
+      if [ "$kind" = quota-exhausted ] && [ "$provider" = gemini ]; then
+        hours=${BASH_REMATCH[2]:-0}; minutes=${BASH_REMATCH[4]:-0}; seconds=${BASH_REMATCH[6]:-0}
+        duration=${BASH_REMATCH[1]}${BASH_REMATCH[3]}${BASH_REMATCH[5]}
+        [ -n "$duration" ] || continue
+        [ "$((10#$minutes))" -lt 60 ] && [ "$((10#$seconds))" -lt 60 ] || continue
+        printf '%s\t%s\t%s\t%s\n' "$kind" "$provider" "$((10#$hours * 3600 + 10#$minutes * 60 + 10#$seconds))" "$duration"
+      elif [ "$kind" = blocked-at-prompt ]; then
+        printf '%s\n' "$recent" | grep -qE '^Do not trust$' || continue
+        printf '%s\t%s\t-\t%s\n' "$kind" "$harness" "$provider"
+      else
+        printf '%s\t%s\t-\tunknown\n' "$kind" "$provider"
+      fi
+      return 0
+    done <<< "$recent"
+  done <<'PATTERNS'
+grok|quota-exhausted|grok|^You hit your weekly limit$
+pi,pi-signed|quota-exhausted|gemini|^Error: Quota reached\. Please wait (([0-9]{1,3})h)?(([0-9]{1,2})m)?(([0-9]{1,2})s)?$
+pi,pi-signed|blocked-at-prompt|trust|^Trust project folder[?]$
+PATTERNS
+  return 1
+}

bin/fm-watch.sh

@@ -27,7 +27,8 @@
 #                          line, since the crew's own log gets no new entry once
 #                          firstmate hands it to a no-mistakes validation. A declared
 #                          external-wait pause or verified captain-held transfer is
-#                          absorbed instead with its own long re-surface cadence,
+#                          normally absorbed with its own long re-surface cadence
+#                          (recognized idle pane stops below take precedence),
 #                          never as a wedge, and that recheck reason names which
 #                          human the wait is on. Only when neither absorb class
 #                          applies does the log's latest recognized status event decide:
@@ -76,6 +77,24 @@
 #                          agent, for human inspection only - never an automatic
 #                          interrupt, signal, or restart of the worker or its
 #                          tool process.
+#   stale: <window> (quota-exhausted: <provider>, observed <UTC>, resets no later than <UTC> (<delay>))
+#   stale: <window> (blocked-at-prompt: <harness> trust)
+#                          recognized idle stops from fm-pane-stop-lib.sh bypass
+#                          ordinary stale/wedge triage, including declared pauses,
+#                          after two unchanged-hash polls. Secondmates and
+#                          away-silenced captain holds are excluded; positive
+#                          working evidence or a dead/missing agent rejects a stop.
+#                          A rendered delay is observed at detection time; that
+#                          time plus the delay is only an upper bound on reset,
+#                          printed with the observation time and raw delay.
+#                          Otherwise the reset is 'unknown', not inferred.
+#                          .pane-stop-<key> stores hash<TAB>busy-generation, so an
+#                          unchanged stop skips repeat probes and wakes. Pane
+#                          churn or busy activity clears it; a new generation
+#                          permits reclassification. No prompt is answered and
+#                          no worker is relaunched. Recovery procedure:
+#                          .agents/skills/stuck-crewmate-recovery/SKILL.md.
+#                          Regression: tests/fm-watch-triage.test.sh.
 #   stale: <window> (unread firstmate instruction: ...)
 #                          the steering-inbox ladder spent its delivery-attempt
 #                          budget on an idle pane without an acknowledgement
@@ -396,6 +415,42 @@ window_backend() {
   echo tmux
 }
 
+# shellcheck source=bin/fm-pane-stop-lib.sh
+. "$SCRIPT_DIR/fm-pane-stop-lib.sh"
+
+pane_stop_stale_check() {
+  local w=$1 task=$2 h=$3 pane=$4 key record parsed provider delay display now observed reset kind reason gen agent_state
+  key=$(window_key "$w")
+  record="$STATE/.pane-stop-$key"
+  parsed=$(fm_pane_stop "$(window_harness "$w")" "$pane") || { rm -f "$record"; return 1; }
+  gen=$(fm_busy_current_gen "$STATE" "$task") || gen=-
+  if [ -f "$record" ] && [ "$(cut -f1 "$record")" = "$h" ] && [ "$(cut -f2 "$record")" = "$gen" ]; then return 0; fi
+  if crew_is_provably_working "$task"; then rm -f "$record"; return 1; fi
+  agent_state=$(fm_backend_agent_state "$(window_backend "$w")" "$w" 2>/dev/null) || agent_state=unreadable
+  case "$agent_state" in dead|missing) rm -f "$record"; return 1 ;; esac
+  IFS=$'\t' read -r kind provider delay display <<< "$parsed"
+  if [ "$delay" != - ]; then
+    now=$(date +%s)
+    reset=$(date -u -r "$((now + delay))" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null \
+      || date -u -d "@$((now + delay))" +%Y-%m-%dT%H:%M:%SZ) || return 1
+    observed=$(date -u -r "$now" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null \
+      || date -u -d "@$now" +%Y-%m-%dT%H:%M:%SZ) || return 1
+    display="observed $observed, resets no later than $reset ($display)"
+  fi
+  if [ "$kind" = quota-exhausted ]; then
+    if [ "$delay" = - ]; then display="resets unknown"; fi
+    reason="stale: $w ($kind: $provider, $display)"
+  else
+    reason="stale: $w ($kind: $provider $display)"
+  fi
+  fm_wake_append stale "$w" "$reason" || exit 1
+  printf '%s\t%s\n' "$h" "$gen" > "$record"
+  printf '%s' "$h" > "$STATE/.stale-$key"
+  rm -f "$STATE/.stale-since-$key" "$STATE/.wedge-escalations-$key"
+  wake "$reason"
+  return 0
+}
+
 window_harness() {
   local w=$1 meta
   meta=$(fm_backend_meta_for_window "$w" "$STATE" 2>/dev/null || true)
@@ -706,7 +761,7 @@ signal_turnend_panes_churned() {  # <file> ...
     return 1
   done
   for key in "${churned_keys[@]}"; do
-    if ! rm -f "$STATE/.stale-$key" "$STATE/.wedge-escalations-$key"; then
+    if ! rm -f "$STATE/.stale-$key" "$STATE/.wedge-escalations-$key" "$STATE/.pane-stop-$key"; then
       for created in "${created_keys[@]+"${created_keys[@]}"}"; do
         rm -f "$STATE/.churn-since-$created"
       done
@@ -1523,7 +1578,7 @@ clear_pause_state() {  # <window-key>
 clear_stale_hash_tracking() {  # <window-key>
   local key=$1
   clear_write_tracking "$key"
-  rm -f "$STATE/.stale-$key" "$STATE/.stale-since-$key" "$STATE/.wedge-escalations-$key" \
+  rm -f "$STATE/.stale-$key" "$STATE/.stale-since-$key" "$STATE/.wedge-escalations-$key" "$STATE/.pane-stop-$key" \
     "$STATE/.waiting-resurfaced-$key"
 }
 
@@ -2757,6 +2812,9 @@ EOF
     # content cannot suppress stale detection. Read once per window per poll and
     # reused below so a busy verdict is consistent within one cycle.
     if window_is_busy "$w" "$tail40"; then busy_now=0; else busy_now=1; fi
+    if [ "$busy_now" -eq 0 ] || [ "$h" != "$prev" ]; then
+      rm -f "$STATE/.pane-stop-$key"
+    fi
     if [ "$h" = "$prev" ]; then
       n=$(( $(cat "$cf" 2>/dev/null || echo 0) + 1 ))
       echo "$n" > "$cf"
@@ -2768,6 +2826,8 @@ EOF
             paused) handle_paused_stale "$w" "$task" "$h" ;;
             *)      clear_pause_tracking "$key" ;;
           esac
+        elif ! captain_held_silenced "$last" && pane_stop_stale_check "$w" "$task" "$h" "$tail40"; then
+          : # Explicit stops bypass the wedge ladder; never answer or relaunch here.
         elif afk_present; then
           # Daemon owns triage: one-shot per distinct stale hash, as before,
           # except that a captain-held pane is never handed over while the

docs/architecture.md

@@ -84,7 +84,7 @@ The deferral is bounded per endpoint by `FM_TURNEND_CHURN_ABSORB_SECS`, tracked
 That bound is load-bearing rather than cosmetic: churn and staleness read the same pane, so a pane that renders continuously - a clock, a spinner, a shell heartbeat, or a harness that leaves a background renderer alive after its agent yields - never reaches the staleness backbone's two-identical-hashes test either, and an unbounded churn absorb would leave a genuinely stopped worker behind such a renderer with no path left to surface it.
 If two metadata records derive the same per-window marker key, including two records that name the same endpoint, that marker is not attributable churn evidence for either task, so the bare turn-ended wake surfaces without changing or migrating existing marker state.
 A `kind=secondmate` task's status signal is the parent-directed reply stream and is never absorbed as provably working; its bare turn-ended signal is absorbed only by the ordinary authoritative working proof because an active secondmate does not enter the staleness backbone that would resurface deferred pane-churn evidence.
-A crew that declares `paused:` for a known external wait, or carries a verified `captain-held` transfer, is separately absorbed while idle and re-surfaced only on the longer pause cadence, rather than being treated as a possible wedge, except that a captain-held transfer is not rechecked while the away-posture record exists.
+Idle declared-wait routing, including the recognized pane-stop precedence exception, is owned by [`bin/fm-watch.sh`](../bin/fm-watch.sh)'s header.
 For an ordinary crew that has stopped, the normal-mode watcher first surfaces one stale wake, then applies that same cadence to an unchanged `paused:` or durable `captain-held` endpoint while attended; the pause classification itself is recovered only when the backend confidently reports its agent dead.
 Live or inconclusive liveness remains fail-open at that initial surface, so a worker genuinely waiting on a decision is never silenced.
 Its later sights are still held to that same bounded cadence rather than re-alarming on every pane-hash change, because the throttle is keyed to the declaration and not to the pane an idle parked worker keeps ticking.

tests/fm-pane-stop.test.sh

@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+# Exact observed stops, duration parsing, and conservative negative cases.
+set -eu
+. "$(dirname "$0")/lib.sh"
+. "$ROOT/bin/fm-pane-stop-lib.sh"
+[ "$(fm_pane_stop grok 'You hit your weekly limit')" = $'quota-exhausted\tgrok\t-\tunknown' ] || fail 'weekly stop'
+[ "$(fm_pane_stop pi 'Error: Quota reached. Please wait 2h29m27s')" = $'quota-exhausted\tgemini\t8967\t2h29m27s' ] || fail 'Gemini reset'
+[ "$(fm_pane_stop pi-signed $'\033[31mError: Quota reached. Please wait 09m05s\033[0m')" = $'quota-exhausted\tgemini\t545\t09m05s' ] || fail 'ANSI and leading zero'
+[ "$(fm_pane_stop pi 'Error: Quota reached. Please wait 1s')" = $'quota-exhausted\tgemini\t1\t1s' ] || fail 'seconds reset'
+for pane in 'idle prompt' 'You hit your weekly limit yesterday' 'You hit your weekly limit.' 'You hit your weekly limit!' 'Error: Quota reached. Please wait 2h29m27s.' 'Error: Quota reached. Please wait 2h29m27s!' '"You hit your weekly limit"' 'Example: You hit your weekly limit' 'Error: Quota reached. Please wait ' 'Error: Quota reached. Please wait 99m' 'Error: Quota reached. Please wait tomorrow'; do
+  ! fm_pane_stop grok "$pane" || fail "false positive: $pane"
+  ! fm_pane_stop pi "$pane" || fail "false positive: $pane"
+done
+! fm_pane_stop claude 'You hit your weekly limit' || fail 'wrong harness'
+! fm_pane_stop grok 'Error: Quota reached. Please wait 2h29m27s' || fail 'wrong provider'
+old=$(printf 'You hit your weekly limit\n'; printf 'normal line\n%.0s' {1..13})
+! fm_pane_stop grok "$old" || fail 'old scrollback'
+while IFS='|' read -r harness text label; do
+  [ "$(fm_pane_stop "$harness" "$text"$'\nDo not trust')" = "$(printf 'blocked-at-prompt\t%s\t-\t%s' "$harness" "$label")" ] || fail "missed $harness dialog"
+  ! fm_pane_stop "$harness" "$text" || fail 'lone heading matched'
+  ! fm_pane_stop "$harness" "Example: $text"$'\nDo not trust' || fail 'quoted dialog matched'
+done <<'DIALOGS'
+pi|Trust project folder?|trust
+pi-signed|Trust project folder?|trust
+DIALOGS
+while IFS='|' read -r harness text label; do
+  ! fm_pane_stop "$harness" "$text" || fail "unsupported $harness dialog matched"
+done <<'DIALOGS'
+gemini|Error: Quota reached. Please wait 1s|quota
+claude|Quick safety check: Is this a project you created or one you trust?|trust
+codex|Do you trust the contents of this directory?|trust
+codex|Hooks need review - 2 hooks are new or changed|hook-review
+agy|Do you trust the contents of this project?|trust
+gemini|Do you trust the files in this folder?|trust
+kimi|Trust this folder?|trust
+muse|Do you trust this workspace?|trust
+DIALOGS
+pass 'observed quota stops and Pi trust dialogs match conservatively'