kunchenguid · keenvc · Sep 20, 2026
diff --git a/.agents/skills/stuck-crewmate-recovery/SKILL.md b/.agents/skills/stuck-crewmate-recovery/SKILL.md
@@ -59,6 +59,13 @@ Never restart, stop, or update the shared daemon on a crewmate's claim.
 It is one instance serving every lane and home, so a restart kills other lanes' in-flight runs.
 Only positive socket refusal or absence is a daemon-down finding; escalate that finding, or a failed run record that names a daemon error, to the captain.
 
+## A worker parked on a provider quota wall
+
+`bin/fm-crew-state.sh` reports `state: quota` when a live harness is stalled on a provider usage-limit retry modal instead of advancing: the process is alive and painting, but the submitted turn cannot run.
+Treat it as neither a wedge nor a declared external wait.
+The retry modal leaves the composer unreadable, so an in-place `fm-control.sh <task-id> relaunch` cannot fix it and a fresh worker on the same provider would hit the same wall.
+Preserve the worktree and its unlanded work, and bring the work back under a new task id chosen for a provider with headroom rather than relaunching in place; a provider limit is not something the fleet can clear, so escalate it to the captain when it blocks delivery.
+
 ## Live-endpoint escalation
 
 Escalate in order:

diff --git a/bin/fm-busy-lib.sh b/bin/fm-busy-lib.sh
@@ -43,10 +43,10 @@
 #   fm-recovery      a documented recovery reset after relaunch
 # Classifier-only sources (never written into a record):
 #   endpoint-gone, herdr-native, grok-regex, rovo-regex, agy-regex, muse-session-log,
-#   cursor-transcript, missing, malformed, gen-mismatch, source-mismatch,
+#   cursor-transcript, quota-wall, missing, malformed, gen-mismatch, source-mismatch,
 #   kimi-unverified, codex-unverified, capture-failed, no-target
 #
-# Classification (fm_busy_classify): busy | idle | unknown | dead, always
+# Classification (fm_busy_classify): busy | idle | unknown | dead | quota, always
 # with the producing source as the second token. Precedence:
 #   1. dead endpoint (fm_busy_classify_live only) -> dead endpoint-gone
 #   2. standalone Kimi before verification       -> unknown kimi-unverified
@@ -57,12 +57,18 @@
 #      Grok/Rovo/AGY temporary regex fallbacks classify a grok, rovo, or agy
 #      task from its rendered tail, then unknown missing
 #   5. malformed, stale, or untrusted records -> unknown, never a fallback
-# Grok, Rovo, and AGY are the ONLY rendered-text classifications that survive the
-# redesign, because none of their structured lifecycles was credited-live-verified
-# in the approved audit (Rovo's clean ACP stopReason lives outside the TUI
-# path firstmate drives, see references/harness/rovo.md; agy 1.2.0 exposes no
-# hook surface at all, see references/harness/agy.md); each is scoped to
-# its own harness= and can never classify another adapter. The delivery
+#   6. any busy verdict over a rendered provider quota wall -> quota quota-wall
+#      (the wall section below owns the two-signal rule)
+# Grok, Rovo, and AGY are the ONLY per-harness rendered-text sources that
+# survive the redesign, because none of their structured lifecycles was
+# credited-live-verified in the approved audit (Rovo's clean ACP stopReason
+# lives outside the TUI path firstmate drives, see references/harness/rovo.md;
+# agy 1.2.0 exposes no hook surface at all, see references/harness/agy.md);
+# each is scoped to its own harness= and can never classify another adapter.
+# The quota wall above is the one cross-harness rendered override: it never
+# invents a verdict from a rendered surface alone, it only DOWNGRADES an
+# already-busy semantic verdict to quota, so a missing or misread signal leaves
+# the semantic verdict intact. The delivery
 # guards in bin/fm-composer-lib.sh match rendered footers for submit
 # acknowledgement and away-mode supervisor injection only; neither is a
 # recorded worker state source.
@@ -867,13 +873,60 @@ fm_busy_agy_tail_busy() {
     | grep -qiE 'esc[[:space:]]+to[[:space:]]+cancel'
 }
 
+# ---------------------------------------------------------------------------
+# Provider quota / usage wall
+#
+# A worker parked on a provider quota wall is the one case where a live,
+# painting harness is NOT advancing: the retry modal keeps the process and the
+# TUI busy while the submitted turn cannot run. The measured fleet incident
+# (workers on one provider, all stopped at the same weekly limit) had every one
+# of them classified busy from its semantic record and reported working, so the
+# wall must override the semantic busy verdict.
+#
+# The signal is rendered and provider-specific, so it is deliberately built
+# from TWO independent wall-phrase families and requires both:
+#   limit - the wall names a spent usage/rate/quota limit
+#   wait  - the wall names a scheduled retry or reset/backoff
+# A single vendor string therefore cannot carry the verdict, and the phrases are
+# wall-shaped rather than bare words (`quota`, `retry`) so a worker writing or
+# discussing a quota-retry feature does not match. The check is scoped to the
+# last few non-empty lines, where a blocking modal renders in the harness
+# chrome the composer otherwise occupies, so scrollback output is never
+# mistaken for a wall.
+FM_BUSY_QUOTA_LIMIT_RE='(usage|rate)[ -]?limit|quota (exceed|exhaust|reached|limit)|exhausted (your )?(capacity|quota)|too many requests|out of (credits|quota)'
+FM_BUSY_QUOTA_WAIT_RE='retry(ing)? (in|after)|will reset|resets? (in|at|after)|try again|attempt #[0-9]|get more access|upgrade your plan'
+
+# fm_busy_quota_tail_wall: consumes a rendered tail on stdin; 0 when the tail
+# shows a provider quota wall (both families present within the bounded tail).
+fm_busy_quota_tail_wall() {
+  local tail
+  tail=$(grep -v '^[[:space:]]*$' | tail -6)
+  [ -n "$tail" ] || return 1
+  printf '%s\n' "$tail" | grep -qiE "$FM_BUSY_QUOTA_LIMIT_RE" || return 1
+  printf '%s\n' "$tail" | grep -qiE "$FM_BUSY_QUOTA_WAIT_RE" || return 1
+  return 0
+}
+
+# fm_busy_quota_wall_verdict: 0 when a busy pane's captured tail shows a quota
+# wall. Captures the pane itself when the caller passed no tail, bounded by
+# fm_backend_capture; every other outcome is a no, never a false wall.
+fm_busy_quota_wall_verdict() {  # <backend> <target> [tail]
+  local backend=$1 target=$2 tail=${3-}
+  if [ -z "$tail" ]; then
+    command -v fm_backend_capture >/dev/null 2>&1 || return 1
+    tail=$(fm_backend_capture "$backend" "$target" 40 2>/dev/null) || return 1
+  fi
+  [ -n "$tail" ] || return 1
+  printf '%s' "$tail" | fm_busy_quota_tail_wall
+}
+
 # fm_busy_classify: semantic classification for a task whose endpoint the
 # caller has already established as present. Prints "<verdict> <source>":
-# busy|idle|unknown plus the producing source (see header). Never probes
+# busy|idle|unknown|quota plus the producing source (see header). Never probes
 # process state. <tail40> is optional pre-captured plain output used only by
 # the grok, rovo, and agy arms; when absent each captures through
 # fm_backend_capture if available, else reports unknown capture-failed.
-fm_busy_classify() {  # <backend> <target> <harness> <id> <state-dir> [tail40]
+fm_busy_classify_raw() {  # <backend> <target> <harness> <id> <state-dir> [tail40]
   local backend=$1 target=$2 harness=$3 id=$4 state=$5 tail40=${6-}
   local out rc r_state r_source native log
   case "$harness" in
@@ -1020,6 +1073,21 @@ fm_busy_classify() {  # <backend> <target> <harness> <id> <state-dir> [tail40]
   printf 'unknown missing'
 }
 
+# fm_busy_classify (public): fm_busy_classify_raw plus the one rendered
+# override - a semantic busy verdict over a provider quota retry modal is not
+# advancing, so it reports `quota` instead of busy. Every other verdict passes
+# through unchanged.
+fm_busy_classify() {  # <backend> <target> <harness> <id> <state-dir> [tail40]
+  local backend=$1 target=$2 verdict tail40=${6-}
+  verdict=$(fm_busy_classify_raw "$@")
+  if [ "${verdict%% *}" = busy ] \
+    && fm_busy_quota_wall_verdict "$backend" "$target" "$tail40"; then
+    printf 'quota quota-wall'
+    return 0
+  fi
+  printf '%s' "$verdict"
+}
+
 # fm_busy_classify_live: fm_busy_classify behind the one process-level
 # override - a gone endpoint is dead, never busy. Requires fm-backend.sh to
 # be sourced for fm_backend_target_exists.
@@ -1055,9 +1123,9 @@ fm_busy_classify_meta() {  # <meta-file> <id> <state-dir> [tail40]
 
 # fm_busy_is_busy: boolean view for callers that only gate on provable
 # activity. 0 iff the classification verdict is exactly busy; idle, unknown,
-# and dead all return 1, so an unknown can never be silently promoted to
-# either boolean pole - callers that must distinguish idle from unknown read
-# the full classification instead.
+# dead, and quota all return 1, so an unknown or a quota-parked worker can
+# never be silently promoted to working - callers that must distinguish idle
+# from unknown or quota read the full classification instead.
 fm_busy_is_busy() {  # <backend> <target> <harness> <id> <state-dir> [tail40]
   local verdict
   verdict=$(fm_busy_classify "$@")

diff --git a/bin/fm-classify-lib.sh b/bin/fm-classify-lib.sh
@@ -1945,6 +1945,9 @@ crew_absorb_class() {  # <id>
     src=${line#*source: }; src=${src%% *}
     case "$src" in run-step|pane) printf 'working'; return ;; esac
   fi
+  # `quota` (a live harness parked on a provider quota wall) is deliberately
+  # NOT absorbed: the worker is neither advancing nor a declared external wait,
+  # so it surfaces for firstmate to preserve and replace.
   printf 'none'
 }
 

diff --git a/bin/fm-crew-state.sh b/bin/fm-crew-state.sh
@@ -22,7 +22,7 @@
 # Output is one stable, parseable, token-tight line firstmate can read every
 # heartbeat:
 #
-#   state: <working|parked|done|blocked|paused|failed|unknown> · source: <run-step|pane|status-log|remote-endpoint|none> · <detail>
+#   state: <working|quota|parked|done|blocked|paused|failed|unknown> · source: <run-step|pane|status-log|remote-endpoint|none> · <detail>
 #
 # Logic, in order:
 #   1. Resolve worktree + backend target + kind from state/<id>.meta. A meta
@@ -90,7 +90,11 @@
 #      proven historical head, or kind=scout): fall back to the recorded
 #      backend's pane busy state, then the resolved status declaration
 #      when its verb maps to a recognized run-state. Decision-only events such as
-#      `resolved` never become current state or detail.
+#      `resolved` never become current state or detail. A busy pane whose text
+#      shows a provider quota wall reports `quota` instead of working
+#      (bin/fm-busy-lib.sh owns the wall signal): the harness is alive but the
+#      agent is not advancing, so recovery is preserve-and-replace under a new
+#      id, never an in-place relaunch into the same unreadable retry modal.
 #   5. Missing meta or torn-down worktree: report unknown · none. If no run is
 #      attributed to this crew, a dead endpoint also reports unknown · none rather
 #      than trusting a stale status log. On tmux and herdr, which own a
@@ -253,12 +257,14 @@ pane_readable() {  # <target>
   esac
 }
 # crew_busy_verdict: the crew's semantic busy state from the one contract
-# owner (bin/fm-busy-lib.sh), as "<busy|idle|unknown> <source>". A converted
-# adapter answers from its own lifecycle record; Grok answers from its
-# isolated rendered-tail fallback; a herdr crew's native `busy` is accepted
+# owner (bin/fm-busy-lib.sh), as "<busy|idle|unknown|quota> <source>". A
+# converted adapter answers from its own lifecycle record; Grok answers from
+# its isolated rendered-tail fallback; a herdr crew's native `busy` is accepted
 # when no record exists, but its native `idle` is NOT, because agent.get
 # reports generation state (idle while a crew blocks on its own long-running
-# foreground tool call) rather than turn state.
+# foreground tool call) rather than turn state. A `quota` verdict is a busy
+# record reclassified by the rendered provider-wall signal, which the contract
+# owner captures itself when the caller passes no tail.
 crew_busy_verdict() {  # <target>
   local tail40=''
   case "$HARNESS" in
@@ -1010,13 +1016,15 @@ fi
 
 # Secondmates idle on their own watcher (idle pane = healthy), so the busy
 # state is not meaningful for them; read their state from the status log only.
-# Only an exact busy verdict reports working here, and only an exact idle
-# verdict permits the status-log fallback below. Missing, malformed, stale, or
-# unverified semantic state remains unknown.
+# Only an exact busy verdict reports working here, an exact quota verdict
+# reports quota (a live harness stalled on a provider wall, not advancing), and
+# only an exact idle verdict permits the status-log fallback below. Missing,
+# malformed, stale, or unverified semantic state remains unknown.
 if [ "$KIND" != secondmate ]; then
   BUSY_VERDICT=$(crew_busy_verdict "$BACKEND_TARGET")
   case "${BUSY_VERDICT%% *}" in
     busy) emit working pane "harness busy (${BUSY_VERDICT#* })" ;;
+    quota) emit quota pane "provider quota wall: not advancing (preserve and replace under a new id, not relaunch)" ;;
     idle) ;;
     *) emit unknown pane "harness state unavailable ($BUSY_VERDICT)" ;;
   esac

diff --git a/bin/fm-test-run.sh b/bin/fm-test-run.sh
@@ -359,6 +359,7 @@ family_for_basename() {
     fm-pr-state-live-e2e.test.sh|\
     fm-sessionstart-hook-live-e2e.test.sh|fm-sessionstart-instruction-refresh-live-e2e.test.sh|\
     fm-quota-array-dispatch-live-e2e.test.sh|fm-send-secondmate-marker-herdr-e2e.test.sh|\
+    fm-quota-wall-live-e2e.test.sh|\
     fm-send-inbox-doorbell-live-e2e.test.sh|\
     fm-calm-claude-mod-plugin.test.sh|fm-calm-claude-mod-live-e2e.test.sh|\
     fm-herdr-submit-confirm-live-e2e.test.sh)

diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md
@@ -2095,3 +2095,30 @@ A throwaway scout was spawned through `bin/fm-spawn.sh --scout --harness omp --m
 6. `bin/fm-control.sh <id> exit` stopped the agent and `bin/fm-teardown.sh` returned the worktree and closed the item.
 
 `FM_OMP_LIVE_E2E=1 tests/fm-omp-primary-live-e2e.test.sh` refreshes the primary evidence; the worker path above is refreshed by repeating the scout dispatch after any omp upgrade.
+
+## Provider quota-wall classification
+
+A worker parked on a provider quota wall must not read as working: the harness is alive and painting a retry modal while the submitted turn cannot advance.
+`bin/fm-busy-lib.sh` classifies that rendered wall as `quota` over an otherwise-busy task, and `bin/fm-crew-state.sh` surfaces it as `state: quota` (source `pane`) instead of `working`, so supervision sees a stalled worker rather than a healthy one.
+The signal is built from two independent rendered families - a limit phrase and a retry/reset phrase - within the last few non-empty lines, so no single vendor string is load-bearing and ordinary worker output does not match.
+
+Verified on 2026-09-20 with opencode 1.18.31 on Linux, driving the real installed OpenCode TUI against a local 429 stub provider so its own retry modal renders with no model tokens spent:
+
+```sh
+bin/fm-test-run.sh tests/fm-quota-wall-live-e2e.test.sh
+```
+
+Observed output:
+
+```text
+ok - a busy OpenCode worker without a rendered wall reads working
+ok - OpenCode 1.18.31 real 429 quota retry modal classifies as quota, not working
+```
+
+The real modal OpenCode painted for the stub's quota error, captured from the pane:
+
+```text
+⬝⬝⬝⬝■■■■ weekly usage limit reached. It will reset in 1 day 14 hours [retrying attempt #1]            esc interrupt
+```
+
+`tests/fm-crew-state.test.sh` pins the logic portably over a synthetic pane transcript, including the divergence cases where only one family, or ordinary worker prose, never reads `quota`.