Skip to content

fix(bin): classify provider quota walls as a distinct quota state - #4996

Open
keenvc wants to merge 1 commit into
kunchenguid:mainfrom
keenvc:fm/fm-quota-wall-reads-as-busy
Open

keenvc wants to merge 1 commit into
kunchenguid:mainfrom
keenvc:fm/fm-quota-wall-reads-as-busy

Conversation

@keenvc

@keenvc keenvc commented Sep 20, 2026

Copy link
Copy Markdown

What

A worker parked on a provider quota wall must not report itself as working.
The measured incident had every worker on one provider stalled at the same weekly limit while bin/fm-crew-state.sh reported each one state: working - source: pane - harness busy.

  • bin/fm-busy-lib.sh gains a rendered quota-wall check as the one cross-harness override that DOWNGRADES a busy semantic verdict to quota; a missing or misread signal leaves the semantic verdict intact.
  • The signal is built from two independent wall-phrase families - a spent usage/rate/quota limit and a scheduled retry/reset - within the last few non-empty lines, so no single vendor string is load-bearing and ordinary worker prose (quota, retry) does not match.
  • bin/fm-crew-state.sh surfaces it as its own state: quota (source pane) with a preserve-and-replace detail, rather than collapsing it into working, paused, or stale.
  • .agents/skills/stuck-crewmate-recovery/SKILL.md states that a quota-parked worker is neither a wedge nor a declared external wait and cannot be relaunched in place.

Why its own state

The recovery differs.
A 429/quota modal leaves the composer unreadable and the control plane correctly refuses to type into it, so an in-place relaunch cannot fix it and a fresh worker on the same provider would hit the same wall; the work is preserved and brought back under a new task id chosen for a provider with headroom.

Tests

Portable regression in tests/fm-crew-state.test.sh, over synthetic pane transcripts with no real harness:

  • five provider wall shapes (opencode-go, Claude, Gemini, Codex, generic 429) all read quota;
  • the divergence cases assert the two-signal rule cannot go quietly vacuous: one family alone, or ordinary worker prose such as adding a usage-limit retry path, still reads working;
  • crew_is_provably_working surfaces a quota-parked worker rather than absorbing it.

Live guard tests/fm-quota-wall-live-e2e.test.sh in the live-harness-optin family (token-free, default-on):
it drives the real installed OpenCode TUI against a local 429 stub provider so OpenCode paints its own retry modal, then proves the same task reads working before the wall and quota after it.
Dated result recorded in docs/verification/runtime-backends.md.

Verification

  • tests/fm-crew-state.test.sh - all pass.
  • tests/fm-quota-wall-live-e2e.test.sh - OpenCode 1.18.31 real 429 quota retry modal classifies as quota, not working.
  • tests/fm-busy-state.test.sh, tests/fm-busy-adapter-wiring.test.sh, tests/fm-watch-triage.test.sh, tests/fm-inactive-reconcile.test.sh, tests/fm-fleet-snapshot-view.test.sh, tests/fm-control.test.sh - all pass.
  • bin/fm-lint.sh clean; bin/fm-doc-audience-check.sh clean.

A worker parked on a provider quota wall kept a live, painting harness
while its turn could not advance, so every one of them read as working
from its semantic busy record. The measured fleet incident had all of
one provider's workers stalled at the same weekly limit while
supervision saw a healthy fleet.

Recognize the wall from the pane text the busy reader already inspects.
The signal is built from two independent rendered families - a
wall-shaped limit phrase and a scheduled retry/reset phrase - within the
last few non-empty lines, so no single vendor string is load-bearing and
ordinary worker prose does not match. A busy verdict over that wall
reports `quota` instead of busy.

fm-crew-state.sh surfaces it as its own `state: quota` rather than
collapsing it into working or a declared pause, because a quota-killed
worker cannot be relaunched in place; the recovery skill now states that
preserve-and-replace under a new id is the path.

The portable regression pins the logic and its divergence cases over
synthetic transcripts. The live guard drives the real installed OpenCode
TUI against a local 429 stub so its own retry modal renders with no
model tokens spent, and proves the same task reads working before the
wall and quota after it.
@kunchenguid

Copy link
Copy Markdown
Owner

Speaking as Kun's firstmate:

First look on HEAD c6d1147fcc0898f35c48ee6f23a931a5db729c5e vs main dd9b2ef21fe6c06846ae74d9070ac2da971ae8e7. Whole thread read (body; no prior comments). Attestation MISMATCH (no no-mistakes-pipeline-attestation block). ~400 LOC: busy-lib quota wall, crew-state state: quota, recovery skill, tests + live e2e.

Contract-class: new-default (own verdict; do not trust PR "fix" framing). Main crew-state vocabulary is working|parked|done|blocked|paused|failed|unknown and busy-lib is busy|idle|unknown|dead only — tip adds a new default-on state: quota / quota quota-wall classification outcome the unconfigured product did not previously emit, plus stuck-crewmate-recovery guidance for preserve-and-replace. Honesty / measured-incident framing does not turn a new always-on state into restore (FM-LEARN-4627 / VISION skill). Two-signal rendered override is carefully fail-closed (missing signal leaves busy intact), but the new vocabulary is still new-default.

CI/NM: tip was action_required. Fork workflows approved this pass after diff review (no workflow file edits; pane-text regex only; no secrets): CI 35482640403, Require no-mistakes 35482640479 — queued. MERGEABLE/UNSTABLE. Missing attestation will keep NM red.

VISION.md (each rule)

  • One captain, one interface: aligns in motive (stop false working) but new surface — quota state reaches fleet views/recovery by default.
  • Authority is explicit: aligns — no new merge/autonomy grant; escalate to captain when blocking.
  • Scripts own the mechanics: aligns.
  • A restart is a non-event: aligns / cannot-tell on fleet durability of the new state alone.
  • Delegation with a spine: aligns (recovery skill documents preserve-and-replace).
  • The fleet outlives any vendor: aligns — two-family phrases, not one vendor string.
  • Scope: aligns (busy/crew-state classification).

Decision: waiting-author — add matching attestation for this HEAD. Not an auto-merge candidate (new-default). When otherwise green+MATCH, Firstmate flag for captain decision — not waiting on author for the class call. No Firstmate flag yet (not otherwise-ready). Security: clean. No Closes/Fixes claims verified.

keenvc added a commit to keenvc/firstmate that referenced this pull request Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants