Skip to content

fix: surface idle quota stops and Pi trust prompts promptly - #5504

Closed
slee029 wants to merge 7 commits into
kunchenguid:mainfrom
slee029:fm/fm-quota-stall-detect
Closed

slee029 wants to merge 7 commits into
kunchenguid:mainfrom
slee029:fm/fm-quota-stall-detect

Conversation

@slee029

@slee029 slee029 commented Sep 24, 2026 •

Copy link
Copy Markdown

Intent

Shouldn't we also have mechanisms that say if something was mid-task but ran out of usage and it's idle that we have some kind of mechanism that's checking in that says, hey, redo that task with a separate model rather than letting that thing sit idle? I think that's what happened when we were using Muse: something sat idle for a really long time. Does Firstmate account for how to fix that in a much quicker, more accurate way?
Context from 2026-09-18: a Grok worker hit "You hit your weekly limit" and a Pi Gemini worker hit "Error: Quota reached. Please wait 2h29m27s". Both sat idle. The watcher only reported generic stale or possible-wedge wakes, with escalating deep-inspection demands, and firstmate had to read the pane, recognise the quota stop and relaunch by hand.

Earlier, Firstmate repair workers were stuck at Pi's folder-trust prompt. How do we prevent that from happening again, whether it's you not knowing about it, or that we avoid getting such a prompt again?

What Changed

  • Recognize rendered Grok and Pi quota stops and Pi folder-trust prompts on idle worker panes, surfacing specific stale wakes instead of generic wedge triage.
  • Deduplicate unchanged stop wakes and report quota reset bounds only when a delay is shown; the watcher does not accept prompts or relaunch workers.
  • Document manual recovery through an eligible model and harness trust handling, with regression coverage for detection and watcher triage.

Risk Assessment

✅ Low: The change is narrowly scoped to recognizing observed idle stops and emitting deduplicated wakes; no material source-backed issue was found.

Testing

The focused classifier test passed. The watcher test reached stale-pane checks but timed out before its quota and trust scenarios; no live Herdr scenario or reviewer-visible artifact was produced. Overall validation is inconclusive.

  • Live validation: ⚠️ inconclusive - 0 of 5 scenarios driven live against the product
Scenario Result Live Evidence
An idle Grok worker hitting its weekly limit promptly wakes Firstmate without an invented reset time ⏸️ untested no No isolated Grok account at its weekly limit was provided. Provide that account and authorization to run it in a named Herdr lab.
An idle Pi Gemini worker hitting quota wakes Firstmate with its raw delay, UTC observation time, and a no-later-than reset bound ⏸️ untested no No isolated Pi Gemini account exhibiting quota exhaustion was provided. Provide one and authorization to run it in a named Herdr lab.
A Pi worker stopped at the project-folder trust dialog receives a prompt-specific wake ⏸️ untested no A provisioned named Herdr lab with an interactive Pi launch and authorization to exercise its trust prompt is needed.
Quoted or unsupported stop text does not trigger quota or trust recovery ⏸️ untested no A provisioned named Herdr lab and an authorized worker to render the adversarial text are needed.
Firstmate resumes a quota-stopped task using a separate model rather than relaunching it automatically ⏸️ untested no An exhausted isolated provider account, authenticated alternate model, and authorization to run recovery are needed.
  • Outcome: ⚠️ 1 warning across 1 run (13m53s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed (3) ✅
  • ⚠️ bin/fm-watch.sh:433 - The UTC reset is calculated from detection time, not the time Pi rendered the delay. If the watcher first sees a Please wait 2h29m27s pane an hour after the error—for example, after watcher downtime—the wake reports a reset an hour late without erroring. An accurate absolute time needs evidence of when the delay was emitted; deciding how to obtain that or whether to withhold the estimate requires authorization, rather than another calculation from detection time.

🔧 Fix applied.
1 warning still open:

  • ⚠️ bin/fm-pane-stop-lib.sh:32 - A Pi worker can print the standalone Trust project folder? line while reading docs/verification/runtime-backends.md, then become idle with that output still in its pane. This matcher reports blocked-at-prompt even though no dialog is open. The existing launch-prompt classifier requires both the heading and the dialog’s Do not trust option; use that paired evidence here too.

🔧 Fix applied.
1 warning still open:

  • ⚠️ bin/fm-pane-stop-lib.sh:33 - Simplification: both quota patterns accept optional . or ! suffixes, although the observed stops and approved narrowed scope require neither variant. Remove the optional punctuation from both patterns so recognition stays limited to the supported rendered stops.

🔧 Fix applied.
✅ Re-checked - no issues remain.

⚠️ **Test** - 1 warning
  • ⚠️ live validation verdict: inconclusive (0 of 5 scenarios were driven live against the product); untested: An idle Grok worker hitting its weekly limit promptly wakes Firstmate without an invented reset time, An idle Pi Gemini worker hitting quota wakes Firstmate with its raw delay, UTC observation time, and a no-later-than reset bound, A Pi worker stopped at the project-folder trust dialog receives a prompt-specific wake, Quoted or unsupported stop text does not trigger quota or trust recovery, Firstmate resumes a quota-stopped task using a separate model rather than relaunching it automatically
  • Live validation: ⚠️ inconclusive - 0 of 5 scenarios driven live against the product
Scenario Result Live Evidence
An idle Grok worker hitting its weekly limit promptly wakes Firstmate without an invented reset time ⏸️ untested no No isolated Grok account at its weekly limit was provided. Provide that account and authorization to run it in a named Herdr lab.
An idle Pi Gemini worker hitting quota wakes Firstmate with its raw delay, UTC observation time, and a no-later-than reset bound ⏸️ untested no No isolated Pi Gemini account exhibiting quota exhaustion was provided. Provide one and authorization to run it in a named Herdr lab.
A Pi worker stopped at the project-folder trust dialog receives a prompt-specific wake ⏸️ untested no A provisioned named Herdr lab with an interactive Pi launch and authorization to exercise its trust prompt is needed.
Quoted or unsupported stop text does not trigger quota or trust recovery ⏸️ untested no A provisioned named Herdr lab and an authorized worker to render the adversarial text are needed.
Firstmate resumes a quota-stopped task using a separate model rather than relaunching it automatically ⏸️ untested no An exhausted isolated provider account, authenticated alternate model, and authorization to run recovery are needed.
  • tests/fm-pane-stop.test.sh
  • tests/fm-watch-triage.test.sh (twice; timed out after 180s and 600s)
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

Test coverage note

tests/fm-watch-triage.test.sh timed out both locally and in the no-mistakes pipeline before reaching its quota-exhausted and trust-prompt scenarios, so those watcher scenarios are proven only by upstream CI.
tests/fm-pane-stop.test.sh (the focused stop classifier, including negative cases) passed.
No live end-to-end scenario was run: it would need isolated provider accounts that are actually quota-exhausted.

@slee029

slee029 commented Sep 24, 2026

Copy link
Copy Markdown
Author

The CI and Require no-mistakes workflow runs for this fork PR are waiting at action_required. Could a maintainer please approve the workflow runs so CI can execute? Thanks.

@kunchenguid

Copy link
Copy Markdown
Owner

Speaking as Kun's firstmate: stamped waiting-ci.

HEAD d9996fecb6e8b494d2f8b05c123f43838db1e9af. Attestation MATCH (body binds tip). MERGEABLE/UNSTABLE. No Closes/Fixes. Brand new (opened 2026-09-24). Author slee029 not blocked. Soft overlap only with same-author #5493 (inbox note wakes; different path).

First-time fork CI: after full diff review I approved workflow runs 35984609348 (CI, now queued), 35984609342 (Require no-mistakes, in_progress), 35984766543 (Require no-mistakes, in_progress). workflow-zero (no .github/workflows/*). Security clean: local pane matchers only, no network, watcher never accepts prompts or relaunches.

contract-class: new-default (own verdict; tip vs main tip 9284978fe931; ignore title "fix").

  • Main has no fm-pane-stop-lib.sh, no pane_stop_stale_check, no quota-exhausted / blocked-at-prompt wake kinds, no .pane-stop-* markers. Idle quota/trust panes today fall through ordinary stale → wedge.
  • Tip adds always-on new wake classes on the unconfigured watcher poll path (bypass wedge after two unchanged hashes), plus always-on .pane-stop-<key> dedup state (ungated). Recovery skill documents manual relaunch / harness trust — watcher still report-only.
  • Per FM-LEARN-4627 always-on wake-class test → new-default, not restore. Peace-of-mind / "honesty already promised" does not convert it. Will not auto-merge even when green.

VISION (per-rule):

  1. One captain, one interface — aligns: specific stop wakes surface honest outcomes; machinery stays below deck (pane_stop_stale_check → wake only; skill owns recovery).
  2. Authority explicit — aligns: watcher never answers prompts or relaunches; skill forbids manufacturing consent / silent model switch (stuck-crewmate-recovery/SKILL.md, header contract).
  3. Scripts/judgment — aligns: exact rendered-stop matchers in fm-pane-stop-lib.sh; recovery judgment stays in the skill.
  4. Restart non-event — aligns: durable .pane-stop-* + wake-queue; AGENTS.md documents the marker.
  5. Delegation with spine — aligns: explicit wake contract + classifier/watcher regression tests (fm-pane-stop.test.sh, test_quota_stale_surfaced). Live E2E left inconclusive by author (no exhausted accounts) — not a VISION miss.
  6. Fleet outlives vendor — aligns: harness-scoped patterns (grok / pi|pi-signed only); unknown wording stays on ordinary stale; negatives cover wrong harness/quoted text.
  7. Scope — aligns: command-layer watcher + recovery skill; no forge/CI/merge-policy growth.
    Closing: aligns with peace-of-mind, but ships as new-default wake surface.

When CI + NM are green: still no auto-merge (new-default). Firstmate-flag deferred until otherwise ready except the captain decision. Not waiting on the author for code right now — waiting on CI.

@kunchenguid

Copy link
Copy Markdown
Owner

Speaking as Kun's firstmate: CI moved this from waiting-ci to waiting-author.

HEAD d9996fecb6e8b494d2f8b05c123f43838db1e9af still MATCH. Require no-mistakes passed. Lint 2 failed on tests/fm-pane-stop.test.sh line 4 (ShellCheck SC1091: not following ./lib.sh). Other behavior jobs on run 35984609348 passed.

Contract-class remains new-default (always-on quota-exhausted / blocked-at-prompt wake classes + ungated .pane-stop-* on the unconfigured watcher path). Will not auto-merge even after green; Firstmate captain card deferred until otherwise ready.

Please fix the lint failure (typical pattern in this repo: shellcheck source directive or source path used by sibling tests) and push. Waiting on you, not the captain.

@slee029

slee029 commented Sep 27, 2026

Copy link
Copy Markdown
Author

Closing this. It adds always-on wake classes on the default watcher path, and turning it into an opt-in flag is not worth the extra surface since nothing on our side depends on it. Thanks for the review.

@slee029 slee029 closed this Sep 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants