Skip to content

fix(bin): report quota-stalled workers as agent gone in crew-state - #5952

Open
mehulbhagwani wants to merge 7 commits into
kunchenguid:mainfrom
mehulbhagwani:fm/fm-bug-quota-stall
Open

mehulbhagwani wants to merge 7 commits into
kunchenguid:mainfrom
mehulbhagwani:fm/fm-bug-quota-stall

Conversation

@mehulbhagwani

@mehulbhagwani mehulbhagwani commented Sep 27, 2026 •

Copy link
Copy Markdown

Intent

Make every open mehulbhagwani PR on kunchenguid/firstmate clean for Kun to merge.

PRs: #5954 #5953 #5952 #5951 #5919 #5709 #4932

For each: checkout that PR head branch on origin/fork, rebase onto current upstream main, fix red CI (behavior portable serial jobs), ensure no-mistakes attestation head_sha MATCHES PR head, push to the PR branch (update existing PR in place — do NOT open new PRs), confirm CI green and mergeable CLEAN.

Priority: 5954 and 5952 already green—verify still clean; then 5953, 5951 red serial jobs; 5919 conflicts; 5709 attestation mismatch; 4932 two serial fails.

Do not merge on Kun's repo (no authority). Leave PRs ready. Report final status table.

Fix all 7 fast.

This run validates and updates existing PR #5952 in place (branch fm/fm-bug-quota-stall on the mehulbhagwani fork): crew-state surfaces agent-free, quota-stalled workers. A readable endpoint holding only its shell (harness exited, for example on a provider usage-limit error) reads unknown/none as agent gone before stale busy or non-terminal status evidence can report working, while a valid ship/scout terminal done: or failed: declaration is still preserved on that shell-only path. A later commit that copied the done/failed override into every unreadable dead/missing/unreachable endpoint branch was reverted: it broke the existing contract that a missing window ignores a stale status log (CI 'dead window -> unknown'), and Greptile flagged that it made a dead secondmate look finished. A stray full duplicate of the crew-state test suite (tests/crewstate_targeted_run.sh) was removed. The branch was rebased onto main, so the push must update the existing PR branch (force with lease) and the PR body attestation must bind to the new head. Do not open a new PR.

What Changed

  • bin/fm-crew-state.sh: when a readable endpoint holds only its shell (the harness exited, for example after a provider usage-limit error), crew-state now reports unknown/none (agent gone) before stale busy or non-terminal status evidence can report working. A valid ship/scout terminal done: or failed: declaration is still honored on that shell-only path. Unreadable dead/missing/unreachable endpoints keep their existing behavior, including ignoring a stale status log when the window is missing.
  • tests/fm-crew-state.test.sh: adds cases covering the shell-only agent-gone path, including that stale busy/non-terminal status does not read as working and that terminal done:/failed: declarations are preserved.

Risk Assessment

✅ Low: The change adds one narrowly gated early check in a read-only status script. It applies only to tmux and herdr endpoints that are readable and positively classified as agent-gone, it excludes secondmates, and it keeps the terminal done/failed handling the existing path uses.

Testing

I ran the crew-state suite, which runs the real fm-crew-state.sh against fixture endpoints. The new tests for the quota-stalled shell-only pane and for terminal status on a shell-only endpoint passed. The second full run had 285 ok and no failures. One earlier run failed once on 'a killed reader reports the same unavailable reader path'. That failure is unrelated to this change and passed on rerun, so I treat it as flaky. I did not launch a live Herdr or tmux harness, so no scenario was driven against the real running product.

  • Live validation: ⚠️ inconclusive - 0 of 2 scenarios driven live against the product
Scenario Result Live Evidence
Worker whose harness died on a usage-limit error (shell-only pane, stale busy record, last status 'working') reads agent gone, not working ⏸️ untested no The prior payload did not establish a live result. It was driven only through fixture endpoints in tests/fm-crew-state.test.sh, not a live harness hitting a real quota error.
Terminal done/failed status recorded before the agent exits is preserved on a shell-only endpoint ⏸️ untested no The prior payload did not establish a live result. It was driven only through the fixture-based test 'terminal status survives a shell-only endpoint', and no real worker was run.
  • Outcome: ⚠️ 1 warning across 1 run (18m55s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

⚠️ **Test** - 1 warning
  • ⚠️ live validation verdict: inconclusive (0 of 2 scenarios were driven live against the product); untested: Worker whose harness died on a usage-limit error (shell-only pane, stale busy record, last status 'working') reads agent gone, not working, Terminal done/failed status recorded before the agent exits is preserved on a shell-only endpoint
  • Live validation: ⚠️ inconclusive - 0 of 2 scenarios driven live against the product
Scenario Result Live Evidence
Worker whose harness died on a usage-limit error (shell-only pane, stale busy record, last status 'working') reads agent gone, not working ⏸️ untested no The prior payload did not establish a live result. It was driven only through fixture endpoints in tests/fm-crew-state.test.sh, not a live harness hitting a real quota error.
Terminal done/failed status recorded before the agent exits is preserved on a shell-only endpoint ⏸️ untested no The prior payload did not establish a live result. It was driven only through the fixture-based test 'terminal status survives a shell-only endpoint', and no real worker was run.
  • bash tests/fm-crew-state.test.sh (twice, fixture-based)
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

@greptile-apps

greptile-apps Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[Medium risk] Changes how worker state is classified when the agent process exits.

The current PR appears safe to merge based on the reviewed changes.

Reviews (5) · Last reviewed commit: "test: drop stray copy of the crew-state ..."

Comment thread bin/fm-crew-state.sh
@mehulbhagwani
mehulbhagwani force-pushed the fm/fm-bug-quota-stall branch 2 times, most recently from 132c3fb to e206c6e Compare September 29, 2026 11:24
@mehulbhagwani mehulbhagwani changed the title fix(bin): classify agent-free quota-stalled workers as gone fix(bin): surface agent-free Pi workers instead of reporting stale working state Sep 29, 2026
@mehulbhagwani
mehulbhagwani force-pushed the fm/fm-bug-quota-stall branch from e206c6e to 51521cd Compare October 1, 2026 11:47
Comment thread bin/fm-crew-state.sh Outdated
Comment on lines +1301 to +1309
tmux:missing|herdr:missing)
# A valid ship/scout terminal declaration outranks the death verdict: the
# harness is gone because the turn finished, not because state is unknown.
case "$LOG_VERB" in
"done")
if [ "$KIND" = ship ]; then
emit_ship_status_done
fi
emit "done" status-log "$(status_line_note "$LOG_LINE")"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Dead Secondmate Appears Finished

When a local secondmate’s endpoint is missing or dead and its status log ends in done: or failed:, this branch reports the terminal status and exits before reporting the lost endpoint. The readable shell-only check excludes secondmates, but this new override does not. Consumers of crew state therefore miss the secondmate’s liveness failure.

Knowledge Base Used: Disable endless escalation for dead agent records

mehulbhagwani and others added 7 commits October 2, 2026 18:07
A shell-only pane with no attributed run was reporting unknown on the
dead-endpoint path before a valid ship/scout done: or failed: status could
be preserved. Check the status log first on every death-class verdict.
This reverts commit 51521cd526d6cd38a27d64da0178dee38b476a86.
A documentation pass committed a full duplicate of tests/fm-crew-state.test.sh
as tests/crewstate_targeted_run.sh; it is not a test entry point and only
duplicates the real suite.
@mehulbhagwani
mehulbhagwani force-pushed the fm/fm-bug-quota-stall branch from 51521cd to 26f878d Compare October 2, 2026 12:58
@mehulbhagwani mehulbhagwani changed the title fix(bin): surface agent-free Pi workers instead of reporting stale working state fix(bin): report quota-stalled workers as agent gone in crew-state Oct 2, 2026
@mehulbhagwani

Copy link
Copy Markdown
Author

The one red job here (Behavior portable serial 4) is tests/fm-remote-delta-read.test.sh, the timing-dependent test from #6363 tracked in #6404; this PR does not touch it. Could you re-run that failed job? I don't have rerun rights on this repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant