fix(bin): sync upstream captain-hold, codex composer, and empty-steer fixes - #1
Merged
Merged
Conversation
…d stop cleanup dropping accents from a held body (kunchenguid#4471) * fix(bin): let captain holds work on hosts with an older JSON::PP Holding a task for the captain, and the cleanup that keeps a captain-held row open, both fail outright on any host whose JSON::PP defaults allow_nonref off - 2.27202 on a Linux desk is one. Both read a task's body back with `decode_json`, but tasks-axi shows a scalar field as a JSON-encoded bare string, and an older library rejects that whole value with "must be object or array". The consequence is fleet-wide on such a host, not one broken command: a worker there cannot formally record a decision for the captain at all. It can only mention the decision in passing in a status line, where it can be missed - which is how a real decision goes unrecorded. The hold reports that the task lost its hold-set stamp; the cleanup cannot return the row to Queued. Both call sites now ask for allow_nonref explicitly rather than inheriting whatever the installed library defaults to. The second one is worth naming: its `/\A"/` guard reads as deliberate, but a leading quote is exactly the bare-string case that fails, so the guard selects for the failing input rather than protecting against it. The regression case forces the older default back off for every perl the commands spawn, then drives both paths - holding a task that carries a body, and tearing down a captain-held row whose deliverable must still be appended. It also probes that the simulation genuinely rejects a bare scalar, so the case cannot pass vacuously on a lenient host. Each half was verified failing on its own unfixed call site with that site's real error message. Suites: fm-captain-hold-lifecycle 51 cases, fm-backlog-atomicity 99 cases, 0 failures. Verification limit: the mechanism is reproduced and tested, but neither fix is verified against a real JSON::PP 2.27202 host, because none is in the loop. This laptop runs 4.06, where the bug does not manifest. `bin/fm-procevent-lavish.sh:471` was checked and left alone - it matches a brace-delimited object before decoding, so allow_nonref never applies. * fix(bin): stop cleanup silently dropping accented characters from a held body Cleanup rewrites a captain-held row's body to append the finished work's deliverable, and the decoder it reads that body with printed decoded characters to a stream with no `:raw` layer. A character at or below U+00FF then came out as one latin-1 byte instead of two UTF-8 ones, so a body reading "café" lost the accent. `fm_backlog_retain` writes that body straight back through `--body-file`, and nothing reported an error - the character was simply gone from a row still waiting on the captain. The decoder now writes bytes, the same `binmode STDOUT, ":raw"` plus `utf8::encode` that the sibling decoder in `bin/fm-captain-hold.sh` already used. Review of the parent commit found this on one of the lines that commit already changed. It predates that change. The test asserts bytes rather than decoded strings, because comparing strings cannot tell latin-1 from UTF-8. It uses two separate rows on purpose: any character above U+00FF makes perl print the whole string as UTF-8, so one body carrying both an accent and an em dash passes even unfixed and proves nothing. Verified failing before the fix on the accented row, passing after. Suites: fm-captain-hold-lifecycle 52 cases, fm-backlog-atomicity 99 cases, 0 failures. * no-mistakes(document): record body-decode regression proofs in captain-hold lifecycle doc * no-mistakes(review): drop whole-file UTF-8 check from retained-body test * no-mistakes(review): correct stale JSON::PP fleet-host claim in lifecycle doc * no-mistakes(review): anchor native-reproduction claims per defect in lifecycle doc
…furniture (kunchenguid#4532) * fix(composer): read codex 0.154's idle starfield and status footer as furniture codex-cli 0.154.0 animates a braille "starfield" around its idle composer: on the row above the bold `›` prompt row, on the `›` row behind the SGR-2 dim `Ask Codex to do anything` placeholder, and on the row below it, then draws a bright status footer (`<model> <effort>[ fast] · <path> · <title>`). The cells are truecolor greys on both sides of the ghost luminance ceiling, so the brighter ones survive ghost stripping, and the rows below the glyph carry no structural edge. The shared classifier selected the bare `›` shape, extended its wrap region over the two rows beneath the glyph, read the survivors and the footer as wrapped typed input, and answered `pending`; the steering doorbell defers on exactly that verdict, so no doorbell ever reached an idle codex 0.154 pane. bin/fm-composer-lib.sh now recognises that furniture by shape, declared once next to the idle placeholders and reached from the two wrap-region boundary points: - a row whose non-whitespace content is entirely braille cells (U+2800..U+28FF, detected byte-exactly under LC_ALL=C) is furniture: it never counts as wrapped typed content and bounds a bare composer's wrap region; braille behind the glyph row's content is stripped before the emptiness decision when nothing else follows the glyph; a row mixing braille with other text stays typed content; - the codex status footer bounds the wrap region exactly as omp's status row does, anchored on the effort token, a spaced middle dot, and a `~` or `/` path cell, so a typed `fix · tests` stays composer input; - `^Ask Codex to do anything$` joins the verified idle-placeholder set; the ghost strip remains what proves that row empty, and the bare-row rule that bright placeholder text is real input is unchanged. Unchanged: the strict blank-row rule, the styled=0 degradation (a plain cmux/orca capture of this screen still reads `unknown`, never `pending`), FM_COMPOSER_GHOST_LUMA_MAX, and every other harness's shape. tests/fm-composer-lib.test.sh carries both live Herdr samples byte-for-byte with the divergence (letters in place of the starfield read `pending`) and the over-stripping negatives; tests/fm-composer-codex-idle-live-e2e.test.sh is the default-on live guard (token-free, skips explicitly without codex or tmux) that launches the installed codex idle and asserts `empty` through both the tmux and the cursorless styled reads, naming codex --version on failure. docs/verification/runtime-backends.md records the dated Herdr evidence: `pending` before, `empty` after, on the captured screen. * no-mistakes(review): drop unreachable codex footer rule and inert placeholder entry --------- Co-authored-by: Todd Billings <todd@usdvcapital.com>
* fix(bin): refuse empty text steers in fm-send A marked secondmate request sent with an empty message delivered only marker and correlation bytes and minted a pending-reply expectation the parent could never see resolved, stalling the fleet with no loud error (kunchenguid#4255). Fail closed on an empty or whitespace-only message on the text path, mirroring the existing --resolve-key refusal. * chore: retain ambient Pi-lens autoformat as its own commit Formatting-only edits produced by ambient Pi-lens autoformat during the msg-loss investigation, kept separate from the behavioural change in c23acba so the fix stays reviewable on its own. AGENTS.md is deliberately excluded: its only autoformat edit stripped the trailing space from the documented FM_OPERATIONAL_PREFIX value, which bin/fm-operational-input.sh:28 defines as "FIRSTMATE_OP: " and line 11 records as permanent compatibility. Documenting that constant without its trailing space makes the doc wrong about the contract, so that one line was restored rather than retained.
brentsec
changed the title
brentsec
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
make sure to also sync the firstmate repo with the upstream firstmate repo before adding my customizations to my fork.. its currently 1 commit behind firstmate's upstream currently.
What Changed
bin/fm-captain-hold.shandbin/fm-backlog-transition-lib.shdecode a shown task body withJSON::PP->new->utf8->allow_nonrefinstead ofdecode_json, and the retention decoder now writes raw bytes viabinmode STDOUT, ":raw"plusutf8::encode, so holds and cleanup work where JSON::PP defaultsallow_nonrefoff and a retained body keeps its non-ASCII characters;tests/fm-captain-hold-lifecycle.test.shadds cases for both paths anddocs/captain-hold-lifecycle.mdrecords them.bin/fm-composer-lib.shaddsfm_composer_strip_brailleand treats braille-only rows as furniture - they bound a bare composer's wrap region and are stripped from behind the glyph row - so codex-cli 0.154's idle starfield screens classify asemptyrather thanpending; covered by new cases intests/fm-composer-lib.test.sh, the newtests/fm-composer-codex-idle-live-e2e.test.sh(registered inbin/fm-test-run.sh), and dated evidence indocs/verification/runtime-backends.md.bin/fm-send.shrefuses an empty or whitespace-only text steer before anything is marked, recorded, or typed, with coverage intests/fm-send-inbox.test.sh;bin/fm-spawn.shandbin/fm-backlog-handoff.shchange only in autoformat output (statement splitting, case-pattern and redirect spacing), with no behavior difference.Risk Assessment
✅ Low: A clean linear upstream sync whose bulk is a verified semantics-neutral reformat, with three narrow behavioral fixes that each carry an executing regression test, and no fork-local changes mixed in.
Testing
I first confirmed the sync itself against the real upstream remote: the branch head resolves to exactly kunchenguid/firstmate main 2da3c5e, the fork's old main is an ancestor, the range holds only the three official commits, and there are no merge commits or conflict markers. I then drove each commit's behaviour against the running product in isolated environments - real fm-send against a live Herdr endpoint, real fm-captain-hold and fm-teardown against a real tasks-axi backlog, and real codex-cli 0.154.0 in a real Herdr pane - and for each defect I re-ran the identical scenario against the pre-sync tree extracted at base aa92177, so every fix has a reproduced failure beside it. The empty-steer stall, the allow_nonref hold failure and the dropped accent all reproduce before and are gone after. The targeted suites for the touched areas pass, and the reformatted fm-spawn.sh still spawns and tears down a real Herdr pane end to end. Two real-Herdr suites failed on the first pass because the Powerlevel10k wizard blocks a fresh pane's shell on this host; I disabled that wizard for the run and both passed, so it is an environment blocker rather than a product one. The one thing I could not drive live is the braille starfield itself: codex 0.154.0 never animated it here across three model and mode combinations including a real attached Herdr viewer, so I report that scenario untested. Visual evidence is a rendered HTML page plus screenshot showing the real codex idle pane and every before/after transcript side by side.
git ls-remote https://github.com/kunchenguid/firstmate.git refs/heads/mainreturned 2da3c5e, identical to local HEAD;git merge-base --is-ancestor aa92177 HEADsuc…bin/fm-send.sh domainagainst a real kind=secondmate task whose endpoint is a live Herdr pane: exit 1, "a text steer requires a nonempty message; nothing was sent", state/domain.inbox absent, no sta…domain "",domain " ",domain $'\t' $'\n') each exited 1 with the same refusal and left state/domain.inbox and state/pending-replies absent - evidence/04-fm-send-live-h…[fm-from-firstmate]corr=e5812b920d04b848(marker + correlation bytes…fm-send.sh domain please re-read the brief and confirmexited 0, wrote state/domain.inbox/001.msg with body[fm-from-firstmate]corr=798a14d993d7d0f5 please re-read the brief and confirmand one pe…bin/fm-captain-hold.sh holdexited 1 on the base tree with the library's own "must be an obje…bin/fm-teardown.shon a real tasks-axi row bodied "The café decision is still open.": base tree producedThe caf� decision(accent lost, no c3 a9), HEAD producedThe café decisionwith byte…fm_backend_herdr_composer_state(the production cursorless styled profile, styled=1 cursor=0 identity=1 rows=20) answeredempty- evidenc…tests/fm-backend-autodetect-smoke.test.shagainst the real installed Herdr: real bin/fm-spawn.sh auto-detected herdr, recorded backend=herdr with herdr_session/workspace/tab/pane in meta, its launch…tests/fm-backlog-handoff.test.shdrives the real bin/fm-backlog-handoff.sh executable against real backlog files (tmux/gh/treehouse stubbed): all cases pass including crash-then-retry durability, co…Evidence: Live validation review page (source HTML)
Source: Live validation review page (source HTML)
Evidence: Raw ANSI bytes of the classified live codex idle frame
Source: Raw ANSI bytes of the classified live codex idle frame
Evidence: Sync fidelity against the real upstream remote
Source: Sync fidelity against the real upstream remote
### upstream ref 2da3c5e2193cb725bf173b7dfcbf8b094c4b862d refs/heads/main ### local HEAD 2da3c5e2193cb725bf173b7dfcbf8b094c4b862d ### base is ancestor of HEAD? yes: aa92177 is an ancestor of HEAD ### commits base..HEAD (exactly the upstream commits, no extras) 2da3c5e2 fix(bin): refuse empty text steers in fm-send (#4259) 8b10b61e fix(bin): read codex 0.154's idle braille starfield rows as composer furniture (#4532) b85e28b5 fix(bin): make captain holds work on hosts with an older JSON::PP, and stop cleanup dropping accents from a held body (#4471) ### merge commits in range (expect none) (end) ### working tree clean / no conflict markers in changed files (end status) (end conflict scan)Evidence: fm-send live drive against a real Herdr endpoint (refusals + happy path)
Source: fm-send live drive against a real Herdr endpoint (refusals + happy path)
Evidence: Pre-fix reproduction of the kunchenguid#4255 empty-steer stall
Source: Pre-fix reproduction of the #4255 empty-steer stall
=== PRE-FIX (base aa92177): empty marked secondmate steer === exit: 0 inbox records: 1 001.msg body -> [fm-from-firstmate]M-bM-^AM-#corr=e5812b920d04b848 pending-reply expectations minted: 1 state/pending-replies/e5812b920d04b848Evidence: Captain hold on an older-JSON::PP host, pre-fix vs HEAD
Source: Captain hold on an older-JSON::PP host, pre-fix vs HEAD
===== PRE-FIX tree (base aa92177) ===== exit: 1 JSON text must be an object or array (but found number, string, true, false or null, use allow_nonref to allow this) at -e line 3. fm-captain-hold: task sample-nonref-body did not retain its hold-set stamp -> hold_kind captain : NO ===== POST-FIX tree (HEAD 2da3c5e) ===== exit: 0 -> hold_kind captain : yesEvidence: Accented captain-held body through cleanup, pre-fix vs HEAD
Source: Accented captain-held body through cleanup, pre-fix vs HEAD
===== PRE-FIX tree (base aa92177) ===== row body renders as : The caf� decision accent survived as UTF-8 (c3 a9) : NO ===== POST-FIX tree (HEAD 2da3c5e) ===== row body renders as : The café decision accent survived as UTF-8 (c3 a9) : yesEvidence: Live codex idle read + starfield hunt
Source: Live codex idle read + starfield hunt
Evidence: Real-Herdr smoke suites re-driven with the host shell blocker removed
Source: Real-Herdr smoke suites re-driven with the host shell blocker removed
ok - real herdr: fm-spawn.sh auto-detects herdr from HERDR_ENV=1 (no explicit config) and prints the loud notice ok - real herdr: auto-detected spawn records backend=herdr and herdr_session/workspace/tab/pane fields in meta ok - real herdr: the auto-detected spawn's launch command actually ran in the herdr pane ok - real herdr: teardown completes the auto-detected spawn/teardown cycle (meta cleared, pane closed) ok - real herdr: isolated lab session removed and default fleet session unchanged FM_TEST_SUMMARY total=2 failed=0 skipped_gate=0Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
bin/fm-send.sh:756- The intent states the fork is "1 commit behind firstmate's upstream", but this branch syncs 3 upstream commits (b85e28b, 8b10b61, 2da3c5e) - ~2,100 lines across 13 files. This still satisfies the stated goal (sync the fork with upstream before adding customizations) and the branch is a strict linear descendant of origin/main with no fork-local commits mixed in, so it is not an intent contradiction. It is worth knowing what lands: the bulk is a semantics-neutral autoformat of bin/fm-spawn.sh and bin/fm-backlog-handoff.sh (verified: both files are identical to their pre-change versions once whitespace, ';' and '&feat(bin): recurring automated review sweep across the fleet kunchenguid/firstmate#39; are normalized away, and a token-level diff shows only case-pattern splits and redirection respacing - no string or heredoc content changed), but one user-visible behavior change rides along: fm-send.sh now exits 1 on an empty or whitespace-only text steer instead of delivering marker-only bytes. Every in-repo caller passes a non-empty constant, so nothing in-tree regresses, but any steer automation you add on top of this fork must send a non-empty message. No action needed on the sync itself.tests/fm-backend-herdr-smoke.test.sh- On this host both real-Herdr suites (tests/fm-backend-herdr-smoke.test.sh, tests/fm-backend-autodetect-smoke.test.sh) fail on a first run because the Powerlevel10k configuration wizard blocks the freshly-created pane's zsh before any command reaches it - the pane sits on "Does this look like >< but taller and fatter? Choice [ynrq]:" so send_text_line never echoes and treehouse never enters a worktree. Unrelated to this sync; both suites pass when the pane shell is launched with POWERLEVEL9K_DISABLE_CONFIGURATION_WIZARD=true. Worth configuring p10k (or exporting that variable) on this machine so local real-Herdr runs are not misread as product failures.git ls-remote https://github.com/kunchenguid/firstmate.git refs/heads/mainreturned 2da3c5e, identical to local HEAD;git merge-base --is-ancestor aa92177 HEADsuc…bin/fm-send.sh domainagainst a real kind=secondmate task whose endpoint is a live Herdr pane: exit 1, "a text steer requires a nonempty message; nothing was sent", state/domain.inbox absent, no sta…domain "",domain " ",domain $'\t' $'\n') each exited 1 with the same refusal and left state/domain.inbox and state/pending-replies absent - evidence/04-fm-send-live-h…[fm-from-firstmate]corr=e5812b920d04b848(marker + correlation bytes…fm-send.sh domain please re-read the brief and confirmexited 0, wrote state/domain.inbox/001.msg with body[fm-from-firstmate]corr=798a14d993d7d0f5 please re-read the brief and confirmand one pe…bin/fm-captain-hold.sh holdexited 1 on the base tree with the library's own "must be an obje…bin/fm-teardown.shon a real tasks-axi row bodied "The café decision is still open.": base tree producedThe caf� decision(accent lost, no c3 a9), HEAD producedThe café decisionwith byte…fm_backend_herdr_composer_state(the production cursorless styled profile, styled=1 cursor=0 identity=1 rows=20) answeredempty- evidenc…tests/fm-backend-autodetect-smoke.test.shagainst the real installed Herdr: real bin/fm-spawn.sh auto-detected herdr, recorded backend=herdr with herdr_session/workspace/tab/pane in meta, its launch…tests/fm-backlog-handoff.test.shdrives the real bin/fm-backlog-handoff.sh executable against real backlog files (tmux/gh/treehouse stubbed): all cases pass including crash-then-retry durability, co…git ls-remote https://github.com/kunchenguid/firstmate.git refs/heads/maincompared against local HEAD,git merge-base --is-ancestor aa92177 HEAD,git log aa92177..HEAD, merge-commit and conflict-marker scan over every changed fileLive: realbin/fm-send.shagainst a live Herdr pane (isolatedbin/fm-herdr-lab.shsession) with a realkind=secondmatetask - empty, empty-string, whitespace-only, tab+newline, a real message, and--key EnterLive regression: the same empty marked secondmate steer against the pre-sync tree extracted from baseaa92177(git archive aa92177 | tar -x), showing exit 0 + marker-only record + minted pending-replyLive: realbin/fm-captain-hold.sh holdon a real tasks-axi backlog with JSON::PP allow_nonref forced off via a PERL5OPT shim, pre-sync tree vs HEADLive: realbin/fm-teardown.shretaining a captain-held row whose body carriescafé, asserting bytes (c3 a9) not decoded strings, pre-sync tree vs HEADLive: realcodex-cli 0.154.0launched in an isolated Herdr pane, captured withfm_backend_herdr_capture_ansiand classified withfm_backend_herdr_composer_state(styled=1 cursor=0 identity=1 rows=20); same frame re-classified with the base-commit composer library for a before/afterStarfield hunt: 750+ sampled frames of live idle codex acrossgpt-daybreak-blue-latest,gpt-6-astra, andgpt-6-astra+ fast mode with a real attached Herdr TUI viewer (fm-herdr-lab.sh viewer start)bin/fm-test-run.sh tests/fm-send-inbox.test.sh tests/fm-captain-hold-lifecycle.test.sh tests/fm-composer-lib.test.sh tests/fm-composer-ghost.test.shbin/fm-test-run.sh tests/fm-backend-herdr-smoke.test.sh tests/fm-backend-autodetect-smoke.test.sh tests/fm-spawn-batch.test.sh tests/fm-spawn-dispatch-profile.test.sh tests/fm-spawn-worktree-settle.test.sh tests/fm-backlog-handoff.test.sh tests/fm-backlog-atomicity.test.shPOWERLEVEL9K_DISABLE_CONFIGURATION_WIZARD=true bin/fm-test-run.sh tests/fm-backend-herdr-smoke.test.sh tests/fm-backend-autodetect-smoke.test.sh(re-drive after fixing the host shell blocker)✅ **Document** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.