Skip to content

fix(bin): variant-shape the zcode interrupt record and log TUI delivery retries - #9

Merged
d-ploutarchos merged 3 commits into
mainfrom
fm/zcode-tui-followups
Sep 15, 2026
Merged

d-ploutarchos merged 3 commits into
mainfrom
fm/zcode-tui-followups

Conversation

@d-ploutarchos

Copy link
Copy Markdown

Intent

Close the two deferred follow-ups from the PR 6 re-review (round-1 N2, deferred by agreement), keeping the zcode TUI variant's delivery-retry behavior observable and its lifecycle semantics honest:

  1. Log delivery-retry attempts to stderr so a swallowing regression (a pointer's Enter being eaten pre-interactivity) is observable in the field, not only in tests.
  2. Make fm_control_interrupt_ends_process conditional for the TUI variant: the headless worker's interrupt legitimately ends the process, but the TUI interrupt cancels the turn and leaves the worker alive, so the unconditional record is wrong for one of the two shapes.

What Changed

  • fm_control_interrupt_ends_process in bin/fm-control-lib.sh now takes the task's recorded zcode_tui meta value: a headless zcode incarnation keeps the process-ends answer, while a recorded TUI variant (zcode_tui=1) answers no, so bin/fm-control.sh interrupt requires the TUI agent alive and refuses a TUI that died under the key instead of reporting the dead state as a landed interrupt.
  • bin/fm-control.sh now settles the post-interrupt agent state through the FM_CONTROL_SETTLE_WAIT window via wait_agent_state (returning early on an observed death) rather than reading it once, so a lingering TUI shutdown is not published as an alive worker.
  • zcode_tui_deliver_brief in bin/fm-spawn.sh logs one stderr line per unconfirmed brief-delivery attempt before the bare-Enter probe and retype, making a swallowed pointer observable in the spawn output; tests in tests/fm-control.test.sh and tests/fm-zcode-harness.test.sh pin the variant-shaped interrupt truth table, the lingering-death settle, and the retry log line, and the zcode harness reference doc is updated to match.

🤖 Generated with Claude Code

Risk Assessment

✅ Low: The change is a two-line semantic tweak plus one stderr log line, both narrowly scoped to the zcode harness: the new variant argument is read from the same task meta the spawn writes and the relaunch rewrites, the only caller passes it, every other harness path is unchanged, and the added tests exercise the real control and spawn binaries against fixtures rather than grepping source.

Testing

Ran the two targeted test files as baseline (fm-control 40 ok, fm-zcode-harness 22 ok), then rebuilt the isolated live lab (staged zcode HOME with a dummy key pointed at a local never-answering provider, isolated tmux servers, isolated firstmate homes) and drove eight scenarios against the real zcode TUI, the real headless worker, real fm-control, and real fm-spawn: busy-TUI interrupt stays alive with the record preserved; the round-1 failure (idle TUI dies under C-c) is now refused with the record left open; headless interrupt ends the process and retires the record; spawn happy path logs no retry; the fm-spawn-created task interrupts to alive; a delayed hook produces exactly one retry line and a landed spawn; an exhausted ladder logs both retries and fails closed with the window removed; and the retry line is on stderr alone. The real ~/.zcode config was verified byte-identical afterwards and every model call stayed on 127.0.0.1. All scenarios passed; the lab and its temp dirs were removed and the worktree is clean.

  • Live validation: ✅ go - 8 of 8 scenarios driven live against the product
Scenario Result Live Evidence
Interrupting a busy zcode TUI task (zcode_tui=1 recorded): fm-control reports verified=agent-alive after the settle window, the TUI shows 'Turn cancelled.' and stays in the foreground, and the busy re… ✅ pass live r2-s1-tui-busy-interrupt-alive.txt
Adversarial (round-1 failure): interrupting an idle zcode TUI task (zcode_tui=1) where C-c exits the TUI ~1.6 s later: fm-control refuses with exit 1 and 'an interrupt must leave the agent running', p… ✅ pass live r2-s2-tui-idle-dies-under-key-refused.txt
Interrupting a headless zcode --prompt worker (no zcode_tui line): fm-control reports verified=agent-ended-by-interrupt as soon as the death is observed, the worker exits 130, and the busy record is r… ✅ pass live r2-s3-headless-interrupt-ends-process.txt
Spawning a zcode TUI scout whose brief lands on the first attempt: fm-spawn records zcode_tui=1, the hook flips the busy record, and no retry line is printed ✅ pass live r2-s4-spawn-tui-happy-path-no-retry-log.txt
Interrupting the task fm-spawn itself created (real meta, real hook-flipped record, TUI mid-turn): agent-alive with the record preserved ✅ pass live r2-s5-spawned-tui-task-interrupt-alive.txt
Spawning a zcode TUI scout whose first delivery is unconfirmed (hook delayed ~2 s): exactly one stderr retry line naming the attempt and the bare-Enter probe, then the spawn lands with the pointer sub… ✅ pass live r2-s6-s7-spawn-tui-retry-log-and-exhaustion.txt
Adversarial: the hook never flips (delayed past zcode's 10 s hook timeout): both retry lines are logged, the spawn fails closed with the delivery error, and the TUI window is closed ✅ pass live r2-s6-s7-spawn-tui-retry-log-and-exhaustion.txt
The delivery-retry line is written to stderr, not stdout: with the streams captured separately, stdout holds only the spawned line and stderr holds the retry line ✅ pass live r2-s8-retry-line-is-on-stderr.txt
Evidence: Round-2 evidence index

Source: Round-2 evidence index

# Round 2 live validation: zcode TUI follow-ups (fm/zcode-tui-followups, 5631259 -> de98782)

Round 1 (files s1..s7, README.md) found one live failure: with zcode_tui=1 recorded, an interrupt
on an IDLE real TUI exits the TUI ~0.6 s after the key, but fm-control read the agent state once,
immediately, and published verified=agent-alive (exit 0) for a dead worker. Commit de98782 replaces
that single read with a bounded settle poll (wait_agent_state over FM_CONTROL_SETTLE_WAIT, 5 s
default) that returns as soon as a death is observed and otherwise holds the alive state.

Everything below was driven in THIS round against the REAL products on this host:
- zcode-app-cli 3.11.2-24 wrapping zcode-runtime 0.16.5
- real tmux 3.4 servers, isolated per lab via TMUX_TMPDIR (never the operator's server)
- bin/fm-control.sh and bin/fm-spawn.sh from the target commit de98782

Isolation: ~/.zcode staged into a throwaway HOME with a DUMMY api key and a baseURL pointing at a
local never-answering HTTP server (r2-lab-provider-requests.txt), so every model call hung locally
and a turn stayed busy as long as needed. The real ~/.zcode config and hook script were verified
byte-identical afterwards (r2-isolation-real-zcode-config-unchanged.txt). FM_GATE_REFUSE_BYPASS=1
is the guard's documented test-harness escape hatch, set only for lab commands.

Files (r2-*):
- r2-s1-tui-busy-interrupt-alive.txt        real TUI mid-turn, zcode_tui=1 -> verified=agent-alive after the full settle window (~6.5 s), "Turn cancelled." on screen, node/zcode-cli still foreground, busy record preserved. PASS
- r2-s2-tui-idle-dies-under-key-refused.txt  the round-1 FAIL re-driven: idle real TUI exits under C-c; 50 ms samples show node/zcode-cli lingering ~1.6 s, then a bare shell; fm-control now REFUSES (exit 1, "an interrupt must leave the agent running"), prints no interrupt-delivered line, busy record left open (not retired). PASS
- r2-s3-headless-interrupt-ends-process.txt  real headless --prompt worker, no zcode_tui line -> verified=agent-ended-by-interrupt in 1.6 s (the poll returned on the first observed death), HEADLESS_EXIT=130, busy record retired. PASS
- r2-s4-spawn-tui-happy-path-no-retry-log.txt  real fm-spawn --zcode-tui scout: hook flip confirmed on attempt 1, zcode_tui=1 recorded in the meta, NO retry line. PASS
- r2-s5-spawned-tui-task-interrupt-alive.txt   fm-control interrupt on the task fm-spawn created (real record, real hook-flipped busy state, TUI mid-turn) -> agent-alive, record preserved. PASS
- r2-s6-s7-spawn-tui-retry-log-and-exhaustion.txt  hook delayed ~2 s: exactly one retry line, bare-Enter probe, spawn lands with the pointer submitted once. Hook delayed past zcode's 10 s hook timeout: both retry lines logged, spawn fails closed, the TUI window is closed. PASS
- r2-s8-retry-line-is-on-stderr.txt          stdout and stderr captured separately: the retry line is on STDERR only; stdout carries just the spawned line. PASS

Targeted automated tests run as baseline (not the full suite):
- tests/fm-control.test.sh   40 ok, includes the new "zcode interrupt postcondition follows the recorded launch variant" test with its lingering-then-dead fixture
- tests/fm-zcode-harness.test.sh   22 ok, includes "a pre-interactivity swallow is recovered by the verify-and-retry delivery, loudly"
Evidence: S2 adversarial: idle TUI dies under C-c, fm-control now refuses (50 ms foreground samples)

Source: S2 adversarial: idle TUI dies under C-c, fm-control now refuses (50 ms foreground samples)

## $ fm-control.sh t1 interrupt (defaults) error: task t1's agent is 'dead' after its interrupt key; an interrupt must leave the agent running exit=1 elapsed=1.80s 0.51s node zcode-cli zcode-node-repl 0.61s node zcode-cli 1.58s node zcode-cli 1.68s bash ## busy files after (a refused interrupt must NOT retire the record) t1.busy-gen t1.busy-state

# Round 2 / Scenario 2 (adversarial, the round-1 FAIL): the same REAL zcode TUI, now IDLE after scenario 1's cancel; meta still records zcode_tui=1.
# On zcode 3.11.2-24 a C-c on the idle composer EXITS the TUI (~0.6s after the key). With the TUI variant recorded, fm-control must REFUSE (exit 1, 'an interrupt must leave the agent running'), never print interrupt-delivered, and must not retire the open busy record.
## pane before
╭─ ◆ ZCODE  v3.11.2-24 ────────────────────────────────────────────────────╮
│ /tmp/fm-zcode-tui-lab-r2.7YZnR3/ws · master                              │
│ Ask a task about this workspace                                          │
╰─ /help commands · /status details ───────────────────────────────────────╯
 › Describe this workspace in one sentence.
 Turn cancelled.
 [ ✓ 14s ]
────────────────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────────────────
 ◈ zai/glm-5.3-flash ─ ◉ yolo ─ ⚡ max ─ ctx 100% left ─ session 0 tokens
## foreground before
    PID    PGID   TPGID COMMAND
 630119  630119  630133 bash
 630133  630133  630133 node
 630150  630133  630133 zcode-cli
 630392  630133  630133 zcode-node-repl
## busy record before
v1 gen=g1789502760.630365.23672 seq=1 state=busy source=fm-spawn event=launch-brief ts=1789502760
## $ fm-control.sh t1 interrupt   (defaults)
error: task t1's agent is 'dead' after its interrupt key; an interrupt must leave the agent running
exit=1
elapsed=1.80s
## foreground-process-group samples every 50ms from just before the interrupt (blank/bash = no foreground agent)
0.00s node zcode-cli zcode-node-repl 
0.12s node zcode-cli zcode-node-repl 
0.22s node zcode-cli zcode-node-repl 
0.31s node zcode-cli zcode-node-repl 
0.42s node zcode-cli zcode-node-repl 
0.51s node zcode-cli zcode-node-repl 
0.61s node zcode-cli 
0.71s node zcode-cli 
0.80s node zcode-cli 
0.90s node zcode-cli 
1.00s node zcode-cli 
1.09s node zcode-cli 
1.19s node zcode-cli 
1.29s node zcode-cli 
1.38s node zcode-cli 
1.48s node zcode-cli 
1.58s node zcode-cli 
1.68s bash 
1.78s bash 
1.87s bash 
1.96s bash 
2.06s bash 
2.16s bash 
2.25s bash 
...
2.74s bash 
2.84s bash 
2.94s bash 
## pane after (tail)
 ◈ zai/glm-5.3-flash ─ ◉ yolo ─ ⚡ max ─ ctx 100% left ─ session 0 tokens
To continue this session, run zcode --resume sess_89d7ac1c-6a58-453c-9ed0-58d292
95f04d
root@vultr:/tmp/fm-zcode-tui-lab-r2.7YZnR3/ws#
## foreground after
    PID    PGID   TPGID COMMAND
 630119  630119  630119 bash
## busy files after (a refused interrupt must NOT retire the record)
t1.busy-gen
t1.busy-state
v1 gen=g1789502760.630365.23672 seq=1 state=busy source=fm-spawn event=launch-brief ts=1789502760
Evidence: S1: busy TUI interrupt stays alive through the settle window

Source: S1: busy TUI interrupt stays alive through the settle window

interrupt-delivered t1 harness=zcode backend=tmux verified=agent-alive cancel=unconfirmed exit=0 elapsed=6.54s › Describe this workspace in one sentence. Turn cancelled. ## foreground process group after (TUI must still be alive) 630133 630133 630133 node 630150 630133 630133 zcode-cli

# Round 2 / Scenario 1: fm-control interrupt on a REAL zcode TUI (zcode-app-cli 3.11.2-24 / zcode-runtime 0.16.5) MID-TURN; task meta records zcode_tui=1.
# Target commit de98782: the interrupt postcondition now SETTLES (bounded poll over FM_CONTROL_SETTLE_WAIT=5s default) instead of a single immediate read.
# Lab: isolated tmux server (TMUX_TMPDIR), isolated FM_HOME, staged zcode HOME with a dummy key pointed at a local never-answering provider (no external calls). FM_GATE_REFUSE_BYPASS=1 is the guard's documented test-harness escape hatch.
## task meta
window=fmlab:fm-t1
endpoint_task_id=t1
worktree=/tmp/fm-zcode-tui-lab-r2.7YZnR3/ws
project=/tmp/fm-zcode-tui-lab-r2.7YZnR3/ws
harness=zcode
kind=ship
mode=no-mistakes
yolo=off
model=default
effort=default
zcode_tui=1
busy_gen=g1789502760.630365.23672
## busy record before
v1 gen=g1789502760.630365.23672 seq=1 state=busy source=fm-spawn event=launch-brief ts=1789502760
## pane before (real tmux capture)
╭─ ◆ ZCODE  v3.11.2-24 ────────────────────────────────────────────────────╮
│ /tmp/fm-zcode-tui-lab-r2.7YZnR3/ws · master                              │
│ Ask a task about this workspace                                          │
╰─ /help commands · /status details ───────────────────────────────────────╯
 › Describe this workspace in one sentence.
 waiting for model… ── [ 🕐 13s ]
────────────────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────────────────
 ◈ zai/glm-5.3-flash ─ ◉ yolo ─ ⚡ max ─ ctx 100% left ─ session 0 tokens
## foreground process group before (pid pgid tpgid comm)
    PID    PGID   TPGID COMMAND
 630119  630119  630133 bash
 630133  630133  630133 node
 630150  630133  630133 zcode-cli
 630392  630133  630133 zcode-node-repl
## $ fm-control.sh t1 interrupt   (defaults: FM_CONTROL_SETTLE_WAIT=5 FM_CONTROL_POLL=0.5)
interrupt-delivered t1 harness=zcode backend=tmux verified=agent-alive cancel=unconfirmed
exit=0
elapsed=6.54s (the alive state must be HELD through the whole settle window, so ~5s is the expected shape)
## pane after
╭─ ◆ ZCODE  v3.11.2-24 ────────────────────────────────────────────────────╮
│ /tmp/fm-zcode-tui-lab-r2.7YZnR3/ws · master                              │
│ Ask a task about this workspace                                          │
╰─ /help commands · /status details ───────────────────────────────────────╯
 › Describe this workspace in one sentence.
 Turn cancelled.
 [ ✓ 14s ]
────────────────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────────────────
 ◈ zai/glm-5.3-flash ─ ◉ yolo ─ ⚡ max ─ ctx 100% left ─ session 0 tokens
## foreground process group after (TUI must still be alive)
    PID    PGID   TPGID COMMAND
 630119  630119  630133 bash
 630133  630133  630133 node
 630150  630133  630133 zcode-cli
 630392  630133  630133 zcode-node-repl
## busy files after (TUI variant must preserve the open record)
t1.busy-gen
t1.busy-state
v1 gen=g1789502760.630365.23672 seq=1 state=busy source=fm-spawn event=launch-brief ts=1789502760
Evidence: S3: headless worker interrupt ends the process and retires the record

Source: S3: headless worker interrupt ends the process and retires the record

interrupt-delivered t2 harness=zcode backend=tmux verified=agent-ended-by-interrupt cancel=unconfirmed exit=0 elapsed=1.55s ^CError: Turn was cancelled. (traceId: ...) HEADLESS_EXIT=130 ## busy files after (headless dead shape must retire the record) (no t2.busy-* files)

# Round 2 / Scenario 3: fm-control interrupt on a REAL HEADLESS zcode worker (--prompt) mid-turn; task meta has NO zcode_tui line (the headless shape keeps the process-ends answer).
# With the settle poll, the dead state must be returned as soon as it is observed (well inside the 5s window) and the busy record retired.
## task meta
window=fmlab:fm-t2
endpoint_task_id=t2
worktree=/tmp/fm-zcode-tui-lab-r2.7YZnR3/ws
project=/tmp/fm-zcode-tui-lab-r2.7YZnR3/ws
harness=zcode
kind=ship
mode=no-mistakes
yolo=off
model=default
effort=default
busy_gen=g1789502826.632159.10929
## busy record before
v1 gen=g1789502826.632159.10929 seq=1 state=busy source=fm-spawn event=launch-brief ts=1789502827
## pane before (tail)
root@vultr:/tmp/fm-zcode-tui-lab-r2.7YZnR3/ws# ZCODE_DISABLE_UPDATE_CHECK=1 FM_Z
CODE_HARNESS=zcode zcode --mode yolo --cwd '/tmp/fm-zcode-tui-lab-r2.7YZnR3/ws'
--prompt 'Describe this workspace in one sentence.'; echo "HEADLESS_EXIT=$?"
## foreground process group before
    PID    PGID   TPGID COMMAND
 632081  632081  632094 bash
 632094  632094  632094 node
 632111  632094  632094 zcode-cli
 632146  632094  632094 node
## $ fm-control.sh t2 interrupt   (defaults)
interrupt-delivered t2 harness=zcode backend=tmux verified=agent-ended-by-interrupt cancel=unconfirmed
exit=0
elapsed=1.55s
## pane after (tail)
--prompt 'Describe this workspace in one sentence.'; echo "HEADLESS_EXIT=$?"
^CError: Turn was cancelled. (traceId: 0ef88681-b11a-4163-9c73-19d392f93871)
HEADLESS_EXIT=130
root@vultr:/tmp/fm-zcode-tui-lab-r2.7YZnR3/ws#
## foreground process group after
    PID    PGID   TPGID COMMAND
 632081  632081  632081 bash
## busy files after (headless dead shape must retire the record)
(no t2.busy-* files)
Evidence: S4/S5: real fm-spawn TUI happy path and interrupt of the spawned task

Source: S4/S5: real fm-spawn TUI happy path and interrupt of the spawned task

# Round 2 / Scenario 5: fm-control interrupt on the task fm-spawn itself created in scenario 4 (real meta with zcode_tui=1 written by fm-spawn, real hook-flipped busy record, real TUI mid-turn).
## meta (zcode_tui line as recorded by fm-spawn)
1:window=firstmate:fm-s1
5:harness=zcode
11:zcode_tui=1
## busy record before
v1 gen=g1789502896.634096.21403 seq=2 state=busy source=zcode-hook event=user-prompt-submit ts=1789502902
## pane before (tail)
 › Read the brief at
 /tmp/fm-zcode-tui-lab-r2.7YZnR3/spawnhome/data/s1/launch-brief.md and follow
 it exactly.
## foreground before
    PID    PGID   TPGID COMMAND
 633981  633981  634460 bash
 633990  633990  634460 treehouse
 634031  634031  634460 bash
 634460  634460  634460 node
 634476  634460  634460 zcode-cli
 634725  634460  634460 zcode-node-repl
## $ fm-control.sh s1 interrupt   (defaults)
interrupt-delivered s1 harness=zcode backend=tmux verified=agent-alive cancel=unconfirmed
exit=0
elapsed=6.44s
## pane after (tail)
 › Read the brief at
 /tmp/fm-zcode-tui-lab-r2.7YZnR3/spawnhome/data/s1/launch-brief.md and follow
 it exactly.
 Turn cancelled.
## foreground after (TUI alive)
    PID    PGID   TPGID COMMAND
 633981  633981  634460 bash
 633990  633990  634460 treehouse
 634031  634031  634460 bash
 634460  634460  634460 node
 634476  634460  634460 zcode-cli
 634725  634460  634460 zcode-node-repl
## busy record after (preserved)
v1 gen=g1789502896.634096.21403 seq=2 state=busy source=zcode-hook event=user-prompt-submit ts=1789502902
Evidence: S6/S7: retry line logged once with a delayed hook, both lines then fail-closed on exhaustion

Source: S6/S7: retry line logged once with a delayed hook, both lines then fail-closed on exhaustion

fm-spawn: task s2: zcode TUI brief delivery attempt 1 of 3 unconfirmed in window firstmate:fm-s2; retry 2 of 3 probes a bare Enter, then retypes the pointer spawned s2 harness=zcode kind=scout window=firstmate:fm-s2 ... exit=0 --- fm-spawn: task s3: zcode TUI brief delivery attempt 1 of 3 unconfirmed in window firstmate:fm-s3; retry 2 of 3 probes a bare Enter, then retypes the pointer fm-spawn: task s3: zcode TUI brief delivery attempt 2 of 3 unconfirmed in window firstmate:fm-s3; retry 3 of 3 probes a bare Enter, then retypes the pointer error: the zcode TUI brief pointer could not be confirmed delivered through the turn hook in window firstmate:fm-s3; inspect window firstmate:fm-s3 exit=1

# Round 2 / Scenario 6: REAL fm-spawn --zcode-tui with a SLOW UserPromptSubmit hook (the staged hook's jq calls are delayed 0.7s each via a PATH shim on the spawn tmux server, ~2s total before the flip), first window = one immediate check.
# Expected: attempt 1 unconfirmed -> ONE stderr retry line naming the attempt and the bare-Enter probe -> the delayed real hook flip confirms delivery -> spawn lands with the pointer submitted once.
## $ FM_ZCODE_TUI_DELIVERY_POLLS=1 FM_ZCODE_TUI_SUBMIT_SLEEP=0 FM_ZCODE_TUI_SUBMIT_SETTLE=0 FM_ZCODE_TUI_SUBMIT_RETRIES=1 fm-spawn.sh s2 <proj> --scout --harness zcode --zcode-tui
fm-spawn: task s2: zcode TUI brief delivery attempt 1 of 3 unconfirmed in window firstmate:fm-s2; retry 2 of 3 probes a bare Enter, then retypes the pointer
spawned s2 harness=zcode kind=scout window=firstmate:fm-s2 worktree=/tmp/fm-zcode-tui-lab-r2.7YZnR3/home/.treehouse/proj-80a7e0/2/proj
exit=0
## retry lines: 1
## busy record
v1 gen=g1789502957.635799.5682 seq=2 state=busy source=zcode-hook event=user-prompt-submit ts=1789502965
## pane (real tmux capture): pointer submitted once; the bare-Enter probe was a no-op on the empty composer
╭─ ◆ ZCODE  v3.11.2-24 ────────────────────────────────────────────────────╮
│ ~/.treehouse/proj-80a7e0/2/proj                                          │
│ Ask a task about this workspace                                          │
╰─ /help commands · /status details ───────────────────────────────────────╯
 › Read the brief at
 /tmp/fm-zcode-tui-lab-r2.7YZnR3/spawnhome/data/s2/launch-brief.md and follow
 it exactly.
 waiting for model… ── [ 🕑 2s ]
────────────────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────────────────
 ◈ zai/glm-5.3-flash ─ ◉ yolo ─ ⚡ max ─ ctx 100% left ─ session 0 tokens

# Round 2 / Scenario 7 (adversarial): same, but the hook's jq calls are delayed 30s (beyond zcode's 10s hook timeout), so the flip NEVER lands: the ladder must log every retry to stderr and fail the spawn closed rather than ride through silently.
## $ FM_ZCODE_TUI_DELIVERY_POLLS=2 FM_ZCODE_TUI_SWALLOW_PROBE_POLLS=2 FM_ZCODE_TUI_SUBMIT_SLEEP=0 FM_ZCODE_TUI_SUBMIT_SETTLE=0 FM_ZCODE_TUI_SUBMIT_RETRIES=1 fm-spawn.sh s3 <proj> --scout --harness zcode --zcode-tui
fm-spawn: task s3: zcode TUI brief delivery attempt 1 of 3 unconfirmed in window firstmate:fm-s3; retry 2 of 3 probes a bare Enter, then retypes the pointer
fm-spawn: task s3: zcode TUI brief delivery attempt 2 of 3 unconfirmed in window firstmate:fm-s3; retry 3 of 3 probes a bare Enter, then retypes the pointer
error: the zcode TUI brief pointer could not be confirmed delivered through the turn hook in window firstmate:fm-s3; inspect window firstmate:fm-s3
warning: leaving task s3's slot claim on /tmp/fm-zcode-tui-lab-r2.7YZnR3/home/.treehouse/proj-80a7e0/3/proj in place; the Treehouse project lock is no longer held, so the next spawn's claim replaces it
exit=1
## retry lines: 2
## status record
failed: the zcode TUI brief pointer could not be confirmed delivered through the turn hook in window firstmate:fm-s3
## busy record (never flipped)
(none)
## windows left on the spawn server (the failed TUI window must have been closed by the spawn failure path)
keep
fm-s1
fm-s2
Evidence: S8: retry line is on stderr only

Source: S8: retry line is on stderr only

## stdout: spawned s4 harness=zcode kind=scout window=firstmate:fm-s4 ... ## stderr: fm-spawn: task s4: zcode TUI brief delivery attempt 1 of 3 unconfirmed in window firstmate:fm-s4; retry 2 of 3 probes a bare Enter, then retypes the pointer

# Round 2 / Scenario 8: the delivery-retry line is written to STDERR (intent item 1), with stdout and stderr captured to SEPARATE files. Same slow-hook setup as scenario 6.
## $ FM_ZCODE_TUI_DELIVERY_POLLS=1 ... fm-spawn.sh s4 <proj> --scout --harness zcode --zcode-tui  >stdout 2>stderr
exit=0
## stdout:
spawned s4 harness=zcode kind=scout window=firstmate:fm-s4 worktree=/tmp/fm-zcode-tui-lab-r2.7YZnR3/home/.treehouse/proj-80a7e0/3/proj
## stderr:
fm-spawn: task s4: zcode TUI brief delivery attempt 1 of 3 unconfirmed in window firstmate:fm-s4; retry 2 of 3 probes a bare Enter, then retypes the pointer
## busy record
v1 gen=g1789502992.638575.19024 seq=2 state=busy source=zcode-hook event=user-prompt-submit ts=1789503000
Evidence: Isolation: real ~/.zcode config unchanged; all model calls local

Source: Isolation: real ~/.zcode config unchanged; all model calls local

# Isolation check: the operator's REAL ~/.zcode config and hook script are byte-identical before and after the whole round-2 lab (sha256).
## before
d19d919c4700a77730e01c92a7910f2f03a720627eaa2318384406680a5c2977  ~/.zcode/cli/config.json
22571b0c56a8633a55cbb768a46741aee769ec5395a33236c5db3d7f52684225  ~/.zcode/cli/fm-turn-end.sh
## after
d19d919c4700a77730e01c92a7910f2f03a720627eaa2318384406680a5c2977  ~/.zcode/cli/config.json
22571b0c56a8633a55cbb768a46741aee769ec5395a33236c5db3d7f52684225  ~/.zcode/cli/fm-turn-end.sh
## verdict
~/.zcode/cli/config.json: OK
~/.zcode/cli/fm-turn-end.sh: OK
## the real registry gained no lab tokens (lab tokens live in the staged home only)
real:   4 tokens
staged: 8 tokens
Evidence: Local provider request log

Source: Local provider request log

# local never-answering provider log: every model call from the round-2 lab went to 127.0.0.1 (no external endpoint, dummy key)
20:05:34 hanging provider listening on 51025
20:06:00 request from 127.0.0.1:58398: POST /api/anthropic/v1/messages HTTP/1.1
20:06:03 request from 127.0.0.1:56942: POST /api/anthropic/v1/messages HTTP/1.1
20:06:18 request from 127.0.0.1:41542: 
20:08:22 request from 127.0.0.1:37390: POST /api/anthropic/v1/messages HTTP/1.1
20:08:25 request from 127.0.0.1:37396: POST /api/anthropic/v1/messages HTTP/1.1
20:08:54 request from 127.0.0.1:42210: 
20:09:25 request from 127.0.0.1:51028: POST /api/anthropic/v1/messages HTTP/1.1
20:09:26 request from 127.0.0.1:51026: 
20:09:26 request from 127.0.0.1:51038: POST /api/anthropic/v1/messages HTTP/1.1
20:09:37 request from 127.0.0.1:60716: POST /api/anthropic/v1/messages HTTP/1.1
20:10:00 request from 127.0.0.1:53596: POST /api/anthropic/v1/messages HTTP/1.1
20:10:01 request from 127.0.0.1:53602: POST /api/anthropic/v1/messages HTTP/1.1
- Outcome: 🔧 2 issues found → auto-fixed ✅ across 2 runs (37m15s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

🔧 **Test** - 2 issues found → auto-fixed ✅
  • ⚠️ bin/fm-control.sh:437 - Live on zcode 3.11.2-24: with zcode_tui=1 recorded, fm-control &lt;id&gt; interrupt on an idle TUI sends C-c, which exits the TUI, yet fm-control prints verified=agent-alive and exits 0, leaving the open busy record for a dead worker (evidence s2-tui-dies-under-key-refused.txt, s2b-tui-dies-under-key-timing.txt: 50 ms samples show node/zcode-cli still in the foreground for ~0.6 s after the key, then a bare shell). verify_interrupt_running reads agent_state once, immediately after delivery, so the TUI-variant refusal this change adds (the zcode-tui-dead cell of test_zcode_interrupt_postcondition_follows_the_recorded_variant) only fires for an instant death and never for the real TUI shutdown. The record logic is correct and the timing gap predates this change, so decide whether to add a bounded settle poll for the alive-required postcondition (mirroring do_exit's EXIT_WAIT loop, with a delayed-death fixture in tests/fm-control.test.sh) in this PR or ship as-is with a follow-up.
  • 🚨 live validation verdict: no-go (6 of 8 scenarios were driven live against the product); failed: Adversarial: the interrupt key actually exits the TUI (idle composer) with zcode_tui=1 recorded: fm-control refuses with 'an interrupt must leave the agent running' instead of reporting a landed interrupt
  • Live validation: ❌ no-go - 6 of 8 scenarios driven live against the product
Scenario Result Live Evidence
Interrupt a zcode TUI task (zcode_tui=1) mid-turn: fm-control reports verified=agent-alive, the turn shows 'Turn cancelled', the TUI process survives, and the open busy record is preserved ✅ pass live s1-tui-interrupt-alive.txt and s6-spawned-tui-task-interrupt-alive.txt (real zcode TUI, real tmux, fm-control.sh)
Adversarial: the interrupt key actually exits the TUI (idle composer) with zcode_tui=1 recorded: fm-control refuses with 'an interrupt must leave the agent running' instead of reporting a landed inter… ❌ fail live s2-tui-dies-under-key-refused.txt, s2b-tui-dies-under-key-timing.txt: output was verified=agent-alive exit 0 while the pane dropped to a bare shell ~0.6 s later; busy record left open
Interrupt a headless zcode --prompt worker (no zcode_tui line) mid-turn: fm-control reports verified=agent-ended-by-interrupt, the worker exits 130, and the busy record is retired ✅ pass live s3-headless-interrupt-ends-process.txt
Headless record still accepts the alive state, and the record function answers process-ends for zcode with no variant, not for zcode_tui=1, and never for claude ⏸️ untested no Not driven against the live product: a real headless zcode worker always exits on C-c, so the headless-alive cell cannot be produced live. The prior run only exercised this through the stubbed lifecyc…
Spawn --zcode-tui scout against the real TUI with the default delivery window: delivery confirmed through the real UserPromptSubmit hook flip, zcode_tui=1 recorded, no retry line in the output ✅ pass live s4-spawn-tui-happy-path-no-retry-log.txt
Spawn --zcode-tui with a slow hook (attempt 1 unconfirmed): exactly one stderr line 'attempt 1 of 3 unconfirmed in window ...; retry 2 of 3 probes a bare Enter' is logged, the probe lands delivery, th… ✅ pass live s5c-s7-spawn-tui-retry-log-and-exhaustion.txt (scenario 5 section); s5-*.txt show the real hook flips before a one-poll window unless delayed
Adversarial: the hook never lands: the ladder logs 'attempt 1 of 3' and 'attempt 2 of 3' retry lines, fails the spawn with the delivery error, records failed: in the status, and closes the TUI window ✅ pass live s5c-s7-spawn-tui-retry-log-and-exhaustion.txt (scenario 7 section)
Fixture-level swallow regression net: a swallowed first pointer is recovered and the retry is logged (stubbed tmux) ⏸️ untested no Not driven against the live product: a genuine pre-interactivity keystroke swallow is timing-dependent and could not be reproduced on demand against the real zcode TUI. The prior run only covered it t…
  • bin/fm-test-run.sh tests/fm-control.test.sh tests/fm-zcode-harness.test.sh (both scripts touched by the change; all pass)
  • Live lab: staged ~/.zcode into a throwaway HOME with a dummy api key and baseURL at a local never-answering HTTP provider (python3), isolated tmux servers via TMUX_TMPDIR, FM_GATE_REFUSE_BYPASS=1 (the guard's documented test-harness escape hatch) for lab commands only
  • FM_HOME=&lt;lab&gt; bin/fm-control.sh t1 interrupt on a real busy zcode TUI with zcode_tui=1 and an armed busy record (s1)
  • FM_HOME=&lt;lab&gt; bin/fm-control.sh t1 interrupt on the same TUI while idle, twice, with 50 ms foreground-process sampling (s2, s2b)
  • FM_HOME=&lt;lab&gt; bin/fm-control.sh t2 interrupt on a real headless zcode --mode yolo --prompt worker (s3)
  • bin/fm-spawn.sh s1 &lt;proj&gt; --scout --harness zcode --zcode-tui against the real tmux backend, real treehouse, real zcode TUI (s4)
  • FM_ZCODE_TUI_DELIVERY_POLLS=1 ... bin/fm-spawn.sh s2/s3 ... --zcode-tui (s5, s5b: hook flipped before the first check)
  • bin/fm-spawn.sh s4 ... --zcode-tui with the staged lab hook wrapped in sleep 2 after install (s5c) and sleep 30 for s5 (s7), hook restored byte-identical afterwards
  • FM_HOME=&lt;spawnhome&gt; bin/fm-control.sh s1 interrupt on the fm-spawn-created TUI task (s6)
  • Post-run: lab servers/provider killed, lab and /tmp/fm-s1..s5 removed, real ~/.zcode config and hook script mtimes unchanged, git status --porcelain clean

🔧 Fix applied.
✅ Re-checked - no issues remain.

  • Live validation: ✅ go - 8 of 8 scenarios driven live against the product
Scenario Result Live Evidence
Interrupting a busy zcode TUI task (zcode_tui=1 recorded): fm-control reports verified=agent-alive after the settle window, the TUI shows 'Turn cancelled.' and stays in the foreground, and the busy re… ✅ pass live r2-s1-tui-busy-interrupt-alive.txt
Adversarial (round-1 failure): interrupting an idle zcode TUI task (zcode_tui=1) where C-c exits the TUI ~1.6 s later: fm-control refuses with exit 1 and 'an interrupt must leave the agent running', p… ✅ pass live r2-s2-tui-idle-dies-under-key-refused.txt
Interrupting a headless zcode --prompt worker (no zcode_tui line): fm-control reports verified=agent-ended-by-interrupt as soon as the death is observed, the worker exits 130, and the busy record is r… ✅ pass live r2-s3-headless-interrupt-ends-process.txt
Spawning a zcode TUI scout whose brief lands on the first attempt: fm-spawn records zcode_tui=1, the hook flips the busy record, and no retry line is printed ✅ pass live r2-s4-spawn-tui-happy-path-no-retry-log.txt
Interrupting the task fm-spawn itself created (real meta, real hook-flipped record, TUI mid-turn): agent-alive with the record preserved ✅ pass live r2-s5-spawned-tui-task-interrupt-alive.txt
Spawning a zcode TUI scout whose first delivery is unconfirmed (hook delayed ~2 s): exactly one stderr retry line naming the attempt and the bare-Enter probe, then the spawn lands with the pointer sub… ✅ pass live r2-s6-s7-spawn-tui-retry-log-and-exhaustion.txt
Adversarial: the hook never flips (delayed past zcode's 10 s hook timeout): both retry lines are logged, the spawn fails closed with the delivery error, and the TUI window is closed ✅ pass live r2-s6-s7-spawn-tui-retry-log-and-exhaustion.txt
The delivery-retry line is written to stderr, not stdout: with the streams captured separately, stdout holds only the spawned line and stderr holds the retry line ✅ pass live r2-s8-retry-line-is-on-stderr.txt
  • bash tests/fm-control.test.sh (40 ok, includes test_zcode_interrupt_postcondition_follows_the_recorded_variant with the FM_FAKE_INTERRUPT_STOPS_AGENT_LATER fixture)
  • bash tests/fm-zcode-harness.test.sh (22 ok, includes the loud retry-ladder test and the variant-shaped ends-process table test)
  • Live: FM_HOME=&lt;lab&gt; fm-control.sh t1 interrupt on a real busy zcode 3.11.2-24 TUI with zcode_tui=1 (r2-s1)
  • Live adversarial: same command on the idle real TUI that exits under C-c, with a 50 ms foreground-process sampler (r2-s2)
  • Live: fm-control.sh t2 interrupt on a real headless zcode --prompt worker with no zcode_tui line (r2-s3)
  • Live: fm-spawn.sh s1 &lt;proj&gt; --scout --harness zcode --zcode-tui against a real isolated tmux server (r2-s4)
  • Live: fm-control.sh s1 interrupt on the task fm-spawn created (r2-s5)
  • Live: FM_ZCODE_TUI_DELIVERY_POLLS=1 ... fm-spawn.sh s2 ... --zcode-tui with the staged hook delayed ~2 s (r2-s6)
  • Live adversarial: FM_ZCODE_TUI_DELIVERY_POLLS=2 FM_ZCODE_TUI_SWALLOW_PROBE_POLLS=2 ... fm-spawn.sh s3 ... --zcode-tui with the hook delayed past zcode's 10 s hook timeout (r2-s7)
  • Live: spawn with stdout and stderr captured to separate files to prove the retry line's stream (r2-s8)
  • sha256 comparison of the real ~/.zcode/cli/config.json and fm-turn-end.sh before and after the lab
  • Local provider request log review (all calls from 127.0.0.1)
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

…ry retries

Close the two deferred PR 6 re-review follow-ups (round-1 N2):

- zcode_tui_deliver_brief logs one best-effort stderr line per delivery
  retry, naming the unconfirmed attempt, the window, and the bare-Enter
  probe it is about to run, so a field swallow regression is observable
  in the spawn's output and not only in tests.

- fm_control_interrupt_ends_process takes the recorded launch variant
  (the task meta's zcode_tui value) from bin/fm-control.sh: a recorded
  TUI incarnation answers no, so the interrupt verification requires
  the agent alive and preserves the open busy record, while a headless
  zcode incarnation keeps today's both-shapes semantics.

Tests extend both directions: the swallow ladder asserts its retry log
(and the happy path asserts silence), and the interrupt truth table
pins headless-alive, TUI-alive, and TUI-dead-refused beside the
existing headless-dead case.
@d-ploutarchos
d-ploutarchos merged commit edd79a0 into main Sep 15, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant