Skip to content

fix: preserve PR monitoring after task relaunch - #58

Merged
MrGTV-love merged 3 commits into
mainfrom
fm/fm-relaunch-tx-breaks-pr-metadata-parse
Oct 9, 2026
Merged

MrGTV-love merged 3 commits into
mainfrom
fm/fm-relaunch-tx-breaks-pr-metadata-parse

Conversation

@MrGTV-love

Copy link
Copy Markdown
Owner

Intent

I would like all bugs to be fixed so tomorrow can be focused entirely to Vernant and not problems preventing Vernant from getting built. A recommended architecture should come with proof.

Context (the backlog record for this bug, as filed): Relaunch writes control_relaunch_tx after pr= in task meta; fm_pr_metadata_identity_parse rejects it and PR monitoring silently stops (repo: firstmate, since 2026-10-06). Root-caused 2026-10-06 by lane d1 on GRE-1842 (https://github.com/MrGTV-love/vernant/pull/3091): an unchanged control_relaunch_tx line appended after pr= in state/.meta makes fm_pr_metadata_identity_parse reject the record; moving the field before pr= restored monitoring (counterfactual ordering passed). Many workers were relaunched on Claude, so any relaunched task with a recorded PR may have silent PR monitoring. Fix the publication order or make the parser order-independent, with a behavioral test (relaunch a task that has pr= recorded, then the PR poll must still parse). Priority: high.

What Changed

  • Publish control_relaunch_tx before the preserved PR metadata block so relaunched task records remain parseable by PR monitoring.
  • Replace existing traceparent entries with a single carrier before the first pr= line, or at the end of records without a PR, and document this ordering.
  • Add a relaunch regression test that checks recorded PR identity parsing with tracing enabled and disabled, including trace carrier preservation or removal.

Risk Assessment

✅ Low: The change narrowly corrects metadata publication ordering, preserves existing carrier and transaction semantics, and introduces no unsupported behavior or substantiated material defects.

Testing

Targeted regressions passed and reproduced both failures with the old writers. Real-Claude stopped-task recovery relaunches passed tracing-off, tracing-on, repeated-carrier, and no-PR scenarios, with CLI and generated-metadata evidence saved. Earlier Herdr and idle-Claude setup attempts could not complete; the supported recovery route succeeded, and all disposable labs were removed.

  • Live validation: ✅ go - 4 of 4 scenarios driven live against the product
Scenario Result Live Evidence
Recover a PR-linked task with tracing disabled; monitoring still accepts its recorded PR ✅ pass live Recovery transcript: tracing-off relaunch succeeded, the parser retained https://github.com/example/repo/pull/90, the watcher snapshot gate accepted it, and carrier count was zero.
Recover a traced PR-linked task with duplicate historic carriers; publish one stable carrier without breaking PR monitoring ✅ pass live Recovery transcript: traced attempt 1 launched a real Claude replacement and published exactly one unchanged traceparent before pr=; both monitoring consumers accepted the PR.
Recover the same traced PR-linked task again; retain its PR and unique trace identity ✅ pass live Recovery transcript: traced attempt 2 succeeded in the preserved endpoint, retained the same carrier exactly once, and passed PR parser and watcher snapshot validation.
Recover a traced task without a recorded PR; append its unique carrier at the end ✅ pass live Recovery transcript: no-PR relaunch succeeded and the generated record ended with exactly one unchanged traceparent line.
Evidence: Real-Claude recovery relaunches: published metadata and watcher validation

Source: Real-Claude recovery relaunches: published metadata and watcher validation

~/tmp/fm-lab.ojtloB
trusted: ~/.no-mistakes/worktrees/32d18ed9638d/01M4EXCNBHH0RJC0P3T69ZP04Q/.no-mistakes/test-tmp/fm-pr-relaunch-tmux.oaAUTI/wt
trusted (project root): ~/.no-mistakes/worktrees/32d18ed9638d/01M4EXCNBHH0RJC0P3T69ZP04Q/.no-mistakes/test-tmp/fm-pr-relaunch-tmux.oaAUTI/proj
Isolated Claude auth: {
  "loggedIn": true,
  "authMethod": "oauth_token",
  "apiProvider": "firstParty"
}
Claude version: 2.1.295 (Claude Code)
Tmux version: tmux 3.5a
starting from a genuine stopped-task shell in private endpoint primary:fm-prrelaunch59644
stopped endpoint: genuine shell, classifier=dead endpoint=primary:fm-prrelaunch59644
monitor parser: accepted URL=https://github.com/example/repo/pull/90
watcher dispatch gate: accepted provider=github URL=https://github.com/example/repo/pull/90 number=90
home session tracing: off

SCENARIO: recorded PR relaunch, tracing off
fm-gate-refuse: gate agent lifecycle permitted only against lab home ~/tmp/fm-lab.ojtloB
fm-gate-refuse: gate agent lifecycle permitted only against lab home ~/tmp/fm-lab.ojtloB
warning: ~/tmp/fm-lab.ojtloB/data/prrelaunch59644/launch-brief.md records no ship branch; defaulting to legacy branch fm/prrelaunch59644
warning: ~/tmp/fm-lab.ojtloB/data/prrelaunch59644/launch-brief.md records no delivery contract line (scaffolded before ship briefs recorded one); launching on the explicit --mode local-only - confirm its definition of done matches
notice: prrelaunch59644 ships mode=local-only while the standing posture for proj is no-mistakes - less rigor than the captain's standing posture; proceed only on a current explicit captain instruction or an intake judgment you can state
skill selection for prrelaunch59644: unavailable - no TypeSafe or OpenRouter key
relaunched prrelaunch59644 harness=claude from=claude model=default effort=default backend=tmux endpoint=primary:fm-prrelaunch59644 worktree=~/.no-mistakes/worktrees/32d18ed9638d/01M4EXCNBHH0RJC0P3T69ZP04Q/.no-mistakes/test-tmp/fm-pr-relaunch-tmux.oaAUTI/wt
monitor parser: accepted URL=https://github.com/example/repo/pull/90
watcher dispatch gate: accepted provider=github URL=https://github.com/example/repo/pull/90 number=90
carrier count=0
Published task metadata:
window=primary:fm-prrelaunch59644
endpoint_task_id=prrelaunch59644
worktree=~/.no-mistakes/worktrees/32d18ed9638d/01M4EXCNBHH0RJC0P3T69ZP04Q/.no-mistakes/test-tmp/fm-pr-relaunch-tmux.oaAUTI/wt
project=~/.no-mistakes/worktrees/32d18ed9638d/01M4EXCNBHH0RJC0P3T69ZP04Q/.no-mistakes/test-tmp/fm-pr-relaunch-tmux.oaAUTI/proj
harness=claude
kind=ship
mode=local-only
yolo=off
branch=fm/prrelaunch59644
tasktmp=/tmp/fm-prrelaunch59644
model=default
effort=default
skill_selection=unavailable
skill_selection_reason=no TypeSafe or OpenRouter key
busy_gen=g1791515070.92353.23710
spawn_gen=s1791515071.84372.19872
control_relaunch_tx=80573.20261009T030426Z.11348
pr=https://github.com/example/repo/pull/90
pr_head=0123456789abcdef0123456789abcdef01234567
x_request=request-90
ok - LIVE: tracing-off relaunch keeps recorded PR accepted by monitoring
stopped endpoint: genuine shell, classifier=dead endpoint=primary:fm-prrelaunch59644
monitor parser: accepted URL=https://github.com/example/repo/pull/90
watcher dispatch gate: accepted provider=github URL=https://github.com/example/repo/pull/90 number=90
home session tracing: on

SCENARIO: traced recorded PR relaunch, attempt 1
fm-gate-refuse: gate agent lifecycle permitted only against lab home ~/tmp/fm-lab.ojtloB
fm-gate-refuse: gate agent lifecycle permitted only against lab home ~/tmp/fm-lab.ojtloB
warning: ~/tmp/fm-lab.ojtloB/data/prrelaunch59644/launch-brief.md records no ship branch; defaulting to legacy branch fm/prrelaunch59644
warning: ~/tmp/fm-lab.ojtloB/data/prrelaunch59644/launch-brief.md records no delivery contract line (scaffolded before ship briefs recorded one); launching on the explicit --mode local-only - confirm its definition of done matches
notice: prrelaunch59644 ships mode=local-only while the standing posture for proj is no-mistakes - less rigor than the captain's standing posture; proceed only on a current explicit captain instruction or an intake judgment you can state
skill selection for prrelaunch59644: unavailable - no TypeSafe or OpenRouter key
relaunched prrelaunch59644 harness=claude from=claude model=default effort=default backend=tmux endpoint=primary:fm-prrelaunch59644 worktree=~/.no-mistakes/worktrees/32d18ed9638d/01M4EXCNBHH0RJC0P3T69ZP04Q/.no-mistakes/test-tmp/fm-pr-relaunch-tmux.oaAUTI/wt
replacement Claude: alive in the preserved endpoint
monitor parser: accepted URL=https://github.com/example/repo/pull/90
watcher dispatch gate: accepted provider=github URL=https://github.com/example/repo/pull/90 number=90
carrier count=1 value=00-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-bbbbbbbbbbbbbbbb-01
Published task metadata:
window=primary:fm-prrelaunch59644
endpoint_task_id=prrelaunch59644
worktree=~/.no-mistakes/worktrees/32d18ed9638d/01M4EXCNBHH0RJC0P3T69ZP04Q/.no-mistakes/test-tmp/fm-pr-relaunch-tmux.oaAUTI/wt
project=~/.no-mistakes/worktrees/32d18ed9638d/01M4EXCNBHH0RJC0P3T69ZP04Q/.no-mistakes/test-tmp/fm-pr-relaunch-tmux.oaAUTI/proj
harness=claude
kind=ship
mode=local-only
yolo=off
branch=fm/prrelaunch59644
tasktmp=/tmp/fm-prrelaunch59644
model=default
effort=default
skill_selection=unavailable
skill_selection_reason=no TypeSafe or OpenRouter key
busy_gen=g1791515079.21440.6587
spawn_gen=s1791515080.13172.14619
control_relaunch_tx=6504.20261009T030437Z.18156
traceparent=00-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-bbbbbbbbbbbbbbbb-01
pr=https://github.com/example/repo/pull/90
pr_head=0123456789abcdef0123456789abcdef01234567
x_request=request-90
ok - LIVE: tracing-on relaunch attempt 1 preserves parseable PR and one stable carrier
stopped endpoint: genuine shell, classifier=dead endpoint=primary:fm-prrelaunch59644

SCENARIO: traced recorded PR relaunch, attempt 2
fm-gate-refuse: gate agent lifecycle permitted only against lab home ~/tmp/fm-lab.ojtloB
fm-gate-refuse: gate agent lifecycle permitted only against lab home ~/tmp/fm-lab.ojtloB
warning: ~/tmp/fm-lab.ojtloB/data/prrelaunch59644/launch-brief.md records no ship branch; defaulting to legacy branch fm/prrelaunch59644
warning: ~/tmp/fm-lab.ojtloB/data/prrelaunch59644/launch-brief.md records no delivery contract line (scaffolded before ship briefs recorded one); launching on the explicit --mode local-only - confirm its definition of done matches
notice: prrelaunch59644 ships mode=local-only while the standing posture for proj is no-mistakes - less rigor than the captain's standing posture; proceed only on a current explicit captain instruction or an intake judgment you can state
skill selection for prrelaunch59644: unavailable - no TypeSafe or OpenRouter key
relaunched prrelaunch59644 harness=claude from=claude model=default effort=default backend=tmux endpoint=primary:fm-prrelaunch59644 worktree=~/.no-mistakes/worktrees/32d18ed9638d/01M4EXCNBHH0RJC0P3T69ZP04Q/.no-mistakes/test-tmp/fm-pr-relaunch-tmux.oaAUTI/wt
replacement Claude: alive in the preserved endpoint
monitor parser: accepted URL=https://github.com/example/repo/pull/90
watcher dispatch gate: accepted provider=github URL=https://github.com/example/repo/pull/90 number=90
carrier count=1 value=00-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-bbbbbbbbbbbbbbbb-01
Published task metadata:
window=primary:fm-prrelaunch59644
endpoint_task_id=prrelaunch59644
worktree=~/.no-mistakes/worktrees/32d18ed9638d/01M4EXCNBHH0RJC0P3T69ZP04Q/.no-mistakes/test-tmp/fm-pr-relaunch-tmux.oaAUTI/wt
project=~/.no-mistakes/worktrees/32d18ed9638d/01M4EXCNBHH0RJC0P3T69ZP04Q/.no-mistakes/test-tmp/fm-pr-relaunch-tmux.oaAUTI/proj
harness=claude
kind=ship
mode=local-only
yolo=off
branch=fm/prrelaunch59644
tasktmp=/tmp/fm-prrelaunch59644
model=default
effort=default
skill_selection=unavailable
skill_selection_reason=no TypeSafe or OpenRouter key
busy_gen=g1791515086.34232.4934
spawn_gen=s1791515087.30647.16573
control_relaunch_tx=28402.20261009T030444Z.25557
traceparent=00-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-bbbbbbbbbbbbbbbb-01
pr=https://github.com/example/repo/pull/90
pr_head=0123456789abcdef0123456789abcdef01234567
x_request=request-90
ok - LIVE: tracing-on relaunch attempt 2 preserves parseable PR and one stable carrier
stopped endpoint: genuine shell, classifier=dead endpoint=primary:fm-prrelaunch59644

ADVERSARIAL: replay old invalid publication order against monitor consumer
old transaction-after-PR record: REJECTED
old carrier-after-PR record: REJECTED

SCENARIO: traced relaunch without recorded PR
fm-gate-refuse: gate agent lifecycle permitted only against lab home ~/tmp/fm-lab.ojtloB
fm-gate-refuse: gate agent lifecycle permitted only against lab home ~/tmp/fm-lab.ojtloB
warning: ~/tmp/fm-lab.ojtloB/data/prrelaunch59644/launch-brief.md records no ship branch; defaulting to legacy branch fm/prrelaunch59644
warning: ~/tmp/fm-lab.ojtloB/data/prrelaunch59644/launch-brief.md records no delivery contract line (scaffolded before ship briefs recorded one); launching on the explicit --mode local-only - confirm its definition of done matches
notice: prrelaunch59644 ships mode=local-only while the standing posture for proj is no-mistakes - less rigor than the captain's standing posture; proceed only on a current explicit captain instruction or an intake judgment you can state
skill selection for prrelaunch59644: unavailable - no TypeSafe or OpenRouter key
relaunched prrelaunch59644 harness=claude from=claude model=default effort=default backend=tmux endpoint=primary:fm-prrelaunch59644 worktree=~/.no-mistakes/worktrees/32d18ed9638d/01M4EXCNBHH0RJC0P3T69ZP04Q/.no-mistakes/test-tmp/fm-pr-relaunch-tmux.oaAUTI/wt
Published no-PR task metadata:
window=primary:fm-prrelaunch59644
endpoint_task_id=prrelaunch59644
worktree=~/.no-mistakes/worktrees/32d18ed9638d/01M4EXCNBHH0RJC0P3T69ZP04Q/.no-mistakes/test-tmp/fm-pr-relaunch-tmux.oaAUTI/wt
project=~/.no-mistakes/worktrees/32d18ed9638d/01M4EXCNBHH0RJC0P3T69ZP04Q/.no-mistakes/test-tmp/fm-pr-relaunch-tmux.oaAUTI/proj
harness=claude
kind=ship
mode=local-only
yolo=off
branch=fm/prrelaunch59644
tasktmp=/tmp/fm-prrelaunch59644
model=default
effort=default
skill_selection=unavailable
skill_selection_reason=no TypeSafe or OpenRouter key
busy_gen=g1791515092.55699.11774
spawn_gen=s1791515093.50016.24167
control_relaunch_tx=46719.20261009T030450Z.10356
traceparent=00-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-bbbbbbbbbbbbbbbb-01
ok - LIVE: traced no-PR relaunch retains one carrier at end of record
stopped endpoint: genuine shell, classifier=dead endpoint=primary:fm-prrelaunch59644
fm-gate-refuse: gate agent lifecycle permitted only against lab home ~/tmp/fm-lab.ojtloB
already-stopped prrelaunch59644 harness=claude backend=tmux endpoint=primary:fm-prrelaunch59644 worktree=~/.no-mistakes/worktrees/32d18ed9638d/01M4EXCNBHH0RJC0P3T69ZP04Q/.no-mistakes/test-tmp/fm-pr-relaunch-tmux.oaAUTI/wt
cleanup: private fm-lab tmux server, marked lab home and worktree fixture removed
Evidence: Targeted regression: passing target and failing original writers

Source: Targeted regression: passing target and failing original writers

Targeted existing regression: test_relaunch_keeps_a_recorded_pr_parseable_for_monitoring
Executed alone using a disposable driver containing the existing test definitions; no repository-wide suite was run.
These hermetic checks use a stubbed runtime and are supporting regression evidence, NOT live-product scenarios.

TARGET
$ bash tests/.pr-relaunch-regression-validation.sh
ok - fm-control relaunch: a recorded PR stays parseable for PR monitoring with tracing off
ok - fm-control relaunch: a recorded PR stays parseable for PR monitoring with tracing on
exit=0

ORIGINAL BASE COMMIT WRITER
$ env FM_TEST_ALT_BIN="$PWD/tests/.pr-relaunch-baseline-bin" bash tests/.pr-relaunch-regression-validation.sh
not ok - PR monitoring must still parse the relaunched task record with tracing off (got: rejected)
exit=1 (expected reproduction)

TRANSACTION ORDER FIX PRESENT, TRACE WRITER RESTORED TO OLD APPEND BEHAVIOR
$ env FM_TEST_ALT_BIN="$PWD/tests/.pr-relaunch-trace-before-bin" bash tests/.pr-relaunch-regression-validation.sh
ok - fm-control relaunch: a recorded PR stays parseable for PR monitoring with tracing off
not ok - PR monitoring must still parse the relaunched task record with tracing on (got: rejected)
exit=1 (expected reproduction of confirmed R1)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed ✅
  • 🚨 bin/fm-spawn.sh:5519 - Tracing-enabled relaunches still break PR monitoring. Concrete sequence: a home has tracing frozen on and a task has a valid recorded PR; relaunch now publishes control_relaunch_tx before the preserved PR block, but subsequently spawn_record_traceparent copies that record and appends traceparent after pr=. fm_pr_metadata_identity_parse rejects that field (bin/fm-pr-lib.sh:393), preventing the watcher from dispatching the PR poll (bin/fm-watch.sh:2982). Relevant sites in the changed files: bin/fm-spawn.sh:5453 removes the prior carrier, :5519 preserves the PR block, :5870 appends the replacement carrier, and :5907 invokes that second publication; tests/fm-control-relaunch.test.sh:742 exercises relaunch without enabling tracing, so its parser assertion at :748 misses this supported path. Complete the publication-order fix in spawn_record_traceparent, the shared writer used by traced relaunches, by placing the carrier before the preserved record. Extend the behavioral regression to cover tracing both on and off.

🔧 Fix applied.
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 4 of 4 scenarios driven live against the product
Scenario Result Live Evidence
Recover a PR-linked task with tracing disabled; monitoring still accepts its recorded PR ✅ pass live Recovery transcript: tracing-off relaunch succeeded, the parser retained https://github.com/example/repo/pull/90, the watcher snapshot gate accepted it, and carrier count was zero.
Recover a traced PR-linked task with duplicate historic carriers; publish one stable carrier without breaking PR monitoring ✅ pass live Recovery transcript: traced attempt 1 launched a real Claude replacement and published exactly one unchanged traceparent before pr=; both monitoring consumers accepted the PR.
Recover the same traced PR-linked task again; retain its PR and unique trace identity ✅ pass live Recovery transcript: traced attempt 2 succeeded in the preserved endpoint, retained the same carrier exactly once, and passed PR parser and watcher snapshot validation.
Recover a traced task without a recorded PR; append its unique carrier at the end ✅ pass live Recovery transcript: no-PR relaunch succeeded and the generated record ended with exactly one unchanged traceparent line.
  • bash tests/.pr-relaunch-regression-validation.sh — isolated execution of test_relaunch_keeps_a_recorded_pr_parseable_for_monitoring; tracing-off and tracing-on cases passed.
  • env FM_TEST_ALT_BIN="$PWD/tests/.pr-relaunch-baseline-bin" bash tests/.pr-relaunch-regression-validation.sh — the original writer reproduced PR parser rejection with tracing off.
  • env FM_TEST_ALT_BIN="$PWD/tests/.pr-relaunch-trace-before-bin" bash tests/.pr-relaunch-regression-validation.sh — restoring only the old trace writer reproduced rejection with tracing on while tracing off passed.
  • bash tests/.pr-relaunch-live-validation.sh — attempted a named Herdr lab; the attempt timed out and bin/fm-herdr-lab.sh teardown fm-lab-pr-relaunch-live-43167-25585 completed.
  • bash tests/.pr-relaunch-tmux-validation.sh and bash tests/.pr-relaunch-tmux-retry-validation.sh — attempted idle-Claude readiness in private tmux labs; import-dialog and subsequent rendering readiness prevented those attempts from reaching relaunch.
  • claude auth status --json | jq '{loggedIn, authMethod, apiProvider}' — confirmed the existing Claude login; the successful disposable configuration also reported authenticated first-party OAuth access.
  • bash tests/.pr-relaunch-stopped-live-validation.sh — drove four successful fm-control.sh recovery relaunches against real Claude 2.1.295 on private tmux 3.5a endpoints.
  • fm_pr_metadata_identity_parse and fm_pr_poll_snapshot_capture — accepted each PR-linked record after live relaunch; duplicate historic carriers became one stable carrier, and repeated relaunch preserved it.
  • fm_pr_metadata_identity_parse — rejected disposable records reproducing both old post-PR publication orders.
  • Removed disposable drivers, fixture repositories, generated task scratch data, and lab homes; confirmed no matching worktree fixtures remained. No full suite, linters, formatters, or static analysis ran.
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

Relaunch appended control_relaunch_tx after the preserved lines, so a task
with pr= recorded ended with a key after its PR identity block.
fm_pr_metadata_identity_parse rejects any such key and PR monitoring stopped
silently. Write the transaction marker before the preserved lines so the pr=
block stays last, and cover it with a relaunch test that parses the result.
@MrGTV-love
MrGTV-love merged commit 2ce74dc into main Oct 9, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant