Skip to content

🐛 fix(kanban): reconcile a crashed provably-done card to review, not gave_up - #80

Merged
cwest merged 1 commit into
cwest/integrationfrom
topic/crash-path-landed-work-reconcile
Jul 26, 2026
Merged

🐛 fix(kanban): reconcile a crashed provably-done card to review, not gave_up#80
cwest merged 1 commit into
cwest/integrationfrom
topic/crash-path-landed-work-reconcile

Conversation

@cwest

@cwest cwest commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Why

A no-PR edit-in-place card whose worker crashed (pid not alive) after
posting a landed-work handoff was false-gave_up and stranded in blocked even
though its deliverable was complete, on disk, and independently re-verified. A
human had to hand-reconcile it.

The prior landed-work proof (_lane_work_provably_done) was wired into the
clean-exit reap path only. It was never consulted on the crash path: a
worker that died a real captured death (signaled — SIGKILL / OOM killer — or
nonzero_exit) fell into the generic crashed branch, which counts a failure
and, on retry-budget exhaustion, emits gave_up. The same card, same
deliverable, same proof, was correctly reconciled on a clean exit but wrongly
stranded on a crash. Crash-vs-clean-exit is orthogonal to whether the work
landed.

What

Wire the existing _lane_work_provably_done proof into the crash/gave_up path
in hermes_cli/kanban_db.py:

  1. detect_crashed_workers now consults the same helper (unchanged — no
    forked predicate) for a would-be crash reap (signaled / nonzero_exit /
    unknown), not just clean_exit.
  2. Proof present → the lane's work landed; the card reconciles forward to
    the owner map's review owner (never done — the deliverable is real but
    UNREVIEWED), rather than gave_up + blocked.
  3. Proof absent → strict current behavior preserved exactly. A genuine crash
    with no deliverable still gave_ups.
  4. PR-requiring cards untouched_card_requires_pr still gates the no-PR
    handoff-comment carve-out, so PR-backed crash behavior is byte-for-byte
    identical. The no-PR edit-in-place card advances to review only on the
    crash path; the clean-exit path keeps releasing it to ready unchanged.

No new config, env var, or schema — reuses the existing window constant and shape
predicate.

Done when

  • A no-PR edit-in-place card whose worker crashes after posting a landed-work
    handoff reconciles to review + the owner-map reviewer, not gave_up. ✅
  • A no-PR card that crashes with NO landed-work proof STILL gave_ups —
    regression-tested explicitly. ✅
  • A PR-requiring card that crashes with a handoff comment but no PR STILL
    gave_ups — the no-PR carve-out does not leak into the PR-backed path. ✅
  • Behavior-contract tests against real kanban_db + temp HERMES_HOME, no
    mocks. ✅
  • Full test_kanban_db.py green (283 passed), ruff clean, 0 regressions. ✅

Scope / Restart-gating

Single-file change in hermes_cli/kanban_db.py plus tests. This fix is inert
until the gateway is restarted
detect_crashed_workers runs inside the
long-lived dispatcher, so a merged change does not take effect on the running
process until it reboots.

Out of scope: why the worker pid dies silently. This stops the crash path from
discarding proven work; it does not prevent the crash.

…gave_up

detect_crashed_workers consulted _lane_work_provably_done only on the
clean-exit / uncaptured-exit reap path. A worker that died a REAL captured
death (signaled: SIGKILL / OOM killer; or nonzero_exit) after its lane work
already landed fell into the generic crashed branch, which counted a failure
and, on retry-budget exhaustion, emitted gave_up and stranded the card in
blocked — even though the deliverable was on disk and re-verifiable.

Crash-vs-clean-exit is orthogonal to whether the work landed: the same
proof the clean-exit path already trusts now gates the crash path too. When
a would-be crash (signaled / nonzero_exit / unknown) has landed-work proof,
it is treated as a benign no-op and the card reconciles FORWARD to its
owner-map review owner (deliverable real but UNREVIEWED — never done). With
no proof, signaled / nonzero_exit fall through to the strict crashed branch
unchanged, so a genuine crash with no deliverable still gives up.

A no-PR edit-in-place card advances to review only on the crash path; the
clean-exit path keeps releasing it to ready unchanged. The shape predicate
keeps a PR-requiring card held to a real PR / completed-run artifact, so the
no-PR handoff-comment carve-out never leaks into the PR-backed crash path.

Behavior-contract tests against real kanban_db + temp HERMES_HOME cover the
reconcile-to-review case and both proof-absent regressions (no-PR and
PR-requiring still gave_up).
@cwest
cwest marked this pull request as ready for review July 26, 2026 04:22

@cwest cwest left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No changes needed.

The fix targets the real asymmetry: detect_crashed_workers only consulted _lane_work_provably_done on the clean_exit/unknown reap path, so a captured worker death (signaled/nonzero_exit) after the lane work landed fell into the generic crashed branch and gave_up on retry-budget exhaustion. Widening the carve-out to the crash kinds and reusing the same proof helper — unchanged — is the right shape.

The gating holds where it matters. Proof 3 (the no-PR handoff-comment) stays scoped by _card_requires_pr, so a PR-requiring worktree card with a handoff comment but no PR never reads as provably-done and still gave_ups. The clean-exit no-PR path still releases to ready (no review lane), and only the crash path advances a proven no-PR card to review — matching #75/#76.

Verified by ground truth in a throwaway clone at the head SHA: full test_kanban_db.py is 283 passed / 0 failed; ruff clean on both files. Reverting the source fix reproduces the incident — test_crash_no_pr_edit_in_place_with_handoff_advances_to_review fails on the exact assertion (crashed card treated as a crash), while the two proof-absent regressions stay green. Dropping the _card_requires_pr gate makes test_crash_pr_requiring_card_with_handoff_no_pr_still_gives_up fail, so the leak guard has teeth. Commit is signed, Conventional, no attribution.

Worth restating what the PR body already flags: this runs inside the long-lived dispatcher, so it is inert until the gateway restarts.

@cwest
cwest merged this pull request into cwest/integration Jul 26, 2026
31 checks passed
@cwest
cwest deleted the topic/crash-path-landed-work-reconcile branch July 26, 2026 04:34
cwest added a commit that referenced this pull request Jul 26, 2026
…gave_up (#80)

detect_crashed_workers consulted _lane_work_provably_done only on the
clean-exit / uncaptured-exit reap path. A worker that died a REAL captured
death (signaled: SIGKILL / OOM killer; or nonzero_exit) after its lane work
already landed fell into the generic crashed branch, which counted a failure
and, on retry-budget exhaustion, emitted gave_up and stranded the card in
blocked — even though the deliverable was on disk and re-verifiable.

Crash-vs-clean-exit is orthogonal to whether the work landed: the same
proof the clean-exit path already trusts now gates the crash path too. When
a would-be crash (signaled / nonzero_exit / unknown) has landed-work proof,
it is treated as a benign no-op and the card reconciles FORWARD to its
owner-map review owner (deliverable real but UNREVIEWED — never done). With
no proof, signaled / nonzero_exit fall through to the strict crashed branch
unchanged, so a genuine crash with no deliverable still gives up.

A no-PR edit-in-place card advances to review only on the crash path; the
clean-exit path keeps releasing it to ready unchanged. The shape predicate
keeps a PR-requiring card held to a real PR / completed-run artifact, so the
no-PR handoff-comment carve-out never leaks into the PR-backed crash path.

Behavior-contract tests against real kanban_db + temp HERMES_HOME cover the
reconcile-to-review case and both proof-absent regressions (no-PR and
PR-requiring still gave_up).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant