Skip to content

fix(kanban): per-run crash grace and scope worker-log exit trailer to the current run - #126604

Open
chryzxc wants to merge 1 commit into
NousResearch:mainfrom
chryzxc:fix/kanban-retry-grace-and-log-trailer
Open

chryzxc wants to merge 1 commit into
NousResearch:mainfrom
chryzxc:fix/kanban-retry-grace-and-log-trailer

Conversation

@chryzxc

@chryzxc chryzxc commented Sep 28, 2026

Copy link
Copy Markdown

Problem

When a worker run ends and the task is retried, two independent issues turn a live worker into a booked protocol violation:

  1. Retries get no crash grace. _reclaim_dead_workers keys the 30 s launch-window grace on tasks.started_at, which is set via COALESCE and therefore keeps the first run's start. Every retry is liveness-checked immediately, with zero grace.
  2. Stale exit trailer from the previous run. The per-task worker log is append-mode across re-runs. A non-parent dispatcher (e.g. per-tick hermes kanban dispatch) reads the last [kanban-worker-exit] rc= trailer; if the current run is killed or misjudged dead, it picks up the previous run's rc=0 and books "exited cleanly without kanban_complete" — quoting the previous run's final output too.

Fix

  • Key the grace on the current run's task_runs.started_at (falling back to tasks.started_at).
  • _default_spawn writes a [kanban-worker-spawn] run=<id> marker before each worker starts; _worker_log_exit_code and _worker_final_output only read the section after the last marker. Legacy logs without a marker behave as before.

Repro (real board, macOS)

A live worker was reclaimed because of the ~1 s start-time drift (#117505). The retry was then checked with no grace, and a CLI dispatcher read the previous run's rc=0 trailer and booked a protocol violation. The still-running worker's kanban_block / kanban_complete then failed with "unknown id or not running". This repeated until the violation budget tripped and the card blocked.

Related

The drift itself is addressed by #118353 / #117091. This PR fixes the separate gaps that amplify a single misjudged liveness check into a protocol-violation loop.

Tests

  • test_previous_runs_trailer_is_not_booked_against_the_current_run and test_retry_gets_the_launch_grace_from_its_own_run fail on main and pass with this change.
  • test_kanban_db.py::test_infrastructure_spawn_refusal_never_charges_the_card already fails on main and is unrelated.

… the current run

Retries were liveness-checked with no launch grace (tasks.started_at keeps
the first run's start), and non-parent dispatchers read the previous run's
rc=0 exit trailer from the append-mode worker log, booking crashes/false
reclaims as protocol violations. Key the grace on the current run and cut
log reads at a per-spawn marker.
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management sweeper:risk-automation Sweeper risk: may affect CI, automerge, label sync, or maintainer automation labels Sep 28, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management P3 Low — cosmetic, nice to have sweeper:risk-automation Sweeper risk: may affect CI, automerge, label sync, or maintainer automation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants