Repository navigation
Conversation
… the current run Retries were liveness-checked with no launch grace (tasks.started_at keeps the first run's start), and non-parent dispatchers read the previous run's rc=0 exit trailer from the append-mode worker log, booking crashes/false reclaims as protocol violations. Key the grace on the current run and cut log reads at a per-spawn marker.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a worker run ends and the task is retried, two independent issues turn a live worker into a booked protocol violation:
_reclaim_dead_workerskeys the 30 s launch-window grace ontasks.started_at, which is set viaCOALESCEand therefore keeps the first run's start. Every retry is liveness-checked immediately, with zero grace.hermes kanban dispatch) reads the last[kanban-worker-exit] rc=trailer; if the current run is killed or misjudged dead, it picks up the previous run'src=0and books "exited cleanly without kanban_complete" — quoting the previous run's final output too.Fix
task_runs.started_at(falling back totasks.started_at)._default_spawnwrites a[kanban-worker-spawn] run=<id>marker before each worker starts;_worker_log_exit_codeand_worker_final_outputonly read the section after the last marker. Legacy logs without a marker behave as before.Repro (real board, macOS)
A live worker was reclaimed because of the ~1 s start-time drift (#117505). The retry was then checked with no grace, and a CLI dispatcher read the previous run's
rc=0trailer and booked a protocol violation. The still-running worker'skanban_block/kanban_completethen failed with "unknown id or not running". This repeated until the violation budget tripped and the card blocked.Related
The drift itself is addressed by #118353 / #117091. This PR fixes the separate gaps that amplify a single misjudged liveness check into a protocol-violation loop.
Tests
test_previous_runs_trailer_is_not_booked_against_the_current_runandtest_retry_gets_the_launch_grace_from_its_own_runfail onmainand pass with this change.test_kanban_db.py::test_infrastructure_spawn_refusal_never_charges_the_cardalready fails onmainand is unrelated.