Repository navigation
Conversation
…never happened _record_fire_ownership_lost wrote "Interrupted by shutdown before terminal completion." into last_error and the executions ledger. Before 132a181 the disjunction it guarded could also fire on an unconfirmed renewal, so the shutdown wording at least had a plausible subject. After 132a181 the function is reached only from store-serialized evidence -- the refused side-effect fence, a transport cancel, or a terminal write the store refused -- none of which is a shutdown. An operator reading last_error was sent looking at gateway restarts for a condition that is not a restart (Sep 2026: 28 executions across 14 jobs carried it while their sessions ended cron_complete). The message now names the fire claim. The gateway-shutdown wording stays with _finish_interrupted_run, its only genuinely shutdown-scoped writer, so the two failure shapes are no longer conflated. The cron-internals doc quoted the old literal verbatim and is updated in the same change. Tests: tests/cron/test_fire_claim_outcome_recording.py pins both directions -- a delivered run whose renewal could not be confirmed is recorded from its own outcome (completed / delivered / no streak), and a refused side-effect fence is still recorded as interrupted, naming the fire claim and never a shutdown.
Review — fix(cron): ownership loss labelSummary: This renames the misleading What changed
Strengths
Findings
Verdict: Looks good to merge. Reviewed using Hermes-Agent |
13 tasks done
Draft
14 tasks done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
_record_fire_ownership_lost()writes"Interrupted by shutdown before terminal completion."intolast_errorand the executions ledger. That wording is now wrong for every path that reaches it.Before #105861 (
132a181d80, merged Sep 17) the guarded condition could also fire on an unconfirmed renewal, so a shutdown-flavoured message at least had a plausible subject. After it, the function is reached only from store-serialized evidence — a refused side-effect fence, a real transport cancel, or a terminal write the store refused. None of those is a shutdown. The gateway-shutdown wording belongs to_finish_interrupted_run, which is now its only genuinely shutdown-scoped writer.At the same time the codebase already uses the honest phrasing one branch over:
_finish_completed_run's CAS-refused path writes exactly"Fire claim ownership lost before terminal completion.". So one failure shape was being recorded under two different labels, and the misleading one is the one that lands inlast_errorand the ledger.Practical cost (Sep 2026): 28 executions across 14 jobs carried the shutdown string while their sessions all ended
cron_completeand their output was delivered, so an operator readinglast_errorwas sent looking at gateway restarts for a condition that is not a restart, and the failure streak/streak nudge accumulated on jobs that never failed.Changes Made
cron/scheduler.py:_OWNERSHIP_LOST_INTERRUPTED→"Fire claim ownership lost before terminal completion.", aligning it with the string_finish_completed_runalready writes; docstring states that an ownership loss is its own failure shape and that shutdown wording belongs to_finish_interrupted_runalone.website/docs/developer-guide/cron-internals.md: the fire-claim section quoted the old literal verbatim; updated to the new string and to the accurate set of paths that reach it.tests/cron/test_fire_claim_outcome_recording.py(new): pins both directions.How to Test
The new file uses a real
jobs.json+ a real execution ledger in temp dirs (no mocks for the bookkeeping):test_delivered_run_is_recorded_from_its_own_outcome— a renewal that cannot be confirmed during a long delivery must not turn an already-delivered run into a recorded failure: executioncompleted,delivery_outcome: delivered,last_status: ok, no failure streak.test_refused_side_effect_fence_is_still_recorded_as_interrupted— the protection stands: a refused fence does not deliver, the execution isfailed, and the recorded error names the fire claim and does not say "shutdown". This is the assertion that is red before this PR (AssertionError: Interrupted by shutdown before terminal completion.).Duplicate check — read this before merging
Searched open and merged PRs for "fire claim" / ownership-loss ledger wording. #108912 ("fix(cron): distinguish fire-claim ownership loss from shutdown in ledger", open since 2026-09-12) works the same area with a broader, different approach. Disclosure, not concealment:
mergeable: CONFLICTINGand has been unmodified for 7 days.132a181d80(Cron agent jobs: successful delivery overwritten with "Interrupted by shutdown before terminal completion." when fire-claim heartbeat lapses mid-run #105861) andeafed27cf0._OWNERSHIP_LOST_INTERRUPTEDas the shutdown string for the pre-delivery loss.This PR is the remaining narrow slice: one constant, so an ownership loss is never labelled a shutdown. If maintainers prefer the #108912 shape, close this one — the underlying defect is what matters. Happy to fold it in instead if that PR is revived.
Related Issue
Fixes # (no issue filed; found by auditing recorded cron failures against their sessions)
Type of Change
Checklist
fix(cron):)scripts/run_tests.shgreen (fulltests/cron/area)cron-internals.mdquoted the literal)