fix(cron): surface missed-fire catch-up lateness in hermes cron list/status - #100445
Merged
Merged
Conversation
…` / `status` The catch-up machinery already re-ran jobs missed during gateway downtime, but the late execution rendered as an ordinary on-time success — no scheduled-vs-actual time, no lateness, no disposition (issue #99879, the visibility half). - Due-scan now persists a `last_dispatch` stamp on every recurring dispatch: scheduled_at, dispatched_at, lateness_seconds, and kind (on_time / late / catch_up, classified against the ticker tolerance and the schedule's catch-up grace window). Manual triggers and one-shots are not stamped (no scheduled instant to be late against / retired beyond grace). - `hermes cron list` renders a per-job Dispatch line; late/catch-up runs show "⚠ catch-up after missed fire: scheduled ..., ran ... (31m late)". - `hermes cron status` calls out jobs whose last dispatch was late or a catch-up, in both the built-in ticker and external provider paths. CLI surface only — no new tools, no policy engine. Addresses the visibility half of #99879.
Contributor
૮ >ﻌ< ა ci reviewran on e60a0e2 — fix(cron): surface missed-fire catch-up lateness in `hermes
|
alt-glitch
added
type/bug
This was referenced Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses the visibility half of #99879 (per the triage comment: item 2, the highest-leverage residual). The catch-up mechanism already existed; this PR makes late catch-up runs visible in the CLI instead of rendering as ordinary on-time successes.
What changed
cron/jobs.py— the due-scan (_get_due_jobs_locked) now persists alast_dispatchstamp on every recurring dispatch:scheduled_at,dispatched_at,lateness_seconds, andkind(on_time/late/catch_up, classified against a 5-min ticker tolerance and the schedule's existing catch-up grace window). Manual triggers aren't stamped (no scheduled instant to be late against); one-shots beyond grace are already retired. The stamp is written under the same jobs lock +save_jobspass the due-scan already holds — no new locking, no schema/state.db change (jobs.json field only).hermes_cli/cron.py—hermes cron listrenders a per-jobDispatch:line: dimon time (scheduled ...)for normal fires, loud⚠ catch-up after missed fire: scheduled 2026-08-31T09:00:00, ran 2026-09-01T08:20:38 (23h 20m late)for late/catch-up fires.hermes cron statusgains a late-jobs callout (⚠ N job(s) last fired late (missed-fire catch-up): ...) in the shared active-jobs summary, so it covers both the built-in ticker and external-provider status paths.Scope discipline: CLI output + the one minimal persisted field needed to display it. No new tools, no missed-run policy engine.
Live repro: real
hermes cron create+ backdatednext_run_at(daily job 23h overdue) + realhermes cron tick/list/statusunder an isolatedHERMES_HOME=/tmp/repro-99879— before (origin/main): the catch-up fired andcron listshowed onlyLast run: ... okwith no hint it was late (catch_up_occurrencescounter incremented but nothing user-visible); after:cron listshowsDispatch: ⚠ catch-up after missed fire: scheduled 2026-08-31T09:00:00-07:00, ran 2026-09-01T08:20:38-07:00 (23h 20m late)andcron statusshows⚠ 1 job(s) last fired late (missed-fire catch-up): ....Tests
tests/cron/test_dispatch_lateness_stamp.py— due-scan stamps catch_up/late/on_time correctly, persists to storage, skips manual triggers, overwrites stale stamps; classification boundary tests.tests/hermes_cli/test_cron_dispatch_visibility.py—cron listDispatch line rendering (loud vs quiet vs absent),cron statuslate-jobs callout, formatting helpers, malformed-stamp tolerance.tests/cron/+ hermes_cli cron tests — 1121 passed, 0 failed.Related: PR #99919 tags the beyond-grace catch-up branch with an in-memory-only
_dispatch_meta; this PR persists the stamp for all recurring dispatches (so on-time vs late is distinguishable cross-process) and adds the actual CLI rendering, which #99919 does not touch.Infographic