Skip to content

Cron dispatcher: a late tick catches up the firings it missed (JOB-001) - #880

Merged
NewWorldVenture merged 1 commit into
mainfrom
claude/cron-dispatch-catch-up-20261003
Oct 3, 2026
Merged

NewWorldVenture merged 1 commit into
mainfrom
claude/cron-dispatch-catch-up-20261003

Conversation

@NewWorldVenture

Copy link
Copy Markdown
Owner

Why

scripts/cron-dispatch.mjs is what gives the app its sub-daily cron cadence on Vercel Hobby: .github/workflows/cron-dispatch.yml asks GitHub to run it every five minutes, and it calls every /api/cron/* route whose schedule fell due. It evaluated a fixed five-minute window ending at the moment it ran, and GitHub starts scheduled workflows hours late (measured gaps of 104–396 minutes, median 209), so every sub-daily firing between two ticks was simply lost. tests/a-late-tick-drops-a-cron.test.ts characterized that; nothing fixed it. Ledger row JOB-001 (🔄 IN PROGRESS, High) names this as the remaining scheduler scope in docs/final-audit/cron-dispatch-cycle.md. finalaudit.md is untouched; this is the repository-side fix with its evidence.

What changes

scripts/cron-dispatch.mjs

  • tickWindow(now, since): the window is (since, now], never narrower than the fixed five minutes (two quick runs cannot lose the minutes between them) and clamped at CATCH_UP_MAX_MINUTES = 24 h. Clamped rather than ignored, because the gaps this exists for are the long ones: a weeks-old boundary after the workflow was disabled still recovers one call per sub-daily route. A missing, unparseable or future CRON_SINCE keeps the fixed window and says why in the log line.
  • dueRoutes(…, since): each route is called at most once per tick however many of its minutes fell in the window; the routes are sweeps, not slot replays.
  • isSubDaily: only routes that fire more than once a day are caught up. The once-a-day ones are exactly what Vercel Hobby accepts and vercel.json mirrors at the same minute, so Vercel already fired them; catching one up would be a second daily run (a second admin-digest email). Inside the fixed window they are evaluated exactly as before.
  • Why 24 h: model-refresh (0 4,16) and autopilot-scan (30 6,18) have 12-hour periods, and every caught-up route has a due minute in any 24 h window, so a wider cap cannot add a route. A test checks the cap against the longest gap in the table, so a slower route cannot be added silently.
  • SCHEDULES is byte-identical; --dry-run stays credential-free; --route ignores the boundary; every existing validation (secret presence, base-URL rules, redirect refusal, bounded responses, deadline, redaction) is unchanged.

.github/workflows/cron-dispatch.yml

  • A step before dispatch asks the Actions API (actions: read, the job's own token) for the previous successful scheduled run of this workflow, excluding the current run, validates the timestamp shape and passes it as CRON_SINCE. Manual single-route runs are not a boundary (they would collapse the next window). A failed lookup logs and falls back to the five-minute window; it never fails the tick. The cron string, sparse checkout, concurrency and repository guard are unchanged.

Tests

  • tests/a-late-tick-drops-a-cron.test.ts → tests/a-late-tick-catches-up.test.ts (git mv; 7 cases; the late tick is now caught up; the measurement paragraph and the corrected severity model are kept).
  • tests/cron-dispatch.test.ts +12: window, cap, clamp, once-per-route, the daily exclusion equals the vercel.json mirror set, the cap against the table's longest gap; "vercel.json ↔ dispatcher table" untouched.
  • tests/cron-dispatch-execution.test.ts +6 CLI cases: catch-up, clamp, fallbacks for bad input, dry run with a boundary, --route ignoring it.
  • Comment-only pointer updates in two tests and one README for the renamed file.

Docs: docs/final-audit/cron-dispatch-cycle.md gains a dated section on the design and its evidence (additive).

Evidence (local; nothing touched GitHub Actions or production)

  • 16 related files, 255 tests pass (the cron suites, the admin-digest files that pin dispatcher lines, a-mirrored-cron-must-be-idempotent, a-paths-filter-must-list-what-it-gates, production-readiness-workflow, test-harness-config-is-what-it-says, health-feature-secrets, cron-auth). ESLint clean; node --check ok; the workflow parses (PyYAML and js-yaml); git diff --check clean.
  • Dry runs: without CRON_SINCE, the log line is byte-identical to before; with a 187-minute boundary the twelve sub-daily routes due in the gap are listed once each and the daily notifications is not; not-a-date and a future timestamp fall back and say why; a 2-minute-old boundary stays at the five-minute window; a boundary older than the cap is clamped and lists all 17 sub-daily routes and no daily one; --route ignores the boundary.
  • The previous-tick step was run under bash -eo pipefail with a fake gh: success writes the boundary; a 403, an empty body, null and a missing gh all log the fallback and exit 0.
  • Mutation controls, each on the script alone: dropping once-per-route fails 4 tests; dropping the cap fails 3; dropping the future check fails 2; dropping the daily exclusion fails 4.

What this does not change

  • The delivered rate: a */5 route still runs once per tick however many of its minutes the tick covered (a-cron-cadence-is-a-promise-nothing-keeps is unchanged). Recovering the rate needs a scheduler that keeps its word, as the workflow's own note says.
  • The known same-minute admin-digest double send between vercel.json and the dispatcher.
  • No schedule, secret, trigger or production setting; the workflow change takes effect only when merged.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KP9rt5rVQ9jDMpZp2xBy3K


Generated by Claude Code

The GitHub Actions dispatcher evaluated a fixed five-minute window ending at
the moment it ran, and GitHub starts the scheduled workflow hours late, so
every sub-daily firing between two ticks was lost (the mechanism that
tests/a-late-tick-drops-a-cron.test.ts characterized; the measured gaps run
104 to 396 minutes).

scripts/cron-dispatch.mjs:
- tickWindow(now, since): the window is (since, now], never narrower than
  the fixed TICK_MINUTES (two quick runs cannot lose the minutes between
  them) and clamped at CATCH_UP_MAX_MINUTES = 24 h. Clamped, not ignored,
  because the gaps this exists for are the long ones: a weeks-old boundary
  after the workflow was disabled still recovers one call per sub-daily
  route. A missing, unparseable or future CRON_SINCE leaves the fixed window
  and says why in the log line.
- dueRoutes(..., since): each route is called at most once per tick however
  many of its minutes fell in the window; the routes are sweeps, not slot
  replays.
- isSubDaily: only routes that fire more than once a day are caught up. The
  once-a-day ones are exactly what Vercel Hobby accepts and vercel.json
  mirrors at the same minute, so Vercel already fired them; catching one up
  would be a second daily run (a second admin-digest email). Inside the
  fixed window they are evaluated exactly as before.
- 24 h, not less: model-refresh (0 4,16) and autopilot-scan (30 6,18) have
  12-hour periods, and every caught-up route has a due minute in any 24 h
  window, so a wider cap cannot add a route. A test checks the cap against
  the longest gap in the table.
- SCHEDULES is byte-identical; --dry-run stays credential-free; --route
  ignores the boundary; every existing validation is unchanged.

.github/workflows/cron-dispatch.yml: a step before dispatch asks the Actions
API (actions: read, the job's own token) for the previous successful
SCHEDULED run of this workflow, excluding the current run, validates the
timestamp and passes it as CRON_SINCE. Manual single-route runs are not a
boundary (they would collapse the next window). A failed lookup logs and
falls back to the five-minute window; it never fails the tick. The cron
string, sparse checkout and concurrency are unchanged.

Tests: tests/a-late-tick-drops-a-cron.test.ts becomes
tests/a-late-tick-catches-up.test.ts (git mv; 7 cases, the late tick now
caught up; the measurement paragraph kept); tests/cron-dispatch.test.ts +12
(window, cap, clamp, once-per-route, daily exclusion equals the vercel.json
mirror set, the cap against the table's longest gap);
tests/cron-dispatch-execution.test.ts +6 CLI cases (catch-up, clamp,
fallbacks, dry run with a boundary, --route ignoring it). 16 related files,
255 tests pass; ESLint clean; the workflow parses; the previous-tick step
was run under bash -eo pipefail with a fake gh for the success, 403, empty
and null cases. Four mutation controls (once-per-route, the cap, the future
check, the daily exclusion) each fail tests.

docs/final-audit/cron-dispatch-cycle.md gains a dated section on the design
and its evidence. What this does not change: the delivered rate (a '*/5'
route still runs once per tick) and the known same-minute admin-digest
double send, both documented as open.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KP9rt5rVQ9jDMpZp2xBy3K
@vercel

vercel Bot commented Oct 3, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
bubaly Ignored Ignored Oct 3, 2026 12:26pm UTC

Request Review

@NewWorldVenture
NewWorldVenture marked this pull request as ready for review October 3, 2026 12:53
@NewWorldVenture
NewWorldVenture merged commit 4b9a136 into main Oct 3, 2026
7 checks passed
NewWorldVenture pushed a commit that referenced this pull request Oct 3, 2026
Brings in 4b9a136 (#873, #877, #880). No overlap with this branch's files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TRY21ZKsFrfB3qtoP972A4
NewWorldVenture added a commit that referenced this pull request Oct 4, 2026
…t one (JOB-001 follow-up) (#902)

* Cron dispatch: a tick that could not find its catch-up boundary is not one (JOB-001 follow-up)

Review 5969190760 on #699, against merged #880: the boundary step swallowed
every lookup outcome into "dispatch the fixed five-minute window and finish
green", and the step that reads boundaries takes only status=success runs.
So a tick whose lookup FAILED — a 403, an outage, an unexpected shape —
dispatched its five minutes, finished green, and was taken by the next tick
as the boundary; every minute between the last real success and that tick
was lost for good. Last covered 03:00; the 09:07 lookup fails and dispatches
five minutes; the 12:07 lookup works and catches up from 09:07.

The step now tells the two look-alikes apart. The API answered and there
was no previous scheduled success — the first tick ever — is `lookup=ok`
with an empty boundary and starts the chain. The API did not answer, or
answered something unusable, is `lookup=failed` with an empty boundary. The
dispatch step runs either way, unconditionally, so nothing due this tick is
skipped. A last step then fails the run when `lookup=failed`, with an
::error:: saying why: the boundary query's existing status=success filter
makes a run failed on purpose invisible to the next tick, which catches up
from the last real success instead — within the 24-hour cap, repeating the
failed tick's five minutes for routes that are idempotent sweeps, losing
nothing. A lookup that keeps failing keeps dispatching and keeps the
workflow red, which is what a broken `actions: read` should look like.

tests/a-tick-that-could-not-find-its-boundary-is-not-one.test.ts extracts
the step's `run:` block from the YAML by indentation and executes it under
`bash -eo pipefail` with a fake `gh` on a path that excludes the real one:
a well-formed start is the boundary; an empty answer starts the chain; a
403, an absent gh and a `null` are each lookup=failed with no boundary and
exit 0. The run-level half is pinned on the YAML: dispatch follows the
boundary step and carries no condition, the failing step follows the
dispatch, is conditioned on lookup == 'failed' and exits 1, and the query
still filters on status=success. With the old shell back (every outcome
green), three cases fail. docs/final-audit/cron-dispatch-cycle.md records
the edge and corrects its earlier "never fails the tick".

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KP9rt5rVQ9jDMpZp2xBy3K

* Cron dispatch: a selected run with an unusable start is a failed lookup, not the start of the chain (review 5970787916, 5970791078)

Two look-alikes were still conflated inside "ok". The --jq expression mapped
a selected run whose run_started_at was null or missing to an empty string,
which the shell read as "no previous run" — the first-tick bootstrap — and
the tick finished green as a boundary. And a shape-valid start in the future
passed as lookup=ok; the dispatcher's tickWindow then ignored it and fell
back to five minutes while the run finished green: the same incomplete
boundary by another route.

The expression now answers one word per outcome — `none` when no run was
selected, the timestamp when it is a string, `unusable` otherwise — and only
`none` starts the chain. The shell applies the two tests tickWindow applies
(a real timestamp per GNU date, not after now), so a boundary the step
accepts is one the dispatcher uses; anything else is lookup=failed. The
step's stderr is no longer discarded, so the API's own error reaches the log.

The test runs the step under /bin/bash with PATH reduced to one directory of
its own: a fake gh that answers from a fixture body the way the API would
(status/event filters applied, the step's --jq evaluated by the real jq) and
symlinks to date, jq and sed, so the machine's gh cannot be reached and the
absent case is proved by the step's own "command not found". New cases: a
null, missing, malformed or future start, a body with no run list and a null
body are each lookup=failed and never "starts the chain"; the step's
accept/refuse decision is checked against tickWindow's on the same values;
and a three-run regression replays the review's chain (A 03:00 success, B
09:07 failed itself, a manual run excluded by event, the 12:07 tick catches
up from A: 547 minutes) with the negative control of a green B. Ordering
pins use at()/between() (ordering-guards-fail-on-absence). Cycle note
updated.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KP9rt5rVQ9jDMpZp2xBy3K

* Cron dispatch: a history of ticks that never succeeded is not a first tick

Review on #902 (5971121597): the boundary lookup asked only for the
previous scheduled SUCCESS, and read an empty answer as the first tick
ever. `status=success` is a filter over the whole history, so that empty
answer is also what a workflow whose every tick so far failed or was
cancelled returns — a 09:07 whose own lookup failed and failed itself, a
tick that died on a blank CRON_SECRET. The 12:07 tick then dispatched its
fixed five minutes, finished green and became the next boundary, and the
three hours after 09:07 were never caught up.

The step now asks twice, in order. The success-only question first; on
`none`, every previous scheduled run of any conclusion (the hundred most
recent). No run at all is the first tick and starts the chain, as before.
Runs with no success among them make the OLDEST usable start the
boundary: no successful tick has covered a minute after it, and a failed
or cancelled run dispatched at most its own five minutes, so this
over-covers and never under-covers (the routes are idempotent sweeps and
the dispatcher caps the window at a day). Runs with no usable start at
all are a failed lookup, and the run fails itself as before. The review
asked for the fallback window with no boundary in that case; that rule
could never be left — no tick can succeed until one has — so the oldest
recorded tick is taken instead, and the next green tick is the boundary.

tests/a-tick-that-could-not-find-its-boundary-is-not-one.test.ts (+6):
the review's chain (09:07 failed, 12:07 catches up from 09:07, three
hours, not "starts the chain"); several failed and cancelled ticks take
the oldest usable start; a manual run is in neither question; failed runs
with no usable start are a failed lookup; a success wins over failures
around it; the first tick ever still starts the chain, and the second
question is asked only inside the `none` branch. The seventeen existing
cases hold unchanged. Mutation control (the `none` branch made to start
the chain again, file restored and checksummed): 5 failures.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KP9rt5rVQ9jDMpZp2xBy3K

* Cron dispatch record: the failed-only-history refinement (review 5971121597)

The dated subsection in docs/final-audit/cron-dispatch-cycle.md records the
two-question boundary lookup, the oldest-usable-start rule for a history
with no success, why "never a boundary" could not be the rule there, and
the six regressions that pin it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KP9rt5rVQ9jDMpZp2xBy3K

* Cron dispatch: the reviews' boundary cases, run against the script that landed

#952 replaced the boundary step's shell with scripts/cron-run-history.mjs
and this branch now carries main's workflow unchanged. The shell-driving
test could no longer run (there is no `run: |` block to extract), so it is
re-aimed at the script's `discoverCatchUpBoundary` through a fake Actions
API that applies `status`, `event` and `created` and pages the way the real
one does.

The cases are the three reviews' cases: the previous success as boundary;
no run or only this run as the start of the chain; a null, missing,
malformed or future start, a body of another shape and a 403 as failed
lookups; accept/refuse checked against tickWindow; the three-run chain with
its negative control; failed-only history taking the oldest run — across
three pages for a full day of ticks, which the shell's single page could
not see — clamped when older than the cap, with manual runs in neither
question; a success winning over failures around it; the first tick's
questions in order. The YAML pins stand: dispatch before the failing step,
the failing step conditioned on a failed lookup, the dispatch step
unconditioned, the boundary step running the script.

The audit record gains a dated note saying the shell was superseded and why.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KP9rt5rVQ9jDMpZp2xBy3K

---------

Co-authored-by: Daniel Hughen <daniel.hughen@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants