Repository navigation
Cron dispatcher: a late tick catches up the firings it missed (JOB-001) - #880
Merged
Merged
Conversation
The GitHub Actions dispatcher evaluated a fixed five-minute window ending at the moment it ran, and GitHub starts the scheduled workflow hours late, so every sub-daily firing between two ticks was lost (the mechanism that tests/a-late-tick-drops-a-cron.test.ts characterized; the measured gaps run 104 to 396 minutes). scripts/cron-dispatch.mjs: - tickWindow(now, since): the window is (since, now], never narrower than the fixed TICK_MINUTES (two quick runs cannot lose the minutes between them) and clamped at CATCH_UP_MAX_MINUTES = 24 h. Clamped, not ignored, because the gaps this exists for are the long ones: a weeks-old boundary after the workflow was disabled still recovers one call per sub-daily route. A missing, unparseable or future CRON_SINCE leaves the fixed window and says why in the log line. - dueRoutes(..., since): each route is called at most once per tick however many of its minutes fell in the window; the routes are sweeps, not slot replays. - isSubDaily: only routes that fire more than once a day are caught up. The once-a-day ones are exactly what Vercel Hobby accepts and vercel.json mirrors at the same minute, so Vercel already fired them; catching one up would be a second daily run (a second admin-digest email). Inside the fixed window they are evaluated exactly as before. - 24 h, not less: model-refresh (0 4,16) and autopilot-scan (30 6,18) have 12-hour periods, and every caught-up route has a due minute in any 24 h window, so a wider cap cannot add a route. A test checks the cap against the longest gap in the table. - SCHEDULES is byte-identical; --dry-run stays credential-free; --route ignores the boundary; every existing validation is unchanged. .github/workflows/cron-dispatch.yml: a step before dispatch asks the Actions API (actions: read, the job's own token) for the previous successful SCHEDULED run of this workflow, excluding the current run, validates the timestamp and passes it as CRON_SINCE. Manual single-route runs are not a boundary (they would collapse the next window). A failed lookup logs and falls back to the five-minute window; it never fails the tick. The cron string, sparse checkout and concurrency are unchanged. Tests: tests/a-late-tick-drops-a-cron.test.ts becomes tests/a-late-tick-catches-up.test.ts (git mv; 7 cases, the late tick now caught up; the measurement paragraph kept); tests/cron-dispatch.test.ts +12 (window, cap, clamp, once-per-route, daily exclusion equals the vercel.json mirror set, the cap against the table's longest gap); tests/cron-dispatch-execution.test.ts +6 CLI cases (catch-up, clamp, fallbacks, dry run with a boundary, --route ignoring it). 16 related files, 255 tests pass; ESLint clean; the workflow parses; the previous-tick step was run under bash -eo pipefail with a fake gh for the success, 403, empty and null cases. Four mutation controls (once-per-route, the cap, the future check, the daily exclusion) each fail tests. docs/final-audit/cron-dispatch-cycle.md gains a dated section on the design and its evidence. What this does not change: the delivered rate (a '*/5' route still runs once per tick) and the known same-minute admin-digest double send, both documented as open. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KP9rt5rVQ9jDMpZp2xBy3K
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
NewWorldVenture
marked this pull request as ready for review
October 3, 2026 12:53
NewWorldVenture
pushed a commit
that referenced
this pull request
Oct 3, 2026
Brings in 4b9a136 (#873, #877, #880). No overlap with this branch's files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TRY21ZKsFrfB3qtoP972A4
This was referenced Oct 3, 2026
NewWorldVenture
added a commit
that referenced
this pull request
Oct 4, 2026
…t one (JOB-001 follow-up) (#902) * Cron dispatch: a tick that could not find its catch-up boundary is not one (JOB-001 follow-up) Review 5969190760 on #699, against merged #880: the boundary step swallowed every lookup outcome into "dispatch the fixed five-minute window and finish green", and the step that reads boundaries takes only status=success runs. So a tick whose lookup FAILED — a 403, an outage, an unexpected shape — dispatched its five minutes, finished green, and was taken by the next tick as the boundary; every minute between the last real success and that tick was lost for good. Last covered 03:00; the 09:07 lookup fails and dispatches five minutes; the 12:07 lookup works and catches up from 09:07. The step now tells the two look-alikes apart. The API answered and there was no previous scheduled success — the first tick ever — is `lookup=ok` with an empty boundary and starts the chain. The API did not answer, or answered something unusable, is `lookup=failed` with an empty boundary. The dispatch step runs either way, unconditionally, so nothing due this tick is skipped. A last step then fails the run when `lookup=failed`, with an ::error:: saying why: the boundary query's existing status=success filter makes a run failed on purpose invisible to the next tick, which catches up from the last real success instead — within the 24-hour cap, repeating the failed tick's five minutes for routes that are idempotent sweeps, losing nothing. A lookup that keeps failing keeps dispatching and keeps the workflow red, which is what a broken `actions: read` should look like. tests/a-tick-that-could-not-find-its-boundary-is-not-one.test.ts extracts the step's `run:` block from the YAML by indentation and executes it under `bash -eo pipefail` with a fake `gh` on a path that excludes the real one: a well-formed start is the boundary; an empty answer starts the chain; a 403, an absent gh and a `null` are each lookup=failed with no boundary and exit 0. The run-level half is pinned on the YAML: dispatch follows the boundary step and carries no condition, the failing step follows the dispatch, is conditioned on lookup == 'failed' and exits 1, and the query still filters on status=success. With the old shell back (every outcome green), three cases fail. docs/final-audit/cron-dispatch-cycle.md records the edge and corrects its earlier "never fails the tick". Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KP9rt5rVQ9jDMpZp2xBy3K * Cron dispatch: a selected run with an unusable start is a failed lookup, not the start of the chain (review 5970787916, 5970791078) Two look-alikes were still conflated inside "ok". The --jq expression mapped a selected run whose run_started_at was null or missing to an empty string, which the shell read as "no previous run" — the first-tick bootstrap — and the tick finished green as a boundary. And a shape-valid start in the future passed as lookup=ok; the dispatcher's tickWindow then ignored it and fell back to five minutes while the run finished green: the same incomplete boundary by another route. The expression now answers one word per outcome — `none` when no run was selected, the timestamp when it is a string, `unusable` otherwise — and only `none` starts the chain. The shell applies the two tests tickWindow applies (a real timestamp per GNU date, not after now), so a boundary the step accepts is one the dispatcher uses; anything else is lookup=failed. The step's stderr is no longer discarded, so the API's own error reaches the log. The test runs the step under /bin/bash with PATH reduced to one directory of its own: a fake gh that answers from a fixture body the way the API would (status/event filters applied, the step's --jq evaluated by the real jq) and symlinks to date, jq and sed, so the machine's gh cannot be reached and the absent case is proved by the step's own "command not found". New cases: a null, missing, malformed or future start, a body with no run list and a null body are each lookup=failed and never "starts the chain"; the step's accept/refuse decision is checked against tickWindow's on the same values; and a three-run regression replays the review's chain (A 03:00 success, B 09:07 failed itself, a manual run excluded by event, the 12:07 tick catches up from A: 547 minutes) with the negative control of a green B. Ordering pins use at()/between() (ordering-guards-fail-on-absence). Cycle note updated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KP9rt5rVQ9jDMpZp2xBy3K * Cron dispatch: a history of ticks that never succeeded is not a first tick Review on #902 (5971121597): the boundary lookup asked only for the previous scheduled SUCCESS, and read an empty answer as the first tick ever. `status=success` is a filter over the whole history, so that empty answer is also what a workflow whose every tick so far failed or was cancelled returns — a 09:07 whose own lookup failed and failed itself, a tick that died on a blank CRON_SECRET. The 12:07 tick then dispatched its fixed five minutes, finished green and became the next boundary, and the three hours after 09:07 were never caught up. The step now asks twice, in order. The success-only question first; on `none`, every previous scheduled run of any conclusion (the hundred most recent). No run at all is the first tick and starts the chain, as before. Runs with no success among them make the OLDEST usable start the boundary: no successful tick has covered a minute after it, and a failed or cancelled run dispatched at most its own five minutes, so this over-covers and never under-covers (the routes are idempotent sweeps and the dispatcher caps the window at a day). Runs with no usable start at all are a failed lookup, and the run fails itself as before. The review asked for the fallback window with no boundary in that case; that rule could never be left — no tick can succeed until one has — so the oldest recorded tick is taken instead, and the next green tick is the boundary. tests/a-tick-that-could-not-find-its-boundary-is-not-one.test.ts (+6): the review's chain (09:07 failed, 12:07 catches up from 09:07, three hours, not "starts the chain"); several failed and cancelled ticks take the oldest usable start; a manual run is in neither question; failed runs with no usable start are a failed lookup; a success wins over failures around it; the first tick ever still starts the chain, and the second question is asked only inside the `none` branch. The seventeen existing cases hold unchanged. Mutation control (the `none` branch made to start the chain again, file restored and checksummed): 5 failures. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KP9rt5rVQ9jDMpZp2xBy3K * Cron dispatch record: the failed-only-history refinement (review 5971121597) The dated subsection in docs/final-audit/cron-dispatch-cycle.md records the two-question boundary lookup, the oldest-usable-start rule for a history with no success, why "never a boundary" could not be the rule there, and the six regressions that pin it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KP9rt5rVQ9jDMpZp2xBy3K * Cron dispatch: the reviews' boundary cases, run against the script that landed #952 replaced the boundary step's shell with scripts/cron-run-history.mjs and this branch now carries main's workflow unchanged. The shell-driving test could no longer run (there is no `run: |` block to extract), so it is re-aimed at the script's `discoverCatchUpBoundary` through a fake Actions API that applies `status`, `event` and `created` and pages the way the real one does. The cases are the three reviews' cases: the previous success as boundary; no run or only this run as the start of the chain; a null, missing, malformed or future start, a body of another shape and a 403 as failed lookups; accept/refuse checked against tickWindow; the three-run chain with its negative control; failed-only history taking the oldest run — across three pages for a full day of ticks, which the shell's single page could not see — clamped when older than the cap, with manual runs in neither question; a success winning over failures around it; the first tick's questions in order. The YAML pins stand: dispatch before the failing step, the failing step conditioned on a failed lookup, the dispatch step unconditioned, the boundary step running the script. The audit record gains a dated note saying the shell was superseded and why. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KP9rt5rVQ9jDMpZp2xBy3K --------- Co-authored-by: Daniel Hughen <daniel.hughen@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
scripts/cron-dispatch.mjsis what gives the app its sub-daily cron cadence on Vercel Hobby:.github/workflows/cron-dispatch.ymlasks GitHub to run it every five minutes, and it calls every/api/cron/*route whose schedule fell due. It evaluated a fixed five-minute window ending at the moment it ran, and GitHub starts scheduled workflows hours late (measured gaps of 104–396 minutes, median 209), so every sub-daily firing between two ticks was simply lost.tests/a-late-tick-drops-a-cron.test.tscharacterized that; nothing fixed it. Ledger row JOB-001 (🔄 IN PROGRESS, High) names this as the remaining scheduler scope indocs/final-audit/cron-dispatch-cycle.md.finalaudit.mdis untouched; this is the repository-side fix with its evidence.What changes
scripts/cron-dispatch.mjstickWindow(now, since): the window is(since, now], never narrower than the fixed five minutes (two quick runs cannot lose the minutes between them) and clamped atCATCH_UP_MAX_MINUTES = 24 h. Clamped rather than ignored, because the gaps this exists for are the long ones: a weeks-old boundary after the workflow was disabled still recovers one call per sub-daily route. A missing, unparseable or futureCRON_SINCEkeeps the fixed window and says why in the log line.dueRoutes(…, since): each route is called at most once per tick however many of its minutes fell in the window; the routes are sweeps, not slot replays.isSubDaily: only routes that fire more than once a day are caught up. The once-a-day ones are exactly what Vercel Hobby accepts andvercel.jsonmirrors at the same minute, so Vercel already fired them; catching one up would be a second daily run (a second admin-digest email). Inside the fixed window they are evaluated exactly as before.model-refresh(0 4,16) andautopilot-scan(30 6,18) have 12-hour periods, and every caught-up route has a due minute in any 24 h window, so a wider cap cannot add a route. A test checks the cap against the longest gap in the table, so a slower route cannot be added silently.SCHEDULESis byte-identical;--dry-runstays credential-free;--routeignores the boundary; every existing validation (secret presence, base-URL rules, redirect refusal, bounded responses, deadline, redaction) is unchanged..github/workflows/cron-dispatch.ymlactions: read, the job's own token) for the previous successful scheduled run of this workflow, excluding the current run, validates the timestamp shape and passes it asCRON_SINCE. Manual single-route runs are not a boundary (they would collapse the next window). A failed lookup logs and falls back to the five-minute window; it never fails the tick. The cron string, sparse checkout, concurrency and repository guard are unchanged.Tests
tests/a-late-tick-drops-a-cron.test.ts→tests/a-late-tick-catches-up.test.ts(git mv; 7 cases; the late tick is now caught up; the measurement paragraph and the corrected severity model are kept).tests/cron-dispatch.test.ts+12: window, cap, clamp, once-per-route, the daily exclusion equals thevercel.jsonmirror set, the cap against the table's longest gap; "vercel.json ↔ dispatcher table" untouched.tests/cron-dispatch-execution.test.ts+6 CLI cases: catch-up, clamp, fallbacks for bad input, dry run with a boundary,--routeignoring it.Docs:
docs/final-audit/cron-dispatch-cycle.mdgains a dated section on the design and its evidence (additive).Evidence (local; nothing touched GitHub Actions or production)
a-mirrored-cron-must-be-idempotent,a-paths-filter-must-list-what-it-gates,production-readiness-workflow,test-harness-config-is-what-it-says,health-feature-secrets,cron-auth). ESLint clean;node --checkok; the workflow parses (PyYAML and js-yaml);git diff --checkclean.CRON_SINCE, the log line is byte-identical to before; with a 187-minute boundary the twelve sub-daily routes due in the gap are listed once each and the dailynotificationsis not;not-a-dateand a future timestamp fall back and say why; a 2-minute-old boundary stays at the five-minute window; a boundary older than the cap is clamped and lists all 17 sub-daily routes and no daily one;--routeignores the boundary.bash -eo pipefailwith a fakegh: success writes the boundary; a 403, an empty body,nulland a missingghall log the fallback and exit 0.What this does not change
*/5route still runs once per tick however many of its minutes the tick covered (a-cron-cadence-is-a-promise-nothing-keepsis unchanged). Recovering the rate needs a scheduler that keeps its word, as the workflow's own note says.vercel.jsonand the dispatcher.🤖 Generated with Claude Code
https://claude.ai/code/session_01KP9rt5rVQ9jDMpZp2xBy3K
Generated by Claude Code