fix(cron): honest fallback/timeout/drift alerts + drift-guard alert-once - #85508
Merged
Conversation
_summarize_cron_failure_for_delivery() unconditionally said 'Fallback chain was exhausted or unavailable.' on every provider failure, even when fallback_providers is empty (the default -- confirmed empty on both the root and cto profile config.yaml). That phrasing implies a fallback was attempted and failed, which sent the operator debugging the wrong thing. Add _fallback_chain_phrase(): reads the effective chain via get_fallback_chain(load_config()) and returns 'No fallback chain configured.' when it's empty, or the original wording when a chain exists. Fails open to the original wording on any config read error. The scheduler's own inactivity-watchdog mislabeling (idle-timeout reported as provider timeout) was already fixed in a prior commit on this branch; this closes the second half of t_29b8da55. Data pull requested by the task (grep errors.log across profiles + root for 'Provider has been unresponsive' + model=, 2026-07-21 to 2026-08-06): 9 stall events total, 5 on claude-sonnet-5, 4 on claude-haiku-4-5, spread across 6 different cron jobs. No material haiku-specific instability -- sonnet-5 stalls at least as often on the cron path in this sample. Reporting per acceptance criteria; not worth a routing change on this evidence.
…lert A cron that dies on a provider timeout with no fallback chain configured now tells the operator exactly how to fix it: `hermes fallback add` for a personal chain, or the cron.model + cron.model_provider fleet defaults for operator-managed fleets. The exhausted-chain branch stays terse — the chain is intact there and no config command applies. Field-reported: users hitting the empty-chain failure could not self-serve from the alert text alone.
A fleet-wide inference config change previously produced one 'Skipped to prevent unintended spend' alert per unpinned job per tick — 40 jobs meant 40 alerts every tick until each was re-pinned (Coatue field report, 2026-08-11). The #44585 guard now reuses the #73506 alert-once shape the preflight path already established: a persisted drift_alerted bit on the job record, a [drift_skip:silent] marker on repeat ticks that suppresses delivery, and the bit clears on the next successful run so a future drift re-alerts. Only the drift branch consults the bit — every other failure keeps alerting per tick. The alert text also now says it is sent once, so operators know the job stays skipped silently until pinned or restored.
The generic failure summarizer caps unrecognized errors at 180 chars, which cut the drift alert off mid-sentence before the pin command. The drift branch now formats its own delivery from the guard's full message, so the one alert the operator gets actually contains the fix.
…cific comments - Widen the scheduler-internal timeout classification to the sibling TERMINAL_CWD lock-wait TimeoutError (#79768), which also matched the generic 'timed out' branch and was delivered as a provider timeout. - Reconcile the drift-guard alert with #72056's lifecycle-aware remediation: finite one-shots are told to recreate the job, not to update a consumed one. - Scrub environment-specific references from comments/docstrings.
Contributor
૮ >ﻌ< ა ci reviewran on 14525ea — fix(docs): remove stray conflict marker in cron.md
|
alt-glitch
added
type/bug
This was referenced Aug 13, 2026
teknium1
added a commit
that referenced
this pull request
Aug 13, 2026
…composed no_agent gate The cherry-picked tests predate #85508's honest fallback-chain phrasing and each other: assertions pinned the old 'exhausted or unavailable' literal and #83188's no_agent fallback-note behavior, which #77648's mode gate supersedes (no provider classification at all for no_agent jobs). Assert the composed contract instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cron failure alerts stop lying about fallback chains, timeouts, and drift — and drift alerts fire once per job instead of once per tick.
Salvage of #85011 (@victor-kyriazakos), which itself carries #81579 (@georgell-ceo); reconciled with #72056 (@sashmatash), the earliest submission of the drift-alert-delivery half. All contributor authorship preserved via cherry-pick.
Changes
cron/scheduler.py:_fallback_chain_phrase()distinguishes "no chain configured" (with the fix commands:hermes fallback add, orcron.model+cron.model_provider) from genuine exhaustion (fix(cron): stop mislabeling inactivity timeouts as provider timeouts + fix fallback-chain wording #81579 + fix(cron): honest fallback-chain alerts + drift-guard alert-once #85011).cron/scheduler.py: the scheduler's own inactivity watchdog (idle for Ns (limit Ns)) is classified before the generic timeout match instead of being reported as a provider timeout (fix(cron): stop mislabeling inactivity timeouts as provider timeouts + fix fallback-chain wording #81579).cron/scheduler.py: sibling site widened — the TERMINAL_CWD lock-waitTimeoutError(Cron inactivity watchdog blind spot: unbounded pre-run blocking (_terminal_cwd_lock, in-process jobs RLock) leaves jobs wedged in 'running' with no logs #79768) also no longer falls through as "provider timeout" (our follow-up; gap flagged in fix(cron): stop mislabeling inactivity timeouts as provider timeouts + fix fallback-chain wording #81579 review).cron/scheduler.py+cron/jobs.py: [Bug]: Cron can inherit temporary paid provider state and continue billing during pause/stop containment #44585 drift-guard skips alert once per job via a persisteddrift_alertedbit with a:silentmarker (same shape as the blocked-config preflight, feat(cron): dead-pin guard — auto-pause + alert on missing script #73506); cleared on the next successful run so future drift re-alerts (fix(cron): honest fallback-chain alerts + drift-guard alert-once #85011).website/docs/user-guide/features/cron.md: drift-guard tip documents alert-once and the one-shot recreation path.Validation
hermes fallback add, or…"cronjob action=updatea dead jobTests: 13 new across 4 files, asserting delivered content through
run_one_job+_deliver_result. Full cron suite: 578 passed, 1 skipped.Closes #85011, closes #81579, closes #72056.
Infographic