fix(cron): honest fallback-chain alerts + drift-guard alert-once - #85011
Closed
victor-kyriazakos wants to merge 5 commits into
Closed
fix(cron): honest fallback-chain alerts + drift-guard alert-once#85011victor-kyriazakos wants to merge 5 commits into
victor-kyriazakos wants to merge 5 commits into
Conversation
_summarize_cron_failure_for_delivery() unconditionally said 'Fallback chain was exhausted or unavailable.' on every provider failure, even when fallback_providers is empty (the default -- confirmed empty on both the root and cto profile config.yaml). That phrasing implies a fallback was attempted and failed, which sent the operator debugging the wrong thing. Add _fallback_chain_phrase(): reads the effective chain via get_fallback_chain(load_config()) and returns 'No fallback chain configured.' when it's empty, or the original wording when a chain exists. Fails open to the original wording on any config read error. The scheduler's own inactivity-watchdog mislabeling (idle-timeout reported as provider timeout) was already fixed in a prior commit on this branch; this closes the second half of t_29b8da55. Data pull requested by the task (grep errors.log across profiles + root for 'Provider has been unresponsive' + model=, 2026-07-21 to 2026-08-06): 9 stall events total, 5 on claude-sonnet-5, 4 on claude-haiku-4-5, spread across 6 different cron jobs. No material haiku-specific instability -- sonnet-5 stalls at least as often on the cron path in this sample. Reporting per acceptance criteria; not worth a routing change on this evidence.
…lert A cron that dies on a provider timeout with no fallback chain configured now tells the operator exactly how to fix it: `hermes fallback add` for a personal chain, or the cron.model + cron.model_provider fleet defaults for operator-managed fleets. The exhausted-chain branch stays terse — the chain is intact there and no config command applies. Field-reported: users hitting the empty-chain failure could not self-serve from the alert text alone.
A fleet-wide inference config change previously produced one 'Skipped to prevent unintended spend' alert per unpinned job per tick — 40 jobs meant 40 alerts every tick until each was re-pinned (Coatue field report, 2026-08-11). The NousResearch#44585 guard now reuses the NousResearch#73506 alert-once shape the preflight path already established: a persisted drift_alerted bit on the job record, a [drift_skip:silent] marker on repeat ticks that suppresses delivery, and the bit clears on the next successful run so a future drift re-alerts. Only the drift branch consults the bit — every other failure keeps alerting per tick. The alert text also now says it is sent once, so operators know the job stays skipped silently until pinned or restored.
The generic failure summarizer caps unrecognized errors at 180 chars, which cut the drift alert off mid-sentence before the pin command. The drift branch now formats its own delivery from the guard's full message, so the one alert the operator gets actually contains the fix.
alt-glitch
added
type/bug
Contributor
|
Merged via PR #85508 (rebase-merge) — your commits were cherry-picked onto current main with your authorship preserved in git log, including the embedded #81579 cherry-pick from @georgell-ceo. On top of your branch we reconciled the drift-alert delivery with #72056 (@sashmatash, the earliest submission of that half — lifecycle-aware remediation for consumed finite one-shots), widened the scheduler-internal timeout classification to the TERMINAL_CWD lock-wait path (#79768), and scrubbed some environment-specific comment references. Thanks for the thorough work — the mutation-probed tests and the alert-once reuse of the preflight shape made this a clean salvage. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cron failure alerts around the fallback chain were lying in three ways, all field-reported from an enterprise fleet running ~40 jobs:
What changed
_fallback_chain_phrase()distinguishes "no chain configured" from genuine exhaustion, and inactivity timeouts are classified before the generic timeout match. 6 tests came with it.hermes fallback add, or thecron.model+cron.model_providerfleet defaults. The exhausted branch stays terse — the chain is intact there, no config command applies.drift_alertedbit, a:silentmarker variant that suppresses re-delivery, cleared on the next successful run so a future drift re-alerts. Only the drift branch consults the bit; every other failure still alerts per tick.Verification
run_one_job+_deliver_result, not helper internals.scripts/run_tests.sh tests/cron/: 47 files, exit 0.Not in this PR (parked as follow-ups)
hermes cron doctorwith bulk re-snapshot/re-pin verbs (a read-only base exists in feat: add cron doctor health check #43729).