Skip to content

fix(cron): name the cron-fleet default in drift-guard remediation - #89292

Open
jackulau wants to merge 1 commit into
NousResearch:mainfrom
jackulau:fix/89242-cron-drift-remediation-fleet-default
Open

fix(cron): name the cron-fleet default in drift-guard remediation#89292
jackulau wants to merge 1 commit into
NousResearch:mainfrom
jackulau:fix/89242-cron-drift-remediation-fleet-default

Conversation

@jackulau

Copy link
Copy Markdown
Contributor

What does this PR do?

The #44585 drift guard fails an unpinned cron job closed when the global provider/model moves under it. Every message it emits offers exactly one remedy: pin this job.

That is correct for one job and wrong for a fleet. #59031 was 34 jobs breaking on a single global switch — 34 per-job edits for one config change. #89242 is the same complaint arriving again, this time with a proposed fix (stop snapshotting inherit-mode jobs) that would disarm the guard for 100% of the population it exists to protect.

The fleet-wide remedy already exists. #73532 added cron.model / cron.model_provider, and _cron_fleet_default_covers_axis makes the guard skip any axis they cover — so one command decouples the whole unpinned fleet from the chat model without giving up spend protection on the axes it doesn't cover. website/docs/user-guide/features/cron.md has said so since #73532:

Setting cron.model (or a per-job pin) is the deliberate way to route cron spend, and the drift guard does not engage for an axis covered by it.

None of the runtime messages did. So the only operators who find it are the ones who go read the docs after being told to make N edits — and #89242 is what happens when they don't. This teaches all three surfaces to name it.

No behaviour change. The guard engages in exactly the cases it engaged in before, and the existing per-job and finite-one-shot remedies are unchanged — this is additive to both.

Related Issue

Fixes #89242

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

cron/scheduler.py

  • The drift-skip remediation gains _fleet_default_hint, appended to both branches — the recurring one and fix(cron): preserve actionable drift recovery alerts #72056's finite-one-shot one. The one-shot path is where it matters most: fix(cron): preserve actionable drift recovery alerts #72056 correctly established that a consumed one-shot can't be fixed by editing it, so until now the only alternative offered there was "create a new one-shot with an explicit provider and model" — still per job.
  • _summarize_cron_failure_for_delivery gains a terse one-sentence version. This is the surface an operator reads unprompted; the log and the alert blob are where they look only after something sent them there. Deliberately shorter than the log wording — that function exists to stay compact, and there's a test asserting the delivery line stays a single line under 500 chars.

hermes_cli/config.py

  • warn_unpinned_cron_jobs_after_model_config_change now offers both remedies. This warning (fix(cron): warn before model config changes trip cron drift guard #73323) fires before any job has failed, which makes it the cheapest place to have omitted the fleet default and the most valuable place to add it.
  • The provider axis names cron.model_provider, not cron.provider. The axis is called provider but the fleet-default key is model_provider, so interpolating the axis name — the obvious way to write this line — produces a key that doesn't exist. There's a dedicated test for that.

tests/cron/test_cron_drift_remediation_names_fleet_default.py — new, 16 tests.

The interesting ones round-trip the advice through the mechanism: they re.findall the hermes config set cron.X command back out of each emitted message and feed X to _cron_fleet_default_covers_axis. A message that advertises a key the guard ignores fails here rather than in somebody's cron log. That's what would have caught this being stale in the first place.

Two tests also pin the exemption itself (a covered axis doesn't drift; an uncovered axis still fails closed), because the advice is only true while that exemption holds. They reuse test_cron_provider_pin.py's own _run_with_current_provider_and_model harness on purpose — a message test that stubbed the guard instead of running it would keep passing after the guard stopped firing.

How to Test

pytest tests/cron/test_cron_drift_remediation_names_fleet_default.py -q   # 16 passed

Manually, with any unpinned agent job:

before: … Skipped to prevent unintended spend … To run on the new config, on the host
        running Hermes pin it explicitly: `hermes cron edit <id> --provider <provider>
        --model <model>` (or pin the original values to keep them).

after:  … (same) … If you want every unpinned cron job on a fixed model regardless of
        your chat model, set the cron-fleet default once instead: `hermes config set
        cron.model <model>` (and `cron.model_provider` if the provider also differs) —
        the guard does not engage for an axis it covers.

Mutation proof — each surface is independently load-bearing:

Reverted Tests that fail
the scheduler hint (_fleet_default_hint"") 4
the delivery-line sentence 1
the config set warning wording 3
_cron_fleet_default_covers_axis → always False 6 — including both exemption pins and both key round-trips

That last row is the one I care about: it's the mutation that #89242's proposed fix is a variant of, and it fails loudly.

Baselinepytest tests/cron/ tests/hermes_cli/test_cron_model_impact.py tests/cli/test_cli_save_config_value.py tests/hermes_cli/test_set_config_value.py -q run serially, with and without the change:

  • 8 failures common to both runs, all pre-existing and all Windows-specific: 6 in test_file_permissions.py (POSIX 0700/0600 mode bits), test_cron_workdir.py::test_tilde_expands, and test_media_delivery_parity.py::TestMediaPolicyEnvBridge::test_bridge_helper_exists_and_applies_config. None is touched by this diff.
  • One extra failure appeared in the without-change run only — test_cleanup_timeout.py::test_run_job_bounds_sessiondb_finalization. It's a wall-clock-bounded finalization test and passes in isolation; I'm calling it flaky rather than claiming this PR fixed something.
  • 867 → 884 passed, the delta being exactly the 16 new tests plus that flake.

tests/cron/test_cron_provider_pin.py, test_cron_drift_alert_once.py and tests/hermes_cli/test_cron_model_impact.py — the files that pin #44585's and #73532's contracts for these same messages — all pass unchanged.

Overlap with open PRs

Ran the duplicate gate over #89242 before starting: no open or merged PR references it, and no open PR touches the drift-guard remediation strings in cron/scheduler.py or hermes_cli/config.py.

The relevant prior art is all merged, and this composes with it rather than competing:

PR What it is Relationship
#44585 the drift guard itself unchanged; this only extends what it says
#73323 (merged) warn at config set time, for #59031 this adds the second remedy to that warning
#73532 (merged) cron.model / cron.model_provider + the guard exemption this is the discoverability half that was never built
#72056 (merged) lifecycle-aware remediation for consumed one-shots preserved verbatim; the hint is appended to it

What this deliberately does not do

I did not implement the fix #89242 proposes, and I'd push back on it. Returning None for both snapshots when both axes are inherit-mode removes the guard from exactly the jobs it was written for — after that change, there is no job left anywhere that it protects. The scenario in the issue's own reproduction is #44585's scenario: an unattended job silently following the global default onto a paid model. The no-snapshot back-compat path the issue cites is narrower than it looks; the code comment calls it "Back-compat" because it's a migration affordance for jobs created before #44585 shipped and for axes whose creation-time resolution failed, not a policy statement.

This was also settled once already. @teknium1 closed #61468 with:

Closing — the batch-refresh need this addressed is now largely mooted by PR #73532: cron.model / cron.model_provider in config.yaml give unpinned cron jobs a fleet-wide default model … (and the #44585 drift guard exempts axes covered by the fleet default). cron.model_drift_guard: false (3a358cb) also exists for fleets that want to follow the changing global default. Thanks for the work — the problem was real, we just solved it one level up.

So the reporter's actual need has two supported answers today (cron.model for "route my fleet deliberately", cron.model_drift_guard: false for "let my fleet track the global"). What was missing is that the runtime never mentions either. That's the defect this PR fixes, and it's message-only precisely so it can't prejudge a policy call that isn't mine to make. If a maintainer does want inherit-mode snapshots reconsidered on the merits, that's a separate change and I've kept it out of here.

I also left the docs alone — they're already correct and already say all three things. The gap was one-directional.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits
  • I searched for existing PRs to make sure this isn't a duplicate — see the Overlap section
  • My PR contains only changes related to this fix (one commit, rebased on main)
  • I've run the cron + config test slice with and without the change, serially (see How to Test). I did not run pytest tests/ -q wholesale: on Windows tests/hermes_cli/ can't be collected (test_doctor_journal_modes.py calls os.geteuid), so a full-suite number from here would be meaningless. CI runs it.
  • I've added tests for my changes — 16, with the mutation proof above
  • I've tested on my platform: Windows 11

Documentation & Housekeeping

  • I've updated relevant documentation — N/A, website/docs/user-guide/features/cron.md already documents all three remedies correctly; this PR brings the runtime up to the docs, not the other way round
  • N/A — no config keys added or changed (cron.model / cron.model_provider / cron.model_drift_guard all already exist)
  • N/A — no architecture or workflow change
  • I've considered cross-platform impact — string formatting only, no platform-specific code
  • N/A — no tool description or schema change

@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/cron Cron scheduler and job management comp/cli CLI entry point, hermes_cli/, setup wizard area/config Config system, migrations, profiles labels Aug 18, 2026
The NousResearch#44585 drift guard fails an unpinned cron job closed when the global
provider/model moves under it, and every message it emits offers exactly one
remedy: pin this job. That is correct for one job and wrong for a fleet.
NousResearch#59031 was 34 jobs breaking on a single global switch, i.e. 34 per-job edits
for one config change, and NousResearch#89242 is the same complaint arriving again with a
proposed fix that would disarm the guard entirely.

The fleet-wide remedy already exists. NousResearch#73532 added `cron.model` /
`cron.model_provider`, and `_cron_fleet_default_covers_axis` makes the guard
skip any axis they cover, so one command decouples the whole unpinned fleet
from the chat model without giving up spend protection on the axes it does not
cover. The cron guide has documented this since NousResearch#73532; none of the runtime
messages did, so the only operators who find it are the ones who read docs
after being told to make N edits.

Teach all three surfaces to name it alongside the pin:

- the scheduler's skip log and alert blob, on both the recurring and the
  NousResearch#72056 finite-one-shot path, naming `cron.model_provider` for the provider
  axis rather than interpolating the axis name into a key that does not exist
- the chat delivery line, which is the surface an operator reads unprompted
  rather than after something sent them to a log; kept to one terse sentence
  so the notification stays one line
- the `hermes config set model.default` warning, which fires before any job
  has failed and is therefore the cheapest place to have omitted it

No behaviour change: the guard engages in exactly the cases it engaged in
before, and the per-job and one-shot remedies are unchanged. The warning is
still suppressed for an axis the fleet default already covers, so an operator
who has taken the advice is never given it.

Tests parse the config key back out of each emitted message and feed it to
`_cron_fleet_default_covers_axis`, so the advice cannot go stale against the
mechanism the way it just did.

Fixes NousResearch#89242
@jackulau
jackulau force-pushed the fix/89242-cron-drift-remediation-fleet-default branch from 66196b0 to ea53e33 Compare August 19, 2026 02:28
@Enough1122

Copy link
Copy Markdown
Contributor

AI code review — automated review for reference; please use your judgment.

  1. tests/cron/test_cron_drift_remediation_names_fleet_default.py — Positive beyond praise: parsing the hermes config set keys back out of the emitted messages and feeding them to _cron_fleet_default_covers_axis turns remediation text into a checked contract ("a remedy in a message is a promise") — the test that would have caught this staleness originally and prevents its return. Pinning the covered-axis exemption is equally sharp: it's the exact canary model_snapshot recorded for inherit-mode cron jobs causes false drift guard failures #89242's snapshot-skip would break for newly created inherit-mode jobs.

  2. Cross-PR observation (no action needed here): model_snapshot recorded for inherit-mode cron jobs causes false drift guard failures #89242 stops recording snapshots for fully-inherit jobs, which shrinks the guard's future population — new inherit-mode jobs will follow cron.model/cron.model_provider silently instead of fail-closing on chat-model drift. That's arguably the intended semantics of inherit mode, but these tests document the philosophical fork (fail-closed vs follow-default) explicitly; worth a sentence in model_snapshot recorded for inherit-mode cron jobs causes false drift guard failures #89242's description linking here so the two changes tell one coherent story.

  3. hermes_cli/config.py:warn_unpinned_cron_jobs_after_model_config_change — Positive: warning earlier at config set time (before N jobs skip) and suppressing it entirely for operators who already set the fleet default avoids both the O(n) pin advice and false alarms for the already-routed; the axis→key interpolation trap (cron.provider doesn't exist) is called out and tested against verbatim output.

@jackulau

Copy link
Copy Markdown
Contributor Author

Thanks — and point 2 is worth acting on rather than filing, so I have taken it
to #89855 (the PR that implements #89242) rather than leaving the observation
here where only this PR's reviewers see it.

Your read of the fork is right, and I would put it slightly more sharply: after
#89855 an inherit-mode job created tomorrow has no snapshot to drift from,
so it follows cron.model / cron.model_provider silently. That is the correct
semantics for inherit mode, and it is also a real behavioural change for anyone
who has been relying on the guard as a blast-radius check on hermes config set model — they lose it for exactly the jobs that never pinned anything. The two
changes are consistent, but only if someone says out loud that "inherit" now
means "follow", not "fail closed on surprise".

That is what these tests document, so the link matters in both directions:
without it, #89855 reads as a bug fix with no policy content, and this PR reads
as remediation advice for a population that is quietly shrinking.

On point 1 — parsing the hermes config set keys back out of the emitted
messages was not the first shape I tried. Asserting on the message text passes
happily while the key inside it is one nobody can run, which is the bug (cron.provider
does not exist). Feeding the parsed key to _cron_fleet_default_covers_axis is
the only version where a remedy that cannot be followed fails the test.

Nothing changed on this branch; it is rebased on current upstream/main,
MERGEABLE, 47 checks green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

model_snapshot recorded for inherit-mode cron jobs causes false drift guard failures

3 participants