fix(cron): harden runtime ownership, supervision, and recovery

[cirwel]

What changed

This PR hardens cron execution as one end-to-end ownership and recovery contract:

• separates declarative cron/jobs.json definitions from mutable cron/runtime.db state;
• migrates existing combined stores without losing schedules, counters, status, or claims;
• gives every fire a durable, token-fenced claim across ticker, provider, and direct-run paths;
• makes execution-ledger transitions crash-safe and terminalizes failed claim/dispatch paths;
• renews ownership before irreversible external delivery so stale workers cannot publish;
• runs agent work in an isolated subprocess with activity-driven inactivity heartbeats;
• terminates complete script/worker trees on timeout or ownership loss, including POSIX descendants that create a new session and Windows taskkill /T /F fallbacks;
• prevents shutdown-queued jobs and check-to-spawn races from beginning side effects, with worker spawn/registration atomic against shutdown;
• makes quick snapshot/restore fail closed when the cron store lock cannot be acquired;
• archives default and named-profile cron definition/runtime pairs from one coherent generation and deletes the whole archive if either pair member cannot be written.

Why

Cron's declarative configuration and scheduler-owned runtime fields previously shared jobs.json. Normal execution dirtied the same artifact operators would back up, review, deploy, or source-control. In parallel, in-process running guards could not prevent two scheduler processes from winning the same fire, and several early-failure, timeout, shutdown, and delivery paths could leave claims, ledgers, or descendant processes behind.

The result was an unsafe boundary: stale workers could continue after ownership changed, mutable state could race backup/restore, and process-group-only cleanup missed descendants that detached with setsid().

Compatibility and recovery

• Existing combined jobs.json files migrate on load; declarations remain in JSON and runtime fields move to SQLite.
• Missing-definition and interrupted-migration recovery paths reconcile from the pending definition journal.
• Runtime tombstones prevent removed jobs from reappearing from stale state.
• Definition digests reject restoring live claims onto changed definitions.
• All storage and execution state remains profile-local.
• POSIX and Windows process cleanup paths have separate regression coverage.
• The live user cron profile was not used or mutated during testing; tests used isolated homes or the repository's hermetic test runner.

How to verify

1. Start with a legacy cron/jobs.json containing schedules, last-run fields, counters, and a claim; load/list jobs and confirm definitions remain in JSON while runtime fields appear in cron/runtime.db.
2. Run two independent processes against the same due job and confirm exactly one wins the fire claim.
3. Force claim loss immediately before delivery and confirm the stale worker produces no external delivery.
4. Run an agent/script that continues producing activity and confirm its parent pulse prevents a false inactivity timeout.
5. Time out a worker or script that spawns both ordinary and start_new_session=True descendants; confirm no marker/descendant survives.
6. Interrupt a queued job and the check-to-worker-spawn window during shutdown; confirm no worker side effect starts and exact claims are released.
7. Mutate default and named-profile cron stores immediately after full-backup staging; confirm the archive contains coherent pre-mutation pairs, then fault one pair-member ZIP write and confirm no partial archive remains.

Test plan

• Canonical hermetic runner: scripts/run_tests.sh -j 4 tests/cron tests/hermes_cli/test_backup.py tests/hermes_cli/test_web_server_cron_profiles.py tests/tools/test_cronjob_run_immediate.py -q — 488 passed, 0 failed across 38 per-file subprocesses.
• ruff check over all 23 changed Python files — passed.
• Python compileall over implementation and relevant test trees — passed.
• git diff --check origin/main — passed.
• Windows-footgun checker over all affected production files — passed.
• Added real multiprocessing contention, crash-ledger, stale-delivery, dispatch-rejection, heartbeat, shutdown-queue, detached-descendant, fail-closed lock, and coherent profile-backup regressions.
• Repository-wide suite — an earlier non-hermetic run reached 4,345 passes but encountered broad e2e/global-state/provider-environment failures and was stopped; the canonical focused suite above is the reliable local evidence, with hosted CI left as the full-suite arbiter.

Risk

Medium-high. This touches scheduler ownership, persistence, process supervision, and backup semantics. The change is deliberately token-fenced and fail-closed: stale owners cannot renew, finalize, or deliver; backup/restore aborts instead of degrading without the cross-process lock; and terminal execution rows are immutable.

Tested on macOS; Windows behavior is covered with mocked taskkill success, timeout, and failure/fallback regressions plus the repository footgun checker.

Related work and credit

• Closes feat(cron): separate declarative definitions from volatile execution state #75607.
• Related to fix(cron): make Trigger now execute immediately and safely #70638 by @smwbev, which covers safe immediate-trigger execution and ownership fencing. This PR preserves that ownership direction while applying it to every fire path and the split runtime store.
• Related to fix(cron): kill the whole process group when a cron script times out #74123 by @jeff-mettel, which identifies cron script orphaning and process-group cleanup. This PR uses the same process-tree safety goal and extends coverage to isolated agent workers and POSIX descendants that escape the original process group.

Review follow-up (2026-08-01)

Addressed both hermes-sweeper findings:

• Terminal/declaration management path: completed (runtime-tombstoned) declarations are now reachable through supported surfaces — opt-in listing (include_completed on the tool, hermes cron list --all), removal by id or name, and revive via a cadence edit (schedule/repeat/enabled) that re-arms the counter and reschedules from now. Live-only actions on a completed job return an explicit terminal error. Reconciliation no longer revives tombstones on non-cadence edits (that path could fire a revived job once past its repeat limit). 8 new end-to-end tool/CLI regressions, RED/GREEN verified.
• Docs: cron-internals.md and the user-guide cron page now document the jobs.json definition / runtime.db runtime split, combined-store migration, digest binding, tombstone lifecycle, completed-job management, and backup-pair coherence.

[teknium1]

Thanks for the thorough ownership and recovery work. The current-main premise is real: cron/scheduler.py:4011 delivers before mark_job_run() at line 4030, and tick() only pre-advances recurring schedules at cron/scheduler.py:4154-4160.

Problems

• cron/jobs.py:1751-1804 hides every runtime_tombstone from get_job, resolve_job_ref, and list_jobs. But tools/cronjob_tools.py:770-795 resolves all management actions before remove or update. A repeat-exhausted declaration retained for reproducibility therefore cannot be listed, removed, edited, or revived through supported surfaces. This contradicts the nearby recovery intent in _reconcile_runtime_state().
• The persistence migration has no documentation update. website/docs/developer-guide/cron-internals.md:36-64 and website/docs/user-guide/features/cron.md:249-259,780-788 still describe mutable lifecycle state as living solely in jobs.json.

Suggested changes

• Add an explicit terminal/declaration management path and end-to-end tool/CLI regressions for list, remove, and revive/edit of a completed job.
• Document the definition/runtime split, migration, and paired-backup behavior.

Automated hermes-sweeper review.

[cirwel]

Both review points addressed in 7034a85 (rebased onto current main):

Terminal/declaration management path. get_job / resolve_job_ref / list_jobs now take an include_terminal opt-in, and the management surfaces route through it:

• list — cronjob(action="list", include_completed=True) and hermes cron list --all show completed declarations as state="completed" with completed_reason / completed_at (the CLI shows the completion badge instead of the stale next-run time).
• remove — works on a completed declaration by id or name and drops it from jobs.json.
• revive/edit — an update that changes cadence (schedule / repeat / enabled) revives the job: the reconciler resets the exhausted counter and the next run is recomputed from now, not the stale pre-completion next_run_at. The response carries revived: true. A non-cadence edit (e.g. prompt) persists the declaration change but keeps the job completed, and says so.
• pause/resume/run on a completed job now return an explicit terminal error with the revive/remove options, instead of "not found".

One deliberate semantics change while closing this: _reconcile_runtime_state() no longer pops the tombstone on non-cadence definition edits. That branch could revive a repeat-exhausted job with completed >= times and a stale next_run_at, firing it once past its limit before re-tombstoning. Revival is now exactly the cadence edit that re-arms the counter, which also resolves the contradiction with the hidden-lookup surfaces flagged in the review.

End-to-end regressions: 8 new tests (tool-layer list/remove-by-id/remove-by-name/revive-by-repeat/revive-by-schedule/prompt-only-stays-completed/live-action-terminal-errors, plus a CLI list --all test asserting the completed badge and reason). RED/GREEN verified: all 8 fail without the fix, pass with it.

Docs. website/docs/developer-guide/cron-internals.md now documents the jobs.json definition / runtime.db state split, the combined-store migration, digest binding, tombstone lifecycle, and backup-pair coherence; the tick-cycle description no longer claims runtime writes go to jobs.json. The user guide's job-storage section covers the split, migration, and a new "Completed jobs" section (list --all, remove, revive via schedule/repeat edit).

Focused suites after the rebase: 159 passed across tests/cron, tests/tools/test_cronjob_tools.py, tests/tools/test_cronjob_run_immediate.py, tests/hermes_cli/test_cron.py (includes the new TestGithubExemptionAbuse from current main); ruff and git diff --check clean. Note: tests/cron/test_cron_inactivity_timeout.py process-tree kill tests are timing-sensitive under parallel local runs — they fail intermittently on unmodified main under load and pass serially with this branch; hosted CI remains the arbiter there.

[cirwel]

Follow-up hardening in the two commits now on the branch (rebased onto current main again): after the earlier fix I ran an adversarial multi-lens review over the terminal-management delta (every finding reproduced by execution before fixing) and closed seven coherence gaps it surfaced:

• Live-first resolution with terminal fallback everywhere (tool actions, remove_job, cron edit): a live job always wins a name tie with a retained completed namesake — previously one finished one-shot could permanently make name-based management of its live namesake ambiguous. Completed declarations stay reachable when nothing shadows them, and ambiguity payloads now carry state so completed twins are distinguishable.
• hermes cron edit / /cron edit can reach completed jobs — the revive path the docs promise now works end-to-end (both surfaces pre-resolved with live-only lookup and returned "Job not found").
• context_from accepts completed upstreams on tool create/update and the dashboard validator: fire-time injection reads the persisted output directory, so a finished one-shot collector is a legitimate chaining source. Previously the tool showed the completed job in listings but rejected chaining to it.
• A final run reports the completed record: a run that exhausts the repeat limit now returns state: completed with completed_reason/completed_at instead of a fabricated schedulable {'id': ...} shape, and execution_success no longer misreports on the final run.
• One-shot revive error is honest: repeat-only revive of a completed one-shot says a new schedule is required instead of the generic "time is in the past".
• Listing coherence: a completed declaration lists on the completed opt-in alone (the disabled filter applies to live jobs only), and tombstone reads are isinstance-guarded against corrupted stores.
• Docs accuracy: digest binding is scoped to the cadence digest (the stored full-definition digest has no comparer), and the architecture + cron-troubleshooting pages now describe the split store alongside the internals/user-guide updates.

12 new regressions across the tool, CLI, and jobs layers (including the gateway-PATCH-only enabled revive leg). Focused suites: 169 passed post-change, plus tests/gateway/test_api_server_jobs.py 16/16; ruff and whitespace clean.

Known limitation, deliberately out of scope: the web dashboard's job list/detail endpoints still use live-only lookup, so completed declarations aren't visible there (its context_from validator IS terminal-aware now, so chained jobs stay editable). Routing the dashboard's list/get through the same opt-in is a small follow-up I'm happy to do separately — kept out of this PR to avoid growing the diff into another subsystem.

[GottZ]

This was generated by AI during triage.

Summary

One PR addresses #75607. #75833 separates declarative cron definitions in jobs.json from volatile state in runtime.db, while also adding legacy migration, reconciliation, fenced execution ownership, coherent backup/restore, documentation, and regression coverage.

Related pull requests

• fix(cron): harden runtime ownership, supervision, and recovery #75833 best fix — (+5269/-448) — keep open with a salvage path: the diff directly addresses the mixed-state cause by partitioning definition and runtime fields, migrating combined stores, and preserving cadence and claims in a profile-local SQLite ledger. This follows the keep_open review on fix(cron): harden runtime ownership, supervision, and recovery #75833: the review's terminal-job-management and stale-documentation findings are addressed in the shown diff through include_terminal lookup and management paths, list/remove/revive regressions, and updated cron user and developer documentation.

Suggested consolidation

Keep #75833 open with a salvage path focused on the definition/runtime separation, lossless migration and reconciliation, token-fenced ownership, coherent backup/restore, terminal-declaration management, and their regression tests. The diff is broad, so the author should keep these end-to-end storage and safety invariants coherent while narrowing unrelated changes where possible; there are no duplicate PRs to close.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I75607(["issue #75607 (open)"])
    P75833["PR #75833 (open)"]
    P75833 -->|best fix| I75607
    class I75607 open
    class P75833 open
    class P75833 best
    class P75833 target
    click I75607 "https://github.com/NousResearch/hermes-agent/issues/75607"
    click P75833 "https://github.com/NousResearch/hermes-agent/pull/75833"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 1 pull request and 1 issue in this complex. Each diff was read against this issue; Assessment working set: 305 kB of PR diffs, 9 kB of issue/PR text, 8 kB of discussion (5 comments), 2 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

[cirwel]

Dogfood follow-up on 3ae839011 (pushed to this branch):

• Exercised the real CLI against isolated HERMES_HOME stores: create, immediate run by id/name, finite-repeat completion, list --all, prompt-only edit, cadence revive, re-completion, and remove-by-name. Runtime-only runs left the jobs.json SHA-256 unchanged.
• Exercised legacy combined-store migration and compared the relevant behavior with the PR base.
• Exercised quick and full backup capture of the cron/jobs.json + cron/runtime.db pair.

The quick-restore pass found one real WAL lifecycle defect in this branch: Python's sqlite3.Connection context manager commits/rolls back but does not close. _journal_snapshot_definitions() could therefore rename the staged runtime.db while its WAL connection was still open, leaving .runtime.db.snap_restore.*-{wal,shm} behind and producing sqlite3.OperationalError: disk I/O error on the published database. Controlled reproduction:

without explicit close: staged WAL/SHM remain; published read -> disk I/O error
with explicit close:     no staged sidecars; pending definition row is readable

3ae839011 fixes both restore-side connections with deterministic contextlib.closing(...) around the transaction context and adds a regression that forces WAL mode and retains the staged connection so garbage collection cannot hide the bug. The new test was RED before the production change and GREEN after it.

Post-fix evidence:

• Installed-source quick snapshot/restore against a real WAL database: restore_ok=True, snapshotted prompt recovered, staged_sidecars=[].
• Focused cron/backup/CLI/tool suite: 604 passed, 0 failed.
• Ownership/claim/heartbeat/shutdown/timeout/recovery stress subset: 117 passed, 0 failed.
• Independent Codex review: no blocking finding; it separately reproduced the WAL failure and ran the full affected backup test file (52 passed).
• Ruff and git diff --check: clean.

GitHub still reports no hosted checks on the branch.

[cirwel]

Final recovery hardening is now on 7621ee917 (rebased onto current main at 56dc01d904).

Review follow-up closed the remaining safety findings:

• owner liveness is host-scoped; foreign, legacy, or unverifiable ownership fails safe;
• runtime schema migration is writer-serialized and retries only bounded transient SQLite locked/busy errors;
• stale public fire/run snapshots cannot clear or replace newer durable claims;
• backup rejects pending definition generations whose base digest does not match the authoritative definitions;
• worker activity is recorded before descendant discovery, and known work is stopped/frozen before reparented-process scanning;
• detached descendants receive bounded TERM/KILL/wait cleanup after timeout or claim loss.

Exact settled patch before commit: SHA-256 be85fa112721afd7d7df1fad290058b640af59ea0b13d781841527d81454682d. Three independent scoped reviews returned VERDICT: APPROVE for persistence, supervision, and backup/integration on that exact generation.

Final local evidence after the rebase:

• tests/cron: 614 passed, 1 skipped;
• backup/gateway/tool matrix: 165 passed;
• concurrent first-migration stress: 10/10 passed;
• Ruff, lockfile, diff hygiene, changed-file compilation, and Windows-footgun scan (944 files) passed;
• website prebuild + production build passed; ASCII guard passed over all 401 docs;
• exact patch Gitleaks scan: 0 findings; full-tree sets were identical to untouched upstream (808 / 808, zero delta).

The canonical repository-wide run is still honestly non-green: 18 failures across 12 unrelated files. Replaying those same files on untouched current upstream reproduced all 18 failures. A branch replay also saw one extra load-sensitive transcription timeout; that source is byte-for-byte unchanged from upstream and the extra timeout did not reproduce there. No cron-repair regression was demonstrated, but this is not being represented as a green full suite.