Skip to content

fix(cron): bound per-job run history — prune old run sessions after each fire - #88331

Open
ayushnangia wants to merge 6 commits into
NousResearch:mainfrom
ayushnangia:fix/88268-cron-run-retention
Open

fix(cron): bound per-job run history — prune old run sessions after each fire#88331
ayushnangia wants to merge 6 commits into
NousResearch:mainfrom
ayushnangia:fix/88268-cron-run-retention

Conversation

@ayushnangia

Copy link
Copy Markdown
Contributor

Summary

Fixes #88268: a busy cron job appends one session row per tick (cron_<jobid>_<timestamp>), growing state.db unbounded (the report: 473 rows from one 15-min job in 5 days) and flooding BOTS/SESSIONS with per-run duplicates.

The per-run rows are the desktop run-history contract (list_cron_job_runs), so the fix is bounded retention, not a single stable row.

Changes

  1. SessionDB.prune_cron_job_runs(job_id, keep) — deletes a job's oldest run sessions beyond the newest keep, scoped by the same [prefix, prefix_hi) id-range scan as list_cron_job_runs. Uses the delete_session cascade shape (delegate children die with the parent, branches orphaned, messages removed, unreferenced prompts cleaned) so FK constraints hold. Returns the count deleted.
  2. Scheduler wiring — after each agent-path run is finalized (end_session, before close), prune keeping cron.run_history_retention (env HERMES_CRON_RUN_HISTORY_RETENTION → config → default 50 — always a superset of the desktop's 20-row window). Best-effort: a prune failure logs and never wedges bookkeeping or delivery.

Validation

  • New tests: keep=3 leaves exactly the newest three runs of the target job (other jobs' runs + non-cron sessions untouched); keep=0 clears the job; FK cascade holds.
  • tests/test_hermes_state.py -k cron: 4 passed · tests/cron/test_cleanup_timeout.py tests/cron/test_jobs.py: 72 passed.
  • Atomic commits: state method → scheduler wiring → tests.

@alt-glitch alt-glitch added type/bug Something isn't working comp/cron Cron scheduler and job management area/sessions Session lifecycle, resume, persistence, history P2 Medium — degraded but workaround exists sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state labels Aug 17, 2026
@ayushnangia
ayushnangia force-pushed the fix/88268-cron-run-retention branch from ae9d38c to a25d317 Compare August 17, 2026 19:35
@Enough1122

Copy link
Copy Markdown
Contributor

AI code review — automated review for reference; please use your judgment.

Reviewed by reviewer-e (AI automated review).

Well-bounded fix for unbounded state.db growth and BOTS/SESSIONS duplicate flooding (#88268): retention resolves env > config > default(50) with invalid values warned about, the prune's id-range scan [prefix, prefix_hi) scales with the excess rather than the whole cron pile, the deletion cascade mirrors delete_session exactly (delegate children → orphan branches → messages → row, plus unreferenced system-prompt cleanup), everything runs inside _execute_write's lock/jitter handling, and a failure can never wedge the run's bookkeeping or delivery. Tests cover newest-kept ordering, cross-job isolation, non-cron-session safety, and the keep=0 edge.

Nit (non-blocking): cron/scheduler.py — the prune executes after every fire even when there's nothing to remove (a 1-minute job under its retention pays one wasted SELECT per tick). A cheap early-out — fetch victim ids first and skip the write transaction when empty — would keep hot jobs at zero write cost in the common case; alternatively only attempt pruning every Nth fire.

SessionDB.prune_cron_job_runs(job_id, keep): deletes a cron job's
oldest run sessions beyond the newest keep, scoped by the same
[prefix, prefix_hi) id-range scan as list_cron_job_runs. Uses the
delete_session cascade shape (delegate children die with the parent,
branches orphaned, messages removed) so FK constraints hold; returns
the count deleted. Bounds state.db growth + BOTS/SESSIONS duplicate
noise from per-tick cron rows (NousResearch#88268).
After the run session is finalized (end_session, before close), prune
the job's old run rows keeping cron.run_history_retention (env
HERMES_CRON_RUN_HISTORY_RETENTION > config > default 50). Best-effort:
a prune failure logs and never wedges bookkeeping or delivery.
- keep=3 leaves exactly the newest three runs of the target job;
  other jobs' runs and non-cron sessions untouched
- keep=0 clears the job's history
- FK cascade: messages removed with the sessions
…esearch#92133)

The bare [prefix, prefix_hi) range leaks runs across jobs when one job
id is an underscore-extension of another (backup vs backup_weekly):
backup_weekly sorts inside backup's range, so a prune could delete
another job's runs. Constrain both the victim SELECT and the keep-set
to rows whose post-prefix remainder matches this run's timestamp shape
(%Y%m%d_%H%M%S via GLOB), which free-form job ids cannot satisfy.
Matches the list-side predicate the NousResearch#92133 PRs add; prune coverage
they leave open.
Enough1122 review on NousResearch#88331: a hot job under its retention paid one
write transaction per fire even when the victim set was empty. The
transaction now opens only when there are rows to delete.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/sessions Session lifecycle, resume, persistence, history comp/cron Cron scheduler and job management P2 Medium — degraded but workaround exists sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Each cron execution spawns a NEW session row instead of reusing a per-job session

3 participants