Skip to content

fix(cron): restore cumulative runtime safeguards on current main - #103426

Open
jayleaton wants to merge 20 commits into
NousResearch:mainfrom
jayleaton:fix/cumulative-cron-safeguards-current-main
Open

jayleaton wants to merge 20 commits into
NousResearch:mainfrom
jayleaton:fix/cumulative-cron-safeguards-current-main

Conversation

@jayleaton

Copy link
Copy Markdown

Summary

Provenance

  • current-main parent: 79445a496c86a19332ad786494b8384d2167e2d0
  • reviewed cumulative parent: 7d030bace2240baaacd6ae603f2ccc57fcfa15e6
  • provider backoff: f40e3977a327b8c5c3f3edad7b957d75a0cf6a19
  • terminal durability lineage: 9b26dcb9fb1f5425b198495b655d7ef54304da3f, 51e201b67397fb596cab6af88d5ac799f8d5122e, 991095136967b0f2776d451bde02dcc05b5ad0ea, d74a8d157a5a4e3fabac716df2d9c14420ec34ae
  • task-bound protocol gate: 97ac4a6c5eb661cc963521cd9180db518cb28e01

Verification

  • RED on current main: provider backoff 16 failures; Kanban guidance 3 failures; fire-fence durability 4 failures plus shutdown-state setup errors
  • focused restored matrix: 318 passed, 0 failed
  • broad cron/protocol/integration matrix: 1421 passed, 0 failed, 2 Windows-only skipped
  • ruff, compileall, compat-pointer checker, and git diff --check pass
  • broad matrix exercised schedule parsing, execution/delivery SQLite DDL, restart-safe external worker paths, scheduled worker claim/fence boundaries, and delivery terminal accounting

Boundaries

  • no model/provider/spend/cap/security configuration changes
  • no merge, gateway start/restart/reload, cron resumption, or product sign-off
  • separately owned agent/kanban_stop.py, tools/delegate_tool*.py, and hermes_cli/kanban_db_dispatch.py are untouched
  • deployment remains gated on independent review and explicit deploy authorization

jayleaton and others added 18 commits September 3, 2026 03:59
A decomposed child with workspace_kind=worktree and no explicit path got
workspace_path=NULL. Dispatch then tried to anchor it on the board's
default_workdir and, when the board had none, failed the spawn with
'no default_workdir set' — twice, tripping the failure-limit circuit
breaker (live: t_ab2a7ce8, t_2dd0f5b7, 2026-09-02).

The root of a decomposed callback graph is usually itself a
dispatcher-materialized worktree <repo>/.worktrees/<root-id>, so the
repo is recoverable from the root row: when the root path is a linked
worktree checkout, stamp each pathless worktree child with the root's
git common-dir parent (the main repo) as its explicit anchor. Children
still get per-task worktrees (never the root's literal checkout), so
sibling isolation is unchanged; a repo that can't be recovered keeps
the anchorless row and the board-default dispatch path; scratch roots
are untouched.

Regression tests cover anchor inheritance + end-to-end child spawn
resolution with no board default_workdir, the unrecoverable-root
fallback, and scratch-root non-leakage.
A task whose title/body carries the durable PRODUCT_SIGNOFF marker defines
its job as reporting back for human sign-off rather than driving to a
terminal lifecycle transition. Repeated clean exits without one are a
property of that workflow, so _account_protocol_violation now classifies
such tasks at accounting time (read from the task row, so the exemption
survives retries, reclaims, and dispatcher restarts) and never auto-blocks
them — the violation is still recorded and the task retried, but the
gave_up breaker never fires across any number of limit-reaching runs.

Review round 1 correction for t_4067fdf1: the one-run PRODUCT_SIGNOFF test
could not reach the breaker; it is replaced by a regression that drives the
task through the full violation limit and proves it stays ready with no
gave_up event, while ordinary coder/QA omissions keep their bounded
retry/block behavior (covered by the existing limit test).
…wnership loss (c-027)

heartbeat_fire_claim() collapsed fire-fence acquisition failure into
False. During slow fenced delivery the worker's own heartbeat thread
cannot acquire the same process-local RLock, so a successfully
delivering run was misclassified as ownership loss and terminalled as
'Interrupted by shutdown before terminal completion.' (TrustMRR exec
4219b48d while RSI recovery exec f502e929 was settling).

Tri-state contract:
- True  = renewed/confirmed owner
- False = authoritatively inspected: owner mismatch / absent claim
- None  = fence unavailable/unconfirmed (contention or store error)

- initial validation stays fail-closed: run starts only on exactly True
- heartbeat loop: None rides the existing grace window (180s prod vs
  30s fence wait); immediate cancel only on explicit False
- post-run probes (_fire_claim_ownership_lost, both interrupted blocks,
  terminal owner-CAS read): None can never adjudicate a confirmed loss;
  uncertain outcomes get distinct ledger errors
- mark_job_run also tri-state: None = fence unavailable (CAS never ran),
  False stays reserved for confirmed owner mismatch
- _side_effect_fence unchanged: still the exactly-once save/delivery
  barrier

Tests (all written red-first, verified failing on unfixed code):
slow_owned_delivery (parent handoff), sustained fence-contention grace,
pre/post-delivery None probes, grace-exhausted None probe, terminal
write None, mark_job_run fence-unavailable None, RSI recovery dispatch
fails closed during original's fenced delivery, TrustMRR slow-owned
delivery terminal success. Managed-gateway restart E2E (systemd scope)
passes on host: one gateway, single side effect, single delivery.
# Conflicts:
#	agent/agent_init.py
#	agent/system_prompt.py
#	cron/jobs.py
#	cron/scheduler.py
#	cron/scheduler_provider.py
#	hermes_cli/kanban_db.py
#	hermes_cli/web_server.py
#	plugins/cron_providers/chronos/__init__.py
#	tools/cronjob_tools.py
Compose reviewed provider reset backoff, task-bound Kanban guidance, and fire-fence terminal durability from cumulative head 7d030ba onto current main 79445a4. Preserve the prior candidate as the second parent while retaining current-main content for separately owned lifecycle and dispatcher workstreams.
@alt-glitch alt-glitch added invalid This doesn't seem right P3 Low — cosmetic, nice to have comp/cron Cron scheduler and job management comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard labels Sep 5, 2026
@Enough1122

Copy link
Copy Markdown

AI code review — automated review for reference; please use your judgment.

Summary: Restores cumulative runtime safeguards on the cron path: task-bound Kanban guidance (fail-closed), clean-worker-exit lifecycle verification with durable read-back, fire-fence contention handling, and a durable provider rate-limit backoff policy. Large PR (~24 files) dominated by new tests (~1.3k lines).

Findings (Non-blocking):

  • agent/system_prompt.py:31-46 — resolve_kanban_worker_guidance requires both the tool and HERMES_KANBAN_TASK and dispatcher-owned context, failing closed on ambiguity. Lifecycle instructions that mutate board state should indeed be task-bound, not merely tool-bound. The agent_init.py + _tool_guidance_block fallback both route through it, so lightweight agents bypassing _load_tools get the same gate. Good.
  • cli.py:77-120 — _ensure_kanban_worker_lifecycle finalizes via one connection and re-reads through a fresh connection before reporting success; missing handoffs become durable protocol_violations instead of silent nominal exits. Retry delays (0/50/200ms) are short but this is a local-DB handshake, not network I/O. Fine.
  • cron/jobs.py:204-222 — _acquire_flock now re-raises non-contention OSErrors instead of treating every lock error as "busy, retry". Previously a permissions/path error would spin until timeout; now only EACCES/EAGAIN/EWOULDBLOCK retry. Correct and strictly better.
  • cron/rate_limit_backoff.py (new) — parses Retry-After / epoch-reset / human date formats with jitter and a 31-day declared-delay cap. Regex-based error-text parsing is inherently heuristic (a message mentioning "429" in passing could trigger backoff), but the consequence is only a delayed retry, and fallback bounds keep it sane. Non-blocking observation.
  • Scope note: I reviewed the agent_init/system_prompt/cli hunks and sampled cron/jobs.py + rate_limit_backoff.py; the scheduler/provider/web-router/test bulk was surveyed by file list only. No blocking issues found in the reviewed portion; the remainder follows the same fail-closed pattern per the test names.

Verdict: Looks good. Nothing blocking.

@jayleaton

jayleaton commented Sep 5, 2026 •

Copy link
Copy Markdown
Author

Cumulative P0 provider-health continuation is now pushed on this existing PR branch at exact head de34ea656013d4365e38be40aa2b78eef89355c0 (implementation commit 9f18f10549b77039c162408b07fafbabb677ac49, parent f50bf4c80f28287b474058fb64b3689695f9946b). It adds one profile-scoped SQLite circuit shared by agent fallback/error/success, cron preflight/outcome, Kanban pre-claim dispatch, and staged reset-aware coder watchdog integration. Verification: 46/46 focused provider/fallback tests; 1406 passed, 3 platform-only skipped across full cron plus dispatch/tool integrations; Ruff, compileall, compat-pointer check, and diff check pass. Deployment remains unauthorized. Manifest: ops/t_5cf9660f-deployment-manifest.json. Hotspot: dispatcher overlap with active task t_ab2d132d must be reconciled before deployment.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management invalid This doesn't seem right P3 Low — cosmetic, nice to have

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants