feat(cron): dead-pin guard — auto-pause + alert on missing script by sycamoregroupltd · Pull Request #73506 · NousResearch/hermes-agent

sycamoregroupltd · 2026-07-28T17:22:16Z

Summary

Carries the cron dead-pin guard (cron script path missing → auto-pause + critical-alert) into main. Source was green in the unmerged wt/t_02f2bb64_control_center worktree but never shipped to the live editable checkout, leaving the running agent unable to detect a dead/missing cron script — exactly the silent-failure class this guard prevents.

Scope is only the 4 guard commits, cherry-picked cleanly onto current origin/main (no conflict; the 3071-file control-center branch is excluded).

Commits

774463d4f feat(cron): reject missing script at create/update time (dead-pin enable-time guard)
fdf4f26b1 feat(cron): fire-time dead-pin alert + auto-pause for missing scripts
184f86309 test(cron): add fire-time dead-pin tests for run_job auto-pause behavior
53ebf7bd9 test(cron): add dead-pin alert-delivery + validator unit tests

Acceptance evidence (pre-merge, on clean checkout of branch tip)

python3 -m pytest tests/cron/test_cron_script.py → 52 passed, 0 failed (dead-pin tests ≥ 14).
python3 -m py_compile cron/scheduler.py → OK; no conflict markers.
grep -rci "dead.pin" --include=*.py → scheduler.py=13, test_cron_script.py=24, cronjob_tools.py=2.
cron/scheduler.py contains _handle_cron_dead_pin (def @2640) and 'dead-pin: script not found:' pause reason (used @2777, @2907).
Branch base is an ancestor of current origin/main → clean, conflict-free merge.

Live verification (AC5 — post-merge/landing)

After this PR lands on the live editable checkout and the gateway restarts, verify the running agent auto-pauses + critical-alerts when a cron script path is missing. (Tracked separately by devops once merged.)

Notes

No credentials/secrets, no provider/model routing, no live trading, no prod trading deploy touched.
Mirrors task t_23fc8d82 (jarvis-os). Parent verdict context: t_b50f908a (sycode-trading, eval-runner independent verdict).

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

…ble-time guard) _validate_cron_script_path now checks the resolved script path actually exists on disk within HERMES_HOME/scripts/ and rejects non-existent files at create/update with a clear tool_error. Empty/None still clears the field. Kept strictly to the enable-time gate: the out-of-scope fire-time auto-pause + #critical-alerts scheduler behavior is intentionally excluded per task scope (no change to other gates/scheduling). Tests: added TestDeadPinGuardEnableTime (create/update rejects missing, subdir missing, existing allowed); seeded scripts in existing tests so they keep passing. All 40 tests in test_cron_script.py pass.

In run_job, when _run_job_script fails with the missing-file class ('Script not found' / 'not a file'), emit a #critical-alerts ping (same target as dqsh_audit_watchdog) AND auto-pause the job via cron.jobs.pause_job (enabled=False, state='paused', paused_reason='dead-pin: script not found: <path>'). Only missing-file failures auto-pause; transient failures (non-zero exit, timeout) alert via the normal error path but keep firing. Wires the LLM path (pre-check) and the no_agent path. Schedule untouched. No credential/secret/ provider changes.

Covers the real run_job entry point (no_agent + LLM paths): - missing script auto-pauses (enabled=False, state='paused', paused_reason set, schedule untouched) - directory path ('not a file') auto-pauses - transient non-zero exit does NOT auto-pause (keeps firing) - LLM path pre-check also auto-pauses on missing script

- TestValidateCronScriptPathDeadPin: unit pins _validate_cron_script_path rejects missing scripts (incl. subdir) and allows existing/empty (req 1). - TestRunJobDeadPinFireTime.test_missing_script_delivers_alert_no_agent: asserts the #critical-alerts ping actually fires on a missing script, not just auto-pause (req 2a). - TestRunJobDeadPinFireTime.test_transient_failure_no_deadpin_alert: asserts a non-missing failure does NOT raise the dead-pin alert and keeps firing (req 3 hardening). Full tests/cron/ = 452 passed, 0 failed.

teknium1

Thanks for covering both creation-time validation and files deleted after scheduling. The missing-path acceptance gap remains on current main, but this needs rework before salvage.

Problems

cron/scheduler.py:2759 hard-codes a Discord channel and bypasses cron's configured delivery routing (cron/scheduler.py:1450-1483). A dead-pin notice includes job and script details, so it must not be sent to a fixed external destination.
cron/scheduler.py:2775 classifies failures by substring. _run_job_script() includes arbitrary stderr/stdout in non-zero-exit failures (cron/scheduler.py:2325-2331), so a script that merely prints not a file would be auto-paused.
The tests stop at run_job, but real execution goes through run_one_job (cron/scheduler.py:3904-4030). For a finite one-shot, mark_job_run() removes the job once its limit is reached (cron/jobs.py:1739-1744), defeating the requested retained-paused state.

Suggested changes

Route alerts only through a user-configured target, not a literal Discord ID.
Use a structured or exact scheduler-owned missing-path signal and test stderr false positives.
Add recurring and one-shot run_one_job coverage proving the paused job survives.

Automated hermes-sweeper review.

teknium1 · 2026-07-30T16:06:59Z

+# Reused by the dead-pin guard below. Do not invent a new channel — this is
+# the one the dqsh_audit_watchdog cron (0ead099b5eac) and other critical
+# alerts already target.
+_CRITICAL_ALERTS_CHANNEL_ID = "1521973787363508325"  # #critical-alerts (Discord)


This literal Discord channel bypasses the job's configured delivery targets and would send one user's cron failure details to a fixed destination whenever Discord is enabled. Please remove the fixed channel and route through a user-configured delivery mechanism.

teknium1 · 2026-07-30T16:06:59Z

+    error behind a paused job.
+    """
+    if not script_output:
+        return False


This substring test also sees arbitrary stderr/stdout from a non-zero script exit. A healthy script that prints "not a file" would be classified as a permanent missing-path failure and auto-paused. Use a structured result or an exact scheduler-owned error prefix, and add that regression case.

…lert-once) Validate a job's configuration BEFORE any agent machinery is constructed: - missing provider API key (AuthError from a read-only resolve_runtime_provider probe; skipped when a fallback_providers chain is configured, since auth-fallback may rescue the run) - attached skill not ready (skill_view readiness_status=setup_needed — missing required env vars / commands / credential files) - delivery platform unknown or unconnected (deliver=local/origin/all are never checked; gateway-config load failures fail open) On a failing check run_job returns a [blocked_config]-marked error without constructing AIAgent/MCP/etc, so a misconfigured job never burns an LLM call. run_one_job records last_status='blocked_config' and delivers the alert exactly ONCE across ticks (persisted preflight_alerted bit — the alert-once shape from the #73506 dead-pin auto-pause); the next healthy run clears the marker so a future break re-alerts. Every preflight check fails open: only an affirmative misconfiguration verdict blocks. Config: cron.preflight (default true); `cron.preflight: false` restores the old fail-during-run behavior. Documented in the cron user guide and config defaults. mark_job_run gains an optional status= override (unblocked call shape unchanged) and drops preflight_alerted on any successful run. Tests: tests/cron/test_preflight_config.py (blocked_config + no agent + single alert across two ticks, healthy job unaffected, recovery clears dedup, fallback-chain rescue, opt-out restores old behavior, skill readiness miss, unknown delivery platform, deliver=local never loads gateway config). Full tests/cron/ + cronjob tool suite green (525 tests). Ported from: paperclipai/paperclip execution-semantics §5 (MIT); in-repo precedent: #27948, #73506

…lert-once) Validate a job's configuration BEFORE any agent machinery is constructed: - missing provider API key (AuthError from a read-only resolve_runtime_provider probe; skipped when a fallback_providers chain is configured, since auth-fallback may rescue the run) - attached skill not ready (skill_view readiness_status=setup_needed — missing required env vars / commands / credential files) - delivery platform unknown or unconnected (deliver=local/origin/all are never checked; gateway-config load failures fail open) On a failing check run_job returns a [blocked_config]-marked error without constructing AIAgent/MCP/etc, so a misconfigured job never burns an LLM call. run_one_job records last_status='blocked_config' and delivers the alert exactly ONCE across ticks (persisted preflight_alerted bit — the alert-once shape from the NousResearch#73506 dead-pin auto-pause); the next healthy run clears the marker so a future break re-alerts. Every preflight check fails open: only an affirmative misconfiguration verdict blocks. Config: cron.preflight (default true); `cron.preflight: false` restores the old fail-during-run behavior. Documented in the cron user guide and config defaults. mark_job_run gains an optional status= override (unblocked call shape unchanged) and drops preflight_alerted on any successful run. Tests: tests/cron/test_preflight_config.py (blocked_config + no agent + single alert across two ticks, healthy job unaffected, recovery clears dedup, fallback-chain rescue, opt-out restores old behavior, skill readiness miss, unknown delivery platform, deliver=local never loads gateway config). Full tests/cron/ + cronjob tool suite green (525 tests). Ported from: paperclipai/paperclip execution-semantics §5 (MIT); in-repo precedent: NousResearch#27948, NousResearch#73506

A fleet-wide inference config change previously produced one 'Skipped to prevent unintended spend' alert per unpinned job per tick — 40 jobs meant 40 alerts every tick until each was re-pinned (Coatue field report, 2026-08-11). The #44585 guard now reuses the #73506 alert-once shape the preflight path already established: a persisted drift_alerted bit on the job record, a [drift_skip:silent] marker on repeat ticks that suppresses delivery, and the bit clears on the next successful run so a future drift re-alerts. Only the drift branch consults the bit — every other failure keeps alerting per tick. The alert text also now says it is sent once, so operators know the job stays skipped silently until pinned or restored.

Copilot AI review requested due to automatic review settings July 28, 2026 17:22

Copilot AI reviewed Jul 28, 2026

alt-glitch added type/feature New feature or request comp/cron Cron scheduler and job management P3 Low — cosmetic, nice to have labels Jul 28, 2026

Hermes Builder added 4 commits July 29, 2026 00:50

sycamoregroupltd force-pushed the fix/cron-dead-pin branch from 53ebf7b to 29333d6 Compare July 28, 2026 23:50

teknium1 reviewed Jul 30, 2026

View reviewed changes

teknium1 mentioned this pull request Aug 7, 2026

feat(cron): pre-dispatch configuration validation (blocked_config + alert-once) #81140

Merged

teknium1 mentioned this pull request Aug 13, 2026

fix(cron): honest fallback/timeout/drift alerts + drift-guard alert-once #85508

Merged

Navigation Menu

Uh oh!

feat(cron): dead-pin guard — auto-pause + alert on missing script - #73506

feat(cron): dead-pin guard — auto-pause + alert on missing script#73506
sycamoregroupltd wants to merge 4 commits into
NousResearch:mainfrom
sycamoregroupltd:fix/cron-dead-pin

sycamoregroupltd commented Jul 28, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

teknium1 left a comment

Uh oh!

teknium1 Jul 30, 2026

Uh oh!

teknium1 Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

sycamoregroupltd commented Jul 28, 2026

Summary

Commits

Acceptance evidence (pre-merge, on clean checkout of branch tip)

Live verification (AC5 — post-merge/landing)

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

teknium1 left a comment

Choose a reason for hiding this comment

Problems

Suggested changes

Uh oh!

teknium1 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

teknium1 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants