feat(cron): auto-pause after consecutive failures + orchestration playbook

[dizhaky]

What does this PR do?

Two related changes:

1. docs/agents-loops-graphs.md — a playbook mapping the agents/loops/graphs orchestration patterns onto this stack's existing capabilities, with a decision matrix for the four orchestration layers (delegate_task, execute_code PTC, cron, Kanban DAG/swarm) and a prioritized gap list.
2. Gap 1 implemented — cron consecutive-failure auto-pause. Previously a recurring job broken by e.g. a rotated credential failed and alerted on every tick forever ("not a retry queue", no failure ceiling). Now mark_job_run tracks a per-job consecutive_failures streak; when it reaches the limit, the job is auto-paused with an explanatory paused_reason, and the scheduler folds the auto-pause notice into the failure delivery of the tripping run — one escalation instead of per-tick spam followed by unexplained silence.

Limit precedence: per-job failure_limit > HERMES_CRON_FAILURE_LIMIT env > default 3; 0 disables. Streak resets on success, resume, or manual trigger. One-shots are exempt (they reach a terminal state on their own). Manual pauses are never overwritten. Mirrors the Kanban dispatcher's existing failure_limit semantics.

Related Issue

Fixes # (none — from a research/build request)

Type of Change

• ✨ New feature (non-breaking change that adds functionality)
• 📝 Documentation update

Changes Made

• cron/jobs.py: consecutive_failures field on job records; streak increment/reset in mark_job_run; auto-pause in the reschedule branch; _consecutive_failure_limit / _auto_pause_pending / public failure_would_pause helpers; streak reset in resume_job and trigger_job
• cron/scheduler.py: _process_job appends the auto-pause notice to the failure delivery when the run trips the limit
• tests/cron/test_failure_auto_pause.py (new): 15 tests — streak accounting, default/env/per-job limits, disable via 0, one-shot exemption, manual-pause preservation, resume/trigger resets, failure_would_pause predicate, paused job not due
• AGENTS.md: documented the new hardening invariant in the scheduled-tasks section
• docs/agents-loops-graphs.md (new): orchestration playbook; Gap 1 marked as built on this branch

How to Test

1. pytest tests/cron/test_failure_auto_pause.py -q — 15 passed
2. pytest tests/cron/ -q — 414 passed; 4 pre-existing environment failures (missing _cffi_backend native module in the test container) reproduce identically on the base commit with this patch stashed
3. Manual: create a recurring job with a failing prompt/model, watch it pause after 3 failures with the escalation line in the delivery, then hermes cron resume <id> and confirm the streak restarts from zero

Checklist

Code

• My commit messages follow Conventional Commits (feat(cron): ...)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature
• I've run the cron test suite; all pass except 4 pre-existing environment failures verified on the base commit
• I've added tests for my changes
• I've tested on my platform: Linux (remote container)

Documentation & Housekeeping

• I've updated relevant documentation (AGENTS.md, docs/)
• cli-config.yaml.example — N/A (env var + per-job field, no config key added)
• I've updated AGENTS.md (scheduled-tasks hardening invariants)
• Cross-platform impact considered — pure-Python logic, no platform-specific calls
• Tool descriptions/schemas — N/A (no tool schema change; failure_limit is settable via update_job)

🤖 Generated with Claude Code

https://claude.ai/code/session_01MW1vwtetBTLSMJcBYp9r1C

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.