feat(cron): auto-pause after consecutive failures + orchestration playbook

.claude/skills/fanout/SKILL.md

@@ -0,0 +1,91 @@
+---
+name: fanout
+description: Run a complex task through the plan → parallel-workers → skeptic → synthesize playbook. Auto-invoke PROACTIVELY whenever a task decomposes into 3+ independent, parallelizable work units (research, analysis, writing, design) AND the estimated serial work exceeds ~15 minutes — especially when time-sensitive. Do NOT trigger for a single work unit, for tasks under ~15 min of serial work, or when the parts are highly interdependent (shared state, sequential dependencies, one step blocks the next). In unattended sessions (cron, loops, scheduled runs with no human to approve a plan), the plan-approval gate is skipped only when every work unit is read-only (research/analysis); if any unit would write (files, external state, sends), abort the fan-out and fall back to serial execution instead.
+---
+
+# Fanout — Parallel Execution Playbook
+
+Run a complex task through the parallel-execution playbook: **plan → parallel subagents execute → a fresh-context skeptic attacks the findings → synthesize what survives.**
+
+In Claude Code, spawn workers with the Agent/Task tool — **all in one message** so they run concurrently. Where the Workflow tool is available (web/Cowork sessions with orchestration enabled), a Workflow with a verify stage is the equivalent for larger fan-outs.
+
+## When to trigger (auto-invoke)
+
+Trigger proactively — without waiting to be asked — when a task meets ALL of:
+
+- Decomposes into **3+ independent work units** (research, analysis, writing, design)
+- Estimated serial work is **more than ~15 minutes** (the cost floor — below this, fan-out overhead isn't worth it)
+- Units don't have hard sequential dependencies or shared mutable state
+
+Do NOT trigger when:
+- The task is a single unit or under the ~15 min cost floor — do it directly
+- Parts are highly interdependent (each step needs the prior step's output) — use sequential agents instead
+- The task is simple enough that decomposition would add overhead without saving wall-clock time
+
+If a task looks parallelizable but doesn't clearly meet the bar, say so and proceed directly rather than forcing a fan-out.
+
+## Unattended-session gate
+
+The plan-approval gate (below) is mandatory in interactive sessions. In **unattended** sessions — cron, loops, scheduled runs, or anywhere there's no human able to approve a plan in real time — apply this rule instead:
+
+- **All work units read-only** (research, analysis, lookups — nothing writes files, calls send-type tools, or mutates external state): skip the approval gate, proceed straight to execution.
+- **Any work unit writes** (files, external systems, messages, commits): abort the fan-out entirely and fall back to serial execution of the task. Do not partially gate — an all-or-nothing rule keeps this predictable.
+
+This waives **only the human plan-approval gate**. The skeptic (step 3) still runs — unattended output has no human reading it before it lands, which is exactly when unchallenged findings are most likely to go unnoticed.
+
+## Procedure
+
+### 1. Plan (gated in interactive sessions)
+Enter plan mode:
+- Decompose into **3–5 discrete, parallelizable work units**, plus dependencies, per-unit effort, and risks.
+- Present the plan. In an interactive session, **wait for the user's approval** — this gate is mandatory; do not spawn agents before it. Exit plan mode only after approval.
+- In an unattended session, apply the unattended-session gate above instead of waiting for approval.
+
+If the task turns out NOT to parallelize cleanly (fewer than 3 independent units, or hard dependencies), say so and recommend doing it directly instead of forcing a fan-out.
+
+### 2. Execute (parallel workers)
+For each approved work unit, spawn one subagent — **all in a single message** so they run concurrently. Give each a self-contained prompt carrying the plan's context for its unit (add a web-search instruction if the unit needs current data).
+
+No dependencies between agents. Wall-clock time ≈ the slowest single unit, not the sum.
+
+### 3. Check (skeptic — required)
+
+**Checking is its own job.** Do not let the agent that writes the answer be the only one that grades it — a synthesizer asked to both merge and critique will reliably rate its own inputs as sound. Spawn one fresh-context skeptic over the combined worker output *before* synthesis:
+
+> You are a skeptic. Your job is to REFUTE, not to summarize. Here are findings from N parallel work units. For each material claim: is it actually supported by evidence, or asserted confidently without proof? Flag (a) unsupported claims, (b) stale or undated evidence, (c) sources that don't say what the finding claims, (d) conflicts between units, (e) anything mistaking correlation, popularity, or pain for significance. Return SURVIVES / WEAK / REFUTED per claim, with a one-line reason. Default to WEAK when evidence is thin — do not be agreeable.
+
+This stage is **required**, not conditional. Fan-out already has a ~15-minute cost floor, so anything reaching it is substantial enough to be worth checking. (The honest exception: work with no falsifiable claims — parallel creative drafts, independent mechanical passes — has nothing to refute. Run the skeptic whenever the output carries factual claims, which is nearly always.)
+
+**First, a mechanical completeness gate.** Before spawning the skeptic, confirm each unit returned *findings*, not narration. A unit that reports "waiting on results" or restates its plan has produced nothing to refute, and a skeptic prompted to attack claims will not flag their *absence*. Re-dispatch it instead.
+
+What this stage does NOT do:
+- The skeptic sees worker **output**, not sources. It judges plausibility and internal consistency; it cannot re-verify a citation it was never given.
+- Nothing structurally forces synthesis to honor the verdicts — re-read the review against the final deliverable if the stakes warrant it.
+- **Check the skeptic too.** It can refute wrongly.
+- Treat a unit's clean sweep with *more* suspicion when it read code rather than running it. Static reading cannot fail a claim the way execution can.
+
+Pass the skeptic's verdicts into synthesis.
+
+### 4. Synthesize
+After the skeptic returns:
+- Build on **surviving** evidence; drop or explicitly caveat REFUTED claims.
+- Resolve conflicts and overlaps across unit results.
+- Produce the final deliverable (report / summary / decision matrix).
+- Cite which unit each finding came from, and carry the skeptic's verdict on any claim that was WEAK or REFUTED.
+- Flag gaps and uncertain areas.
+
+### 5. Save
+- Write the plan to `plan.md` (audit trail).
+- Write the skeptic's verdicts to `review.md` — the paper trail for *what was challenged and what survived* is as useful later as the findings themselves.
+- Save the synthesis as the final deliverable, linked back to `plan.md`.
+
+## Checklist
+- [ ] Task has 3+ independent units and >~15 min estimated serial work (else do it directly)
+- [ ] Interactive session: plan approved before any worker spawns
+- [ ] Unattended session: all units read-only (gate skipped) OR fan-out aborted (any unit writes)
+- [ ] All workers dispatched in one message
+- [ ] Every unit returned actual findings, not narration (re-dispatch if not)
+- [ ] Skeptic ran over the combined output **before** synthesis
+- [ ] Skeptic's own verdicts spot-checked — it can be wrong too
+- [ ] Synthesis builds on surviving evidence; REFUTED claims dropped or caveated
+- [ ] `plan.md` + `review.md` + final deliverable saved

AGENTS.md

@@ -1184,6 +1184,11 @@ Hardening invariants:
   across processes.
 - Cron sessions pass `skip_memory=True` by default; memory providers
   intentionally do not run during cron.
+- **Consecutive-failure auto-pause**: a recurring job that fails N times
+  in a row (per-job `failure_limit` > `HERMES_CRON_FAILURE_LIMIT` env >
+  default 3; `0` disables) is paused instead of failing forever, with the
+  auto-pause notice folded into the final failure delivery. `resume` /
+  `trigger` and any successful run reset the streak. One-shots are exempt.
 
 Cron deliveries are **not** mirrored into the target gateway session —
 they land in their own cron session with a header/footer frame so the

cron/jobs.py

@@ -237,6 +237,52 @@ def _oneshot_run_claim_ttl_seconds() -> float:
     )
 
 
+# Recurring jobs that fail this many times in a row are auto-paused instead
+# of firing (and alerting) on every subsequent tick forever — the classic
+# failure mode being a rotated credential. Operators fix the cause and
+# `hermes cron resume <id>`; resume/trigger reset the streak. Precedence:
+# per-job "failure_limit" > HERMES_CRON_FAILURE_LIMIT env > this default.
+# 0 or negative disables auto-pause.
+DEFAULT_CONSECUTIVE_FAILURE_LIMIT = 3
+
+
+def _consecutive_failure_limit(job: Dict[str, Any]) -> int:
+    raw = job.get("failure_limit")
+    if raw is None:
+        raw = os.getenv("HERMES_CRON_FAILURE_LIMIT", "").strip() or None
+    if raw is None:
+        return DEFAULT_CONSECUTIVE_FAILURE_LIMIT
+    try:
+        return int(raw)
+    except (TypeError, ValueError):
+        return DEFAULT_CONSECUTIVE_FAILURE_LIMIT
+
+
+def _auto_pause_pending(job: Dict[str, Any], additional_failures: int = 0) -> bool:
+    """True when the job's failure streak (plus ``additional_failures`` not
+    yet recorded) has reached the auto-pause limit.
+
+    Only recurring schedules qualify — one-shots reach a terminal state on
+    their own and must not be re-labeled "paused".
+    """
+    if job.get("schedule", {}).get("kind") not in {"cron", "interval"}:
+        return False
+    limit = _consecutive_failure_limit(job)
+    if limit <= 0:
+        return False
+    return int(job.get("consecutive_failures") or 0) + additional_failures >= limit
+
+
+def failure_would_pause(job: Dict[str, Any]) -> bool:
+    """Return True when recording one more failure would auto-pause ``job``.
+
+    The scheduler uses this to fold the auto-pause notice into the failure
+    delivery of the run that trips the limit, so the operator gets one
+    escalation instead of an unexplained silence.
+    """
+    return _auto_pause_pending(job, additional_failures=1)
+
+
 def _job_running_in_this_process(job_id: str) -> bool:
     """Return True when the scheduler in THIS process is still running ``job_id``.
 
@@ -1422,6 +1468,7 @@ def create_job(
         "last_status": None,
         "last_error": None,
         "last_delivery_error": None,
+        "consecutive_failures": 0,
         # Delivery configuration
         "deliver": deliver,
         "origin": origin,  # Tracks where job was created for "origin" delivery
@@ -1655,6 +1702,9 @@ def resume_job(job_id: str) -> Optional[Dict[str, Any]]:
             "state": "scheduled",
             "paused_at": None,
             "paused_reason": None,
+            # Resuming is an operator statement that the cause is fixed —
+            # restart the auto-pause streak from zero.
+            "consecutive_failures": 0,
             "next_run_at": next_run_at,
         },
     )
@@ -1672,6 +1722,8 @@ def trigger_job(job_id: str) -> Optional[Dict[str, Any]]:
             "state": "scheduled",
             "paused_at": None,
             "paused_reason": None,
+            # Manual trigger, like resume, resets the auto-pause streak.
+            "consecutive_failures": 0,
             "next_run_at": _hermes_now().isoformat(),
         },
     )
@@ -1719,6 +1771,13 @@ def mark_job_run(job_id: str, success: bool, error: Optional[str] = None,
                 job["last_run_at"] = now
                 job["last_status"] = "ok" if success else "error"
                 job["last_error"] = error if not success else None
+                # Consecutive-failure streak: reset on success, else grow.
+                # Absent key (pre-existing records) reads as 0. Drives the
+                # recurring-job auto-pause below.
+                if success:
+                    job["consecutive_failures"] = 0
+                else:
+                    job["consecutive_failures"] = int(job.get("consecutive_failures") or 0) + 1
                 # Track delivery failures separately — cleared on successful delivery
                 job["last_delivery_error"] = delivery_error
                 # Clear any external-fire claim so a re-armed recurring job can
@@ -1798,7 +1857,23 @@ def mark_job_run(job_id: str, success: bool, error: Optional[str] = None,
                         job["enabled"] = False
                         job["state"] = "completed"
                 elif job.get("state") != "paused":
-                    job["state"] = "scheduled"
+                    if not success and _auto_pause_pending(job):
+                        job["enabled"] = False
+                        job["state"] = "paused"
+                        job["paused_at"] = now
+                        job["paused_reason"] = (
+                            f"auto-paused after {job['consecutive_failures']} "
+                            "consecutive failures"
+                        )
+                        logger.warning(
+                            "Job '%s' (%s) auto-paused after %d consecutive "
+                            "failures; fix the cause and resume with "
+                            "`hermes cron resume %s`.",
+                            job.get("name", "?"), job["id"],
+                            job["consecutive_failures"], job["id"],
+                        )
+                    else:
+                        job["state"] = "scheduled"
 
                 save_jobs(jobs)
                 return

cron/scheduler.py

@@ -282,7 +282,7 @@ def _resolve_cron_enabled_toolsets(job: dict, cfg: dict) -> list[str] | None:
     "QQBOT_HOME_CHANNEL": "QQ_HOME_CHANNEL",
 }
 
-from cron.jobs import get_due_jobs, mark_job_run, save_job_output, advance_next_runs, claim_dispatch, heartbeat_run_claim
+from cron.jobs import get_due_jobs, mark_job_run, save_job_output, advance_next_runs, claim_dispatch, heartbeat_run_claim, failure_would_pause
 from cron.jobs import get_ticker_heartbeat_age
 from cron.executions import create_execution, finish_execution, mark_execution_running
 
@@ -4035,6 +4035,17 @@ def run_one_job(job: dict, *, adapters=None, loop=None, verbose: bool = False) -
             # If the agent responded with [SILENT], skip delivery (but
             # output is already saved above).  Failed jobs always deliver.
             deliver_content = final_response if success else _summarize_cron_failure_for_delivery(job, error)
+            # When this failure trips the consecutive-failure limit,
+            # mark_job_run below will auto-pause the job — say so in the
+            # same delivery, so the operator gets one escalation instead of
+            # per-tick failure spam followed by unexplained silence.
+            if not success and failure_would_pause(job):
+                streak = int(job.get("consecutive_failures") or 0) + 1
+                deliver_content += (
+                    f"\n⏸️ Auto-pausing this job after {streak} consecutive "
+                    f"failures. It will not run again until resumed: "
+                    f"`hermes cron resume {job['id']}`"
+                )
             # Treat whitespace-only final responses the same as empty
             # responses: do not deliver a blank message, and let the
             # empty-response guard below mark the run as a soft failure.