From b4aecef44cd7d216e806ec03b39d6822cc8629ec Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 10:17:39 +0000 Subject: [PATCH 1/5] docs: add agents/loops/graphs orchestration playbook for this stack Maps the agent/loop/graph patterns onto existing Hermes capabilities (delegate_task, execute_code PTC, cron, Kanban DAG + swarm), documents the seams between layers, and prioritizes four reliability gaps to close (cron failure auto-pause, output gates, pre_verify rubrics, delegation defaults). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01MW1vwtetBTLSMJcBYp9r1C --- docs/agents-loops-graphs.md | 219 ++++++++++++++++++++++++++++++++++++ 1 file changed, 219 insertions(+) create mode 100644 docs/agents-loops-graphs.md diff --git a/docs/agents-loops-graphs.md b/docs/agents-loops-graphs.md new file mode 100644 index 0000000000000..2339db43cdf51 --- /dev/null +++ b/docs/agents-loops-graphs.md @@ -0,0 +1,219 @@ +# Agents, Loops, and Graphs — Playbook for This Stack + +A mapping of the three orchestration patterns (autonomous agents, self-checking +loops, dependency-graph fan-outs) onto what this stack actually runs today, and +a prioritized list of what to build next. Written against the fork at +`dizhaky/hermes-agent`, the Claude Code environment, and the live fleet +(mfc1 gateway, memory-gateway, vault daemons, Linear-driven ops). + +The short version: **all three patterns already exist here.** The work is not +building them — it is (1) using the right layer for the right job, (2) closing +four specific reliability gaps, and (3) adopting two conventions that make +loops and graphs trustworthy: real pass/fail checks and fresh-context +verifiers. + +--- + +## 1. What already exists (do not rebuild) + +### Agents — this stack is already at "Level 4" + +| Capability | Where it lives | +|---|---| +| Tool-calling agent runtime | Hermes core (`run_agent.py`, `agent/`) and Claude Code | +| Cross-session memory | Memory providers (`plugins/memory/` — memgw, honcho, etc.), FTS5 session search (`tools/session_search_tool.py`), Obsidian vault, Linear | +| Learning loop | Background review (`agent/background_review.py`) auto-creates/patches skills; Curator (`agent/curator.py`) archives stale ones | +| Triggered/scheduled autonomy | `hermes cron`, webhook subscriptions, gateway delivery to Telegram/Slack/etc. | + +The "paste a checkpoint prompt" advice for the memory problem is mostly +obsolete here. The durable-state conventions already in use are better: +Linear issues for work state, `docs/system-log/` for daily operational +history, and the vault for knowledge. **Convention to adopt:** any agent task +expected to outlive one session gets a Linear issue at start (the checkpoint) +and a system-log line at end. That is the checkpoint pattern, made durable +and searchable instead of pasted. + +Similarly, "reusable system prompts" are skills in this stack. A recurring +agent role (researcher, data analyst, code agent) should be codified as a +Hermes skill or Claude Code skill — not a prompt kept in a note. Skills are +versioned, curator-managed, loadable per cron job (`--skill`), and +progressively disclosed instead of burning context. + +### Loops — several already in production + +- **Cron jobs** (`cron/`): four schedule shapes, per-job model pinning, + script pre-injection, `no_agent` script-only watchdog mode, `context_from` + chaining, model-policy guard for legal/finance workloads, prompt-injection + scan of assembled prompts. +- **verify-on-stop** (`agent/verification_stop.py`): a *real* check — edited + code must have fresh passing verification evidence (exit code 0 recorded in + SQLite) before the agent may finish. This is the article's "check that can + actually fail," already built. +- **CI auto-healer / auto-fix / escalation detector** (`.github/workflows/`): + a production fix-until-green loop on this fork. +- **Fleet loops**: vault healer, env-guard watchdog, config-integrity + watchdog, ugw-health-check — all existing loop-shaped automation. + +### Graphs — Kanban is the DAG scheduler + +The under-advertised finding: **Hermes already has a dependency-graph +scheduler.** The Kanban subsystem (`tools/kanban_tools.py`, +`hermes_cli/kanban_db.py`) supports: + +- `parents: [...]` edges; a task stays in `todo` until every parent is `done`, + then auto-promotes to `ready` (fan-in for free) +- cycle rejection via topological sort +- a dispatcher inside the gateway that spawns one OS subprocess per task, + with `max_in_progress`, per-profile caps, per-task `max_runtime_seconds`, + and auto-block after `failure_limit` consecutive failures +- per-task model/provider/toolset overrides (cheap models for cheap nodes) +- git-worktree workspaces for parallel code mutation without conflicts +- **`hermes kanban swarm`** — a prebuilt diamond: parallel workers → verifier + → synthesizer, with a shared blackboard on the root task + +That last item is exactly the article's fan-out/converge diamond *plus* its +fresh-context checker, as one command. + +On the Claude Code side, the Workflow tool provides the same shape for +repo-scale work (parallel finders → adversarial verifiers → synthesis), and +subagents cover flat fan-out. + +--- + +## 2. The decision matrix — which layer for which job + +The stack has four orchestration layers that do not share a scheduler. Route +work by durability and shape: + +| Job shape | Use | Why | +|---|---|---| +| Multi-angle work inside a chat, results needed this turn | `delegate_task` batch mode | Flat parallel fan-out, isolated contexts, summaries return as one message | +| Mechanical multi-step pipeline (loop over 40 files, retry-with-backoff, filter/aggregate) | `execute_code` (PTC) | Intermediate results never enter context; plain code instead of tokens — the article's "REDUCE — no model, no tokens" step | +| Durable DAG, overnight or unattended, mixed models, code mutation | Kanban (`hermes kanban swarm`, `kanban_create` with `parents`) | Survives restarts, real dependency edges, failure auto-block, worktree isolation | +| Scheduled monitoring/reporting | `hermes cron` — with `no_agent` + script whenever the check is mechanical | LLM-free watchdogs cost nothing and cannot hallucinate a pass | +| Repo-scale code review/migration/audit in Claude Code | Workflow tool / subagent fan-out | Worktree isolation, schema-validated agent outputs, adversarial verify built into the pattern | + +Known seams to respect (verified in code, not guesses): + +- `delegate_task` children have the kanban toolset **stripped** — a subagent + cannot enqueue DAG work. Orchestration across the seam must be done by the + parent or via `execute_code`/CLI. +- `delegate_task` has **no dependency edges** — it is flat fan-out that joins + on all children. Anything with stage-2-needs-stage-1 structure belongs in + Kanban or a PTC script. +- `execute_code` RPC calls are serialized (global lock) — no tool-call + parallelism inside a script, and it cannot spawn subagents. +- Cron sessions get a hard interrupt and skip memory providers by design — + do not put memory-dependent reasoning in cron prompts. + +--- + +## 3. The four gaps worth closing (prioritized) + +### Gap 1 — Cron has no retry and no consecutive-failure auto-pause ⚠ highest leverage + +`cron/executions.py` states it plainly: "not a retry queue." A failed run +delivers a one-line failure and waits for the next tick; nothing counts +consecutive failures; a job broken by a rotated credential fails quietly +forever. This is the root of the standing "Infrastructure Health & Alerts" +noise and the open cron-cleanup/health-check Linear projects. + +**Build:** track `consecutive_failures` in the job record in +`cron/scheduler.py`; after N (default 3), auto-pause the job and deliver one +escalation (Slack + Linear issue via the existing delivery path) instead of +per-tick failure spam. Mirror of Kanban's existing `failure_limit` semantics +— the pattern is already in the codebase, just not in cron. + +### Gap 2 — Cron outputs have no pass/fail gate + +The `[SILENT]` sentinel is self-reported by the model; nothing validates that +a "weekly digest" contains a digest. Two mitigations, no new infrastructure: + +- **Prefer `no_agent` script jobs for every check that is mechanical.** + Script exit semantics are the gate. Reserve the LLM for jobs that need + reasoning, fed by `--script` output. +- **For LLM jobs that matter, chain a checker**: a second cron job with + `context_from` pointing at the producer, running a *different* (cheap) + model, whose only task is "does this output meet the criteria — reply + PASS or a one-line failure." The worker never grades its own homework. + +### Gap 3 — verify-on-stop covers code only + +The evidence gate keys on file mutations and detected verify commands, and +caps at 2 attempts. Research/writing/analysis turns have no equivalent. The +`pre_verify` hook surface (`agent/verify_hooks.py`) ships empty — it is the +designed extension point. **Build (optional, later):** a `pre_verify` hook +that enforces per-task rubrics for non-code deliverables (accounting workpapers +already have this culturally via the audit-QC skill; the hook mechanizes it). + +### Gap 4 — Delegation defaults are conservative for this hardware + +`delegation.max_concurrent_children` defaults to 3 and +`max_spawn_depth` to 1, config-only. For mfc1-class hardware, set in +`~/.hermes/config.yaml`: + +```yaml +delegation: + max_concurrent_children: 5 # from 3 + max_spawn_depth: 1 # keep flat; raise to 2 only for a real orchestrator need +``` + +Leave depth at 1 until a concrete task needs nesting — depth is where +runaway costs live. + +--- + +## 4. Two conventions that make all of it trustworthy + +**1. A check that can fail, or it is not a loop.** Every recurring automation +must name its gate: a script exit code (`no_agent`), a verify command +(verify-on-stop), CI status (auto-healer), or a chained checker job. If the +gate is "the model says it's done," it is a draft generator, not a loop. +Track the **keep rate** — outputs acted on ÷ outputs produced. Below ~50%, +the automation costs more than doing the task by hand; fix the gate or kill +the job. + +**2. Worker and checker never share a context.** Kanban swarm's verifier, +Claude Code's adversarial-verify stage, and the chained-checker cron pattern +all enforce this structurally. When composing ad hoc (e.g. `delegate_task`), +spawn the checker as a *separate* child that receives only the finding — +never the worker's conversation. + +--- + +## 5. Applying it to live projects + +- **Infra health (System Health Remediator, cron cleanup):** convert + LLM-based health checks to `no_agent` scripts; land Gap 1 so broken jobs + pause-and-escalate once instead of alerting forever. This directly + retires standing alert noise. +- **Month-end close (3 entities):** a Kanban swarm — three parallel + per-entity reconciliation workers → audit-QC verifier (fresh context) → + synthesis into the close package. The entities are genuinely independent; + the convergence genuinely needs all three. Textbook diamond. +- **OneDrive reorg loop:** already loop-shaped; add the keep-rate metric and + a failure ceiling per run so it self-reports when classification quality + drops instead of grinding on. +- **M&A / diligence work:** fan out the existing skill suite (commercial DD, + legal DD, valuation) as parallel Kanban workers or Claude Code subagents; + converge through a verifier before the synthesis memo. +- **Morning brief / digests:** cron `context_from` chains — collectors feed + a composer; composer output optionally gated by a cheap checker (Gap 2 + pattern). + +--- + +## 6. Suggested build order + +1. **Gap 1** — cron consecutive-failure auto-pause + single escalation + (small patch, `cron/scheduler.py` + `cron/jobs.py`, mirrors Kanban's + `failure_limit`; closes real open Linear work). +2. **Gap 2 conventions** — sweep existing cron jobs: mechanical checks → + `no_agent` scripts; add chained checkers to the LLM jobs that feed + decisions. +3. **Gap 4** — one-line config bump on the gateway host. +4. **First Kanban swarm in anger** — run one real diamond (month-end close + or a diligence sprint) end-to-end; capture what worked as a skill so the + background-review loop compounds it. +5. **Gap 3** — `pre_verify` rubric hook, only if non-code loop quality + becomes a felt problem. From e7e49e50b59f1028bedd1088ce17c590b452ed65 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 10:33:26 +0000 Subject: [PATCH 2/5] feat(cron): auto-pause recurring jobs after consecutive failures MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A recurring job that fails N times in a row is now paused instead of firing (and alerting) on every subsequent tick forever — the classic failure mode being a rotated credential. mark_job_run tracks a consecutive_failures streak per job (reset on success, resume, or manual trigger); when the streak reaches the limit, the job is paused with an explanatory paused_reason, and the scheduler folds the auto-pause notice into the failure delivery of the tripping run so the operator gets one escalation instead of per-tick spam followed by unexplained silence. Limit precedence: per-job failure_limit > HERMES_CRON_FAILURE_LIMIT env > default 3; zero or negative disables. One-shots are exempt — they reach a terminal state on their own. Manual pauses are never overwritten. Mirrors the Kanban dispatcher's existing failure_limit semantics. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01MW1vwtetBTLSMJcBYp9r1C --- AGENTS.md | 5 + cron/jobs.py | 77 +++++++++++- cron/scheduler.py | 13 +- docs/agents-loops-graphs.md | 12 +- tests/cron/test_failure_auto_pause.py | 174 ++++++++++++++++++++++++++ 5 files changed, 274 insertions(+), 7 deletions(-) create mode 100644 tests/cron/test_failure_auto_pause.py diff --git a/AGENTS.md b/AGENTS.md index e8096816eedb0..cd08abbefdc19 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1184,6 +1184,11 @@ Hardening invariants: across processes. - Cron sessions pass `skip_memory=True` by default; memory providers intentionally do not run during cron. +- **Consecutive-failure auto-pause**: a recurring job that fails N times + in a row (per-job `failure_limit` > `HERMES_CRON_FAILURE_LIMIT` env > + default 3; `0` disables) is paused instead of failing forever, with the + auto-pause notice folded into the final failure delivery. `resume` / + `trigger` and any successful run reset the streak. One-shots are exempt. Cron deliveries are **not** mirrored into the target gateway session — they land in their own cron session with a header/footer frame so the diff --git a/cron/jobs.py b/cron/jobs.py index 23ce8c184175e..6f78079442eed 100644 --- a/cron/jobs.py +++ b/cron/jobs.py @@ -237,6 +237,52 @@ def _oneshot_run_claim_ttl_seconds() -> float: ) +# Recurring jobs that fail this many times in a row are auto-paused instead +# of firing (and alerting) on every subsequent tick forever — the classic +# failure mode being a rotated credential. Operators fix the cause and +# `hermes cron resume `; resume/trigger reset the streak. Precedence: +# per-job "failure_limit" > HERMES_CRON_FAILURE_LIMIT env > this default. +# 0 or negative disables auto-pause. +DEFAULT_CONSECUTIVE_FAILURE_LIMIT = 3 + + +def _consecutive_failure_limit(job: Dict[str, Any]) -> int: + raw = job.get("failure_limit") + if raw is None: + raw = os.getenv("HERMES_CRON_FAILURE_LIMIT", "").strip() or None + if raw is None: + return DEFAULT_CONSECUTIVE_FAILURE_LIMIT + try: + return int(raw) + except (TypeError, ValueError): + return DEFAULT_CONSECUTIVE_FAILURE_LIMIT + + +def _auto_pause_pending(job: Dict[str, Any], additional_failures: int = 0) -> bool: + """True when the job's failure streak (plus ``additional_failures`` not + yet recorded) has reached the auto-pause limit. + + Only recurring schedules qualify — one-shots reach a terminal state on + their own and must not be re-labeled "paused". + """ + if job.get("schedule", {}).get("kind") not in {"cron", "interval"}: + return False + limit = _consecutive_failure_limit(job) + if limit <= 0: + return False + return int(job.get("consecutive_failures") or 0) + additional_failures >= limit + + +def failure_would_pause(job: Dict[str, Any]) -> bool: + """Return True when recording one more failure would auto-pause ``job``. + + The scheduler uses this to fold the auto-pause notice into the failure + delivery of the run that trips the limit, so the operator gets one + escalation instead of an unexplained silence. + """ + return _auto_pause_pending(job, additional_failures=1) + + def _job_running_in_this_process(job_id: str) -> bool: """Return True when the scheduler in THIS process is still running ``job_id``. @@ -1422,6 +1468,7 @@ def create_job( "last_status": None, "last_error": None, "last_delivery_error": None, + "consecutive_failures": 0, # Delivery configuration "deliver": deliver, "origin": origin, # Tracks where job was created for "origin" delivery @@ -1655,6 +1702,9 @@ def resume_job(job_id: str) -> Optional[Dict[str, Any]]: "state": "scheduled", "paused_at": None, "paused_reason": None, + # Resuming is an operator statement that the cause is fixed — + # restart the auto-pause streak from zero. + "consecutive_failures": 0, "next_run_at": next_run_at, }, ) @@ -1672,6 +1722,8 @@ def trigger_job(job_id: str) -> Optional[Dict[str, Any]]: "state": "scheduled", "paused_at": None, "paused_reason": None, + # Manual trigger, like resume, resets the auto-pause streak. + "consecutive_failures": 0, "next_run_at": _hermes_now().isoformat(), }, ) @@ -1719,6 +1771,13 @@ def mark_job_run(job_id: str, success: bool, error: Optional[str] = None, job["last_run_at"] = now job["last_status"] = "ok" if success else "error" job["last_error"] = error if not success else None + # Consecutive-failure streak: reset on success, else grow. + # Absent key (pre-existing records) reads as 0. Drives the + # recurring-job auto-pause below. + if success: + job["consecutive_failures"] = 0 + else: + job["consecutive_failures"] = int(job.get("consecutive_failures") or 0) + 1 # Track delivery failures separately — cleared on successful delivery job["last_delivery_error"] = delivery_error # Clear any external-fire claim so a re-armed recurring job can @@ -1798,7 +1857,23 @@ def mark_job_run(job_id: str, success: bool, error: Optional[str] = None, job["enabled"] = False job["state"] = "completed" elif job.get("state") != "paused": - job["state"] = "scheduled" + if not success and _auto_pause_pending(job): + job["enabled"] = False + job["state"] = "paused" + job["paused_at"] = now + job["paused_reason"] = ( + f"auto-paused after {job['consecutive_failures']} " + "consecutive failures" + ) + logger.warning( + "Job '%s' (%s) auto-paused after %d consecutive " + "failures; fix the cause and resume with " + "`hermes cron resume %s`.", + job.get("name", "?"), job["id"], + job["consecutive_failures"], job["id"], + ) + else: + job["state"] = "scheduled" save_jobs(jobs) return diff --git a/cron/scheduler.py b/cron/scheduler.py index 2cafa88ed1cf1..d6e65e864d1fa 100644 --- a/cron/scheduler.py +++ b/cron/scheduler.py @@ -282,7 +282,7 @@ def _resolve_cron_enabled_toolsets(job: dict, cfg: dict) -> list[str] | None: "QQBOT_HOME_CHANNEL": "QQ_HOME_CHANNEL", } -from cron.jobs import get_due_jobs, mark_job_run, save_job_output, advance_next_runs, claim_dispatch, heartbeat_run_claim +from cron.jobs import get_due_jobs, mark_job_run, save_job_output, advance_next_runs, claim_dispatch, heartbeat_run_claim, failure_would_pause from cron.jobs import get_ticker_heartbeat_age from cron.executions import create_execution, finish_execution, mark_execution_running @@ -4035,6 +4035,17 @@ def run_one_job(job: dict, *, adapters=None, loop=None, verbose: bool = False) - # If the agent responded with [SILENT], skip delivery (but # output is already saved above). Failed jobs always deliver. deliver_content = final_response if success else _summarize_cron_failure_for_delivery(job, error) + # When this failure trips the consecutive-failure limit, + # mark_job_run below will auto-pause the job — say so in the + # same delivery, so the operator gets one escalation instead of + # per-tick failure spam followed by unexplained silence. + if not success and failure_would_pause(job): + streak = int(job.get("consecutive_failures") or 0) + 1 + deliver_content += ( + f"\n⏸️ Auto-pausing this job after {streak} consecutive " + f"failures. It will not run again until resumed: " + f"`hermes cron resume {job['id']}`" + ) # Treat whitespace-only final responses the same as empty # responses: do not deliver a blank message, and let the # empty-response guard below mark the run as a soft failure. diff --git a/docs/agents-loops-graphs.md b/docs/agents-loops-graphs.md index 2339db43cdf51..8279bb31e5baf 100644 --- a/docs/agents-loops-graphs.md +++ b/docs/agents-loops-graphs.md @@ -118,11 +118,13 @@ consecutive failures; a job broken by a rotated credential fails quietly forever. This is the root of the standing "Infrastructure Health & Alerts" noise and the open cron-cleanup/health-check Linear projects. -**Build:** track `consecutive_failures` in the job record in -`cron/scheduler.py`; after N (default 3), auto-pause the job and deliver one -escalation (Slack + Linear issue via the existing delivery path) instead of -per-tick failure spam. Mirror of Kanban's existing `failure_limit` semantics -— the pattern is already in the codebase, just not in cron. +**Built (this branch):** `mark_job_run` in `cron/jobs.py` now tracks +`consecutive_failures` per job; after N in a row (per-job `failure_limit` > +`HERMES_CRON_FAILURE_LIMIT` env > default 3, `0` disables) a recurring job +is auto-paused, and the scheduler folds the auto-pause notice into the final +failure delivery — one escalation instead of per-tick spam. `resume`, +`trigger`, and any success reset the streak; one-shots are exempt. Mirrors +Kanban's existing `failure_limit` semantics. ### Gap 2 — Cron outputs have no pass/fail gate diff --git a/tests/cron/test_failure_auto_pause.py b/tests/cron/test_failure_auto_pause.py new file mode 100644 index 0000000000000..d6507bca5fd9b --- /dev/null +++ b/tests/cron/test_failure_auto_pause.py @@ -0,0 +1,174 @@ +"""Tests for the consecutive-failure auto-pause on recurring cron jobs. + +A recurring job that fails ``failure_limit`` times in a row (per-job field > +``HERMES_CRON_FAILURE_LIMIT`` env > default 3) is paused by ``mark_job_run`` +instead of firing and alerting on every subsequent tick forever. Resume and +manual trigger reset the streak; success resets the streak; one-shots are +exempt (they reach a terminal state on their own). +""" + +import pytest + +from cron.jobs import ( + create_job, + get_job, + get_due_jobs, + mark_job_run, + resume_job, + trigger_job, + update_job, + failure_would_pause, +) + + +@pytest.fixture() +def tmp_cron_dir(tmp_path, monkeypatch): + """Redirect cron storage to a temp directory.""" + monkeypatch.setattr("cron.jobs.CRON_DIR", tmp_path / "cron") + monkeypatch.setattr("cron.jobs.JOBS_FILE", tmp_path / "cron" / "jobs.json") + monkeypatch.setattr("cron.jobs.OUTPUT_DIR", tmp_path / "cron" / "output") + return tmp_path + + +def _recurring_job(**kwargs): + return create_job(prompt="Watch the thing", schedule="every 1h", **kwargs) + + +class TestFailureStreak: + def test_failure_increments_and_success_resets(self, tmp_cron_dir): + job = _recurring_job() + mark_job_run(job["id"], success=False, error="boom") + assert get_job(job["id"])["consecutive_failures"] == 1 + + mark_job_run(job["id"], success=False, error="boom") + assert get_job(job["id"])["consecutive_failures"] == 2 + + mark_job_run(job["id"], success=True) + assert get_job(job["id"])["consecutive_failures"] == 0 + + def test_missing_key_reads_as_zero(self, tmp_cron_dir): + """Records written before the field existed must not crash or pause early.""" + job = _recurring_job() + update_job(job["id"], {"consecutive_failures": None}) + mark_job_run(job["id"], success=False, error="boom") + assert get_job(job["id"])["consecutive_failures"] == 1 + + +class TestAutoPause: + def test_recurring_job_pauses_at_default_limit(self, tmp_cron_dir): + job = _recurring_job() + for _ in range(2): + mark_job_run(job["id"], success=False, error="boom") + stored = get_job(job["id"]) + assert stored["state"] == "scheduled" + assert stored["enabled"] is True + + mark_job_run(job["id"], success=False, error="boom") + stored = get_job(job["id"]) + assert stored["state"] == "paused" + assert stored["enabled"] is False + assert "auto-paused after 3 consecutive failures" in stored["paused_reason"] + assert stored["paused_at"] is not None + + def test_paused_job_is_not_due(self, tmp_cron_dir): + job = _recurring_job() + for _ in range(3): + mark_job_run(job["id"], success=False, error="boom") + due_ids = [j["id"] for j in get_due_jobs()] + assert job["id"] not in due_ids + + def test_success_between_failures_prevents_pause(self, tmp_cron_dir): + job = _recurring_job() + for _ in range(2): + mark_job_run(job["id"], success=False, error="boom") + mark_job_run(job["id"], success=True) + for _ in range(2): + mark_job_run(job["id"], success=False, error="boom") + assert get_job(job["id"])["state"] == "scheduled" + + def test_per_job_limit_overrides_default(self, tmp_cron_dir): + job = _recurring_job() + update_job(job["id"], {"failure_limit": 1}) + mark_job_run(job["id"], success=False, error="boom") + assert get_job(job["id"])["state"] == "paused" + + def test_env_limit_overrides_default(self, tmp_cron_dir, monkeypatch): + monkeypatch.setenv("HERMES_CRON_FAILURE_LIMIT", "5") + job = _recurring_job() + for _ in range(4): + mark_job_run(job["id"], success=False, error="boom") + assert get_job(job["id"])["state"] == "scheduled" + mark_job_run(job["id"], success=False, error="boom") + assert get_job(job["id"])["state"] == "paused" + + def test_zero_limit_disables_auto_pause(self, tmp_cron_dir, monkeypatch): + monkeypatch.setenv("HERMES_CRON_FAILURE_LIMIT", "0") + job = _recurring_job() + for _ in range(6): + mark_job_run(job["id"], success=False, error="boom") + stored = get_job(job["id"]) + assert stored["state"] == "scheduled" + assert stored["consecutive_failures"] == 6 + + def test_manual_pause_reason_is_not_overwritten(self, tmp_cron_dir): + """A failure recorded against an already-paused job keeps the manual pause.""" + from cron.jobs import pause_job + + job = _recurring_job() + pause_job(job["id"], reason="operator hold") + mark_job_run(job["id"], success=False, error="boom") + stored = get_job(job["id"]) + assert stored["state"] == "paused" + assert stored["paused_reason"] == "operator hold" + + +class TestOneShotExemption: + def test_one_shot_never_reports_pending_pause(self, tmp_cron_dir): + job = create_job(prompt="Once", schedule="30m") + update_job(job["id"], {"consecutive_failures": 99}) + assert failure_would_pause(get_job(job["id"])) is False + + def test_failed_one_shot_ends_completed_not_paused(self, tmp_cron_dir): + job = create_job(prompt="Once", schedule="30m") + mark_job_run(job["id"], success=False, error="boom") + stored = get_job(job["id"]) + assert stored["state"] == "completed" + assert stored["enabled"] is False + + +class TestStreakReset: + def test_resume_resets_streak(self, tmp_cron_dir): + job = _recurring_job() + for _ in range(3): + mark_job_run(job["id"], success=False, error="boom") + assert get_job(job["id"])["state"] == "paused" + + resumed = resume_job(job["id"]) + assert resumed["state"] == "scheduled" + assert resumed["consecutive_failures"] == 0 + # Two more failures must not immediately re-pause (limit is 3). + for _ in range(2): + mark_job_run(job["id"], success=False, error="boom") + assert get_job(job["id"])["state"] == "scheduled" + + def test_trigger_resets_streak(self, tmp_cron_dir): + job = _recurring_job() + for _ in range(2): + mark_job_run(job["id"], success=False, error="boom") + triggered = trigger_job(job["id"]) + assert triggered["consecutive_failures"] == 0 + + +class TestFailureWouldPause: + def test_true_only_on_the_tripping_failure(self, tmp_cron_dir): + job = _recurring_job() + assert failure_would_pause(get_job(job["id"])) is False + for _ in range(2): + mark_job_run(job["id"], success=False, error="boom") + assert failure_would_pause(get_job(job["id"])) is True + + def test_false_when_disabled(self, tmp_cron_dir, monkeypatch): + monkeypatch.setenv("HERMES_CRON_FAILURE_LIMIT", "0") + job = _recurring_job() + update_job(job["id"], {"consecutive_failures": 10}) + assert failure_would_pause(get_job(job["id"])) is False From 2b6d649e03827502fb50c5a2526d27aeb96317ad Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 12:19:06 +0000 Subject: [PATCH 3/5] feat(skills): bundle fanout parallel-execution skill for web sessions Adds the plan -> parallel-workers -> skeptic -> synthesize playbook as a repo-level Claude Code skill so remote/web sessions on this repo can invoke /fanout. Canonical source lives in dotfiles ~/.claude/skills; this copy is generalized (no machine-local refs paths). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01MW1vwtetBTLSMJcBYp9r1C --- .claude/skills/fanout/SKILL.md | 91 ++++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) create mode 100644 .claude/skills/fanout/SKILL.md diff --git a/.claude/skills/fanout/SKILL.md b/.claude/skills/fanout/SKILL.md new file mode 100644 index 0000000000000..3651cab84fefc --- /dev/null +++ b/.claude/skills/fanout/SKILL.md @@ -0,0 +1,91 @@ +--- +name: fanout +description: Run a complex task through the plan → parallel-workers → skeptic → synthesize playbook. Auto-invoke PROACTIVELY whenever a task decomposes into 3+ independent, parallelizable work units (research, analysis, writing, design) AND the estimated serial work exceeds ~15 minutes — especially when time-sensitive. Do NOT trigger for a single work unit, for tasks under ~15 min of serial work, or when the parts are highly interdependent (shared state, sequential dependencies, one step blocks the next). In unattended sessions (cron, loops, scheduled runs with no human to approve a plan), the plan-approval gate is skipped only when every work unit is read-only (research/analysis); if any unit would write (files, external state, sends), abort the fan-out and fall back to serial execution instead. +--- + +# Fanout — Parallel Execution Playbook + +Run a complex task through the parallel-execution playbook: **plan → parallel subagents execute → a fresh-context skeptic attacks the findings → synthesize what survives.** + +In Claude Code, spawn workers with the Agent/Task tool — **all in one message** so they run concurrently. Where the Workflow tool is available (web/Cowork sessions with orchestration enabled), a Workflow with a verify stage is the equivalent for larger fan-outs. + +## When to trigger (auto-invoke) + +Trigger proactively — without waiting to be asked — when a task meets ALL of: + +- Decomposes into **3+ independent work units** (research, analysis, writing, design) +- Estimated serial work is **more than ~15 minutes** (the cost floor — below this, fan-out overhead isn't worth it) +- Units don't have hard sequential dependencies or shared mutable state + +Do NOT trigger when: +- The task is a single unit or under the ~15 min cost floor — do it directly +- Parts are highly interdependent (each step needs the prior step's output) — use sequential agents instead +- The task is simple enough that decomposition would add overhead without saving wall-clock time + +If a task looks parallelizable but doesn't clearly meet the bar, say so and proceed directly rather than forcing a fan-out. + +## Unattended-session gate + +The plan-approval gate (below) is mandatory in interactive sessions. In **unattended** sessions — cron, loops, scheduled runs, or anywhere there's no human able to approve a plan in real time — apply this rule instead: + +- **All work units read-only** (research, analysis, lookups — nothing writes files, calls send-type tools, or mutates external state): skip the approval gate, proceed straight to execution. +- **Any work unit writes** (files, external systems, messages, commits): abort the fan-out entirely and fall back to serial execution of the task. Do not partially gate — an all-or-nothing rule keeps this predictable. + +This waives **only the human plan-approval gate**. The skeptic (step 3) still runs — unattended output has no human reading it before it lands, which is exactly when unchallenged findings are most likely to go unnoticed. + +## Procedure + +### 1. Plan (gated in interactive sessions) +Enter plan mode: +- Decompose into **3–5 discrete, parallelizable work units**, plus dependencies, per-unit effort, and risks. +- Present the plan. In an interactive session, **wait for the user's approval** — this gate is mandatory; do not spawn agents before it. Exit plan mode only after approval. +- In an unattended session, apply the unattended-session gate above instead of waiting for approval. + +If the task turns out NOT to parallelize cleanly (fewer than 3 independent units, or hard dependencies), say so and recommend doing it directly instead of forcing a fan-out. + +### 2. Execute (parallel workers) +For each approved work unit, spawn one subagent — **all in a single message** so they run concurrently. Give each a self-contained prompt carrying the plan's context for its unit (add a web-search instruction if the unit needs current data). + +No dependencies between agents. Wall-clock time ≈ the slowest single unit, not the sum. + +### 3. Check (skeptic — required) + +**Checking is its own job.** Do not let the agent that writes the answer be the only one that grades it — a synthesizer asked to both merge and critique will reliably rate its own inputs as sound. Spawn one fresh-context skeptic over the combined worker output *before* synthesis: + +> You are a skeptic. Your job is to REFUTE, not to summarize. Here are findings from N parallel work units. For each material claim: is it actually supported by evidence, or asserted confidently without proof? Flag (a) unsupported claims, (b) stale or undated evidence, (c) sources that don't say what the finding claims, (d) conflicts between units, (e) anything mistaking correlation, popularity, or pain for significance. Return SURVIVES / WEAK / REFUTED per claim, with a one-line reason. Default to WEAK when evidence is thin — do not be agreeable. + +This stage is **required**, not conditional. Fan-out already has a ~15-minute cost floor, so anything reaching it is substantial enough to be worth checking. (The honest exception: work with no falsifiable claims — parallel creative drafts, independent mechanical passes — has nothing to refute. Run the skeptic whenever the output carries factual claims, which is nearly always.) + +**First, a mechanical completeness gate.** Before spawning the skeptic, confirm each unit returned *findings*, not narration. A unit that reports "waiting on results" or restates its plan has produced nothing to refute, and a skeptic prompted to attack claims will not flag their *absence*. Re-dispatch it instead. + +What this stage does NOT do: +- The skeptic sees worker **output**, not sources. It judges plausibility and internal consistency; it cannot re-verify a citation it was never given. +- Nothing structurally forces synthesis to honor the verdicts — re-read the review against the final deliverable if the stakes warrant it. +- **Check the skeptic too.** It can refute wrongly. +- Treat a unit's clean sweep with *more* suspicion when it read code rather than running it. Static reading cannot fail a claim the way execution can. + +Pass the skeptic's verdicts into synthesis. + +### 4. Synthesize +After the skeptic returns: +- Build on **surviving** evidence; drop or explicitly caveat REFUTED claims. +- Resolve conflicts and overlaps across unit results. +- Produce the final deliverable (report / summary / decision matrix). +- Cite which unit each finding came from, and carry the skeptic's verdict on any claim that was WEAK or REFUTED. +- Flag gaps and uncertain areas. + +### 5. Save +- Write the plan to `plan.md` (audit trail). +- Write the skeptic's verdicts to `review.md` — the paper trail for *what was challenged and what survived* is as useful later as the findings themselves. +- Save the synthesis as the final deliverable, linked back to `plan.md`. + +## Checklist +- [ ] Task has 3+ independent units and >~15 min estimated serial work (else do it directly) +- [ ] Interactive session: plan approved before any worker spawns +- [ ] Unattended session: all units read-only (gate skipped) OR fan-out aborted (any unit writes) +- [ ] All workers dispatched in one message +- [ ] Every unit returned actual findings, not narration (re-dispatch if not) +- [ ] Skeptic ran over the combined output **before** synthesis +- [ ] Skeptic's own verdicts spot-checked — it can be wrong too +- [ ] Synthesis builds on surviving evidence; REFUTED claims dropped or caveated +- [ ] `plan.md` + `review.md` + final deliverable saved From edcf8577ebfaae9869dd14ba5afd4ee445d9059b Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 12:48:11 +0000 Subject: [PATCH 4/5] feat(claude): pre-approve MCP servers for web sessions Web/remote sessions on this repo repeatedly stall on 'requires approval' for MCP tools (claude-code-remote triggers, gateway calls). Pre-approve the servers unattended GitHub-task sessions rely on so autonomous runs never block on a permission prompt nobody is watching. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01MW1vwtetBTLSMJcBYp9r1C --- .claude/settings.json | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.claude/settings.json b/.claude/settings.json index 444f9d022c150..d93dd765c90fa 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -1,4 +1,12 @@ { + "permissions": { + "allow": [ + "mcp__github__*", + "mcp__claude-code-remote__*", + "mcp__Unified_Gateway__*", + "mcp__Linear__*" + ] + }, "hooks": { "PreToolUse": [ { From 0f461f62eee6ecb1bbfffd91dd2a0a455406a5a2 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 12:52:49 +0000 Subject: [PATCH 5/5] revert(claude): drop repo-level permissions block per repo guard MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit test_no_repository_local_claude_permissions_file deliberately forbids a permissions block in tracked .claude/settings.json — on a public repo it would grant tool access to every agent that opens a clone. User-level settings are the right home for MCP pre-approval. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01MW1vwtetBTLSMJcBYp9r1C --- .claude/settings.json | 8 -------- 1 file changed, 8 deletions(-) diff --git a/.claude/settings.json b/.claude/settings.json index d93dd765c90fa..444f9d022c150 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -1,12 +1,4 @@ { - "permissions": { - "allow": [ - "mcp__github__*", - "mcp__claude-code-remote__*", - "mcp__Unified_Gateway__*", - "mcp__Linear__*" - ] - }, "hooks": { "PreToolUse": [ {