diff --git a/.egg-state/agent-outputs/coder/brc-memory-issue-3312-v2.md b/.egg-state/agent-outputs/coder/brc-memory-issue-3312-v2.md index cde5b8e66e..c410fba7f1 100644 --- a/.egg-state/agent-outputs/coder/brc-memory-issue-3312-v2.md +++ b/.egg-state/agent-outputs/coder/brc-memory-issue-3312-v2.md @@ -1,31 +1,121 @@ -# Coder BRC memory — issue-3312-v2, slice-1 (decompose orchestrator/models.py, closes #3450) - -## Change model (CLAIM — verify against live git log) -- **Pattern:** domain-split (data-dominated pydantic-models module), NOT method-modules-on-class. -- **Two commits:** `8e8ed492e` pure `git mv` baseline (models.py → models/__init__.py, byte-identical), - then `8a6af6ae6` extraction into 6 domain submodules + barrel + Dockerfile COPY + allowlist drop + CLAUDE.md seam. -- **Submodules** (all under both caps; largest `_config.py` 593 lines / 27KB): - `_enums` (status StrEnums + LIVE_POD_STATUSES + AgentRole re-export), - `_decisions` (HITLDecision/OperatorDirective/IterationSummary), - `_execution` (ReviewVerdict/AggregatedReviewResult/ContainerInfo/AgentExecution/CycleTiming/AgentExitInfo/PhaseExecution/_REMOVED_ROLE_MIGRATION), - `_config` (PipelineConfig + resolve_consensus_timeout_minutes + PHASE_CONSENSUS_TIMEOUT_DEFAULTS_MIN), - `_pipeline` (RepoSpec/Pipeline/resolve_slice_repo), - `_events` (PipelineEvent/ProgressEvent). -- **Import DAG (no cycles):** _enums → _decisions → {_config, _execution} → _pipeline → _events. Siblings import relative; external deps absolute (matches state_store). -- **Barrel** = stable public API: explicit per-symbol re-exports; also re-exports sibling-pkg symbols consumers pull through models: `PipelinePhase`, `Slice` (egg_contracts.models), `AgentRole` (egg_contracts.agent_roles), plus OVERSEER_TIER_MODELS / SLICE_ID_PATTERN. `__all__` lists 31 symbols. -- **No `patch("models.")` module-global seams exist** (audited) → submodules import collaborators directly, no `import models as _pkg` indirection needed. -- **Python 3.14.6:** annotations lazy (PEP 649/749) → the pre-split forward ref (resolve_consensus_timeout_minutes → PipelineConfig) is fine; kept intra-module in _config regardless. - -## Audit (task-1-1) — CLAIM -- ~197 files reference the module; dominant style bare `from models import X`. `PipelinePhase as PipelineModelsPhase` (routes loop) is load-bearing. -- Edge symbols (resolve_consensus_timeout_minutes, ReviewerType, AgentExitInfo, _REMOVED_ROLE_MIGRATION, OVERSEER_TIER_MODELS, SLICE_ID_PATTERN): 0 problematic external refs; re-exported anyway. -- No `__module__`-sensitive consumer. - -## Verdict / verification (my proposal SHA 8a6af6ae6) -- **Repo-wide `pytest --collect-only`: 16,041 tests, 0 import errors** → every models importer resolves post-split. -- **orchestrator/tests/test_models.py: 124/124 pass.** ruff check + format clean. file-size ratchet exit 0 (models.py gone from allowlist; every submodule under caps). -- Full orchestrator suite: 7,471 passed; **143 non-passing are ALL sandbox env failures** — `git init not supported in the container` / CalledProcessError / k8s `list_namespaced_pod` unreachable — in git/worktree/session/k8s modules (gateway_client, agent_salvage, kubernetes_spawner, cli, reconcile*, slice_diff*, commit_statefiles*, ...). NONE reference models/import/attribute. Identical env class documented by prior landed slices. -- **NOT executable in this sandbox:** Dockerfile COPY smoke-check (`docker build` + `import models` in image) — no docker/network. COPY line added per exact established pattern (mirrors gateway_client etc.). Flag for reviewer_code if image-build verification is required. +# Coder BRC memory — issue-3312-v2, slice-4 (decompose orchestrator/routes/pipelines.py, closes the file-size program) + +## Operator directive (cq-3, RESOLVED 2026-07-05) — BINDING +- One-shot model is FIXED (build eca7ca740): worktree re-attach preserves clean unpushed commits that descend from the slice base. **Work INCREMENTALLY across invocations toward the single PR.** +- Commit after each cohesive cluster extraction; commits persist across invocations. NO re-slicing — scope/structure stay as contracted. +- Direct `git push` is BLOCKED in pipeline sessions (only `mcp__brc__propose` pushes). So durability = COMMITS, not pushes. Do NOT propose until the decomposition is COMPLETE (barrel under cap + allowlist empty) — a partial propose gets NACKed. +- Recovery: baseline `0228f4a9f` and the extraction chain are in the object store; fast-forward-recover the newest clean tip rather than restarting from baseline. + +## Current state (HEAD after the LATEST invocation) +- Branch `egg/issue-3312-v2-slice-4-coder/work`; commits PERSIST via the re-attach fix. **⚠️ RECOVERY GOTCHA (bit me this invocation):** the worktree CAN start reset to the slice base `64fa30773` (reflog: `reset: moving to origin/egg/issue-3312-v2/slice-4`), losing the chain from HEAD — but the full chain SURVIVES in the object store. FIRST STEP every invocation: `git log --oneline -3`; if HEAD==base and `orchestrator/routes/pipelines/` is a monolith again, find the newest tip in `git reflog` and `git merge --ff-only ` to recover (it's a clean fast-forward descendant of base; ff-only is on-lineage so the gateway permits it). Newest tip as of now = commit **`923a2b351`** (GIANT #3 bite 9 — plan/PENDING/IMPLEMENT blocks extracted, **`_run_pipeline` = 1,467L = UNDER CAP**; barrel 2,932L, over only because `_run_pipeline` still lives in it). NOTE the real newest tip is the memory commit after this; recovery finds it via reflog. Scratch `giant*_*.py`/`blockflow.py` are UNTRACKED → wiped by the reset (`slice4_xtract.py` IS committed/persists; re-create `blockflow.py` from its logic below if needed). NOTE: `git merge --ff-only ` recovers cleanly every invocation — proven 3×. +- **⚠️ TOOL-CLOBBER LESSON (bit me bite 2):** `slice4_xtract.py ... sym` **OVERWRITES** the target submodule file — it does NOT append. When extracting a 2nd+ symbol into an EXISTING submodule (e.g. adding `_map_host_repos` to `_run_pipeline_setup.py` which already held `_sync_contract_setup`), the tool clobbers the prior symbol → ImportError on the barrel re-export. FIX: after running the tool, if the submodule pre-existed, restore the prior symbols: `git show HEAD:` + append the newly-tool-prefixed `def` from the clobbered file (I did this via a merge script). BETTER for future: extract to a barrel-top helper, then MANUALLY append it to the existing submodule + add to the barrel `from ._x import (...)` re-export, skipping the tool for same-submodule 2nd+ adds. +- Barrel `orchestrator/routes/pipelines/__init__.py`. **GIANT #1 + GIANT #2 DONE.** GIANT #3 (`_run_pipeline`) IN PROGRESS: **~2,797L** (bites 1-4 landed). `_run_pipeline_setup.py` (4 helpers: `_sync_contract_setup`, `_map_host_repos`, `_start_phase_setup`, `_sync_source_branch_drafts`). After GIANT #3 the barrel drops under cap → remove the WIP allowlist entry (files: map EMPTY = terminal criterion). +- **bite 4 DONE ✅ (c4f715cce):** source_branch block → `_sync_source_branch_drafts` — PURE SIDE-EFFECT helper (0 returns, pipeline not reassigned, path locals used only within-block → NO return value). Merge-flow proven for 4th same-submodule add. _run_pipeline 2,853→2,797. **KEY LIVE-CHECK method (use before each block extraction):** for each blockflow "OUTPUT", grep `ast.walk(rp)` for Load nodes with lineno > block-end — if 0 reads-after, the output is DEAD (safe to not thread; but beware F841 if it's assigned-and-unused *within the helper* — see worktree caveat). +- **bite 5 DONE ✅ (da48f871a):** worktree block → `_resolve_worktree_repo` via (pipeline, done). current_phase was assigned-AND-used *within* the block (loads at L1254/1277/1292) → stays a clean helper-local, NO F841 (the earlier F841 worry was unfounded once I checked within-block loads). _run_pipeline 2,797→2,711. **ALL 5 SETUP BLOCKS DONE.** `_run_pipeline_setup.py` = 727L (5 helpers, under cap). +- **bite 6 DONE ✅ (8e013b6e7):** lifted the 2 health-monitor closures (`_on_health_escalation`, `_health_monitor_poll`) → `_run_pipeline_support.py` (62L) via functools.partial (GIANT #2 recipe). `_run_pipeline` 2,711→2,684. `_make_overseer_teardown_hook` stays NESTED (deferred — its `nonlocal phase_overseer_active` would need boxing across every use-site for only 37L; NOT worth it, and it's harmless nested since the while-loop split is what gets under cap). +- **bite 7 DONE ✅ (7f49ee07f):** extracted the 708L HITL-gate converge block (`if current_phase.value in _HITL_GATE_PHASES and not pipeline.config.hitl_gates:`) → `_run_hitl_gate.py::_run_hitl_gate_converge` (722L). **PROVEN control-flow-signal recipe for while-loop blocks:** the block's 4 outer-while `continue`s → `return pipeline, "continue"` (preserve trailing comments); fall-through → `return pipeline, None`; caller `pipeline, _act = _helper(...); if _act == "continue": continue`. OUT=pipeline only. `d` was a blockflow false-positive (comprehension-local, no store before block) → NOT a param. Added `_run_hitl_gate_converge` to test `_EXTRACTED_HELPERS`. `_run_pipeline` 2,684→**1,988L**. Extraction-script guard: assert `all continues in block == the known set` + no continue inside a nested for/while (else it's an inner-loop continue you must NOT convert). Blocks with ONLY continues (no break/return) are the safest while-loop extractions. +- **bite 8 DONE ✅ (95dc11c25):** extracted the 296L phase-execution `if True: while ...:` block → `_run_phase.py::_run_phase_execution` (331L). **KEY LESSON: the block's 4 `break`s belonged to its OWN inner `while` (L1721-2015 spans the whole if-body) — they are INNER-loop breaks that STAY verbatim; only the 1 escaping bare `return` needed the signal.** The blockflow/naive break-scan is misleading — ALWAYS check if breaks sit inside a nested loop within the block (the guard `for loop in ast.walk(blk): assert no break inside` catches it). Threaded in+out: pipeline, phase_execution, phase_failed (all pre-init'd before the block). `e` = except-name reuse (excluded). Dropped the now-dead `tester_gap_summary` pre-init from the giant (0 loads; helper re-inits its own). `_run_pipeline` 1,988→**1,712L**. +- **bite 9 DONE ✅ (923a2b351):** extracted plan (133L, `_run_plan_advance`, 1 outer-while break → "break" signal, `phase_overseer_active` bool threaded in+out) + PENDING (81L, `_run_pending_phase_init`, pure) + IMPLEMENT (66L, `_run_implement_advance`, pure) → `_run_phase_blocks.py` (338L). Reusable tool: `.egg-state/agent-outputs/coder/giant3_loopblocks.py` (finds block by test-sig+size since line#s shift; handles threaded I/O + break-signal). Added the 3 to test `_EXTRACTED_HELPERS`. `_run_pipeline` 1,712→**1,467L (UNDER CAP)**. +- **✅ FINAL BITE DONE — TERMINAL COMPLETE (this invocation). The decomposition PROGRAM is finished; PROPOSE issued.** + 1. **Moved `_run_pipeline` → `_run_pipeline.py` (1,483L)** via `slice4_xtract.py`; barrel dropped to **1,466L (UNDER cap)**. All 16 orphaned barrel imports were verified genuine `_pkg.` seam refs (each ≥2 uses in submodules) → retained with `# noqa: F401`, none deleted. `patch("routes.pipelines._run_pipeline")` (test_start_pipeline) + `_pkg._run_pipeline` (_drivers) resolve via the barrel re-export. Commit `cced2d2c3`. + 2. **Terminal:** dropped the last allowlist entry → **`files:` map EMPTY** (verified `check-file-sizes.py` exit 0, no hard-cap errors). Deleted scratch `slice4_xtract.py` (git rm). **⚠️ ROLE BOUNDARY (bit me at propose): `orchestrator/CLAUDE.md` is a DOCUMENTER-owned restricted path — coder CANNOT push it** (gateway: `role 'coder' cannot modify restricted paths: orchestrator/CLAUDE.md`, alternative_role=documenter). The routes/pipelines/ seam subsection (task-4-5's doc half) is the DOCUMENTER's deliverable. I authored+reverted it, then had to soft-reset to drop the CLAUDE.md commit ENTIRELY from history (a net-zero modify+revert still fails — the gateway inspects per-commit changes, not just the net diff). Coder's terminal = allowlist EMPTY + scratch delete only. + 3. **Test-mechanical (task-4-6):** fixed 3 source-introspection helpers in `test_advance_phase_thread.py` broken by the move: `_auto_advance_block`/`_run_pipeline_source` strip the `_pkg.` prefix (the moved fn's barrel globals are now `_pkg.`, breaking bare `datetime.now(UTC)` / the `_sync_worktree_reconciling_divergence` try/except regex); `_recover_advance_block` now introspects `_start_pipeline_body` (the real body — `start_pipeline` is a thin @route wrapper post decision-8). 15/15 pass. Also formatted a stale-format line in `test_ble001_narrowing_audit.py` (pre-split fallback branch) left by an earlier in-slice edit. + 4. **Verification (make lint/test-all's venv-sync is NETWORK-BLOCKED in this sandbox — ran tools directly from `.venv/bin`):** ruff check + format GREEN tree-wide; `check-file-sizes.py` GREEN with EMPTY allowlist; **whole-repo collect-only = 16,757 tests, 0 import errors**; all `_run_pipeline`-coupled seam tests GREEN (start_pipeline patch, advance_phase source-introspection, slice_loop_import_seam, origin_main, pipelines_apply); dense pipeline seam coverage GREEN (consensus_polling/brc_nack/concurrent/slice_run_loop). **Documented NON-regressions (pre-existing, NOT split-induced):** (a) sandbox `git init` env failures in test setup (git init "not supported in the container"); (b) `test_concurrent_status.py` 2 message-store-pollution flakes that PASS in isolation (37/37) — an earlier batch test leaves 3 messages; the test doesn't mock the store. Neither touches my change surface. + 5. **PROPOSED** via `mcp__brc__propose` (push to origin happens through the gateway — also satisfies the operator's end-of-invocation push directive). +- **GIANT #3 bite 2 DONE ✅ (commit 082b0da19):** extracted host-repo block (`if host_repo_map:` L1218-1327) → `_map_host_repos`. repo_volumes/worktree_repo_path pre-init at L1212-1213 → threaded input+output, NO sentinel (0 returns). `GatewayError` kept barrel-side as retained-for-seam re-export. `_run_pipeline` 3,085→2,987. **REMAINING setup blocks:** worktree `if worktree_repo_path!=repo_path:` L1341-1437 (2 ret; OUT current_phase[first-assigned INSIDE at L1346, NOT pre-init → check if read before the loop L2035 reassigns it]/pipeline) — TRICKIER; source_branch L1464-1527 (0 ret; OUT analysis_rel/drafts_dir/plan_rel are NOT pre-init and have no else → only safe if read later under same guard, VERIFY before extracting); start_phase `if start_phase=='implement':` (~L1500s now, 2 ret, OUT pipeline only = SAFE, mirrors contract-sync). Do start_phase next (safe), then verify worktree/source_branch guard-usage before touching them. +- **GIANT #3 bite 1 DONE ✅ (commit 6a70a0031):** extracted the contract-sync setup block (`if not pipeline.contract_synced:`) → `_run_pipeline_setup.py::_sync_contract_setup` (276L). Established the **`(pipeline, done)` early-return convention** for mid-function block extraction: the block's bare `return`s (error-exit paths) become `return pipeline, True`; helper ends `return pipeline, False`; caller does `pipeline, _done = _helper(...); if _done: return`. `_run_pipeline` 3,324→3,085. **Data-flow tool `.egg-state/agent-outputs/coder/blockflow.py`** (use-def analyzer: `blockflow.py _run_pipeline ` → INPUTS/OUTPUTS; treats `except..as` names + local imports as block-local; ignore `*_err` names in OUTPUTS = exception-name reuse). **SOURCE-INTROSPECTION TEST PATTERN:** extracting from `_run_pipeline` trips tests in `test_advance_phase_thread.py` that `inspect.getsource(_run_pipeline)` and count call-sites/markers. Fixed `_run_pipeline_source()` to append extracted helpers via an `_EXTRACTED_HELPERS = (...)` tuple — **APPEND each new setup/phase helper name to that tuple as you extract** so the `_commit_statefiles_to_worktree` count-pin (5) + try/except regexes stay valid. Other introspection tests to watch: `TestAutoAdvanceRespawnsThread` (`# TEST_MARKER: auto_advance_block`), the origin_main call-site test. +- **GIANT #2 DONE ✅ (this invocation):** `_run_concurrent_phase` split → `_run_concurrent.py` (1,439L) + `_run_concurrent_retry.py` (215L, the impasse-retry wrapper) + `_run_concurrent_support.py` (422L). Lifted the 5 nested closures (`_superseded_by_restart`/`_record_container_exit`/`_stop_running_containers`/`_latest_proposal_ts`/`_update_agents_complete`) to `*_impl` fns + `functools.partial` bindings (all call sites verbatim); ALSO lifted the "Record spawned containers" block (`_record_spawned_agents_impl`) and the "Phase-level spawn retry" block (`_retry_transient_spawn_failures_impl`) to get the giant under cap. Bodies byte-verbatim modulo `_pkg.`-prefix + docstring re-indent. NO nonlocal; partial early-binding == closure late-binding (captured locals never reassigned after closure def, verified via reassign.py). ruff clean; import OK; **423 seam tests pass**. LESSON: hand-added support helpers MUST be added to the barrel `from ._run_concurrent_support import (...)` re-export or `_pkg.` raises AttributeError at call time (bit me: 54 failures until I added `_retry_transient_spawn_failures_impl` to the barrel re-export). Scratch scripts: `.egg-state/agent-outputs/coder/giant2_{lift,record,final}.py` (one-shot, delete at terminal). +- **Pre-existing NON-regression test failures (do NOT chase; fix mechanically at task-4-6):** (a) `test_advance_phase_thread.py::TestRecoverPipelineClearsConcurrentState` (2) — source-introspection `inspect.getsource(start_pipeline)` looks for `TEST_MARKER: recover_advance_clear` which now lives in `_routes_lifecycle.py:658` after the prior route-body extraction (start_pipeline is a thin wrapper); update the test to glob the package dir. (b) `test_slice_phase_restart_hardening.py::TestPersistContractStatefiles` (2) — sandbox `git init` env failure in `_init_repo` setup, before pipelines code runs (documented non-issue). +- 37 submodules now (added `_run_implement` + `_run_implement_support`). +- **ALL 16 @pipelines_bp.route BODIES are extracted (decision-8, this invocation)** into `_routes_read` `_routes_crud` `_routes_restart` `_routes_status` `_routes_lifecycle` `_routes_stream`. The barrel keeps each route's decorator(s) + signature as a THIN WRAPPER: `return __body(args)` (BARE call — barrel IS _pkg; body re-exported into barrel namespace; also keeps patch("routes.pipelines.__body") + route-name seams). The tool gained a **`--routes` mode** for this: `.venv/bin/python .egg-state/.../slice4_xtract.py --routes "" route1 route2 ...`. +- 35 submodules extracted so far, ALL under 1500 lines / 100KB: + `_criteria` `_drafts` `_reviews` `_context_pr` `_brc_history` `_statefiles` `_worktree_sync` `_alerts` `_overseer` `_slice_state` `_drivers` `_decisions` `_pod_liveness` `_ledger` `_populate` `_prompt_review` `_prompt_agent` `_prompt_phase` `_prompt_reviewer` `_status_wait` `_resolve` (identifiers/resolution/event-emit) `_salvage` `_hitl_rerun` `_status_view` (`_get_pr_info`/`_consensus_block`/`_get_concurrent_status`/`_build_slice_diff_summary`) `_first_principles` `_stacked_pr` (`_start_stacked_pr_reconciler`) `_slice_completion` `_lifecycle_helpers` (submission/gateway-mode/cleanup/terminate helpers) `_run_support` (`_spawn_and_wait`/`_parse_resolution`/`_clear_stale_impasses_for_producers`/`_pipeline_superseded_by_restart`). +- **Verification: `pytest --collect-only` = ~16,197 tests, 0 import errors.** Per-cluster tests green. `make lint`/ruff clean on every touched file. +- **NEXT (5th invocation): execute the PROVEN-SAFE giant-split recipe below, starting with `_run_implement_phase_slices` (smallest). The risky analysis is DONE — it's now mechanical. Do it with fresh context; verify with test_slice_run_loop_integration + test_slice_phase_restart_hardening + collect-only before each commit.** +- NEW tool fix (in the committed tool): `_local_bound_names` now recognizes FUNCTION-LOCAL imports as local bindings. Before, a moved fn with `from models import ContainerInfo` (local) + bare `ContainerInfo(...)` got its usage `_pkg.`-rewritten (ContainerInfo is barrel-top) leaving the local import orphaned (F401). Now local-imported names stay bare. If you still see an orphaned-local-import F401 in a freshly-extracted submodule, just delete the dead local import (all its uses became `_pkg.` — behaviour identical). + +## Gotchas LEARNED (apply on every remaining extraction) +- **Tool byte-offset fix (already in the committed tool):** ast `col_offset` is a UTF-8 BYTE offset; the `_pkg.` insertion now slices on encoded bytes. Before the fix, a rewritten name AFTER an em-dash/curly-quote on the same line got `_pkg.` spliced mid-identifier (surfaced as F821 `_X_pkg`). If you ever see an F821 like `_L_pkg`, the tool regressed — check the byte-slice logic. +- **Module CONSTANT that references a moved class/enum AT DEFINITION TIME must move WITH it.** The tool only moves def/class, leaving constants in the barrel — but a barrel constant like `_FOREST_REASON_TO_OUTCOME = {..: PopulateOutcome.X}` is evaluated at import BEFORE the bottom re-export → `NameError: PopulateOutcome not defined`. Fix: cut the constant, append it to the submodule (after its enum/class, bare refs), and add it to the barrel `from ._x import (...)` re-export so `_pkg.<CONST>` still resolves. (Runtime refs to moved symbols INSIDE barrel functions are fine — resolved at call time via the re-export.) +- **NamedTuple/StrEnum base classes:** the tool rewrites `class X(NamedTuple)`→`class X(_pkg.NamedTuple)`; this WORKS (verified `_fields`/members intact) but leaves the barrel's `NamedTuple`/typing import F401-unused if the barrel has no other user → keep it with `# noqa: F401`. (`StrEnum` usually stays used by barrel-resident `ContextPrCreationReason`.) +- **Retained-for-seam barrel imports:** moving the sole barrel user of a name (patched getter like `get_container_spawner`, or a models type like `HITLDecision` reached via `_pkg.` in a submodule annotation) leaves it F401-unused in the barrel but it MUST stay imported → `# noqa: F401 — retained for _pkg re-export / patch seam`. Only DELETE a now-unused barrel import when NO submodule references it via `_pkg.` AND it's not a patch target (e.g. the overseer forward-ref TYPE_CHECKING imports, or `ContractSlice` which moved to `_populate`'s TYPE_CHECKING block). +- **Source-introspection tests break on decomposition (mechanical in-slice fix, task-4-6):** e.g. `test_pipelines_origin_main_parameterization::test_helper_has_multiple_call_sites` did `inspect.getsource(pipelines_module)` (barrel only) and counted `_resolve_origin_ref(` call sites; after extraction the callers live in submodules, so it now globs the whole `routes/pipelines/` package dir. Watch for other tests that `inspect.getsource` the barrel or assert on barrel-only source. + +## Extraction convention (matches LANDED slice-15 routes/signals; see orchestrator/CLAUDE.md seam tables) +- Flask blueprint: `@pipelines_bp.route` decorators + thin wrappers STAY in the barrel (decision-8). Only helper/handler bodies move. +- Submodule reaches barrel-resident + test-patched globals via `import routes.pipelines as _pkg` → `_pkg.<name>`. Bodies VERBATIM (only free barrel-global refs gain a `_pkg.` prefix). +- Barrel re-exports every moved symbol at the bottom via `from ._x import (...) # noqa: E402,F401` (keeps the 64 `patch("routes.pipelines.<name>")` seams resolving — all re-exported). +- Module-level CONSTANTS stay barrel-resident (referenced via `_pkg.`); do NOT move them (the tool only moves def/class). +- `global` statements: NONE exist in the barrel → no shared-mutable-state rebinding hazard (state is mutated in place via `_pkg.`). + +## The extraction TOOL — REUSE IT: `.egg-state/agent-outputs/coder/slice4_xtract.py` +- Run from repo root: `.venv/bin/python .egg-state/agent-outputs/coder/slice4_xtract.py <submodule> "<title>" sym1 sym2 ...` +- AST/scope-aware; rewrites free barrel-global Name refs → `_pkg.` (text-position insert, verbatim), skips builtins/locals/function-local-imports, captures barrel names bound inside top-level try/except + `if TYPE_CHECKING`. `Literal`/`Annotated` are NOT prefixed (would break ruff forward-ref special-casing) and are auto-imported from typing. +- Post-run manual fixups per cluster: (a) add a `TYPE_CHECKING:` block for any string forward-ref types the moved fns use that pyflakes flags F821 (e.g. `ContainerSpawner` via `..container_spawner` try/except; overseer also needed `CorrectiveExecutor`/`AdjudicationVerdict`/`SpawnedContainer`); (b) drop now-unused forward-ref imports from the BARREL (they moved out → F401); (c) `ruff check --fix --select I001` + `ruff format` both files; (d) `PYTHONPATH=orchestrator .venv/bin/python -c "import routes.pipelines"` + targeted pytest; (e) commit. +- Verify per cluster: `ruff check --select F821 <submodule>` should be empty after fixups. + +## Remaining work — ONLY the 3 giants left (+ terminal cleanup) +- ALL misc helper clusters AND all 16 route bodies DONE. Do NOT re-extract. + +### PROVEN-SAFE RECIPE for the giant internal split (analysis done this invocation — EXECUTE mechanically) +The giants are closure-heavy. Key de-risking result: **NONE of the giants use `nonlocal` except `_run_pipeline` (one: `phase_overseer_active` in `_make_overseer_teardown_hook`)**, AND (verified via AST) the enclosing-locals a giant's nested closures capture are NOT reassigned after the closure is defined. THEREFORE lifting a nested closure to a module-level fn + replacing it with `functools.partial(pre-bound captured locals)` is **behaviour-EXACT** (partial's early-binding == the closure's late-binding when the captured var never changes). This keeps ALL call sites VERBATIM. +- Reusable analysis tools written this invocation: `/tmp/freevars.py <outer_fn> <nested...>` (symtable: prints each nested closure's captures-from-outer + globals) and `/tmp/reassign.py <outer_fn> <names...>` (AST: prints real assignments to those names in the outer scope, excluding nested scopes). RE-CREATE them if /tmp is gone (they're ~30 lines each; symtable `is_free()`/`is_local()` + ast Store-Name walk skipping nested FunctionDefs). +- **Generic per-giant procedure**: (1) run freevars.py to get each nested closure's true enclosing-local captures (module-importable names like `get_pipeline_state_lock`/`load_contract`/`save_contract` are barrel-level → become `_pkg.` when moved, NOT params; only genuine locals like pipeline_id/store/pipeline/spawner become params; names the outer fn imports LOCALLY and aren't barrel-level, e.g. `SliceStatus`, become a function-local import in the lifted fn). (2) run reassign.py to CONFIRM none of those locals are reassigned after the closure def (if any IS, that one must be threaded live, not partial-bound). (3) In the barrel, cut the nested `def`s, paste them as barrel TOP-LEVEL fns (dedent 4-8 spaces; add `*, <captured-locals>` kwargs to the signature; bodies otherwise VERBATIM — bare barrel refs resolve at barrel level, add local imports for non-barrel names). (4) Replace each removed nested def with `name = functools.partial(name_toplevel, <captured>=<captured>, ...)` at the same spot (add `import functools` to barrel if missing). A closure that calls a sibling closure (e.g. `_persist_slice_status_complete` calls `_commit_and_push_slice_statefiles`) takes the sibling as a param `commit_and_push` and the partial passes the other partial. (5) Verify barrel imports + targeted pytest (call sites unchanged so this should be green). (6) Use the tool to move the lifted top-level fns into a support submodule (e.g. `_run_implement_support.py`) — tool auto-rewrites bare barrel refs → `_pkg.`. (7) Use the tool to move the now-under-cap giant into its own submodule (e.g. `_run_implement.py`). Re-export everything. + +### GIANT #1 DONE ✅ (commit 2ed6ef8e9) — `_run_implement_phase_slices` split → `_run_implement.py` (1,496L) + `_run_implement_support.py` (242L). Barrel 8,364 → 6,694. +- Lifted 3 closures (`_commit_and_push_slice_statefiles`, `_persist_slice_status_complete`, `_contract_loader`) to `*_impl` fns in `_run_implement_support.py`; replaced with `functools.partial(...)` INSIDE the giant. All call sites verbatim. Then tool-moved the giant to `_run_implement.py`. +- **CORRECTION to the recipe (bit me, verification caught it): `load_contract`/`save_contract`/`SliceStatus` were FUNCTION-LOCAL imports in the giant, NOT barrel-level** — the lifted `*_impl` fns must re-import them locally (`from egg_contracts.loader import load_contract, save_contract`; `from egg_contracts.models import SliceStatus`). ALWAYS run the freevars UNRESOLVED_GLOBALS check (symtable: a lifted top-level fn's `is_global()` names must all be module-bound-or-builtin) BEFORE trusting — it flags names that aren't actually barrel-level. +- The submodule header pushed the giant to 1,503 (3 over) → shaved by lifting `_contract_loader` too AND trimming the submodule docstring to 1 line (1,496). Barrel keeps `import functools` + `import concurrent.futures` with `# noqa: F401 — retained for _pkg.<x> re-export` (reached via `_pkg.functools`/`_pkg.concurrent` from `_run_implement.py`). +- Verified: make lint clean; 183 slice-loop/pipeline tests pass; 16,203 collect 0 import errors. + +### GIANT #2 — REFINED PLAN (this invocation found closures ALONE are insufficient) +- **Exact sizes (AST): `_run_concurrent_phase` = lines 1165-2901 = 1,737L. `_run_concurrent_phase_with_impasse_retry` = 960-1162 = 203L. `_run_pipeline` = 3025-6348 = 3,324L.** +- The 5 closures total only **211L** (`_superseded_by_restart` 1665-1675/11L; `_record_container_exit` 1680-1749/70L; `_stop_running_containers` 1751-1758/8L; `_latest_proposal_ts` 1770-1790/21L; `_update_agents_complete` 1792-1892/101L). Lifting all 5 → giant ≈ **1,544L, STILL ~44 OVER cap.** So closure-lift is NECESSARY BUT NOT SUFFICIENT. +- **MUST ALSO extract one more ~50-70L block** to get `_run_concurrent_phase` under 1500. Candidate seams (section-header comments, offsets relative to 1165): "Build per-role prompts" (~+98..+235), "Phase-level retry for transient spawn failures" (~+288..+358, ~70L), "Record spawned containers/agents in pipeline state" (~+359..+418). Extract ONE cohesive setup block as a module-level helper (thread its inputs as params, return its outputs) — verify the data-flow (inputs→outputs) carefully; this is the higher-risk part. Then giant + retry-wrapper: giant → `_run_concurrent.py` (must be <1500), retry-wrapper (203L) → same file only if it fits, else its OWN `_run_concurrent_retry.py`; lifted closures + the setup-block helper → `_run_concurrent_support.py`. +- **Per-closure param lists (genuine local-var captures → `*, kwargs`; the rest are re-imports)**: `_superseded_by_restart(*, pipeline_id, run_epoch, store)`; `_record_container_exit(exec_info, final_info, *, docker_client, _logs_lock, has_failures, all_logs, store, phase_str, pipeline_id)`; `_stop_running_containers(*, active_executions, exited_containers, docker_client)`; `_latest_proposal_ts(_pid, _sid)` [no local-var captures]; `_update_agents_complete(*, store, phase_str, pipeline_id, slice_id)`. +- **Local re-imports the lifted fns need** (aliased/local imports in the outer fn, NOT barrel-level — freevars UNRESOLVED check confirms): `from models import AgentExecution as StateAgentExecution, AgentExecutionStatus as StateAgentStatus` (used by `_record_container_exit` + `_update_agents_complete`); `_get_brc_tracker` — re-import `from peer_consensus import get_peer_consensus_tracker as _get_brc_tracker` (None-or-fn try/except, used by `_latest_proposal_ts` + `_update_agents_complete`). `PipelinePhase`/`ContainerStatus`/`datetime`/`UTC`/`AgentExitInfo`/`get_pipeline_state_lock`/`logger` ARE barrel-level → resolve bare→`_pkg.` on tool-move (but RUN the freevars UNRESOLVED check to confirm — the outer fn may also locally-import PipelinePhase/ContainerStatus; if flagged, re-import them too). Before removing the outer `_get_brc_tracker` block (1762-1768), grep-confirm the OUTER fn body doesn't use `_get_brc_tracker` directly (only the 2 closures do per freevars) → then it can be deleted. +- Concurrency-safe (from prior analysis): `has_failures = [False]` mutable box; `all_logs`/`active_executions`/`exited_containers` mutated in place → pass refs as params, partial-safe. NO nonlocal. Keep bodies BYTE-verbatim. Verify: test_consensus_polling, test_brc_nack_iteration, test_concurrent_*, and collect-only. + +### GIANT #2 ANALYSIS (original) — `_run_concurrent_phase` + `_run_concurrent_phase_with_impasse_retry` +- 5 nested closures to lift (all partial-safe — verified via reassign.py that every captured local is assigned ONLY before its closure's def; NO nonlocal): `_superseded_by_restart` (caps: pipeline_id, run_epoch, store), `_record_container_exit(exec_info, final_info)` (caps: PipelinePhase, StateAgentStatus, _logs_lock, all_logs, docker_client, has_failures, phase_str, pipeline_id, store — it's a CONTAINER-EXIT CALLBACK passed to the monitor), `_stop_running_containers` (caps: active_executions, docker_client, exited_containers), `_latest_proposal_ts(_pid, _sid)` (caps: _get_brc_tracker), `_update_agents_complete` (caps: ContainerStatus, PipelinePhase, StateAgentStatus, _get_brc_tracker, phase_str, pipeline_id, slice_id, store). +- **Concurrency-safe confirmed**: `has_failures = [False]` is an explicit MUTABLE BOX (mutated in-place `has_failures[0] = True`, never rebound) → partial-binding the same list ref is behaviour-exact. `all_logs`/`active_executions`/`exited_containers` likewise mutated in place. reviewer_concurrency scrutinizes this — keep bodies BYTE-verbatim, note the box-passing in the commit. +- `_get_brc_tracker` is NOT a closure — it's a local `from peer_consensus import get_peer_consensus_tracker as _get_brc_tracker` (None-or-fn, try/except). Lifted `_latest_proposal_ts_impl`/`_update_agents_complete_impl` should re-import it locally (replicate the None-or-import), not thread it as a param. +- Sizes: after lifting the 5 closures (~150L total) the giant drops well under cap. Put lifted fns in `_run_concurrent_support.py`, move the giant + its retry-wrapper to `_run_concurrent.py`. Run the freevars UNRESOLVED_GLOBALS check on each lifted fn (watch for more function-local imports like StateAgentStatus/StateAgentExecution/docker_client that aren't barrel-level → local re-import). Verify: test_consensus_polling, test_brc_nack_iteration, test_concurrent_*. + +### CONCRETE: `_run_implement_phase_slices` (1,678L → ~1,485L after lifting 2 closures; smallest giant, do FIRST) [DONE — see GIANT #1 above] +- Lift these 2 nested closures to a support module `_run_implement_support.py`: + - `_commit_and_push_slice_statefiles(message, *, pipeline_id, worktree_repo_path, pipeline, store, spawner, gateway_mode, issue_number)` — barrel body at 1114-1188. Uses `_pkg.get_pipeline_state_lock`, `_pkg._commit_statefiles_to_worktree`, `_pkg._pipeline_identifier`, `_pkg.logger`. 3 call sites (all `_commit_and_push_slice_statefiles(msg)`). + - `_persist_slice_status_complete(slice_id, *, pipeline_id, worktree_repo_path, commit_and_push, pr_number=None, pr_url=None, basis=None, commit_to_branch=True)` — barrel body at 1190-1315. Uses `_pkg.get_pipeline_state_lock`, `_pkg.load_contract`, `_pkg.save_contract`, `_pkg._validate_slice_completion_basis`, `_pkg.SliceCompletionInvariantError`, `_pkg.logger`; add local `from egg_contracts.models import SliceStatus`; its call to `_commit_and_push_slice_statefiles` becomes `commit_and_push(...)`. 5 call sites. +- Replace the 2 nested defs (in the giant, after the local imports ~line 1113) with: + `_commit_and_push_slice_statefiles = functools.partial(_pkg._commit_and_push_slice_statefiles, pipeline_id=pipeline_id, worktree_repo_path=worktree_repo_path, pipeline=pipeline, store=store, spawner=spawner, gateway_mode=gateway_mode, issue_number=issue_number)` and + `_persist_slice_status_complete = functools.partial(_pkg._persist_slice_status_complete, pipeline_id=pipeline_id, worktree_repo_path=worktree_repo_path, commit_and_push=_commit_and_push_slice_statefiles)`. + (In the barrel these are bare `_commit_and_push_slice_statefiles`/etc.; after moving the giant to `_run_implement.py` the tool rewrites to `_pkg.`.) Reassignment check: only `issue_number` is assigned (L1019, BEFORE the closures) → all safe. `_contract_loader`/`_bootstrap_check_one` stay nested (small). For extra margin also lift `_contract_loader` if needed. +- `_run_concurrent_phase` (1,859L): nested closures `_superseded_by_restart`, `_record_container_exit`, `_stop_running_containers`, `_latest_proposal_ts`, `_update_agents_complete` (abs ~3343-3480) — same partial-lift; NO nonlocal, verify reassignment. reviewer_concurrency will scrutinize — keep bodies byte-verbatim. +- `_run_pipeline` (3,325L): the hard one. Has ONE `nonlocal phase_overseer_active` in `_make_overseer_teardown_hook` — that closure can't be trivially partial-lifted (it rebinds an outer local); either keep it nested or use a 1-element list/box for the flag. Mandate (non-negotiable #7): split the phase-transition loop into per-phase handlers + thin loop. This one likely needs a small shared context (dataclass or a mutable box) for the phase-loop state; do it LAST, most carefully, dense seam coverage. +- **task-4-3 (non-negotiable #7): the THREE giant functions** are all that keep the barrel over cap. Current barrel positions (re-grep for exact lines): + - `_run_implement_phase_slices` (~1,678L — barely over cap) + - `_run_concurrent_phase_with_impasse_retry` (~205L — UNDER cap, a thin wrapper around `_run_concurrent_phase`; can move normally with the tool, but keep it WITH `_run_concurrent_phase` for cohesion) + - `_run_concurrent_phase` (~1,859L — barely over cap) + - `_run_pipeline` (~3,325L — the big one) + Each must be INTERNALLY SPLIT (a submodule holding one whole is still over cap). Approach per giant: first identify internal seams (big self-contained blocks / per-phase sections) and extract them as helper fns (either into the same new submodule as sub-helpers, or a nested sub-package), leaving the giant itself under cap. For `_run_pipeline`: split the phase-transition loop into per-phase handlers (`_run_refine_phase`/`_run_plan_phase`/`_run_pr_phase` + the existing `_run_implement_phase_slices`) + a thin `_run_loop`, preserving transition ordering EXACTLY. HIGHEST-RISK — do in a dedicated invocation with full budget; dense seam coverage: test_consensus_polling, test_brc_nack_iteration, test_concurrent_*, test_advance_phase_*, test_slice_run_loop_integration. +- **task-4-3 (non-negotiable #7): the THREE giant functions must be INTERNALLY SPLIT, not just moved** (each already exceeds the 1500 hard cap so a submodule holding one is still over cap): + - `_run_pipeline` (~3,300L) → split into per-phase handlers `_run_refine_phase`/`_run_plan_phase`/`_run_implement_phase_slices`(exists)/`_run_pr_phase` + thin `_run_loop`. Preserve transition ordering EXACTLY. Dense seam coverage: test_consensus_polling, test_brc_nack_iteration, test_concurrent_*, test_advance_phase_*. + - `_run_concurrent_phase` (~1,740L) and `_run_implement_phase_slices` (~1,680L) → internal split into their own submodule(s), recursive barrel if a cluster further-splits. +- **task-4-5/4-6 terminal**: drop the WIP `pipelines/__init__.py` allowlist entry (files: map EMPTY); add a concrete `routes/pipelines/` seam subsection to orchestrator/CLAUDE.md (mirror the slice-15 routes/signals table); verify all 4 CLAUDE.md seam tables current; DELETE `.egg-state/agent-outputs/coder/slice4_xtract.py` (scratch tool, not a deliverable) before the terminal propose. **Dockerfile: NO change needed** — `pipelines/` stays under the recursive `COPY orchestrator/routes/ ./routes/` (orchestrator/Dockerfile:45); confirmed by grounding. +- Then `make lint` + `make test-all` green, and `mcp__brc__propose`. + +### GIANT #3 — CONCRETE STRUCTURAL MAP (analysis done this invocation; EXECUTE next, dedicated fresh context) +- `_run_pipeline` L1083-4406 (**3,324L**). Body = tiny setup (L1083-1120) + ONE outer `try:` L1121-4406 (28 body stmts, 2 except handlers, 7 finally stmts). +- **3 nested closures** (lift or keep-nested): `_make_overseer_teardown_hook` L1129-1165 (37L, contains inner `_hook` L1152-1163 with the ONLY `nonlocal phase_overseer_active` at L1153 — to lift, convert the flag to a 1-element box `phase_overseer_active = [False]` and mutate `[0]`, OR just KEEP this closure nested since it is only 37L); `_on_health_escalation` L1965-1967 (3L, keep nested); `_health_monitor_poll` L1979-2014 (36L). +- **Setup blocks inside the try (extract to `_run_pipeline_setup.py` helpers — thread inputs/outputs explicitly; each is a self-contained `if`): ~666L total →** host_repo_map `if host_repo_map:` L1218-1327 (110L); worktree `if worktree_repo_path != repo_path:` L1341-1437 (97L); source_branch `if pipeline.source_branch and not (...)` L1464-1527 (64L); contract-sync `if not pipeline.contract_synced:` L1531-1782 (**252L**, biggest single block); start_phase `if pipeline.config.start_phase == "implement":` L1796-1938 (143L); plus a try L1950-2033 (84L, overseer/health-monitor setup — pairs with the 2 health closures). +- **THE PHASE-TRANSITION STATE MACHINE = `while True:` L2035-4024 (1,990L)** — this is what non-negotiable #7 targets. Phase branches inside: `current_phase == PipelinePhase.IMPLEMENT` at L2126 and L3073; plan→populate-contract L2817; refine/plan HITL-sync L2974; converge-before-advance (#3392) L3141 / L3677 / L3731; auto-advance respawn L3951 (**has `# TEST_MARKER: auto_advance_block` L3951 — load-bearing, TestAutoAdvanceRespawnsThread asserts on it via inspect.getsource; if the block moves to a submodule, update that test to glob the pkg dir, task-4-6**); `current_phase == PipelinePhase.APPLY` L3985. +- **finally block** L4277-4406: overseer teardown `if overseer_container_id:` L4277-4295 (19L); cleanup try L4300-4376 (77L); restart-detection `if not pipeline_was_restarted:` L4383-4406 (24L) — extract cleanup to a `_run_pipeline_finally_impl` helper. +- **Recommended execution order (each a commit-able checkpoint):** (1) lift the 3 health/overseer closures + the L1950-2033 setup try → `_run_pipeline_support.py` (box the nonlocal flag). (2) Extract the 5 setup `if` blocks → `_run_pipeline_setup.py` (careful data-flow: many locals in/out — return a small result object or tuple; verify each with collect-only + targeted tests). **STATUS: contract-sync (block 4) DONE (bite 1). REMAINING setup blocks (line numbers UNCHANGED for the 3 before contract-sync; start_phase shifted up ~240): host_repo `if host_repo_map:` L1218-1327 (0 ret, OUT repo_volumes/worktree_repo_path); worktree `if worktree_repo_path!=repo_path:` L1341-1437 (2 ret, OUT current_phase/pipeline); source_branch L1464-1527 (0 ret, OUT analysis_rel/drafts_dir/plan_rel); start_phase `if start_phase=='implement':` (~L1556-1698 now, 2 ret, OUT pipeline).** ⚠️ **CONDITIONAL-ASSIGNMENT CAVEAT:** unlike contract-sync (OUT=pipeline, always-defined), host_repo/source_branch/worktree assign their OUTPUTS *inside* the `if` — if the guard is false the output is undefined. Before extracting, check whether each output is (a) also assigned before the block [thread as input+output default], (b) has an `else` branch, or (c) only read later under the SAME guard [then it's not a must-return output]. Run `blockflow.py` + READ the block's if/else to confirm; do NOT assume the `(x, done)` pattern transfers blindly. (3) Extract the finally-cleanup → helper. After (1)-(3) the giant ≈ 3,324 − 90 − 666 − 96 ≈ **2,470L, STILL over** → (4) the hard part: split the `while True:` loop's per-phase sections into `_run_refine_phase`/`_run_plan_phase`/`_run_pr_phase` handlers + a THIN loop, preserving transition ordering EXACTLY. The loop shares MANY locals across iterations → thread loop-state via a small mutable context (dataclass/SimpleNamespace) or return-and-rebind the handful of state vars each iteration. HIGHEST RISK. Dense seam coverage: test_consensus_polling, test_brc_nack_iteration, test_advance_phase_*, test_concurrent_*, test_slice_run_loop_integration, TestAutoAdvanceRespawnsThread. Then giant → `_run_pipeline.py` (must be <1500), re-export all. +- After GIANT #3: barrel ≈ 1,440L → under cap → **task-4-5/4-6 TERMINAL**: drop the `pipelines/__init__.py` allowlist entry (files map EMPTY); add the concrete `routes/pipelines/` seam subsection to orchestrator/CLAUDE.md; fix the 4 pre-existing source-introspection/env test failures (see Current state); DELETE `slice4_xtract.py` + `giant2_*.py` scratch; `make lint` + `make test-all`; then `mcp__brc__propose`. + +## Known NON-issues (do not chase) +- Sandbox env failures: `git init` returns non-zero ("not supported in the container") and gateway_client tests erroring on gateway git policy — identical class the recovered chain documented (~143 non-passing). NOT split-induced. They fail in test SETUP before pipelines code runs. +- My naive `python -c "import orchestrator.routes.pipelines"` fails on `agent_salvage` (parent-of-routes module needs `...` not `..`, but the try/except flat fallback resolves under `PYTHONPATH=orchestrator`, which is how the suite runs). Pre-existing from the pure-move; use `PYTHONPATH=orchestrator .venv/bin/python -c "import routes.pipelines"` for smoke checks. ## Open NACK responses -(none yet) +(none — not yet proposed; decomposition incomplete) diff --git a/.egg-state/agent-outputs/reviewer_code/brc-memory-issue-3312-v2.md b/.egg-state/agent-outputs/reviewer_code/brc-memory-issue-3312-v2.md new file mode 100644 index 0000000000..4a8f18b7ab --- /dev/null +++ b/.egg-state/agent-outputs/reviewer_code/brc-memory-issue-3312-v2.md @@ -0,0 +1,27 @@ +## Codebase / change model + +<!-- enrichment (claims, not ground truth); re-verify vs the live git-log delta — #3189 anchors are authoritative --> +- + +## Per-producer assessment + +<!-- summaries are SHA-stamped claims; stale when enrichment_sha != the producer's current proposal SHA --> + +### coder + +- producer: coder +- last_reviewed_commit_sha: - +- prior_verdict: NACK +- prior_nack_reasons: blocking +- prior_conditional_obligation: - +- enrichment_sha: - +- summary_of_assessment: blocking + +## Decision log + +- 2026-07-05T19:12:31Z ack coder: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx [a.py] +- 2026-07-05T19:12:31Z nack coder: blocking +- 2026-07-05T19:14:11Z ack coder: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx [a.py] +- 2026-07-05T19:14:11Z nack coder: blocking +- 2026-07-05T19:15:05Z ack coder: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx [a.py] +- 2026-07-05T19:15:05Z nack coder: blocking diff --git a/.egg-state/brc-history/issue-3312-v2-implement-slice-4.json b/.egg-state/brc-history/issue-3312-v2-implement-slice-4.json new file mode 100644 index 0000000000..c0b8d16847 --- /dev/null +++ b/.egg-state/brc-history/issue-3312-v2-implement-slice-4.json @@ -0,0 +1,12117 @@ +[ + { + "id": "47f4435a-d820-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:29:43.194505+00:00", + "phase": "implement" + }, + { + "id": "362f36aa-9cc9-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:29:43.863197+00:00", + "phase": "implement" + }, + { + "id": "76aaa56a-05a4-43", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:29:49.855074+00:00", + "phase": "implement" + }, + { + "id": "0d5997bc-7552-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:29:51.647632+00:00", + "phase": "implement" + }, + { + "id": "e268e790-7ca8-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter deliverable is the orchestrator/CLAUDE.md routes/pipelines/ seam-table row (task-4-5), which must document the coder's CONCRETE landed submodule layout \u2014 same pattern as slices 1-3 where each seam table was authored against the landed split, not a guessed one. pipelines.py is still a single 30,520-line file on both the coder work branch and this branch (HEAD 711b0de43); coder producer_phase=WORKING, no CONSENSUS_PROPOSE for issue-3312-v2/slice-4 yet. Blocked on the coder landing the pipelines/ decomposition before I can author an accurate seam table. Not proposing speculative docs.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:30:57.648474+00:00", + "phase": "implement" + }, + { + "id": "c4881f52-67a3-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=80604 util=0.08 cache_hit=0.94 decision=no_warm_session", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:31:25.218988+00:00", + "phase": "implement" + }, + { + "id": "ecd395c2-2afa-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:31:30.358287+00:00", + "phase": "implement" + }, + { + "id": "792b0126-3c64-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:31:36.026976+00:00", + "phase": "implement" + }, + { + "id": "78e2e40c-4e55-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter still blocked on coder. BRC state unchanged: coder producer_phase=WORKING, no CONSENSUS_PROPOSE for issue-3312-v2/slice-4; pipelines.py not yet decomposed. Additionally the local worktree /home/egg/repos/egg is currently empty / re-provisioning (mount recreated 07:31, not a git repo this instant), so no tree to author the routes/pipelines/ seam table against even if I wanted to. Deliverable (orchestrator/CLAUDE.md pipelines/ seam row, task-4-5) must document the coder's CONCRETE landed submodule layout \u2014 same pattern as slices 1-3. Not proposing speculative docs; exiting cleanly.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:32:18.878391+00:00", + "phase": "implement" + }, + { + "id": "9fd47a32-0fb8-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=86500 util=0.09 cache_hit=0.97 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:32:27.568396+00:00", + "phase": "implement" + }, + { + "id": "5a1509cf-53da-43", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:32:33.329834+00:00", + "phase": "implement" + }, + { + "id": "b9f0e6e9-b7b1-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:32:43.722617+00:00", + "phase": "implement" + }, + { + "id": "68c2f656-7514-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:32:56.130833+00:00", + "phase": "implement" + }, + { + "id": "12f6135b-7f50-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter blocked on coder (3rd consecutive spawn, no state change). coder producer_phase=WORKING, no CONSENSUS_PROPOSE for issue-3312-v2/slice-4; pipelines.py still undecomposed, HEAD 711b0de43 on coder+documenter branches. My deliverable (orchestrator/CLAUDE.md routes/pipelines/ seam row, task-4-5) documents the coder's CONCRETE landed submodule layout \u2014 cannot author until the split lands. The orchestrator keeps deriving documenter/propose, but the correct predecessor is coder/propose; documentation is strictly downstream of the split. Exiting cleanly, no speculative proposal.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:33:24.157597+00:00", + "phase": "implement" + }, + { + "id": "5e453853-da82-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=91649 util=0.09 cache_hit=0.97 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:33:30.535592+00:00", + "phase": "implement" + }, + { + "id": "88ea2526-a7e0-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:35:11.933330+00:00", + "phase": "implement" + }, + { + "id": "02dd03ab-cd91-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:37:19.942566+00:00", + "phase": "implement" + }, + { + "id": "e3e01da3-ce65-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:39:27.837486+00:00", + "phase": "implement" + }, + { + "id": "adb1d89c-8ad2-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:41:48.699065+00:00", + "phase": "implement" + }, + { + "id": "496dd8de-c214-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:46:26.878921+00:00", + "phase": "implement" + }, + { + "id": "e6cc7a28-e9f6-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "slice-4 pipelines.py decomposition: task-4-2 pure-move baseline committed (ce433c299) + verified (imports clean, ruff clean, ratchet exit 0, 142/142 targeted tests; 4 test_compose_event_prompt failures proven pre-existing/environmental via stash). Baseline is local-only (direct push gateway-blocked; branch persists across events). Remaining task-4-3/4/5/6 (extraction + _run_pipeline split + drop last allowlist entry) planned in durable BRC memory for continuation; propose only when the whole slice is green. Not proposing this event (slice incomplete).", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:46:44.176060+00:00", + "phase": "implement" + }, + { + "id": "54ea6acd-810a-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=175936 util=0.18 cache_hit=0.99 decision=no_warm_session", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:47:07.720854+00:00", + "phase": "implement" + }, + { + "id": "289b8c1b-8121-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:47:16.337805+00:00", + "phase": "implement" + }, + { + "id": "e37d8bb0-f2d7-47", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:47:25.809024+00:00", + "phase": "implement" + }, + { + "id": "2d65fa63-5299-40", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=189601 util=0.19 cache_hit=0.96 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:49:18.864662+00:00", + "phase": "implement" + }, + { + "id": "83067c54-5c8f-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:49:25.308728+00:00", + "phase": "implement" + }, + { + "id": "e032f0d4-9b40-47", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:49:36.848698+00:00", + "phase": "implement" + }, + { + "id": "b0098ca8-139c-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "slice-4 coder still BLOCKED on the same unmounted/read-only worktree infra fault (see standing OVERSEER_ALERT d6ef622d): /home/egg/repos/egg has 0 files, writes return Permission denied, HEAD still 711b0de43 (verified baseline ce433c299 still orphaned off-branch). No code/state write is possible; no coder action can clear this. Awaiting infra repair: remount a writable tree + `git update-ref refs/heads/egg/issue-3312-v2-slice-4-coder/work ce433c299`. Not re-broadcasting full alert to avoid noise.", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:49:54.422342+00:00", + "phase": "implement" + }, + { + "id": "466c68cc-ddbd-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=193252 util=0.19 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T07:50:02.346263+00:00", + "phase": "implement" + }, + { + "id": "864658c8-e445-41", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T08:03:39.379484+00:00", + "phase": "implement" + }, + { + "id": "5a78d9ed-805f-41", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T08:03:44.756817+00:00", + "phase": "implement" + }, + { + "id": "9d93d54e-79c7-44", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter blocked on coder (4th consecutive spawn, no state change). coder WORKING, no proposal; pipelines.py undecomposed at HEAD 711b0de43. Raised a medium overseer alert this round flagging the stall + empty-worktree infra symptom. Not proposing speculative docs; exiting cleanly.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T08:04:25.936202+00:00", + "phase": "implement" + }, + { + "id": "1573068c-9600-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=98451 util=0.10 cache_hit=0.95 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T08:04:33.933603+00:00", + "phase": "implement" + }, + { + "id": "5b738d82-dd25-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T08:20:14.273864+00:00", + "phase": "implement" + }, + { + "id": "1adb48a8-99f6-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T08:20:20.325827+00:00", + "phase": "implement" + }, + { + "id": "82efa9bf-68dd-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=194608 util=0.19 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T08:20:32.709283+00:00", + "phase": "implement" + }, + { + "id": "77f5abc8-05ea-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T08:33:42.357469+00:00", + "phase": "implement" + }, + { + "id": "3fe94bb8-00a1-47", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T08:33:47.800560+00:00", + "phase": "implement" + }, + { + "id": "0a37f6e8-b276-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter blocked on coder (5th spawn, no change since prior). coder WORKING, no CONSENSUS_PROPOSE; pipelines.py undecomposed at HEAD 711b0de43. Overseer alert already raised at 08:04 (stall + empty-worktree infra symptom) \u2014 not re-alerting to avoid noise. Deliverable (orchestrator/CLAUDE.md routes/pipelines/ seam row) is downstream of the coder's split; not proposing speculative docs. Exiting cleanly.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T08:34:02.783389+00:00", + "phase": "implement" + }, + { + "id": "ea99ae3b-36c0-46", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=101650 util=0.10 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T08:34:08.615482+00:00", + "phase": "implement" + }, + { + "id": "73c3fcae-3c79-47", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T08:50:16.360412+00:00", + "phase": "implement" + }, + { + "id": "5abf4733-f614-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T08:50:34.760190+00:00", + "phase": "implement" + }, + { + "id": "d8ea461f-767c-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=196019 util=0.20 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T08:50:49.444771+00:00", + "phase": "implement" + }, + { + "id": "b4d94758-8486-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T09:03:46.027227+00:00", + "phase": "implement" + }, + { + "id": "da03a679-df84-48", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T09:03:50.329010+00:00", + "phase": "implement" + }, + { + "id": "e3c6eff7-7ec0-41", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter blocked on coder (6th spawn, no change). coder WORKING, no CONSENSUS_PROPOSE; pipelines.py undecomposed at HEAD 711b0de43. Overseer alerted at 08:04 (stall + empty-worktree infra symptom); holding on re-alert. Deliverable is downstream of the coder's split \u2014 no speculative docs. Exiting cleanly.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T09:04:06.244544+00:00", + "phase": "implement" + }, + { + "id": "8f8903cb-1b79-47", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=104793 util=0.10 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T09:04:11.196869+00:00", + "phase": "implement" + }, + { + "id": "49bedeca-9274-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T09:20:21.829943+00:00", + "phase": "implement" + }, + { + "id": "58255e35-c1e2-40", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T09:20:31.060471+00:00", + "phase": "implement" + }, + { + "id": "43ab27cd-5c78-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=197538 util=0.20 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T09:20:37.502824+00:00", + "phase": "implement" + }, + { + "id": "7969d8f1-3d1a-47", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T09:33:50.281018+00:00", + "phase": "implement" + }, + { + "id": "3462b592-a07c-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T09:33:54.970410+00:00", + "phase": "implement" + }, + { + "id": "5ac49061-f5a2-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter blocked on coder (7th spawn, ~1.5h stalled, no change). coder WORKING, no proposal; pipelines.py undecomposed at HEAD 711b0de43. Escalated overseer alert to HIGH this round (persistent stall + empty-worktree infra symptom unaddressed since 08:04 medium). No speculative docs; exiting cleanly.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T09:34:22.306134+00:00", + "phase": "implement" + }, + { + "id": "fe47d917-5f73-46", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=110495 util=0.11 cache_hit=0.96 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T09:34:29.412616+00:00", + "phase": "implement" + }, + { + "id": "03065faf-bb5c-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T09:50:21.861845+00:00", + "phase": "implement" + }, + { + "id": "abe3db33-cb1a-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T09:50:29.182585+00:00", + "phase": "implement" + }, + { + "id": "7387738f-ada6-41", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=200079 util=0.20 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T09:51:11.581464+00:00", + "phase": "implement" + }, + { + "id": "9ced9c23-339b-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T10:03:51.080295+00:00", + "phase": "implement" + }, + { + "id": "e5aee041-8ea2-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T10:03:55.860384+00:00", + "phase": "implement" + }, + { + "id": "1774666d-f362-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter blocked on coder (8th spawn, no change). coder WORKING, no proposal; pipelines.py undecomposed at HEAD 711b0de43. Escalation ladder complete (medium 08:04, high 09:34) \u2014 holding on further alerts. No speculative docs; exiting cleanly.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T10:04:03.833561+00:00", + "phase": "implement" + }, + { + "id": "36c735c1-0cf1-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=113151 util=0.11 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T10:04:09.391580+00:00", + "phase": "implement" + }, + { + "id": "9ea5823b-5172-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T10:20:25.897992+00:00", + "phase": "implement" + }, + { + "id": "333d9c17-1ff2-41", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T10:20:34.874151+00:00", + "phase": "implement" + }, + { + "id": "46b3ce05-38c1-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=202120 util=0.20 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T10:20:40.994979+00:00", + "phase": "implement" + }, + { + "id": "f963d9d7-a433-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T10:33:53.229725+00:00", + "phase": "implement" + }, + { + "id": "5bda66ea-1d25-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T10:33:57.520360+00:00", + "phase": "implement" + }, + { + "id": "881787e5-6f53-47", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter blocked on coder (9th spawn, no change). coder WORKING, no proposal; pipelines.py undecomposed at HEAD 711b0de43. Escalation ladder complete (medium 08:04, high 09:34). No speculative docs; exiting cleanly.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T10:34:04.980819+00:00", + "phase": "implement" + }, + { + "id": "7d5855ce-35b7-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=115762 util=0.12 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T10:34:09.957814+00:00", + "phase": "implement" + }, + { + "id": "c7148438-9ea9-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T10:50:27.302920+00:00", + "phase": "implement" + }, + { + "id": "1864cdec-7c7a-48", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T10:50:33.811843+00:00", + "phase": "implement" + }, + { + "id": "5e0ccd9f-9572-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=203072 util=0.20 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T10:50:39.905941+00:00", + "phase": "implement" + }, + { + "id": "06c094d4-4b17-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T22:57:30.487834+00:00", + "phase": "implement" + }, + { + "id": "c9fea317-2b54-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T22:57:30.738604+00:00", + "phase": "implement" + }, + { + "id": "a2726b3f-53d5-43", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T22:57:37.518441+00:00", + "phase": "implement" + }, + { + "id": "9c2d41e2-128f-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T22:57:38.764975+00:00", + "phase": "implement" + }, + { + "id": "a31a6c6c-281d-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T22:59:44.312915+00:00", + "phase": "implement" + }, + { + "id": "55b9afda-5960-48", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T22:59:55.048980+00:00", + "phase": "implement" + }, + { + "id": "005d0100-f267-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter cannot propose yet \u2014 my only deliverable (task-4-5) is the concrete orchestrator/CLAUDE.md `pipelines/` submodule-layout seam row, which is strictly downstream of the coder's decomposition. Grounded on the live tree (HEAD=711b0de43): orchestrator/routes/pipelines.py is STILL the 1.44MB monolith on-branch, no orchestrator/routes/pipelines/ package exists, and the coder's pure-move baseline ce433c299 is orphaned off-branch (coder tip=711b0de43). Writing a seam row now would fabricate a layout that does not exist (correctly NACKable); no_changes_needed is also false (real deliverable). Waiting on coder to land the pipelines/ decomposition on-branch; then I read the concrete layout from the tree, author the seam row, and propose. Documenter worktree is writable and healthy; blocker is upstream (coder infra faults, OVERSEER_ALERTs d6ef622d / b694dadc). Verdict recorded in durable BRC memory.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:00:02.493700+00:00", + "phase": "implement" + }, + { + "id": "d71d0a1c-9c43-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=97209 util=0.10 cache_hit=0.99 decision=no_warm_session", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:00:14.531892+00:00", + "phase": "implement" + }, + { + "id": "2245e123-3396-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:00:19.789599+00:00", + "phase": "implement" + }, + { + "id": "018658be-b823-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:00:45.695385+00:00", + "phase": "implement" + }, + { + "id": "226205ec-4a21-47", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: NO legal move this event. Worktree is now empty + read-only (0 files; git checkout HEAD -- . restores 0; writes to worktree AND .egg-state denied) \u2014 same unmounted-worktree infra fault the coder flagged (d6ef622d/b694dadc), now hitting documenter at base 64fa30773. Cannot write orchestrator/CLAUDE.md seam row (task-4-5), cannot update durable memory, cannot report_impasse (all write .egg-state). Independently, my deliverable is downstream of the coder's pipelines.py decomposition, which is not on-branch (coder tip=711b0de43, baseline ce433c299 orphaned). Raised OVERSEER_ALERT 72762213 with the infra-repair recipe. Alive; blocked on infra repair + coder. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:01:42.863792+00:00", + "phase": "implement" + }, + { + "id": "5f5ca10d-37b5-46", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=105755 util=0.11 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:01:57.134456+00:00", + "phase": "implement" + }, + { + "id": "d7bf9a4a-bc9a-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:02:05.964328+00:00", + "phase": "implement" + }, + { + "id": "33d04a28-0330-47", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:02:17.137245+00:00", + "phase": "implement" + }, + { + "id": "33656349-4a6c-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:02:20.394245+00:00", + "phase": "implement" + }, + { + "id": "f3d16428-291a-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter still BLOCKED, state unchanged: worktree empty + read-only (0 files, HEAD=64fa30773, writes to worktree/.egg-state denied); pipelines.py decomposition not on-branch (coder tip=711b0de43). No legal FS move \u2014 cannot write orchestrator/CLAUDE.md seam row, update durable memory, or report_impasse. Standing OVERSEER_ALERT 72762213 holds the full infra-repair recipe (remount writable trees @64fa30773 \u2192 restore ce433c299 \u2192 coder lands decomposition \u2192 documenter writes seam row). Not re-broadcasting the full alert to avoid noise. Alive; awaiting infra repair + coder. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:02:28.902910+00:00", + "phase": "implement" + }, + { + "id": "5df55ab1-e356-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=109169 util=0.11 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:02:36.724298+00:00", + "phase": "implement" + }, + { + "id": "c81d1b43-7171-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=117069 util=0.12 cache_hit=0.99 decision=no_warm_session", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:02:51.769902+00:00", + "phase": "implement" + }, + { + "id": "75ffe6f8-a917-48", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:02:58.513539+00:00", + "phase": "implement" + }, + { + "id": "117061bc-4422-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:03:06.827067+00:00", + "phase": "implement" + }, + { + "id": "a272a689-80d8-44", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=120064 util=0.12 cache_hit=0.98 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:03:46.402074+00:00", + "phase": "implement" + }, + { + "id": "2e0a8852-7551-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:03:54.909756+00:00", + "phase": "implement" + }, + { + "id": "7a7db4c7-c2b8-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:04:00.193215+00:00", + "phase": "implement" + }, + { + "id": "58976c8b-7928-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=122363 util=0.12 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:04:07.406970+00:00", + "phase": "implement" + }, + { + "id": "0ed7ae06-18fb-43", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:32:44.922625+00:00", + "phase": "implement" + }, + { + "id": "d947fb72-cb8d-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:32:53.280569+00:00", + "phase": "implement" + }, + { + "id": "8dff876d-b851-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: INFRA REPAIRED \u2014 worktree is populated + writable again (durable memory refreshed). Still cannot propose: the pipelines/ decomposition is NOT on-branch (orchestrator/routes/pipelines.py is still the 1.44MB monolith; no pipelines/ package; allowlist still holds the single pipelines.py entry). My task-4-5 seam row is strictly downstream of that decomposition, so writing it now would fabricate a non-existent layout (NACKable); no_changes_needed is false (real deliverable). NOTE for coder/operator: coder branch tip is now 64fa30773 (slice-4 base) \u2014 the earlier pure-move baseline ce433c299 was NOT restored via update-ref and remains orphaned off-branch (still in object store); coder must re-land/cherry-pick it and complete the extraction + _run_pipeline split + drop the LAST allowlist entry. Infra is no longer my blocker; sole remaining dependency = coder landing the decomposition. Alive; will author the seam row and propose once it lands. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:33:47.012251+00:00", + "phase": "implement" + }, + { + "id": "b7c82127-58f3-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=117397 util=0.12 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:33:56.675749+00:00", + "phase": "implement" + }, + { + "id": "2a2efc14-edb7-47", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:34:16.835185+00:00", + "phase": "implement" + }, + { + "id": "74c74218-65ab-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:34:23.526468+00:00", + "phase": "implement" + }, + { + "id": "2923b950-de12-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=125885 util=0.13 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:34:29.685446+00:00", + "phase": "implement" + }, + { + "id": "bad922f2-f14c-47", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:36:23.173981+00:00", + "phase": "implement" + }, + { + "id": "0598923d-bdcd-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:36:23.547319+00:00", + "phase": "implement" + }, + { + "id": "a0c0ffe0-59c8-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:36:27.590110+00:00", + "phase": "implement" + }, + { + "id": "03122658-18e4-43", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:36:31.521514+00:00", + "phase": "implement" + }, + { + "id": "88d8b9ba-a556-43", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change since last event. Infra healthy (worktree writable), but the pipelines/ decomposition is still NOT on-branch \u2014 orchestrator/routes/pipelines.py remains the 1.44MB monolith, no pipelines/ package, allowlist still holds the single pipelines.py entry, coder tip=64fa30773. My task-4-5 seam row is downstream of that decomposition; cannot propose without fabricating a non-existent layout, and no_changes_needed is false. Sole dependency = coder landing the decomposition (must re-land/cherry-pick orphaned baseline ce433c299 first). Durable memory current. Alive; will author the seam row and propose once the code lands. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:36:40.220411+00:00", + "phase": "implement" + }, + { + "id": "fc6a52ae-8525-41", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=120724 util=0.12 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:36:45.847222+00:00", + "phase": "implement" + }, + { + "id": "aef698ce-ff39-44", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:38:36.818138+00:00", + "phase": "implement" + }, + { + "id": "1813ded1-911f-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:40:50.385850+00:00", + "phase": "implement" + }, + { + "id": "bb757af9-d9d6-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:43:17.433119+00:00", + "phase": "implement" + }, + { + "id": "e2086c3e-762f-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:45:32.428225+00:00", + "phase": "implement" + }, + { + "id": "8b263ded-70e6-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=215534 util=0.22 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-04T23:45:54.167764+00:00", + "phase": "implement" + }, + { + "id": "f3219c5b-60a9-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:06:58.564283+00:00", + "phase": "implement" + }, + { + "id": "babdeaa6-c688-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:07:20.679979+00:00", + "phase": "implement" + }, + { + "id": "64b3f123-992b-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: CODER PROGRESS observed. Coder branch tip advanced 64fa30773 \u2192 0228f4a9f = \"Move pipelines.py to pipelines/__init__.py (pure move baseline)\" (task-4-2): pipelines.py is gone, pipelines/__init__.py exists. BUT this is only the pure-move baseline \u2014 pipelines/ holds ONLY __init__.py; the EXTRACTION (task-4-3 _run_pipeline split / task-4-4 cluster submodules) has NOT landed yet, and the coder hasn't proposed a complete slice. My task-4-5 seam row documents the concrete submodule layout, which appears only after extraction \u2014 writing it now (whole file = single __init__.py) would be premature/inaccurate and NACKable; no_changes_needed is false. Still blocked on coder, but now on the EXTRACTION step, not the move. Durable memory refreshed (worktree re-provisioning wiped the prior copy). Will author the seam row + propose once the extraction lands. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:08:25.022511+00:00", + "phase": "implement" + }, + { + "id": "c7df0fcd-a195-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=133525 util=0.13 cache_hit=0.97 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:08:34.883085+00:00", + "phase": "implement" + }, + { + "id": "2111b8b1-fc84-48", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:16:03.238247+00:00", + "phase": "implement" + }, + { + "id": "be6d0abb-826e-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:16:13.393518+00:00", + "phase": "implement" + }, + { + "id": "71976105-a8e6-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:18:53.375608+00:00", + "phase": "implement" + }, + { + "id": "effe37fd-31e3-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:20:46.908968+00:00", + "phase": "implement" + }, + { + "id": "62219cfb-7cb9-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:20:55.369790+00:00", + "phase": "implement" + }, + { + "id": "079e615c-b1cf-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:21:07.321537+00:00", + "phase": "implement" + }, + { + "id": "54d25411-e6d6-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=248824 util=0.25 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:21:20.439000+00:00", + "phase": "implement" + }, + { + "id": "a9719fb9-5faf-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:21:28.883684+00:00", + "phase": "implement" + }, + { + "id": "f1a64283-9a86-41", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:21:39.700751+00:00", + "phase": "implement" + }, + { + "id": "8efa4951-a954-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=252628 util=0.25 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:21:49.369006+00:00", + "phase": "implement" + }, + { + "id": "e0eb37f8-a59e-46", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: ROOT CAUSE isolated from the coder branch reflog and reported (OVERSEER_ALERT b4ca0796). The coder's pure-move baseline is committed then ORPHANED every spawn by `git reset --hard origin/egg/issue-3312-v2/slice-4` at container setup (0228f4a9f and earlier ce433c299 both orphaned; origin still @64fa30773). Coder can't push (gated behind slice-complete propose) so it can never build the extraction on its baseline \u2192 infinite restart, zero integration. This is the day-long-stall wedge; the earlier read-only-worktree fault is now REPAIRED. My task-4-5 seam row is strictly downstream \u2014 cannot propose (would fabricate a non-existent layout; no_changes_needed is false), and I cannot fix the branch reset (role/push-gated). Awaiting operator branch-persistence fix \u2192 coder extraction \u2192 then I author the orchestrator/CLAUDE.md pipelines/ seam row and propose. Durable memory refreshed. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:22:25.110330+00:00", + "phase": "implement" + }, + { + "id": "6557f9dc-7f5f-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=149631 util=0.15 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:22:38.071529+00:00", + "phase": "implement" + }, + { + "id": "94c38dc7-5cfd-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:51:57.457418+00:00", + "phase": "implement" + }, + { + "id": "33fc0f0d-fb03-47", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:52:02.913098+00:00", + "phase": "implement" + }, + { + "id": "6d2625e0-80df-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=257729 util=0.26 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:52:15.044177+00:00", + "phase": "implement" + }, + { + "id": "f9384fa8-a0c4-46", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:52:47.960624+00:00", + "phase": "implement" + }, + { + "id": "590a600d-127f-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:52:58.353390+00:00", + "phase": "implement" + }, + { + "id": "502d7ef0-385c-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change. coder_tip=64fa30773, origin/egg/issue-3312-v2/slice-4=64fa30773, no new coder commits, pipelines/ extraction still not integrated. The root-cause branch-persistence wedge (coder baseline orphaned by per-spawn `git reset --hard origin`) is captured in standing OVERSEER_ALERT b4ca0796 with the fix recipe; awaiting operator action. My task-4-5 seam row is downstream \u2014 cannot propose (would fabricate a non-existent layout; no_changes_needed false) and cannot fix the branch reset (role/push-gated). Not re-raising the alert to avoid noise. Alive; will author the seam row + propose once the extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:53:15.200923+00:00", + "phase": "implement" + }, + { + "id": "f1ea83a4-42d1-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=153894 util=0.15 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T00:53:23.140806+00:00", + "phase": "implement" + }, + { + "id": "abe6c581-0f3f-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T01:22:00.857244+00:00", + "phase": "implement" + }, + { + "id": "7a022e9f-2608-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T01:22:05.580625+00:00", + "phase": "implement" + }, + { + "id": "93207bc1-72a3-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=261056 util=0.26 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T01:22:10.099208+00:00", + "phase": "implement" + }, + { + "id": "3a752618-98e8-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T01:22:51.888905+00:00", + "phase": "implement" + }, + { + "id": "18aedb7b-53e1-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T01:23:00.258555+00:00", + "phase": "implement" + }, + { + "id": "7bf937b9-5525-41", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (HEAD=coder_tip=origin/slice-4=64fa30773; no pipelines/ extraction integrated). Stable deadlock on the branch-persistence wedge \u2014 coder's baseline orphaned by per-spawn `git reset --hard origin`; fix recipe in standing OVERSEER_ALERT b4ca0796, awaiting operator. My task-4-5 seam row is downstream; no legal producer move (can't fabricate a non-existent layout; no_changes_needed false; can't fix the reset \u2014 role/push-gated). Alive; will author the seam row + propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T01:23:15.149176+00:00", + "phase": "implement" + }, + { + "id": "4df99b0e-3088-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=157457 util=0.16 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T01:23:21.141616+00:00", + "phase": "implement" + }, + { + "id": "dee3c345-e1ec-48", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T01:52:01.872732+00:00", + "phase": "implement" + }, + { + "id": "4fb20364-716c-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T01:52:06.922180+00:00", + "phase": "implement" + }, + { + "id": "72cc682f-0e1d-41", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=264428 util=0.26 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T01:52:13.209784+00:00", + "phase": "implement" + }, + { + "id": "128a19be-bd93-44", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T01:52:53.750243+00:00", + "phase": "implement" + }, + { + "id": "df536d0d-db89-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T01:53:02.769631+00:00", + "phase": "implement" + }, + { + "id": "49d3aa11-b9ec-44", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERT b4ca0796, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose the seam row once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T01:53:15.770053+00:00", + "phase": "implement" + }, + { + "id": "bfc4d455-de29-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=160565 util=0.16 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T01:53:21.150999+00:00", + "phase": "implement" + }, + { + "id": "8e6d0f12-b01b-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T02:22:04.721860+00:00", + "phase": "implement" + }, + { + "id": "35cf7fd9-bb4f-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T02:22:09.892846+00:00", + "phase": "implement" + }, + { + "id": "d827efe9-e20f-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=267651 util=0.27 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T02:22:19.372055+00:00", + "phase": "implement" + }, + { + "id": "f1e02f60-834c-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T02:22:56.495584+00:00", + "phase": "implement" + }, + { + "id": "5a884e3e-d871-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T02:23:05.234749+00:00", + "phase": "implement" + }, + { + "id": "fa06b444-551a-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERT b4ca0796, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T02:23:13.729408+00:00", + "phase": "implement" + }, + { + "id": "2aacd2c4-72af-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=163444 util=0.16 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T02:23:18.771063+00:00", + "phase": "implement" + }, + { + "id": "9919a32d-36b9-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T02:52:07.855364+00:00", + "phase": "implement" + }, + { + "id": "3799040a-efdd-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T02:52:12.487685+00:00", + "phase": "implement" + }, + { + "id": "2b7a081c-c70f-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=271174 util=0.27 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T02:52:16.409987+00:00", + "phase": "implement" + }, + { + "id": "5e41ac4e-b027-46", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T02:52:58.846923+00:00", + "phase": "implement" + }, + { + "id": "78494e01-2326-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T02:53:05.805023+00:00", + "phase": "implement" + }, + { + "id": "a5841912-bb9b-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERT b4ca0796, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T02:53:17.542890+00:00", + "phase": "implement" + }, + { + "id": "0a6d85b5-8634-48", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=166243 util=0.17 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T02:53:22.452737+00:00", + "phase": "implement" + }, + { + "id": "755dbd7d-9bd3-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T03:22:10.529636+00:00", + "phase": "implement" + }, + { + "id": "badf6ea3-ecb9-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T03:22:14.888617+00:00", + "phase": "implement" + }, + { + "id": "b1f5b008-4ea0-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=274364 util=0.27 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T03:22:21.449751+00:00", + "phase": "implement" + }, + { + "id": "cac669e3-8162-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T03:23:01.768580+00:00", + "phase": "implement" + }, + { + "id": "9a0838dc-870d-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T03:23:08.278765+00:00", + "phase": "implement" + }, + { + "id": "cad764ac-d245-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERT b4ca0796, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T03:23:14.876343+00:00", + "phase": "implement" + }, + { + "id": "a3f0382c-2954-46", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=168762 util=0.17 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T03:23:19.142428+00:00", + "phase": "implement" + }, + { + "id": "60bb90fe-67b2-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T03:52:13.909803+00:00", + "phase": "implement" + }, + { + "id": "acfcb3ed-2246-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T03:52:22.075396+00:00", + "phase": "implement" + }, + { + "id": "d1f2ff87-a01e-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=277688 util=0.28 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T03:52:26.141355+00:00", + "phase": "implement" + }, + { + "id": "1f99ce83-9ac8-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T03:53:06.921749+00:00", + "phase": "implement" + }, + { + "id": "4dc0781d-1efa-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T03:53:13.608580+00:00", + "phase": "implement" + }, + { + "id": "fbfc6246-e90d-43", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERT b4ca0796, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T03:53:20.568145+00:00", + "phase": "implement" + }, + { + "id": "ca09c002-52ff-41", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=171182 util=0.17 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T03:53:24.881042+00:00", + "phase": "implement" + }, + { + "id": "a4e1d36f-c7fb-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T04:22:16.857573+00:00", + "phase": "implement" + }, + { + "id": "33f8740c-a4da-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T04:22:21.680012+00:00", + "phase": "implement" + }, + { + "id": "f8566b65-66fe-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=280978 util=0.28 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T04:22:43.580632+00:00", + "phase": "implement" + }, + { + "id": "34c96d54-fba8-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T04:23:09.196303+00:00", + "phase": "implement" + }, + { + "id": "68396e64-a3d7-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T04:23:18.159839+00:00", + "phase": "implement" + }, + { + "id": "d34c36be-5b0a-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERT b4ca0796, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T04:23:25.524806+00:00", + "phase": "implement" + }, + { + "id": "5d57d0a8-b00c-41", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=173784 util=0.17 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T04:23:30.348742+00:00", + "phase": "implement" + }, + { + "id": "8e437634-decd-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T04:52:19.953948+00:00", + "phase": "implement" + }, + { + "id": "535493b4-6044-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T04:52:24.478634+00:00", + "phase": "implement" + }, + { + "id": "98990f84-8234-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=285326 util=0.29 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T04:52:28.440728+00:00", + "phase": "implement" + }, + { + "id": "0afd3df0-7877-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T04:53:11.214034+00:00", + "phase": "implement" + }, + { + "id": "87281758-b865-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T04:53:17.877518+00:00", + "phase": "implement" + }, + { + "id": "2bf879f7-5734-44", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERT b4ca0796, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T04:53:23.995202+00:00", + "phase": "implement" + }, + { + "id": "97399cf9-88ba-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=176165 util=0.18 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T04:53:28.557402+00:00", + "phase": "implement" + }, + { + "id": "86451b3e-1923-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T05:22:22.769665+00:00", + "phase": "implement" + }, + { + "id": "7255e804-263a-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T05:22:27.231939+00:00", + "phase": "implement" + }, + { + "id": "5951300d-c2fc-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=288486 util=0.29 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T05:23:01.195647+00:00", + "phase": "implement" + }, + { + "id": "e5c57edd-33dd-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T05:23:13.683425+00:00", + "phase": "implement" + }, + { + "id": "592b4d56-b3d2-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T05:23:20.550752+00:00", + "phase": "implement" + }, + { + "id": "c9428aa8-451a-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERT b4ca0796, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T05:23:26.665824+00:00", + "phase": "implement" + }, + { + "id": "b210d5cc-7735-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=178541 util=0.18 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T05:23:32.294498+00:00", + "phase": "implement" + }, + { + "id": "f7bef729-a06c-44", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T05:52:24.666966+00:00", + "phase": "implement" + }, + { + "id": "09d84989-4143-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T05:52:29.572696+00:00", + "phase": "implement" + }, + { + "id": "9f8783db-9642-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=293493 util=0.29 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T05:53:07.492824+00:00", + "phase": "implement" + }, + { + "id": "7c45f450-6482-41", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T05:53:15.834585+00:00", + "phase": "implement" + }, + { + "id": "0daf6351-ab23-44", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T05:53:22.528013+00:00", + "phase": "implement" + }, + { + "id": "b7f460d6-1f34-41", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERT b4ca0796, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T05:53:28.761698+00:00", + "phase": "implement" + }, + { + "id": "ee8ce83e-a765-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=181052 util=0.18 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T05:53:33.392057+00:00", + "phase": "implement" + }, + { + "id": "f668bc5e-98d0-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T06:22:27.498151+00:00", + "phase": "implement" + }, + { + "id": "0cae3ded-3c6a-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T06:22:32.971135+00:00", + "phase": "implement" + }, + { + "id": "9a11c724-291e-47", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=298874 util=0.30 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T06:22:58.382800+00:00", + "phase": "implement" + }, + { + "id": "1f9b7fa8-01c0-43", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T06:23:19.952347+00:00", + "phase": "implement" + }, + { + "id": "189d1dca-edf8-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T06:23:33.185344+00:00", + "phase": "implement" + }, + { + "id": "269066c4-5544-47", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction). Deadlock ~5.5h unaddressed; re-surfaced the root-cause branch-persistence alert (a3b4d28a, ref b4ca0796) with the operator fix recipe. Task-4-5 seam row downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T06:24:04.607876+00:00", + "phase": "implement" + }, + { + "id": "429294e6-7f31-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=187198 util=0.19 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T06:24:11.055361+00:00", + "phase": "implement" + }, + { + "id": "014f76de-6d34-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T06:52:27.851369+00:00", + "phase": "implement" + }, + { + "id": "61ba6cfe-57a8-47", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T06:52:33.105466+00:00", + "phase": "implement" + }, + { + "id": "856ce034-b04c-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=303452 util=0.30 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T06:52:53.984473+00:00", + "phase": "implement" + }, + { + "id": "c82af57d-6f0b-41", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T06:53:19.860450+00:00", + "phase": "implement" + }, + { + "id": "7757bbf5-c28b-43", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T06:53:26.538567+00:00", + "phase": "implement" + }, + { + "id": "cdf0c1f7-72ea-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T06:53:32.950809+00:00", + "phase": "implement" + }, + { + "id": "269838c6-8610-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=189883 util=0.19 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T06:53:39.012984+00:00", + "phase": "implement" + }, + { + "id": "8f268a96-49a3-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T07:22:31.217910+00:00", + "phase": "implement" + }, + { + "id": "72f17fc2-5133-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T07:22:40.103448+00:00", + "phase": "implement" + }, + { + "id": "7deee7a6-d946-47", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=307619 util=0.31 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T07:23:04.675892+00:00", + "phase": "implement" + }, + { + "id": "093caa45-2da7-47", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T07:23:22.541552+00:00", + "phase": "implement" + }, + { + "id": "1c587993-94c6-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T07:23:28.943096+00:00", + "phase": "implement" + }, + { + "id": "9c7a5c16-ea3d-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T07:23:39.027898+00:00", + "phase": "implement" + }, + { + "id": "41237b8c-2e54-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=192299 util=0.19 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T07:23:43.969269+00:00", + "phase": "implement" + }, + { + "id": "76fe9b23-fea3-41", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T07:52:34.763758+00:00", + "phase": "implement" + }, + { + "id": "0b8cfc9f-133f-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T07:52:40.082296+00:00", + "phase": "implement" + }, + { + "id": "329a777c-365f-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=312014 util=0.31 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T07:53:01.311366+00:00", + "phase": "implement" + }, + { + "id": "effbd66e-6245-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T07:53:26.683577+00:00", + "phase": "implement" + }, + { + "id": "8606be4a-302d-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T07:53:34.943720+00:00", + "phase": "implement" + }, + { + "id": "a41ce9e1-69cd-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T07:53:42.115361+00:00", + "phase": "implement" + }, + { + "id": "8f04f605-0c72-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=194697 util=0.19 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T07:53:46.340946+00:00", + "phase": "implement" + }, + { + "id": "c69a88ef-e6c2-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T08:22:38.133944+00:00", + "phase": "implement" + }, + { + "id": "90669dc7-048d-48", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T08:22:42.657116+00:00", + "phase": "implement" + }, + { + "id": "552204bc-11f6-44", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=316283 util=0.32 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T08:22:53.782524+00:00", + "phase": "implement" + }, + { + "id": "b25645cc-640d-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T08:23:28.881439+00:00", + "phase": "implement" + }, + { + "id": "b8900d70-e052-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T08:23:37.366050+00:00", + "phase": "implement" + }, + { + "id": "a73d5e4d-9711-47", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T08:23:43.319599+00:00", + "phase": "implement" + }, + { + "id": "ec590891-6196-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=197222 util=0.20 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T08:23:49.070067+00:00", + "phase": "implement" + }, + { + "id": "07f73b25-7c19-41", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T08:52:39.881037+00:00", + "phase": "implement" + }, + { + "id": "2a2a4945-889c-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T08:52:46.230428+00:00", + "phase": "implement" + }, + { + "id": "b3e5f5ec-c374-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=319444 util=0.32 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T08:53:17.057248+00:00", + "phase": "implement" + }, + { + "id": "46104acd-b821-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T08:53:32.958087+00:00", + "phase": "implement" + }, + { + "id": "434fdab0-3c65-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T08:53:39.694540+00:00", + "phase": "implement" + }, + { + "id": "ec35c25c-1900-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T08:53:45.662980+00:00", + "phase": "implement" + }, + { + "id": "761dd245-1c60-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=199618 util=0.20 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T08:53:50.630214+00:00", + "phase": "implement" + }, + { + "id": "b61e7f7c-3906-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T09:22:44.925487+00:00", + "phase": "implement" + }, + { + "id": "91d70c3e-ae35-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T09:22:50.302451+00:00", + "phase": "implement" + }, + { + "id": "4813a4f0-8c5b-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=324327 util=0.32 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T09:22:53.886604+00:00", + "phase": "implement" + }, + { + "id": "fddfde26-581f-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T09:23:36.200485+00:00", + "phase": "implement" + }, + { + "id": "c6d3cd1d-3fec-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T09:23:42.574624+00:00", + "phase": "implement" + }, + { + "id": "0e4e165e-cd33-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T09:23:48.682849+00:00", + "phase": "implement" + }, + { + "id": "c9d1bc20-8687-44", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=202144 util=0.20 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T09:23:54.360487+00:00", + "phase": "implement" + }, + { + "id": "8b085687-0843-41", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T09:52:47.280005+00:00", + "phase": "implement" + }, + { + "id": "22a0a3e0-f8d4-47", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T09:52:52.238441+00:00", + "phase": "implement" + }, + { + "id": "10918e42-5fd4-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=327489 util=0.33 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T09:52:56.542819+00:00", + "phase": "implement" + }, + { + "id": "6669f69e-8081-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T09:53:38.558060+00:00", + "phase": "implement" + }, + { + "id": "9da77389-23ac-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T09:53:45.013631+00:00", + "phase": "implement" + }, + { + "id": "34366636-fec9-43", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T09:53:53.760648+00:00", + "phase": "implement" + }, + { + "id": "3ec5557b-d42f-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=204729 util=0.20 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T09:53:58.320730+00:00", + "phase": "implement" + }, + { + "id": "dd381638-b70e-41", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T10:22:48.876248+00:00", + "phase": "implement" + }, + { + "id": "3bdba7d5-acf9-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T10:22:54.766376+00:00", + "phase": "implement" + }, + { + "id": "db87639c-cbee-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=330636 util=0.33 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T10:23:12.025407+00:00", + "phase": "implement" + }, + { + "id": "5328b94a-70b8-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T10:23:41.900472+00:00", + "phase": "implement" + }, + { + "id": "5ff24b50-e754-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T10:23:48.230556+00:00", + "phase": "implement" + }, + { + "id": "187a6597-c847-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T10:23:56.851602+00:00", + "phase": "implement" + }, + { + "id": "7214b210-37f6-41", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=207285 util=0.21 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T10:24:01.508420+00:00", + "phase": "implement" + }, + { + "id": "de36b07f-6b6e-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T10:52:52.773634+00:00", + "phase": "implement" + }, + { + "id": "5a76126d-5651-40", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T10:52:57.397234+00:00", + "phase": "implement" + }, + { + "id": "e81c9a01-6d54-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=334570 util=0.33 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T10:53:01.856538+00:00", + "phase": "implement" + }, + { + "id": "3b6eec9a-2997-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T10:53:43.773348+00:00", + "phase": "implement" + }, + { + "id": "edb41444-3377-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T10:53:51.895005+00:00", + "phase": "implement" + }, + { + "id": "889ed6a2-d2b2-47", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T10:54:00.766547+00:00", + "phase": "implement" + }, + { + "id": "5b510d44-f137-46", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=209911 util=0.21 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T10:54:05.111999+00:00", + "phase": "implement" + }, + { + "id": "ee31aa30-9c85-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T11:22:53.928674+00:00", + "phase": "implement" + }, + { + "id": "07a921a5-2ad2-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T11:22:58.697991+00:00", + "phase": "implement" + }, + { + "id": "86759aba-7e4b-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=337717 util=0.34 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T11:23:38.739980+00:00", + "phase": "implement" + }, + { + "id": "b6e2e307-6671-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T11:23:46.151574+00:00", + "phase": "implement" + }, + { + "id": "c0b4d225-41ba-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T11:23:54.588859+00:00", + "phase": "implement" + }, + { + "id": "962bb3dc-5153-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T11:24:00.885610+00:00", + "phase": "implement" + }, + { + "id": "47650646-9b6f-48", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=212313 util=0.21 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T11:24:06.793020+00:00", + "phase": "implement" + }, + { + "id": "e9707a7d-1fc9-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T11:52:57.833698+00:00", + "phase": "implement" + }, + { + "id": "67c78d44-51e7-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T11:53:02.914312+00:00", + "phase": "implement" + }, + { + "id": "1897eb42-333f-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=343141 util=0.34 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T11:53:24.742784+00:00", + "phase": "implement" + }, + { + "id": "2d5fb923-66fa-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T11:53:48.870303+00:00", + "phase": "implement" + }, + { + "id": "e64f2083-e091-41", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T11:53:57.573292+00:00", + "phase": "implement" + }, + { + "id": "ad14db31-b673-46", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T11:54:09.449572+00:00", + "phase": "implement" + }, + { + "id": "a7ddac3d-af94-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=215184 util=0.22 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T11:54:14.001220+00:00", + "phase": "implement" + }, + { + "id": "441a1e22-d31f-40", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T12:23:00.244449+00:00", + "phase": "implement" + }, + { + "id": "70e1c52e-1864-48", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T12:23:07.492802+00:00", + "phase": "implement" + }, + { + "id": "2745b73f-9284-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=347194 util=0.35 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T12:23:10.814521+00:00", + "phase": "implement" + }, + { + "id": "466799b3-33ed-41", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T12:23:50.918333+00:00", + "phase": "implement" + }, + { + "id": "06f4e022-06bb-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T12:23:58.031144+00:00", + "phase": "implement" + }, + { + "id": "d1ea9e5d-37fe-47", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (~12h deadlock; all refs @64fa30773; no pipelines/ extraction). Re-surfaced the root-cause branch-persistence alert (b8f628cf, ref b4ca0796/a3b4d28a) with the operator fix recipe. Task-4-5 seam row downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T12:24:22.754954+00:00", + "phase": "implement" + }, + { + "id": "ba61a204-d088-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=220689 util=0.22 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T12:24:29.006158+00:00", + "phase": "implement" + }, + { + "id": "7bdb2314-9cd8-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T12:53:04.829033+00:00", + "phase": "implement" + }, + { + "id": "9d5e15a7-931f-41", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T12:53:11.704353+00:00", + "phase": "implement" + }, + { + "id": "8485a9b2-5f07-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=350408 util=0.35 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T12:53:15.678681+00:00", + "phase": "implement" + }, + { + "id": "e54c1c76-1ca2-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T12:53:55.004868+00:00", + "phase": "implement" + }, + { + "id": "94c80c4c-c9e8-48", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T12:54:03.633672+00:00", + "phase": "implement" + }, + { + "id": "b6b3b69f-eadb-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a / b8f628cf, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T12:54:12.407827+00:00", + "phase": "implement" + }, + { + "id": "6628d766-fa73-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=223209 util=0.22 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T12:54:17.318365+00:00", + "phase": "implement" + }, + { + "id": "bfef8fd1-9654-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T13:23:05.864548+00:00", + "phase": "implement" + }, + { + "id": "bb2f00d2-60c7-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T13:23:10.417868+00:00", + "phase": "implement" + }, + { + "id": "b3738189-ee0d-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=353555 util=0.35 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T13:23:14.119605+00:00", + "phase": "implement" + }, + { + "id": "6d636c40-02db-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T13:23:57.209563+00:00", + "phase": "implement" + }, + { + "id": "015bef5b-e275-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T13:24:10.827350+00:00", + "phase": "implement" + }, + { + "id": "5fb27226-53b3-48", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a / b8f628cf, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T13:24:19.809450+00:00", + "phase": "implement" + }, + { + "id": "6275736c-6990-41", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=226027 util=0.23 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T13:24:24.863100+00:00", + "phase": "implement" + }, + { + "id": "c76c444a-d51d-40", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T13:53:08.935559+00:00", + "phase": "implement" + }, + { + "id": "ecd4d467-cfa1-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T13:53:18.173087+00:00", + "phase": "implement" + }, + { + "id": "d664e5b4-bb6d-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=356811 util=0.36 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T13:53:23.926843+00:00", + "phase": "implement" + }, + { + "id": "573ffb40-635b-46", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T13:54:00.850538+00:00", + "phase": "implement" + }, + { + "id": "ba68c9e9-e1b8-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T13:54:07.527396+00:00", + "phase": "implement" + }, + { + "id": "f80c0f8a-575c-43", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a / b8f628cf, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T13:54:15.719598+00:00", + "phase": "implement" + }, + { + "id": "f5187a99-9804-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=228462 util=0.23 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T13:54:22.139852+00:00", + "phase": "implement" + }, + { + "id": "e9541d04-03c9-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T14:23:11.190224+00:00", + "phase": "implement" + }, + { + "id": "39f1c429-8a0f-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T14:23:19.187370+00:00", + "phase": "implement" + }, + { + "id": "2fd19059-714f-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=360087 util=0.36 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T14:23:24.549687+00:00", + "phase": "implement" + }, + { + "id": "949cdfeb-c181-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T14:24:02.743400+00:00", + "phase": "implement" + }, + { + "id": "7b3847b2-c214-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T14:24:09.377138+00:00", + "phase": "implement" + }, + { + "id": "09dfe6fb-7b02-43", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a / b8f628cf, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T14:24:15.793297+00:00", + "phase": "implement" + }, + { + "id": "b34f4ac8-18a3-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=231027 util=0.23 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T14:24:22.362855+00:00", + "phase": "implement" + }, + { + "id": "31991124-06be-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T14:53:12.918442+00:00", + "phase": "implement" + }, + { + "id": "e7790621-fab4-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T14:53:20.215914+00:00", + "phase": "implement" + }, + { + "id": "f52b3f63-b51f-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=363234 util=0.36 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T14:53:27.842473+00:00", + "phase": "implement" + }, + { + "id": "bda4a821-2651-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T14:54:05.508099+00:00", + "phase": "implement" + }, + { + "id": "abad561c-dd10-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T14:54:14.127619+00:00", + "phase": "implement" + }, + { + "id": "46f853ea-9908-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a / b8f628cf, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T14:54:22.247229+00:00", + "phase": "implement" + }, + { + "id": "0fd56fc7-e682-47", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=233461 util=0.23 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T14:54:26.780842+00:00", + "phase": "implement" + }, + { + "id": "fa49ff75-3bd7-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T15:23:16.577572+00:00", + "phase": "implement" + }, + { + "id": "4cf4a7f9-6b6c-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T15:23:21.502271+00:00", + "phase": "implement" + }, + { + "id": "b7b08204-5c79-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=366381 util=0.37 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T15:23:27.227660+00:00", + "phase": "implement" + }, + { + "id": "af7e634a-ba5f-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T15:24:07.908131+00:00", + "phase": "implement" + }, + { + "id": "06577c31-c0b3-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T15:24:16.322334+00:00", + "phase": "implement" + }, + { + "id": "cc0e34e4-6899-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a / b8f628cf, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T15:24:24.563513+00:00", + "phase": "implement" + }, + { + "id": "e377a126-d9ac-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=236088 util=0.24 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T15:24:30.468077+00:00", + "phase": "implement" + }, + { + "id": "577cbdd2-e634-44", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T15:53:18.885121+00:00", + "phase": "implement" + }, + { + "id": "3fe2c1b3-a1ad-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T15:53:26.717926+00:00", + "phase": "implement" + }, + { + "id": "bcaeed43-0bb3-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=369708 util=0.37 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T15:53:30.325060+00:00", + "phase": "implement" + }, + { + "id": "7cde76a7-9f56-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T15:54:10.159509+00:00", + "phase": "implement" + }, + { + "id": "25ddb449-a32b-47", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T15:54:18.589806+00:00", + "phase": "implement" + }, + { + "id": "bd15da5f-5f83-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a / b8f628cf, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T15:54:25.323660+00:00", + "phase": "implement" + }, + { + "id": "75fef3cd-aef6-43", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=238524 util=0.24 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T15:54:31.868491+00:00", + "phase": "implement" + }, + { + "id": "bc385a62-1c1e-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T16:23:21.169700+00:00", + "phase": "implement" + }, + { + "id": "d83b2bbd-1a2a-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T16:23:32.455853+00:00", + "phase": "implement" + }, + { + "id": "3560702c-ac29-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=372855 util=0.37 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T16:23:36.711663+00:00", + "phase": "implement" + }, + { + "id": "a75327bb-a51e-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T16:24:12.250946+00:00", + "phase": "implement" + }, + { + "id": "6e5b7172-4734-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T16:24:18.857317+00:00", + "phase": "implement" + }, + { + "id": "361b084b-381d-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a / b8f628cf, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T16:24:26.396897+00:00", + "phase": "implement" + }, + { + "id": "0705ecce-6a78-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=241026 util=0.24 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T16:24:31.727756+00:00", + "phase": "implement" + }, + { + "id": "eb4c9f21-4f58-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T16:53:23.873675+00:00", + "phase": "implement" + }, + { + "id": "4df48a2b-d1d2-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T16:53:32.785908+00:00", + "phase": "implement" + }, + { + "id": "09e4bfff-b155-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=376131 util=0.38 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T16:53:39.459584+00:00", + "phase": "implement" + }, + { + "id": "006c3cbb-03e9-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T16:54:14.845985+00:00", + "phase": "implement" + }, + { + "id": "29c31897-2ae5-44", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T16:54:21.181313+00:00", + "phase": "implement" + }, + { + "id": "b7829587-6495-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a / b8f628cf, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T16:54:38.471450+00:00", + "phase": "implement" + }, + { + "id": "9e51aaf2-a5e0-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=243588 util=0.24 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T16:54:45.648592+00:00", + "phase": "implement" + }, + { + "id": "9707ec9e-51a6-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:23:25.644192+00:00", + "phase": "implement" + }, + { + "id": "26157777-91ab-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:23:35.615318+00:00", + "phase": "implement" + }, + { + "id": "6914ec04-7e6d-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=379440 util=0.38 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:23:39.557400+00:00", + "phase": "implement" + }, + { + "id": "c2f7d430-8264-43", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:24:18.552632+00:00", + "phase": "implement" + }, + { + "id": "4d6970ff-56f4-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:24:25.271315+00:00", + "phase": "implement" + }, + { + "id": "2a2404e1-35ba-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a / b8f628cf, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:24:31.703032+00:00", + "phase": "implement" + }, + { + "id": "6b92cede-fbc3-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=246024 util=0.25 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:24:38.255855+00:00", + "phase": "implement" + }, + { + "id": "a79d12dc-c794-48", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:35:23.681885+00:00", + "phase": "implement" + }, + { + "id": "ff06da5d-182f-43", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:35:24.861139+00:00", + "phase": "implement" + }, + { + "id": "33de4de1-28f6-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:35:29.658312+00:00", + "phase": "implement" + }, + { + "id": "fa491e5a-7331-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:35:32.380191+00:00", + "phase": "implement" + }, + { + "id": "b2cdfce4-b8ed-41", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=382587 util=0.38 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:35:34.615507+00:00", + "phase": "implement" + }, + { + "id": "14a72194-f644-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:35:41.569487+00:00", + "phase": "implement" + }, + { + "id": "dd748bcb-987e-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a / b8f628cf, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:35:42.006034+00:00", + "phase": "implement" + }, + { + "id": "be094132-f3ac-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:35:48.874322+00:00", + "phase": "implement" + }, + { + "id": "45317d19-8438-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=248761 util=0.25 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:35:52.349391+00:00", + "phase": "implement" + }, + { + "id": "8987bc0b-ec03-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=385810 util=0.39 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:35:53.939896+00:00", + "phase": "implement" + }, + { + "id": "257268e8-37a1-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:36:02.633819+00:00", + "phase": "implement" + }, + { + "id": "4b61bb02-9f97-44", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:36:03.227441+00:00", + "phase": "implement" + }, + { + "id": "69d51ddf-c5be-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:36:06.817283+00:00", + "phase": "implement" + }, + { + "id": "c98db4af-9f06-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:36:11.303946+00:00", + "phase": "implement" + }, + { + "id": "067e3fac-bf26-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=388957 util=0.39 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:36:11.673802+00:00", + "phase": "implement" + }, + { + "id": "a0c25e1d-581e-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (~18h deadlock; all refs @64fa30773; no pipelines/ extraction). Re-surfaced the root-cause branch-persistence alert (dddae924, ref b4ca0796) with the operator fix recipe. Task-4-5 seam row downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:36:33.668717+00:00", + "phase": "implement" + }, + { + "id": "1857b93d-c086-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=254167 util=0.25 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:36:39.773021+00:00", + "phase": "implement" + }, + { + "id": "c9fd3b6f-ae74-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:36:49.859517+00:00", + "phase": "implement" + }, + { + "id": "682ae240-c054-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:36:58.299811+00:00", + "phase": "implement" + }, + { + "id": "ac2c63bc-2165-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / dddae924, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:37:05.474013+00:00", + "phase": "implement" + }, + { + "id": "56883515-2e15-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=256639 util=0.26 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:37:12.291281+00:00", + "phase": "implement" + }, + { + "id": "f2aed5e1-8c15-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:37:27.907210+00:00", + "phase": "implement" + }, + { + "id": "69961b6a-07c4-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:37:28.269100+00:00", + "phase": "implement" + }, + { + "id": "931565a2-3480-48", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:37:33.234195+00:00", + "phase": "implement" + }, + { + "id": "bc1aa2ef-d7ad-46", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:37:36.258723+00:00", + "phase": "implement" + }, + { + "id": "e07acb4d-544c-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / dddae924, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:37:44.182717+00:00", + "phase": "implement" + }, + { + "id": "361340cf-2b87-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=259167 util=0.26 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:37:50.503907+00:00", + "phase": "implement" + }, + { + "id": "5711583d-c2be-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:39:43.975895+00:00", + "phase": "implement" + }, + { + "id": "48581051-0fb9-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:42:18.607254+00:00", + "phase": "implement" + }, + { + "id": "2d4013a3-cc64-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:44:27.262738+00:00", + "phase": "implement" + }, + { + "id": "7ff74740-b4bc-44", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=466064 util=0.47 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T17:45:24.312962+00:00", + "phase": "implement" + }, + { + "id": "0a9d5834-f3d6-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T18:07:31.861015+00:00", + "phase": "implement" + }, + { + "id": "d3274261-4a9c-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T18:07:32.198822+00:00", + "phase": "implement" + }, + { + "id": "b19b332f-6471-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T18:07:40.634476+00:00", + "phase": "implement" + }, + { + "id": "bcadc00e-aae4-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T18:07:45.294576+00:00", + "phase": "implement" + }, + { + "id": "e47e5730-e60d-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / dddae924, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T18:07:47.997181+00:00", + "phase": "implement" + }, + { + "id": "124ff2e1-ce90-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=261557 util=0.26 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T18:07:54.827933+00:00", + "phase": "implement" + }, + { + "id": "b44f39d4-fa93-44", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T18:09:55.596208+00:00", + "phase": "implement" + }, + { + "id": "f597b35a-c3a8-44", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T18:12:17.189833+00:00", + "phase": "implement" + }, + { + "id": "c531bc71-2f6b-40", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T18:14:28.838998+00:00", + "phase": "implement" + }, + { + "id": "10bce558-483d-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T18:16:35.114570+00:00", + "phase": "implement" + }, + { + "id": "778c2844-2e8c-48", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T18:18:37.981610+00:00", + "phase": "implement" + }, + { + "id": "0367d353-3065-40", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=560036 util=0.56 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T18:19:03.080391+00:00", + "phase": "implement" + }, + { + "id": "965acc0e-5ee0-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T18:37:36.253679+00:00", + "phase": "implement" + }, + { + "id": "fba31fc1-18db-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T18:37:36.323281+00:00", + "phase": "implement" + }, + { + "id": "637860cc-8356-44", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T18:37:43.212733+00:00", + "phase": "implement" + }, + { + "id": "5dae9347-1931-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / dddae924, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T18:37:52.324001+00:00", + "phase": "implement" + }, + { + "id": "a0732625-823f-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=264076 util=0.26 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T18:37:56.883454+00:00", + "phase": "implement" + }, + { + "id": "c34d9b8a-caff-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T18:38:17.592273+00:00", + "phase": "implement" + }, + { + "id": "dfcc545d-e6bb-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T18:40:22.858703+00:00", + "phase": "implement" + }, + { + "id": "2140f133-d296-40", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=625098 util=0.63 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T18:42:32.371792+00:00", + "phase": "implement" + }, + { + "id": "dccf21f4-d09a-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T19:07:36.196039+00:00", + "phase": "implement" + }, + { + "id": "683c7c40-866c-41", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T19:07:36.318948+00:00", + "phase": "implement" + }, + { + "id": "3bccaac7-7391-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T19:07:43.340819+00:00", + "phase": "implement" + }, + { + "id": "8e24c785-2639-46", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / dddae924, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T19:07:50.724956+00:00", + "phase": "implement" + }, + { + "id": "2d3c7fcc-2c50-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=266465 util=0.27 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T19:07:55.194399+00:00", + "phase": "implement" + }, + { + "id": "e002de07-15e7-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T19:07:55.651365+00:00", + "phase": "implement" + }, + { + "id": "65e30ed4-81dd-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T19:10:03.168215+00:00", + "phase": "implement" + }, + { + "id": "ca303506-4c06-40", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T19:12:05.687132+00:00", + "phase": "implement" + }, + { + "id": "502a0a1c-5ade-41", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T19:14:12.819277+00:00", + "phase": "implement" + }, + { + "id": "40f30a9b-b1ca-41", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=704858 util=0.70 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T19:16:12.299342+00:00", + "phase": "implement" + }, + { + "id": "ca0f7bd6-58e8-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T19:37:40.197655+00:00", + "phase": "implement" + }, + { + "id": "2fff5b1b-2854-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T19:37:40.479829+00:00", + "phase": "implement" + }, + { + "id": "5afc2762-bd74-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T19:37:48.122103+00:00", + "phase": "implement" + }, + { + "id": "eae7e717-13d9-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / dddae924, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T19:37:54.393693+00:00", + "phase": "implement" + }, + { + "id": "4fe04f81-f21c-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T19:37:55.612022+00:00", + "phase": "implement" + }, + { + "id": "d8a8598d-75a6-48", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=268856 util=0.27 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T19:37:58.930030+00:00", + "phase": "implement" + }, + { + "id": "ce511560-47c5-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T19:39:59.761683+00:00", + "phase": "implement" + }, + { + "id": "844d85eb-d673-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T19:42:22.805085+00:00", + "phase": "implement" + }, + { + "id": "f9e1b182-f12e-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=772422 util=0.77 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T19:43:04.611474+00:00", + "phase": "implement" + }, + { + "id": "0a01d460-e692-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:07:41.833334+00:00", + "phase": "implement" + }, + { + "id": "610fac13-87e7-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:07:41.848060+00:00", + "phase": "implement" + }, + { + "id": "2a9c52a7-804b-43", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:07:48.475234+00:00", + "phase": "implement" + }, + { + "id": "a2f7a920-e457-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / dddae924, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:07:54.505841+00:00", + "phase": "implement" + }, + { + "id": "81ca3088-d978-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=271376 util=0.27 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:07:59.044985+00:00", + "phase": "implement" + }, + { + "id": "29f7bff2-42bc-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:08:03.826039+00:00", + "phase": "implement" + }, + { + "id": "c65e7a17-5ca6-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:10:04.810208+00:00", + "phase": "implement" + }, + { + "id": "e89b176a-3819-48", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=832192 util=0.83 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:11:22.069401+00:00", + "phase": "implement" + }, + { + "id": "772bd119-1878-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:37:44.704476+00:00", + "phase": "implement" + }, + { + "id": "81012cb7-67f6-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:37:45.158056+00:00", + "phase": "implement" + }, + { + "id": "32535f67-21ad-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:37:51.172342+00:00", + "phase": "implement" + }, + { + "id": "9dd6efd0-5dd2-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:37:51.357597+00:00", + "phase": "implement" + }, + { + "id": "34c4f1c0-b0a1-41", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge \u2014 fix recipe in standing OVERSEER_ALERTs b4ca0796 / dddae924, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:37:58.481011+00:00", + "phase": "implement" + }, + { + "id": "d30bf925-50fb-46", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=273834 util=0.27 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:38:04.065797+00:00", + "phase": "implement" + }, + { + "id": "93e189b4-27d4-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:39:54.500161+00:00", + "phase": "implement" + }, + { + "id": "561ef392-a7a5-48", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:42:15.230885+00:00", + "phase": "implement" + }, + { + "id": "ee4a3c04-469a-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:44:28.329302+00:00", + "phase": "implement" + }, + { + "id": "c4d552e8-aff9-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:47:16.119451+00:00", + "phase": "implement" + }, + { + "id": "1fdfb015-ecbb-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:50:04.347202+00:00", + "phase": "implement" + }, + { + "id": "c0812fb1-95c4-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:52:15.762734+00:00", + "phase": "implement" + }, + { + "id": "bbb5fc11-41fe-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:54:35.874152+00:00", + "phase": "implement" + }, + { + "id": "f9f1815f-6742-48", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:56:40.073410+00:00", + "phase": "implement" + }, + { + "id": "79425114-940d-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T20:58:44.430140+00:00", + "phase": "implement" + }, + { + "id": "e88cae6f-dfea-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:00:49.101537+00:00", + "phase": "implement" + }, + { + "id": "71de496e-56d1-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:02:52.673626+00:00", + "phase": "implement" + }, + { + "id": "a66eda14-dcbf-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:04:56.903433+00:00", + "phase": "implement" + }, + { + "id": "1cf95067-3a05-47", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:07:06.794100+00:00", + "phase": "implement" + }, + { + "id": "2469ff13-dde4-46", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:07:47.812866+00:00", + "phase": "implement" + }, + { + "id": "9e7cb2c8-a546-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:07:54.744872+00:00", + "phase": "implement" + }, + { + "id": "901db6f1-aa2f-48", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: PROGRESS \u2014 branch-persistence wedge RESOLVED. Coder commits now persist across spawns; coder branch tip advanced 64fa30773 \u2192 c7f676c27 with 12 commits (pure-move baseline + __file__-path fix + 10 extraction commits creating pipelines/ with __init__.py + 11 submodules: _criteria/_drafts/_reviews/_context_pr/_brc_history/_statefiles/_worktree_sync/_alerts/_overseer/_slice_state/_drivers). BUT extraction is still IN PROGRESS and NOT integrated: work is on the coder branch only (my HEAD + origin/slice-4 still 64fa30773), and the allowlist was re-keyed to pipelines/__init__.py grandfathering the still-oversized barrel (dropped in the FINAL commit once under cap). My task-4-5 seam row documents the FINAL layout \u2014 authoring now (barrel still over cap, more commits expected) would be inaccurate/NACKable. Still blocked on coder to finish the extraction + drop the LAST allowlist entry; then I author the seam row and propose. Durable memory updated. Alive. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:08:56.947621+00:00", + "phase": "implement" + }, + { + "id": "3ad35c65-7559-48", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=285794 util=0.29 cache_hit=0.98 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:09:07.546318+00:00", + "phase": "implement" + }, + { + "id": "fe111071-4926-41", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:09:10.820531+00:00", + "phase": "implement" + }, + { + "id": "0c5ac636-cc0c-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:11:26.781166+00:00", + "phase": "implement" + }, + { + "id": "1dc4cd09-7385-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:13:36.178274+00:00", + "phase": "implement" + }, + { + "id": "bac4c874-b53e-44", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:15:49.406407+00:00", + "phase": "implement" + }, + { + "id": "1c2621b6-3fbd-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=296514 util=0.30 cache_hit=0.14 decision=at_or_above_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:16:30.742462+00:00", + "phase": "implement" + }, + { + "id": "a1595d9d-2537-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:16:40.187884+00:00", + "phase": "implement" + }, + { + "id": "a73f43bc-f97f-41", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:16:49.103526+00:00", + "phase": "implement" + }, + { + "id": "bef64360-c07d-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:19:07.587135+00:00", + "phase": "implement" + }, + { + "id": "453c8674-69c3-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:21:10.781216+00:00", + "phase": "implement" + }, + { + "id": "3249b522-4929-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:23:11.154838+00:00", + "phase": "implement" + }, + { + "id": "e81de50c-41c1-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:25:14.505448+00:00", + "phase": "implement" + }, + { + "id": "2178e718-e644-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:27:28.602342+00:00", + "phase": "implement" + }, + { + "id": "1127c1d4-0b70-48", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:29:51.731505+00:00", + "phase": "implement" + }, + { + "id": "0c43c41a-513e-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:31:54.917574+00:00", + "phase": "implement" + }, + { + "id": "ac6d6e0a-d83e-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=421714 util=0.42 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:33:57.250673+00:00", + "phase": "implement" + }, + { + "id": "f4f46eab-796e-44", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:37:51.800525+00:00", + "phase": "implement" + }, + { + "id": "fa8cf3ac-dff7-46", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:38:01.624138+00:00", + "phase": "implement" + }, + { + "id": "3af564d6-ec86-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: coder extraction ACTIVELY PROGRESSING (wedge stays resolved). Coder tip 0e7510ae4; barrel pipelines/__init__.py now 14,167 lines (from 30,520) \u2014 still well over the 1,500-line cap, allowlist entry still grandfathering it. More submodules landed (_decisions/_pod_liveness/_ledger/_populate/_prompt_review/_prompt_agent/_prompt_phase/_prompt_reviewer). NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed a complete slice. My task-4-5 seam row documents the FINAL layout \u2014 still premature. Blocked on coder to finish extraction (barrel under cap + drop the LAST allowlist entry); then I author the seam row and propose. Alive. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:38:18.926300+00:00", + "phase": "implement" + }, + { + "id": "1b0d5edd-7d64-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=290571 util=0.29 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:38:27.225686+00:00", + "phase": "implement" + }, + { + "id": "296b1a76-1e8c-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:46:44.533056+00:00", + "phase": "implement" + }, + { + "id": "6c8b3682-4552-47", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:46:54.139352+00:00", + "phase": "implement" + }, + { + "id": "6ef44646-50c4-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:49:08.476349+00:00", + "phase": "implement" + }, + { + "id": "f6ac7414-cd91-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:51:33.670942+00:00", + "phase": "implement" + }, + { + "id": "a9aa87c3-318e-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:53:34.740910+00:00", + "phase": "implement" + }, + { + "id": "6b088822-9fb3-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:56:16.359082+00:00", + "phase": "implement" + }, + { + "id": "73d6a388-7ce5-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T21:58:19.999091+00:00", + "phase": "implement" + }, + { + "id": "b7e779ca-02a1-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:00:32.334015+00:00", + "phase": "implement" + }, + { + "id": "9147a9c6-958d-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:02:33.510844+00:00", + "phase": "implement" + }, + { + "id": "b7d3ba57-d788-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=540407 util=0.54 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:02:59.041350+00:00", + "phase": "implement" + }, + { + "id": "ff456deb-c276-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:07:54.780108+00:00", + "phase": "implement" + }, + { + "id": "5e8ce1aa-6c07-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:08:03.207090+00:00", + "phase": "implement" + }, + { + "id": "78a378c0-3587-48", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: coder extraction still progressing well. Tip 99ca6a72b; barrel pipelines/__init__.py down to 11,853 lines (from 30,520), 29 submodules landed, coder notes \"only route-bodies + 3 giants remain.\" Still over the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout \u2014 still premature. Blocked on coder to finish extraction (barrel under cap + drop the LAST allowlist entry); then I author the seam row and propose. Alive. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:08:11.690004+00:00", + "phase": "implement" + }, + { + "id": "47233e30-7850-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=293841 util=0.29 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:08:18.365863+00:00", + "phase": "implement" + }, + { + "id": "b7d3b666-fb19-47", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:16:47.889280+00:00", + "phase": "implement" + }, + { + "id": "a9bf5815-7ef4-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:16:58.398130+00:00", + "phase": "implement" + }, + { + "id": "29a39786-2bea-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:19:14.393974+00:00", + "phase": "implement" + }, + { + "id": "e651d39b-8236-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:21:22.530877+00:00", + "phase": "implement" + }, + { + "id": "a0ddde9d-1161-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:23:23.839516+00:00", + "phase": "implement" + }, + { + "id": "f2d34dce-f7d3-40", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:26:41.948062+00:00", + "phase": "implement" + }, + { + "id": "313cbbc5-360f-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:31:40.486676+00:00", + "phase": "implement" + }, + { + "id": "80bd9f3e-271b-47", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=604444 util=0.60 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:33:40.142911+00:00", + "phase": "implement" + }, + { + "id": "9f907422-27cc-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:37:56.780539+00:00", + "phase": "implement" + }, + { + "id": "fd86d4db-2516-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:38:04.955681+00:00", + "phase": "implement" + }, + { + "id": "947bb8a1-c6e8-44", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: coder extraction nearing completion. Tip 6d0e3f942; barrel pipelines/__init__.py down to 8,364 lines (from 30,520); all route bodies extracted (decision-8: @route decorators stay in barrel); coder notes \"only 3 giants remain.\" Still over the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout \u2014 still premature until the barrel is under cap. Blocked on coder to finish + drop the LAST allowlist entry; then I author the seam row and propose. Alive. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:38:19.496651+00:00", + "phase": "implement" + }, + { + "id": "fecbbabe-1060-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=296834 util=0.30 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:38:25.771526+00:00", + "phase": "implement" + }, + { + "id": "2316b5b9-3037-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:46:50.795342+00:00", + "phase": "implement" + }, + { + "id": "37e29a85-39c0-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:47:02.380454+00:00", + "phase": "implement" + }, + { + "id": "55402420-557e-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:49:03.173632+00:00", + "phase": "implement" + }, + { + "id": "fdee66de-0c2b-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:51:15.687422+00:00", + "phase": "implement" + }, + { + "id": "3f467cd1-a182-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:54:10.055255+00:00", + "phase": "implement" + }, + { + "id": "464325ed-2bde-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=693145 util=0.69 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T22:56:14.376833+00:00", + "phase": "implement" + }, + { + "id": "dde476c9-ce73-47", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T23:08:00.454923+00:00", + "phase": "implement" + }, + { + "id": "8849c498-43be-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T23:08:08.952657+00:00", + "phase": "implement" + }, + { + "id": "49b5ce8f-ebfb-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: coder extraction near-final. Tip dbee33571 (last commit a BRC-memory note: giant-split recipe ready for mechanical execution next invocation). Barrel pipelines/__init__.py still 8,364 lines \u2014 3 giant helpers remain to split under the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout \u2014 premature until the barrel is under cap. Blocked on coder to finish the giant-split + drop the LAST allowlist entry; then I author the seam row and propose. Alive. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T23:08:17.585285+00:00", + "phase": "implement" + }, + { + "id": "1e1c95bc-ffc0-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=299981 util=0.30 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T23:08:23.447514+00:00", + "phase": "implement" + }, + { + "id": "6f960fe3-615a-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T23:16:53.810028+00:00", + "phase": "implement" + }, + { + "id": "81d1aa66-2bdc-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T23:17:07.795098+00:00", + "phase": "implement" + }, + { + "id": "1888dad3-f563-40", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T23:19:09.048202+00:00", + "phase": "implement" + }, + { + "id": "abfc770b-2565-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T23:21:38.718102+00:00", + "phase": "implement" + }, + { + "id": "e5832998-e530-40", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T23:23:48.495172+00:00", + "phase": "implement" + }, + { + "id": "1c57ac4c-d117-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T23:26:01.959754+00:00", + "phase": "implement" + }, + { + "id": "1a10c8fc-397b-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T23:28:17.207957+00:00", + "phase": "implement" + }, + { + "id": "2bf85b75-1bef-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=788405 util=0.79 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T23:28:30.551140+00:00", + "phase": "implement" + }, + { + "id": "875f38c6-f40b-48", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T23:38:01.897181+00:00", + "phase": "implement" + }, + { + "id": "144bd546-b8a9-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T23:38:10.777648+00:00", + "phase": "implement" + }, + { + "id": "154cc3d9-1bfc-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: coder giant-split progressing. Tip 0d2a78a48; barrel pipelines/__init__.py down to 6,694 lines (from 30,520); giant #1 split done, giants #2/#3 remain. Still over the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout \u2014 premature until the barrel is under cap. Blocked on coder to finish giants #2/#3 + drop the LAST allowlist entry; then I author the seam row and propose. Alive. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T23:38:18.256321+00:00", + "phase": "implement" + }, + { + "id": "ea1c4e5d-3342-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=302872 util=0.30 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T23:38:23.342163+00:00", + "phase": "implement" + }, + { + "id": "01ad47a9-84ff-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T23:46:55.665754+00:00", + "phase": "implement" + }, + { + "id": "206ff74e-a045-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T23:47:10.055499+00:00", + "phase": "implement" + }, + { + "id": "bc0a5412-dd16-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T23:49:48.050195+00:00", + "phase": "implement" + }, + { + "id": "fe06a59e-a288-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T23:51:56.315364+00:00", + "phase": "implement" + }, + { + "id": "1d936646-1728-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=859804 util=0.86 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-05T23:52:33.012565+00:00", + "phase": "implement" + }, + { + "id": "932b1b10-dee2-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:08:04.528645+00:00", + "phase": "implement" + }, + { + "id": "e8d04227-8974-48", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:08:16.112982+00:00", + "phase": "implement" + }, + { + "id": "3caa8ebc-3778-47", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: coder working giant #2 split. Tip 331ad4e7f (last commit a BRC-memory planning note refining the giant #2 recipe); barrel pipelines/__init__.py still 6,694 lines. Giants #2/#3 remain to bring the barrel under the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout \u2014 premature until under cap. Blocked on coder to finish + drop the LAST allowlist entry; then I author the seam row and propose. Alive. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:08:33.267138+00:00", + "phase": "implement" + }, + { + "id": "c865f5a2-ab0b-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=305779 util=0.31 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:08:40.291892+00:00", + "phase": "implement" + }, + { + "id": "5d74061b-f34a-47", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:16:57.547883+00:00", + "phase": "implement" + }, + { + "id": "635513f8-54d5-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:17:07.753876+00:00", + "phase": "implement" + }, + { + "id": "c89a7c6e-ca33-41", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:19:18.113403+00:00", + "phase": "implement" + }, + { + "id": "fd35d1a5-235d-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:22:13.780590+00:00", + "phase": "implement" + }, + { + "id": "728dfdc9-fdfb-44", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:24:38.639450+00:00", + "phase": "implement" + }, + { + "id": "1472e2cb-36e9-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:26:55.477225+00:00", + "phase": "implement" + }, + { + "id": "3843e939-b898-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:28:58.067434+00:00", + "phase": "implement" + }, + { + "id": "c8d8f397-2ba2-40", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:31:00.510125+00:00", + "phase": "implement" + }, + { + "id": "9f93e484-2829-40", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:33:28.660812+00:00", + "phase": "implement" + }, + { + "id": "ba1aa72d-3716-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:35:58.427174+00:00", + "phase": "implement" + }, + { + "id": "f2a0d188-0ac3-43", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:38:05.922781+00:00", + "phase": "implement" + }, + { + "id": "af4129e1-8525-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:38:14.115755+00:00", + "phase": "implement" + }, + { + "id": "aa1f5429-1833-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change since last event (coder tip 331ad4e7f; barrel pipelines/__init__.py still 6,694 lines; coder mid-work on giant #2 split). Still over the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout \u2014 premature until under cap. Blocked on coder to finish giants #2/#3 + drop the LAST allowlist entry; then I author the seam row and propose. Alive. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:38:23.197596+00:00", + "phase": "implement" + }, + { + "id": "f67f3eaa-3507-47", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=308749 util=0.31 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:38:28.418255+00:00", + "phase": "implement" + }, + { + "id": "2d749f68-707c-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:38:33.777720+00:00", + "phase": "implement" + }, + { + "id": "d0ab8998-078c-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:40:36.516507+00:00", + "phase": "implement" + }, + { + "id": "2bba8484-7a22-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=241543 util=0.24 cache_hit=0.99 decision=at_or_above_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:42:42.212872+00:00", + "phase": "implement" + }, + { + "id": "4d0d2710-5c3f-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:47:00.594728+00:00", + "phase": "implement" + }, + { + "id": "70fb78f8-442d-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:47:14.977938+00:00", + "phase": "implement" + }, + { + "id": "89db5438-b559-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:49:16.532897+00:00", + "phase": "implement" + }, + { + "id": "ae07ce5a-f1df-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:51:21.655547+00:00", + "phase": "implement" + }, + { + "id": "9261cd0e-ba54-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:54:05.182394+00:00", + "phase": "implement" + }, + { + "id": "533d016d-dfdf-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:56:38.410789+00:00", + "phase": "implement" + }, + { + "id": "f2014278-7b66-40", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T00:58:48.663387+00:00", + "phase": "implement" + }, + { + "id": "7e48a573-c114-48", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=344432 util=0.34 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T01:00:50.293586+00:00", + "phase": "implement" + }, + { + "id": "1682ffaf-8891-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T01:08:10.856460+00:00", + "phase": "implement" + }, + { + "id": "63c7d46e-3300-41", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T01:08:19.479721+00:00", + "phase": "implement" + }, + { + "id": "cc69b1dc-66fa-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: coder on giant #3 (_run_pipeline split, task-4-3). Tip 6641929ab; barrel pipelines/__init__.py down to 4,529 lines (from 30,520). Still over the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout \u2014 premature until under cap. Blocked on coder to finish the _run_pipeline split + drop the LAST allowlist entry; then I author the seam row and propose. Alive. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T01:08:27.094503+00:00", + "phase": "implement" + }, + { + "id": "6e3553bf-1170-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=311634 util=0.31 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T01:08:32.575572+00:00", + "phase": "implement" + }, + { + "id": "f5979677-456f-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T01:17:04.730683+00:00", + "phase": "implement" + }, + { + "id": "724310bb-d4ee-41", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T01:17:17.899581+00:00", + "phase": "implement" + }, + { + "id": "b147374a-3599-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T01:19:19.645800+00:00", + "phase": "implement" + }, + { + "id": "84f94859-1efc-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T01:21:26.842261+00:00", + "phase": "implement" + }, + { + "id": "46948ac7-6b71-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T01:23:47.542245+00:00", + "phase": "implement" + }, + { + "id": "98ea1ff9-246b-41", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=422843 util=0.42 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T01:25:21.760789+00:00", + "phase": "implement" + }, + { + "id": "a4c9c5f9-6ec9-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T01:38:12.754549+00:00", + "phase": "implement" + }, + { + "id": "d43a836b-35c4-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T01:38:22.388331+00:00", + "phase": "implement" + }, + { + "id": "03dd2c1b-dd5b-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: coder still splitting giant #3 (_run_pipeline, task-4-3). Tip adeb2f2ef; barrel pipelines/__init__.py down to 4,299 lines; _run_pipeline now 2,853L. Still over the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout \u2014 premature until under cap. Blocked on coder to finish the _run_pipeline split + drop the LAST allowlist entry; then I author the seam row and propose. Alive. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T01:38:31.228974+00:00", + "phase": "implement" + }, + { + "id": "6a119770-6e08-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=314686 util=0.31 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T01:38:38.692756+00:00", + "phase": "implement" + }, + { + "id": "0dbdaa7c-6ddf-44", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T01:47:05.949997+00:00", + "phase": "implement" + }, + { + "id": "5bc9695c-81e6-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T01:47:17.478312+00:00", + "phase": "implement" + }, + { + "id": "17a7a21d-794f-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T01:49:22.320740+00:00", + "phase": "implement" + }, + { + "id": "2d5f95ff-a035-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=482865 util=0.48 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T01:51:20.464233+00:00", + "phase": "implement" + }, + { + "id": "48538e00-3008-43", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T02:08:13.895359+00:00", + "phase": "implement" + }, + { + "id": "4015c5a2-578d-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T02:08:22.638115+00:00", + "phase": "implement" + }, + { + "id": "dee55669-fefe-48", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: coder still splitting giant #3 (_run_pipeline). Tip 596472736; barrel pipelines/__init__.py 4,244 lines; _run_pipeline now 2,797L. Still over the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout \u2014 premature until under cap. Blocked on coder to finish the _run_pipeline split + drop the LAST allowlist entry; then I author the seam row and propose. Alive. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T02:08:33.971885+00:00", + "phase": "implement" + }, + { + "id": "89d097b5-71a6-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=317658 util=0.32 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T02:08:40.560980+00:00", + "phase": "implement" + }, + { + "id": "c5945b92-ad13-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T02:17:08.488316+00:00", + "phase": "implement" + }, + { + "id": "3ae355e8-7a5f-44", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T02:17:21.167271+00:00", + "phase": "implement" + }, + { + "id": "0b3cf6dd-ada5-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T02:19:23.226156+00:00", + "phase": "implement" + }, + { + "id": "c6512c28-d167-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=535928 util=0.54 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T02:20:11.756349+00:00", + "phase": "implement" + }, + { + "id": "e0b8731b-7b28-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T02:38:16.972418+00:00", + "phase": "implement" + }, + { + "id": "335dceef-7b91-44", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T02:38:25.628216+00:00", + "phase": "implement" + }, + { + "id": "a346a10a-d63f-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: coder still on giant #3 (_run_pipeline). Tip 233533d86; barrel pipelines/__init__.py 4,159 lines; _run_pipeline 2,711L (all 5 setup blocks extracted; while-loop split next). Still over the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout \u2014 premature until under cap. Blocked on coder to finish the _run_pipeline split + drop the LAST allowlist entry; then I author the seam row and propose. Alive. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T02:38:33.812965+00:00", + "phase": "implement" + }, + { + "id": "628d0e63-9cca-47", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=320602 util=0.32 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T02:38:40.543827+00:00", + "phase": "implement" + }, + { + "id": "9481fe4e-d621-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T02:47:09.807471+00:00", + "phase": "implement" + }, + { + "id": "4a5968a4-5090-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T02:47:21.268096+00:00", + "phase": "implement" + }, + { + "id": "37bb51df-81b3-41", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T02:50:04.717696+00:00", + "phase": "implement" + }, + { + "id": "a23f6523-78cf-44", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=596600 util=0.60 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T02:51:53.540691+00:00", + "phase": "implement" + }, + { + "id": "d91f62c4-7163-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T03:08:21.842253+00:00", + "phase": "implement" + }, + { + "id": "8cbf2520-3aa3-46", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T03:08:30.186547+00:00", + "phase": "implement" + }, + { + "id": "34cefba5-caeb-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: coder near end of giant #3 (_run_pipeline). Tip d3c0c182c; barrel pipelines/__init__.py 4,136 lines; _run_pipeline 2,684L; coder notes only the while-loop split remains. Still over the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout \u2014 premature until under cap. Blocked on coder to finish + drop the LAST allowlist entry; then I author the seam row and propose. Alive. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T03:08:39.358412+00:00", + "phase": "implement" + }, + { + "id": "a0903ff7-c26c-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=323647 util=0.32 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T03:08:45.412857+00:00", + "phase": "implement" + }, + { + "id": "9ac856c5-52b0-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T03:17:10.929762+00:00", + "phase": "implement" + }, + { + "id": "eeaf1063-5be1-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T03:17:24.267905+00:00", + "phase": "implement" + }, + { + "id": "e563e4e3-c355-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T03:19:28.902193+00:00", + "phase": "implement" + }, + { + "id": "76403aed-3b0a-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T03:21:41.598419+00:00", + "phase": "implement" + }, + { + "id": "7587fa8c-a4eb-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T03:23:42.560186+00:00", + "phase": "implement" + }, + { + "id": "15f1b7fb-89da-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=669065 util=0.67 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T03:25:20.598588+00:00", + "phase": "implement" + }, + { + "id": "b7bec6b1-22ca-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T03:38:25.747441+00:00", + "phase": "implement" + }, + { + "id": "621de6bc-bba8-48", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T03:38:36.930049+00:00", + "phase": "implement" + }, + { + "id": "3466b765-b118-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: coder closing on giant #3 (_run_pipeline). Tip c62d83064; barrel pipelines/__init__.py 3,445 lines; _run_pipeline 1,988L (down from 2,684; 708L HITL-gate block extracted). Still over the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout \u2014 premature until under cap. Blocked on coder to finish + drop the LAST allowlist entry; then I author the seam row and propose. Alive. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T03:38:45.848734+00:00", + "phase": "implement" + }, + { + "id": "d43b7a1e-acf7-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=326574 util=0.33 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T03:38:51.570130+00:00", + "phase": "implement" + }, + { + "id": "1b6c8e72-1729-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T03:47:14.839120+00:00", + "phase": "implement" + }, + { + "id": "4f08080f-e591-40", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T03:47:35.130257+00:00", + "phase": "implement" + }, + { + "id": "c32e460d-e7a9-40", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T03:49:55.639016+00:00", + "phase": "implement" + }, + { + "id": "854f47dc-e09e-48", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T03:52:23.367539+00:00", + "phase": "implement" + }, + { + "id": "e4f8b667-55a4-40", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T03:54:47.452322+00:00", + "phase": "implement" + }, + { + "id": "a10b14a7-2fb3-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=747536 util=0.75 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T03:56:59.706344+00:00", + "phase": "implement" + }, + { + "id": "378dc786-807c-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T04:08:30.221023+00:00", + "phase": "implement" + }, + { + "id": "445c77c0-21bf-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T04:08:41.374316+00:00", + "phase": "implement" + }, + { + "id": "4117c4cd-31bd-47", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: coder nearly done with giant #3. Tip a5506d41a; barrel pipelines/__init__.py 3,172 lines; _run_pipeline 1,712L (~212L over cap); coder notes ONE block left then the final move + terminal allowlist drop. Still over the 1,500-line cap; allowlist entry still present; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout \u2014 premature until under cap. Blocked on coder to finish + drop the LAST allowlist entry (files: \u2192 EMPTY); then I author the seam row and propose. Alive. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T04:08:50.632456+00:00", + "phase": "implement" + }, + { + "id": "91ebcdee-6f29-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=329707 util=0.33 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T04:08:58.190760+00:00", + "phase": "implement" + }, + { + "id": "11c9f36d-e5d5-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T04:17:14.873087+00:00", + "phase": "implement" + }, + { + "id": "14663c18-a1e5-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T04:17:28.845549+00:00", + "phase": "implement" + }, + { + "id": "251eb429-1a8b-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T04:20:09.573409+00:00", + "phase": "implement" + }, + { + "id": "e63d7c4f-28bf-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T04:22:17.731622+00:00", + "phase": "implement" + }, + { + "id": "6e28314f-7a63-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T04:24:20.778973+00:00", + "phase": "implement" + }, + { + "id": "4441e30a-3bd4-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=817250 util=0.82 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T04:24:59.292483+00:00", + "phase": "implement" + }, + { + "id": "6edd4521-146a-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T04:38:29.792054+00:00", + "phase": "implement" + }, + { + "id": "dce23de2-9fbf-47", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T04:38:39.244608+00:00", + "phase": "implement" + }, + { + "id": "deab6f64-b4e3-47", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: milestone \u2014 _run_pipeline now UNDER cap (1,467L). Tip e953a9525; but the barrel pipelines/__init__.py is still 2,932 lines (over the 1,500-line cap); coder notes only the final move+terminal+propose bite remains. Allowlist entry still present; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout \u2014 premature until the barrel is under cap. Blocked on coder to finish the final move + drop the LAST allowlist entry (files: \u2192 EMPTY); then I author the seam row and propose. Alive. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T04:38:54.866173+00:00", + "phase": "implement" + }, + { + "id": "32203083-ad46-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=333073 util=0.33 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T04:39:01.158686+00:00", + "phase": "implement" + }, + { + "id": "800384c6-50bf-43", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T04:47:18.920313+00:00", + "phase": "implement" + }, + { + "id": "b28d20d8-12d7-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T04:47:26.714555+00:00", + "phase": "implement" + }, + { + "id": "73584fae-90a6-47", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T04:49:29.418321+00:00", + "phase": "implement" + }, + { + "id": "b1776cf2-af46-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T04:51:42.045746+00:00", + "phase": "implement" + }, + { + "id": "256eb04e-0ebd-44", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T04:53:43.000619+00:00", + "phase": "implement" + }, + { + "id": "ed5148f7-980f-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T04:56:17.778896+00:00", + "phase": "implement" + }, + { + "id": "c77b410a-63cf-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T04:58:24.693969+00:00", + "phase": "implement" + }, + { + "id": "df661424-62b9-47", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T05:00:34.994976+00:00", + "phase": "implement" + }, + { + "id": "fcaf3a9b-32c0-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T05:02:50.716504+00:00", + "phase": "implement" + }, + { + "id": "bd58bde3-8178-41", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T05:08:35.194329+00:00", + "phase": "implement" + }, + { + "id": "53fb6dcf-7ca0-41", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T05:08:43.688138+00:00", + "phase": "implement" + }, + { + "id": "7a909b66-1b5a-41", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change since last event (coder tip e953a9525; barrel pipelines/__init__.py still 2,932 lines, over the 1,500-line cap; coder on the final move+terminal+propose bite). _run_pipeline is under cap (1,467L) but the barrel isn't yet. Allowlist entry still present; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout \u2014 premature until the barrel is under cap. Blocked on coder to finish + drop the LAST allowlist entry (files: \u2192 EMPTY); then I author the seam row and propose. Alive. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T05:08:54.165052+00:00", + "phase": "implement" + }, + { + "id": "1a0b450e-cc56-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=336037 util=0.34 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T05:08:59.735373+00:00", + "phase": "implement" + }, + { + "id": "5df00ad0-9eca-48", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T05:38:36.165217+00:00", + "phase": "implement" + }, + { + "id": "4b780632-180a-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T05:38:47.552077+00:00", + "phase": "implement" + }, + { + "id": "f61b241d-809a-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (coder tip e953a9525; barrel pipelines/__init__.py still 2,932 lines, over cap; coder on final move+terminal+propose bite; _run_pipeline under cap at 1,467L). Allowlist entry still present; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout \u2014 premature until the barrel is under cap. Blocked on coder to finish + drop the LAST allowlist entry (files: \u2192 EMPTY); then I author the seam row and propose. Alive. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T05:38:54.132658+00:00", + "phase": "implement" + }, + { + "id": "bef4fc30-ac16-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=338963 util=0.34 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T05:38:59.917581+00:00", + "phase": "implement" + }, + { + "id": "32358fa5-96d1-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T06:08:38.803518+00:00", + "phase": "implement" + }, + { + "id": "67b991c7-6574-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T06:08:49.523198+00:00", + "phase": "implement" + }, + { + "id": "7ff127e8-b41d-48", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: no change (coder tip e953a9525; barrel pipelines/__init__.py still 2,932 lines, over cap; coder on final move+terminal+propose bite; _run_pipeline under cap 1,467L). Allowlist entry still present; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout \u2014 premature until the barrel is under cap. Blocked on coder to finish + drop the LAST allowlist entry (files: \u2192 EMPTY); then I author the seam row and propose. Alive. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T06:08:57.380047+00:00", + "phase": "implement" + }, + { + "id": "60d74919-3b9a-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=341806 util=0.34 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T06:09:02.861398+00:00", + "phase": "implement" + }, + { + "id": "c745486a-6fea-44", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T06:38:40.862616+00:00", + "phase": "implement" + }, + { + "id": "2dd54a7e-4105-43", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T06:38:50.438704+00:00", + "phase": "implement" + }, + { + "id": "bd7d3399-61ed-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: extraction still progressing but INTERMITTENT reset-churn observed. Coder tip e953a9525; reflog shows a reset\u219264fa30773 then fast-forward back to e953a9525 that recovered most commits but ORPHANED 923a2b351 (\"Split _run_pipeline while-loop \u2192 _run_phase_blocks.py; giant under cap\", task-4-3/non-negotiable #7) \u2014 it's a sibling of the tip (both children of a5506d41a), reachable in the object store but off-branch. NOTE: even at 923a2b351 the barrel pipelines/__init__.py is still 2,932 lines (over the 1,500-line cap) and the allowlist entry is still present, so it is NOT the terminal commit \u2014 the final barrel-shrink + allowlist-drop still remain regardless. Fast-forward recovery is mostly working (barrel 30,520\u21922,932 has persisted), so this is a milder/recovering recurrence of the earlier branch-persistence wedge (b4ca0796), not a full stall \u2014 watching, not re-alerting yet. Still NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row is premature until the barrel is under cap. Blocked on coder to finish; then I author the seam row and propose. Alive. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T06:39:57.913856+00:00", + "phase": "implement" + }, + { + "id": "de6baaec-ea9b-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=353340 util=0.35 cache_hit=0.98 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T06:40:08.977177+00:00", + "phase": "implement" + }, + { + "id": "d11bd035-7565-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:08:44.164597+00:00", + "phase": "implement" + }, + { + "id": "d1ab8100-4a29-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:08:57.252581+00:00", + "phase": "implement" + }, + { + "id": "76f0797f-7d75-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: RECURRENCE escalated (OVERSEER_ALERT 8642ccd3, ref b4ca0796). Coder tip stuck at e953a9525 / barrel 2,932L (over cap) for ~2.5h; reflog shows per-spawn reset-to-origin bouncing the coder to base 64fa30773 (origin/slice-4 never advanced past base \u2192 coder re-does early work each spawn instead of committing the final barrel-shrink + allowlist-drop). Durable fix = push coder progress to origin/slice-4 (or fast-forward origin+branch to e953a9525/923a2b351). No agent-side move can clear it; my task-4-5 seam row is downstream. Alive; will author the seam row + propose once the extraction completes and integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:10:18.114676+00:00", + "phase": "implement" + }, + { + "id": "b89cdafa-3249-44", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=368612 util=0.37 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:10:28.940061+00:00", + "phase": "implement" + }, + { + "id": "65d2ca73-3fd6-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:15:12.021401+00:00", + "phase": "implement" + }, + { + "id": "29c20049-d501-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:15:13.296104+00:00", + "phase": "implement" + }, + { + "id": "6a14435c-0b17-40", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:15:17.196240+00:00", + "phase": "implement" + }, + { + "id": "d7ba5591-ca9d-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:15:25.037696+00:00", + "phase": "implement" + }, + { + "id": "a686f628-5331-45", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: still stalled at coder tip e953a9525 / barrel 2,932L (over cap); no advance since last event. Root cause = per-spawn reset-to-origin bouncing the coder to base (origin/slice-4 still 64fa30773); escalated in standing OVERSEER_ALERT 8642ccd3 (ref b4ca0796) with the push-to-origin fix. Allowlist entry still present; NOT integrated; coder has not proposed. My task-4-5 seam row is downstream. Alive; will author the seam row + propose once the extraction completes and integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:15:35.737620+00:00", + "phase": "implement" + }, + { + "id": "61c97220-5c0e-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=371754 util=0.37 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:15:42.137298+00:00", + "phase": "implement" + }, + { + "id": "42e802da-debb-49", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:15:52.833233+00:00", + "phase": "implement" + }, + { + "id": "c5f0cb68-a3f3-46", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:16:02.466066+00:00", + "phase": "implement" + }, + { + "id": "0c97ed41-cbb2-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: still stalled at coder tip e953a9525 / barrel 2,932L (over cap); no advance. Root cause = per-spawn reset-to-origin bouncing the coder to base (origin/slice-4 still 64fa30773); escalated in standing OVERSEER_ALERT 8642ccd3 (ref b4ca0796), awaiting operator push-to-origin fix. Allowlist entry still present; NOT integrated; coder has not proposed. My task-4-5 seam row is downstream. Alive; will author the seam row + propose once the extraction completes and integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:16:12.475219+00:00", + "phase": "implement" + }, + { + "id": "692fafa3-a5b1-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=374699 util=0.37 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:16:18.341741+00:00", + "phase": "implement" + }, + { + "id": "34518009-12a5-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:16:24.900952+00:00", + "phase": "implement" + }, + { + "id": "34d91d44-4fd3-47", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:16:35.936215+00:00", + "phase": "implement" + }, + { + "id": "7bc79cde-1f88-43", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WAITING_ON_ROLE", + "body": "slice-4 documenter: still stalled at coder tip e953a9525 / barrel 2,932L (over cap); no advance. Root cause = per-spawn reset-to-origin (origin/slice-4 still 64fa30773); escalated in standing OVERSEER_ALERT 8642ccd3 (ref b4ca0796), awaiting operator push-to-origin fix. Allowlist entry still present; NOT integrated; coder has not proposed. My task-4-5 seam row is downstream. Alive; will author the seam row + propose once the extraction completes and integrates. Exiting.", + "metadata": { + "state": "WAITING_ON_ROLE", + "waiting_on": "coder", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:16:42.698585+00:00", + "phase": "implement" + }, + { + "id": "7cd8aa46-6b3a-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=377478 util=0.38 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:16:49.704364+00:00", + "phase": "implement" + }, + { + "id": "49e7eebb-2853-49", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:17:28.912063+00:00", + "phase": "implement" + }, + { + "id": "66dc83ec-d40b-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:19:34.636218+00:00", + "phase": "implement" + }, + { + "id": "b930386c-9369-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:21:36.804164+00:00", + "phase": "implement" + }, + { + "id": "8be2ea35-43ca-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:23:54.861387+00:00", + "phase": "implement" + }, + { + "id": "bf091234-16bb-46", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:26:11.685935+00:00", + "phase": "implement" + }, + { + "id": "9d099c8f-4d91-44", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:28:34.534608+00:00", + "phase": "implement" + }, + { + "id": "ec966853-636b-45", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:30:47.505483+00:00", + "phase": "implement" + }, + { + "id": "00841cec-3cf0-40", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:36:10.972900+00:00", + "phase": "implement" + }, + { + "id": "90515dc7-ab2d-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:38:23.791091+00:00", + "phase": "implement" + }, + { + "id": "c26596a5-f284-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:43:30.262982+00:00", + "phase": "implement" + }, + { + "id": "44307c28-668b-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:45:52.345131+00:00", + "phase": "implement" + }, + { + "id": "3fadcc1f-6df9-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:46:59.997393+00:00", + "phase": "implement" + }, + { + "id": "43df70fd-f8ba-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:47:09.741618+00:00", + "phase": "implement" + }, + { + "id": "b469a23b-f96a-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:48:01.135406+00:00", + "phase": "implement" + }, + { + "id": "946ff580-846a-41", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "CONSENSUS_PROPOSE", + "subject": "Proposal from coder", + "body": "Slice-4 COMPLETE \u2014 orchestrator/routes/pipelines.py (30,520L structural outlier, over both caps) is fully decomposed into routes/pipelines/ (barrel + 46 underscore-prefixed submodules, ALL under the 1,500-line / 100KB cap). This invocation landed the final bite: moved the last giant _run_pipeline (the per-phase driver, non-negotiable #7) out of the barrel into _run_pipeline.py, dropping the barrel to 1,466L (under cap). With that, scripts/file-size-allowlist.yaml's files: map is now EMPTY \u2014 the terminal acceptance criterion of the whole #3312/#3450/#3447 file-size decomposition program: no orchestrator source file remains over the global cap. Non-negotiable #7 addressed head-on: _run_pipeline's phase-transition state machine was split across prior bites into _run_pipeline_setup, _run_pipeline_support (health closures via functools.partial), _run_phase/_run_phase_blocks (per-phase execution + plan/PENDING/IMPLEMENT advance via (pipeline, signal) returns), and _run_hitl_gate \u2014 transition ordering preserved exactly. Pure refactor: bodies byte-verbatim modulo _pkg. barrel-ref prefixing + docstring re-indent. All 16 orphaned barrel imports verified as genuine _pkg. seam refs (each >=2 submodule uses) -> retained with noqa F401, none deleted; patch(\"routes.pipelines._run_pipeline\") + _pkg._run_pipeline resolve via barrel re-export. Fixed 3 source-introspection seam tests broken by the move (strip _pkg. prefix; introspect _start_pipeline_body vs the thin @route wrapper) + a stale ruff-format line. ROLE-BOUNDARY NOTE: the routes/pipelines/ seam subsection for orchestrator/CLAUDE.md (task-4-5's doc half) is a documenter-owned restricted path, so it is intentionally NOT in this coder proposal \u2014 the documenter authors that seam row. Coder deliverables (code split, EMPTY allowlist, test fixes) are complete.", + "metadata": { + "payload": { + "summary": "Slice-4 COMPLETE \u2014 orchestrator/routes/pipelines.py (30,520L structural outlier, over both caps) is fully decomposed into routes/pipelines/ (barrel + 46 underscore-prefixed submodules, ALL under the 1,500-line / 100KB cap). This invocation landed the final bite: moved the last giant _run_pipeline (the per-phase driver, non-negotiable #7) out of the barrel into _run_pipeline.py, dropping the barrel to 1,466L (under cap). With that, scripts/file-size-allowlist.yaml's files: map is now EMPTY \u2014 the terminal acceptance criterion of the whole #3312/#3450/#3447 file-size decomposition program: no orchestrator source file remains over the global cap. Non-negotiable #7 addressed head-on: _run_pipeline's phase-transition state machine was split across prior bites into _run_pipeline_setup, _run_pipeline_support (health closures via functools.partial), _run_phase/_run_phase_blocks (per-phase execution + plan/PENDING/IMPLEMENT advance via (pipeline, signal) returns), and _run_hitl_gate \u2014 transition ordering preserved exactly. Pure refactor: bodies byte-verbatim modulo _pkg. barrel-ref prefixing + docstring re-indent. All 16 orphaned barrel imports verified as genuine _pkg. seam refs (each >=2 submodule uses) -> retained with noqa F401, none deleted; patch(\"routes.pipelines._run_pipeline\") + _pkg._run_pipeline resolve via barrel re-export. Fixed 3 source-introspection seam tests broken by the move (strip _pkg. prefix; introspect _start_pipeline_body vs the thin @route wrapper) + a stale ruff-format line. ROLE-BOUNDARY NOTE: the routes/pipelines/ seam subsection for orchestrator/CLAUDE.md (task-4-5's doc half) is a documenter-owned restricted path, so it is intentionally NOT in this coder proposal \u2014 the documenter authors that seam row. Coder deliverables (code split, EMPTY allowlist, test fixes) are complete.", + "attestation": {}, + "artifacts": [ + "orchestrator/routes/pipelines/__init__.py", + "orchestrator/routes/pipelines/_run_pipeline.py", + "scripts/file-size-allowlist.yaml", + "orchestrator/tests/test_advance_phase_thread.py", + "orchestrator/tests/test_ble001_narrowing_audit.py", + "aa6d2b619" + ], + "risk_considered": "Load-bearing back-compat surface (~137 referencing files, ~64 patch seams). De-risked: whole-repo collect-only = 16,757 tests / 0 import errors; barrel re-exports every external symbol; patch targets + _pkg. seams verified resolving. Pure refactor (byte-verbatim). Documented pre-existing NON-regressions unaffected: sandbox git-init env failures (test setup) and test_concurrent_status message-store pollution flakes (pass in isolation). CLAUDE.md doc row deferred to documenter (restricted path); no commit in the pushed range touches it.", + "commit_sha": "aa6d2b619", + "files_changed": [ + "orchestrator/routes/pipelines/__init__.py", + "orchestrator/routes/pipelines/_run_pipeline.py", + "scripts/file-size-allowlist.yaml", + "orchestrator/tests/test_advance_phase_thread.py", + "orchestrator/tests/test_ble001_narrowing_audit.py" + ], + "tests_run": [], + "tasks_satisfied": [ + "task-4-1", + "task-4-2", + "task-4-3", + "task-4-4", + "task-4-5", + "task-4-6" + ], + "no_changes_needed": false, + "no_changes_reason": "" + }, + "version": 1, + "commit_sha": "aa6d2b619", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:48:48.559854+00:00", + "phase": "implement" + }, + { + "id": "def64281-75cb-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "tester", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=propose (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:48:54.095866+00:00", + "phase": "implement" + }, + { + "id": "f025d1cc-07d6-40", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=ack (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:48:54.130801+00:00", + "phase": "implement" + }, + { + "id": "b2dd4417-91cd-47", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code_holistic", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=ack (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:48:55.461273+00:00", + "phase": "implement" + }, + { + "id": "426f760a-2917-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_contract", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=ack (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:48:59.009206+00:00", + "phase": "implement" + }, + { + "id": "95f0efd2-8fee-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_security", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=ack (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:49:00.744588+00:00", + "phase": "implement" + }, + { + "id": "b6f16eb7-b519-40", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_concurrency", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=ack (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:49:01.771967+00:00", + "phase": "implement" + }, + { + "id": "40cdcabb-08c0-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "tester", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:49:05.154662+00:00", + "phase": "implement" + }, + { + "id": "947bee0a-81f9-45", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:49:07.265934+00:00", + "phase": "implement" + }, + { + "id": "725bdc13-2c3c-44", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_contract", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:49:09.158294+00:00", + "phase": "implement" + }, + { + "id": "61f84bfa-3c54-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code_holistic", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:49:09.280218+00:00", + "phase": "implement" + }, + { + "id": "2b8dd57d-2103-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_security", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:49:13.087351+00:00", + "phase": "implement" + }, + { + "id": "028694cf-c3dd-44", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_concurrency", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:49:15.076270+00:00", + "phase": "implement" + }, + { + "id": "f85ca296-0a4a-42", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=229524 util=0.23 cache_hit=1.00 decision=at_or_above_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:49:22.648138+00:00", + "phase": "implement" + }, + { + "id": "f9a60613-e57b-43", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:49:40.209931+00:00", + "phase": "implement" + }, + { + "id": "9b956f53-76e7-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code", + "to_role": "coder", + "message_type": "CONSENSUS_ACK", + "subject": "ACK from reviewer_code for coder", + "body": "Terminal slice-4 decomposition verified clean against live tree. check-file-sizes.py exits 0 with the allowlist files: map EMPTY (terminal acceptance criterion); every pipelines/ barrel+submodule is under the 1500-line/100KB hard cap (largest _run_implement.py 1496L). All seams resolve (_run_pipeline, _start_pipeline_body, start_pipeline, 4 extracted helpers). Both modified seam-test files pass 19/19 \u2014 the source-introspection fixes are correct: strip the _pkg. decomposition prefix and introspect the real _start_pipeline_body vs the thin @route wrapper. R3 Dockerfile parity holds (routes/ ships via recursive COPY line 45; no Dockerfile change needed). test_ble001 change is cosmetic ruff-format churn. Deferred orchestrator/CLAUDE.md seam subsection is a documenter-owned restricted path, correctly out of coder scope. Pure refactor, behavior-preserving.", + "metadata": { + "payload": { + "artifact_references": [ + "scripts/file-size-allowlist.yaml", + "orchestrator/routes/pipelines/__init__.py", + "orchestrator/routes/pipelines/_run_pipeline.py", + "orchestrator/tests/test_advance_phase_thread.py", + "orchestrator/tests/test_ble001_narrowing_audit.py", + "orchestrator/Dockerfile" + ], + "reason": "Terminal slice-4 decomposition verified clean against live tree. check-file-sizes.py exits 0 with the allowlist files: map EMPTY (terminal acceptance criterion); every pipelines/ barrel+submodule is under the 1500-line/100KB hard cap (largest _run_implement.py 1496L). All seams resolve (_run_pipeline, _start_pipeline_body, start_pipeline, 4 extracted helpers). Both modified seam-test files pass 19/19 \u2014 the source-introspection fixes are correct: strip the _pkg. decomposition prefix and introspect the real _start_pipeline_body vs the thin @route wrapper. R3 Dockerfile parity holds (routes/ ships via recursive COPY line 45; no Dockerfile change needed). test_ble001 change is cosmetic ruff-format churn. Deferred orchestrator/CLAUDE.md seam subsection is a documenter-owned restricted path, correctly out of coder scope. Pure refactor, behavior-preserving.", + "ack_version": 1, + "attestation": { + "files_reviewed": [ + "scripts/file-size-allowlist.yaml", + "orchestrator/routes/pipelines/__init__.py", + "orchestrator/routes/pipelines/_run_pipeline.py", + "orchestrator/tests/test_advance_phase_thread.py", + "orchestrator/tests/test_ble001_narrowing_audit.py", + "orchestrator/Dockerfile" + ], + "issues_found": 0, + "checks_run": [ + "check-file-sizes.py exit 0", + "pipelines pkg all under hard cap", + "seam import resolution", + "pytest 19/19", + "Dockerfile recursive COPY parity" + ], + "verdict": "clean_pure_refactor" + } + }, + "version": 1, + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:50:32.530268+00:00", + "phase": "implement" + }, + { + "id": "b278a1c1-9707-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=86817 util=0.09 cache_hit=0.95 decision=no_warm_session", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:50:42.397242+00:00", + "phase": "implement" + }, + { + "id": "96ca518d-a3d7-44", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_concurrency", + "to_role": "coder", + "message_type": "CONSENSUS_ACK", + "subject": "ACK from reviewer_concurrency for coder", + "body": "Concurrency review clean. Terminal commit aa6d2b619 touches no production concurrency code (tests + empty allowlist + scratch delete only). The test_advance_phase_thread.py retargeting is correct: stripping the `_pkg.` decomposition prefix RESTORES the pre-split structural regexes (moved bodies reference barrel globals as `_pkg.<name>`, 166x in _run_pipeline), and repointing to `_start_pipeline_body` (decision-8 thin @route wrapper) resolves the real body carrying `TEST_MARKER: recover_advance_clear`. Verified the introspection guards do NOT pass vacuously \u2014 markers present, invariants still asserted. Whole-slice concurrency surface green: 15/15 advance-phase-thread (auto-advance thread respawn, recover-pipeline concurrent-state clearing, post-BRC error swallowing), 82 consensus-polling + slice-run-loop, 37/37 concurrent-status in isolation. _run_pipeline decomposition preserves thread semantics (verbatim bodies; continue/break->return-signal threading covered by passing thread tests).", + "metadata": { + "payload": { + "artifact_references": [ + "orchestrator/tests/test_advance_phase_thread.py", + "orchestrator/tests/test_ble001_narrowing_audit.py", + "scripts/file-size-allowlist.yaml", + "orchestrator/routes/pipelines/__init__.py", + "orchestrator/routes/pipelines/_run_pipeline.py", + "orchestrator/routes/pipelines/_routes_lifecycle.py" + ], + "reason": "Concurrency review clean. Terminal commit aa6d2b619 touches no production concurrency code (tests + empty allowlist + scratch delete only). The test_advance_phase_thread.py retargeting is correct: stripping the `_pkg.` decomposition prefix RESTORES the pre-split structural regexes (moved bodies reference barrel globals as `_pkg.<name>`, 166x in _run_pipeline), and repointing to `_start_pipeline_body` (decision-8 thin @route wrapper) resolves the real body carrying `TEST_MARKER: recover_advance_clear`. Verified the introspection guards do NOT pass vacuously \u2014 markers present, invariants still asserted. Whole-slice concurrency surface green: 15/15 advance-phase-thread (auto-advance thread respawn, recover-pipeline concurrent-state clearing, post-BRC error swallowing), 82 consensus-polling + slice-run-loop, 37/37 concurrent-status in isolation. _run_pipeline decomposition preserves thread semantics (verbatim bodies; continue/break->return-signal threading covered by passing thread tests).", + "ack_version": 1 + }, + "version": 1, + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:51:03.169218+00:00", + "phase": "implement" + }, + { + "id": "04007146-5748-41", + "pipeline_id": "issue-3312-v2", + "from_role": "tester", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:51:09.225138+00:00", + "phase": "implement" + }, + { + "id": "eb3741e5-64a0-42", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_concurrency", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=81576 util=0.08 cache_hit=0.99 decision=no_warm_session", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:51:12.873176+00:00", + "phase": "implement" + }, + { + "id": "f2785071-c4f9-44", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_security", + "to_role": "coder", + "message_type": "CONSENSUS_ACK", + "subject": "ACK from reviewer_security for coder", + "body": "Security ACK (terminal slice-4 proposal aa6d2b619). Pure mechanical decomposition of routes/pipelines.py giant \u2014 no behavioral change. Security-critical checks all pass: (1) @require_lifecycle_secret auth guards preserved 9\u21929 (origin/main had 9, decomposed package has 9: 8 barrel wrappers + 1 in _criteria.py); none dropped, decorators still gate every originally-gated route (verified barrel imports cleanly, routes resolve). (2) No new secrets/eval/exec/subprocess/shell/verify=False/http introduced \u2014 grep of +diff found only pre-existing security infra moved verbatim. (3) Trust boundary intact. (4) Terminal changes (empty allowlist = stricter policy, test source-introspection fixes, scratch-tool delete) have zero security impact. (5) check-file-sizes.py green with empty allowlist.", + "metadata": { + "payload": { + "artifact_references": [ + "orchestrator/routes/pipelines/__init__.py", + "orchestrator/routes/pipelines/_run_pipeline.py", + "orchestrator/routes/pipelines/_criteria.py", + "scripts/file-size-allowlist.yaml", + "orchestrator/tests/test_advance_phase_thread.py", + "orchestrator/tests/test_ble001_narrowing_audit.py" + ], + "reason": "Security ACK (terminal slice-4 proposal aa6d2b619). Pure mechanical decomposition of routes/pipelines.py giant \u2014 no behavioral change. Security-critical checks all pass: (1) @require_lifecycle_secret auth guards preserved 9\u21929 (origin/main had 9, decomposed package has 9: 8 barrel wrappers + 1 in _criteria.py); none dropped, decorators still gate every originally-gated route (verified barrel imports cleanly, routes resolve). (2) No new secrets/eval/exec/subprocess/shell/verify=False/http introduced \u2014 grep of +diff found only pre-existing security infra moved verbatim. (3) Trust boundary intact. (4) Terminal changes (empty allowlist = stricter policy, test source-introspection fixes, scratch-tool delete) have zero security impact. (5) check-file-sizes.py green with empty allowlist.", + "ack_version": 1 + }, + "version": 1, + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:51:14.113501+00:00", + "phase": "implement" + }, + { + "id": "a427c50a-004a-42", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_security", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:51:14.551806+00:00", + "phase": "implement" + }, + { + "id": "baa4b67a-794c-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code_holistic", + "to_role": "coder", + "message_type": "CONSENSUS_ACK", + "subject": "ACK from reviewer_code_holistic for coder", + "body": "Holistic ACK of the slice-4 terminal (routes/pipelines.py giant decomposed into ~45 under-cap submodules + program terminal). Verified: (1) Terminal acceptance criterion met \u2014 file-size-allowlist.yaml files: map is EMPTY and check-file-sizes.py exits 0 (barrel __init__.py 1466L < 1500 hard cap; remaining entries are soft-cap warnings only). (2) Behavior-preserving: barrel imports cleanly and all patch-target re-exports resolve (_run_pipeline, _start_pipeline_body, start_pipeline, _run_concurrent_phase, _run_implement_phase_slices) so test patches on routes.pipelines.<name> keep intercepting. (3) Test-seam fixes are principled \u2014 source-introspection helpers strip the decomposition-only _pkg. prefix and introspect the real _start_pipeline_body rather than the thin @route wrapper; test_ble001 change is a benign ruff-format normalization in the not-taken pre-split fallback branch. (4) R3 Dockerfile parity holds: COPY orchestrator/routes/ ./routes/ ships the new pipelines/ subpackage recursively (no Dockerfile change needed \u2014 claim confirmed); models/ and event_loop/ have explicit COPY lines from prior slices. (5) 7612 orchestrator tests collect with 0 import errors; all 317 touched/seam tests pass (advance_phase_thread, ble001, start_pipeline, slice_loop_import_seam, pipelines_apply, origin_main, overseer_model, role_to_reviewer_mapping, handlers_brc); ruff clean on the pipelines package. (6) Scratch extraction tool deleted. The orchestrator/CLAUDE.md routes/pipelines/ seam subsection is correctly deferred to the documenter (restricted path) \u2014 a legitimate role boundary, not a code defect. No holistic correctness or architecture concerns.", + "metadata": { + "payload": { + "artifact_references": [ + "scripts/file-size-allowlist.yaml", + "orchestrator/routes/pipelines/__init__.py", + "orchestrator/routes/pipelines/_run_pipeline.py", + "orchestrator/tests/test_advance_phase_thread.py", + "orchestrator/tests/test_ble001_narrowing_audit.py", + "orchestrator/Dockerfile" + ], + "reason": "Holistic ACK of the slice-4 terminal (routes/pipelines.py giant decomposed into ~45 under-cap submodules + program terminal). Verified: (1) Terminal acceptance criterion met \u2014 file-size-allowlist.yaml files: map is EMPTY and check-file-sizes.py exits 0 (barrel __init__.py 1466L < 1500 hard cap; remaining entries are soft-cap warnings only). (2) Behavior-preserving: barrel imports cleanly and all patch-target re-exports resolve (_run_pipeline, _start_pipeline_body, start_pipeline, _run_concurrent_phase, _run_implement_phase_slices) so test patches on routes.pipelines.<name> keep intercepting. (3) Test-seam fixes are principled \u2014 source-introspection helpers strip the decomposition-only _pkg. prefix and introspect the real _start_pipeline_body rather than the thin @route wrapper; test_ble001 change is a benign ruff-format normalization in the not-taken pre-split fallback branch. (4) R3 Dockerfile parity holds: COPY orchestrator/routes/ ./routes/ ships the new pipelines/ subpackage recursively (no Dockerfile change needed \u2014 claim confirmed); models/ and event_loop/ have explicit COPY lines from prior slices. (5) 7612 orchestrator tests collect with 0 import errors; all 317 touched/seam tests pass (advance_phase_thread, ble001, start_pipeline, slice_loop_import_seam, pipelines_apply, origin_main, overseer_model, role_to_reviewer_mapping, handlers_brc); ruff clean on the pipelines package. (6) Scratch extraction tool deleted. The orchestrator/CLAUDE.md routes/pipelines/ seam subsection is correctly deferred to the documenter (restricted path) \u2014 a legitimate role boundary, not a code defect. No holistic correctness or architecture concerns.", + "ack_version": 1 + }, + "version": 1, + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:51:16.348049+00:00", + "phase": "implement" + }, + { + "id": "e2b05fe5-276f-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code_holistic", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:51:16.755975+00:00", + "phase": "implement" + }, + { + "id": "3541a777-6fc4-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code_holistic", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=89035 util=0.09 cache_hit=0.95 decision=no_warm_session", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:51:24.758517+00:00", + "phase": "implement" + }, + { + "id": "e3f5cc2a-4cf7-40", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_security", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=79512 util=0.08 cache_hit=0.99 decision=no_warm_session", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:51:25.031743+00:00", + "phase": "implement" + }, + { + "id": "0cba55fa-5c22-47", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_contract", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:51:27.180206+00:00", + "phase": "implement" + }, + { + "id": "2d74ffe4-3e08-42", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:52:26.771081+00:00", + "phase": "implement" + }, + { + "id": "90b8046f-1fc9-46", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_contract", + "to_role": "coder", + "message_type": "CONSENSUS_ACK", + "subject": "ACK from reviewer_contract for coder", + "body": "Contract verification PASS for coder slice-4 terminal (aa6d2b619). TERMINAL CRITERION MET: scripts/file-size-allowlist.yaml files: map is EMPTY ({}) and check-file-sizes.py exits 0 \u2014 every previously-allowlisted giant decomposed, all pipelines/ submodules under the hard cap (barrel __init__.py 1466L, _run_pipeline.py 1483L, _run_implement.py 1496L \u2014 all <=1500L/<=100KB). task-4-3: _run_pipeline decomposed into _run_pipeline.py + _run_hitl_gate/_run_phase/_run_phase_blocks/_run_pipeline_setup blocks; 337 loop-seam tests green (test_consensus_polling, test_brc_nack, test_concurrent_*, test_slice_run_loop_integration) \u2014 no transition-ordering change. task-4-1/4-4: barrel re-exports resolve (import routes.pipelines OK; patch('routes.pipelines._run_pipeline') and _start_pipeline_body intercept via barrel). R3 Dockerfile parity: COPY orchestrator/routes/ ./routes/ recursively ships pipelines/, no Dockerfile change needed. task-4-6: 144 touched/seam + 337 loop tests pass; terminal test-mechanical fixes in test_advance_phase_thread.py (strip _pkg. prefix; introspect _start_pipeline_body) and test_ble001 format line are legitimate seam repairs. The 2 collect-only errors (test_compose_event_prompt, test_brc_preamble_collapsed) are a pre-existing orchestrator.-prefix PYTHONPATH quirk in slice-6 event_prompt tests, NOT split-induced. task-4-5 doc-half (CLAUDE.md routes/pipelines/ seam section) remains pending on the documenter (restricted path, correctly deferred by coder) \u2014 verified separately against documenter's proposal, not a coder-side blocker.", + "metadata": { + "payload": { + "artifact_references": [ + "scripts/file-size-allowlist.yaml", + "orchestrator/routes/pipelines/__init__.py", + "orchestrator/routes/pipelines/_run_pipeline.py", + "orchestrator/tests/test_advance_phase_thread.py", + "orchestrator/tests/test_ble001_narrowing_audit.py", + "orchestrator/Dockerfile" + ], + "reason": "Contract verification PASS for coder slice-4 terminal (aa6d2b619). TERMINAL CRITERION MET: scripts/file-size-allowlist.yaml files: map is EMPTY ({}) and check-file-sizes.py exits 0 \u2014 every previously-allowlisted giant decomposed, all pipelines/ submodules under the hard cap (barrel __init__.py 1466L, _run_pipeline.py 1483L, _run_implement.py 1496L \u2014 all <=1500L/<=100KB). task-4-3: _run_pipeline decomposed into _run_pipeline.py + _run_hitl_gate/_run_phase/_run_phase_blocks/_run_pipeline_setup blocks; 337 loop-seam tests green (test_consensus_polling, test_brc_nack, test_concurrent_*, test_slice_run_loop_integration) \u2014 no transition-ordering change. task-4-1/4-4: barrel re-exports resolve (import routes.pipelines OK; patch('routes.pipelines._run_pipeline') and _start_pipeline_body intercept via barrel). R3 Dockerfile parity: COPY orchestrator/routes/ ./routes/ recursively ships pipelines/, no Dockerfile change needed. task-4-6: 144 touched/seam + 337 loop tests pass; terminal test-mechanical fixes in test_advance_phase_thread.py (strip _pkg. prefix; introspect _start_pipeline_body) and test_ble001 format line are legitimate seam repairs. The 2 collect-only errors (test_compose_event_prompt, test_brc_preamble_collapsed) are a pre-existing orchestrator.-prefix PYTHONPATH quirk in slice-6 event_prompt tests, NOT split-induced. task-4-5 doc-half (CLAUDE.md routes/pipelines/ seam section) remains pending on the documenter (restricted path, correctly deferred by coder) \u2014 verified separately against documenter's proposal, not a coder-side blocker.", + "ack_version": 1, + "attestation": { + "tasks_verified": [ + "task-4-1", + "task-4-2", + "task-4-3", + "task-4-4", + "task-4-5", + "task-4-6" + ], + "notes": "task-4-5 coder-half (allowlist EMPTY) verified; doc-half (orchestrator/CLAUDE.md routes/pipelines/ seam) is a documenter-owned restricted path, verified against documenter proposal not coder." + } + }, + "version": 1, + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:52:59.169943+00:00", + "phase": "implement" + }, + { + "id": "4ba0c4bd-858a-48", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_contract", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:53:35.726570+00:00", + "phase": "implement" + }, + { + "id": "8229c3d9-2e60-40", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_contract", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=101554 util=0.10 cache_hit=0.98 decision=no_warm_session", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:53:49.302236+00:00", + "phase": "implement" + }, + { + "id": "338d665a-3148-45", + "pipeline_id": "issue-3312-v2", + "from_role": "tester", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:53:54.132934+00:00", + "phase": "implement" + }, + { + "id": "bc60573c-6b7d-46", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:54:51.365446+00:00", + "phase": "implement" + }, + { + "id": "eba83283-38f0-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "CONSENSUS_PROPOSE", + "subject": "Proposal from documenter", + "body": "Slice-4 documenter deliverable (task-4-5): authored the concrete `routes/pipelines/` decomposition seam row in orchestrator/CLAUDE.md, rebased onto the origin slice-4 tip (aa6d2b619, the coder's terminal commit). The row documents the 46-submodule layout, the decision-8 routes-in-barrel convention (16 @pipelines_bp.route thin wrappers stay in the barrel delegating to _routes_* bodies), the _run_pipeline per-phase split (non-negotiable #7 / task-4-3: thin loop \u2192 _run_pipeline_setup/_run_phase/_run_phase_blocks/_run_implement/_run_concurrent/_run_hitl_gate), packaging-neutral recursive `COPY orchestrator/routes/`, and the terminal criterion. This is the FINAL slice of the #3312 program: pipelines.py (30,520L/1.44MB, the last allowlisted file) is decomposed into routes/pipelines/ with the barrel __init__.py at 1,466 lines (under cap) and every submodule under both caps; scripts/file-size-allowlist.yaml's files: map is now EMPTY (terminal acceptance criterion), closing #3312. check-file-sizes.py ratchet exit 0 with the empty allowlist; this documenter commit (92beb1769) changes ONLY orchestrator/CLAUDE.md, stacked directly on the origin slice-4 tip. Verified all four CLAUDE.md seam tables (orchestrator/gateway/sandbox/shared) present and current.", + "metadata": { + "payload": { + "summary": "Slice-4 documenter deliverable (task-4-5): authored the concrete `routes/pipelines/` decomposition seam row in orchestrator/CLAUDE.md, rebased onto the origin slice-4 tip (aa6d2b619, the coder's terminal commit). The row documents the 46-submodule layout, the decision-8 routes-in-barrel convention (16 @pipelines_bp.route thin wrappers stay in the barrel delegating to _routes_* bodies), the _run_pipeline per-phase split (non-negotiable #7 / task-4-3: thin loop \u2192 _run_pipeline_setup/_run_phase/_run_phase_blocks/_run_implement/_run_concurrent/_run_hitl_gate), packaging-neutral recursive `COPY orchestrator/routes/`, and the terminal criterion. This is the FINAL slice of the #3312 program: pipelines.py (30,520L/1.44MB, the last allowlisted file) is decomposed into routes/pipelines/ with the barrel __init__.py at 1,466 lines (under cap) and every submodule under both caps; scripts/file-size-allowlist.yaml's files: map is now EMPTY (terminal acceptance criterion), closing #3312. check-file-sizes.py ratchet exit 0 with the empty allowlist; this documenter commit (92beb1769) changes ONLY orchestrator/CLAUDE.md, stacked directly on the origin slice-4 tip. Verified all four CLAUDE.md seam tables (orchestrator/gateway/sandbox/shared) present and current.", + "attestation": {}, + "artifacts": [ + "orchestrator/CLAUDE.md" + ], + "risk_considered": "Docs-only change on documenter-owned orchestrator/CLAUDE.md (check_file_restriction can_write=true). Rebased onto origin/egg/issue-3312-v2/slice-4 (aa6d2b619) so the push is a clean fast-forward; diff vs origin tip is exactly one file (orchestrator/CLAUDE.md). Seam row verified against the landed pipelines/ package (46 submodules, exact line counts, 16 route decorators in barrel, barrel 1,466L under cap). Ratchet exit 0 with empty allowlist; remaining check-file-sizes output is pre-existing soft-cap advisories on unrelated shared/ files, not hard-cap failures.", + "commit_sha": "92beb1769", + "files_changed": [ + "orchestrator/CLAUDE.md" + ], + "tests_run": [], + "tasks_satisfied": [ + "task-4-5" + ], + "no_changes_needed": false, + "no_changes_reason": "" + }, + "version": 1, + "commit_sha": "92beb1769", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:55:47.953299+00:00", + "phase": "implement" + }, + { + "id": "073dc69f-59bb-45", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=ack (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:55:54.914129+00:00", + "phase": "implement" + }, + { + "id": "259d20ae-e091-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_contract", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=ack (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:55:55.257781+00:00", + "phase": "implement" + }, + { + "id": "dc07ff70-87aa-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:56:01.852161+00:00", + "phase": "implement" + }, + { + "id": "490f4e0c-cc4f-45", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_contract", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:56:03.996491+00:00", + "phase": "implement" + }, + { + "id": "02f67d4b-5c73-42", + "pipeline_id": "issue-3312-v2", + "from_role": "tester", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:56:08.910599+00:00", + "phase": "implement" + }, + { + "id": "483cae4f-69fe-40", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: PROPOSED", + "body": "slice-4 documenter: PROPOSED \u2705 (task-4-5 delivered). Authored the orchestrator/CLAUDE.md routes/pipelines/ decomposition seam row (46-submodule layout, decision-8 routes-in-barrel, _run_pipeline per-phase split, packaging-neutral recursive COPY, terminal criterion) in commit 92beb1769 \u2014 changes ONLY orchestrator/CLAUDE.md, rebased onto the origin slice-4 terminal tip aa6d2b619 so the push fast-forwarded. mcp__brc__propose: status=proposed, version=1, reviewers=[reviewer_contract, reviewer_code]. The branch-persistence wedge (b4ca0796/8642ccd3) is resolved \u2014 the coder's completed extraction reached origin (barrel 1,466L under cap; allowlist files: EMPTY = terminal criterion, #3312 complete). Awaiting reviewer ACKs; will address any NACK on orchestrator/CLAUDE.md and re-propose, else confirm once all ACK. Exiting.", + "metadata": { + "state": "PROPOSED", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:56:23.963598+00:00", + "phase": "implement" + }, + { + "id": "20c579d8-6d38-41", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=443673 util=0.44 cache_hit=1.00 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:56:37.368543+00:00", + "phase": "implement" + }, + { + "id": "4a17d2e4-aa05-47", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code", + "to_role": "documenter", + "message_type": "CONSENSUS_ACK", + "subject": "ACK from reviewer_code for documenter", + "body": "orchestrator/CLAUDE.md routes/pipelines/ seam subsection verified accurate against the landed code. Every quantitative claim checks out: \"46 submodules\" (47 .py files minus the barrel), exactly 16 @pipelines_bp.route decorators in the barrel (decision-8), and all spot-checked cited line counts exact (_run_implement.py 1,496 largest, _run_pipeline.py 1,483, barrel 1,466, _run_concurrent.py 1,439, _populate.py 1,460, _routes_status.py 572, _run_pipeline_setup.py 727). All 20 spot-checked submodule filenames exist. The decision-8 (route decorators in barrel), non-negotiable #7 (_run_pipeline per-phase split), packaging-neutral recursive COPY (Dockerfile:45, verified in the coder review), and terminal-criterion (allowlist files: map EMPTY, verified) prose are all correct. This is the documenter half the coder correctly deferred as a restricted path. Pure documentation, no behaviour change.", + "metadata": { + "payload": { + "artifact_references": [ + "orchestrator/CLAUDE.md" + ], + "reason": "orchestrator/CLAUDE.md routes/pipelines/ seam subsection verified accurate against the landed code. Every quantitative claim checks out: \"46 submodules\" (47 .py files minus the barrel), exactly 16 @pipelines_bp.route decorators in the barrel (decision-8), and all spot-checked cited line counts exact (_run_implement.py 1,496 largest, _run_pipeline.py 1,483, barrel 1,466, _run_concurrent.py 1,439, _populate.py 1,460, _routes_status.py 572, _run_pipeline_setup.py 727). All 20 spot-checked submodule filenames exist. The decision-8 (route decorators in barrel), non-negotiable #7 (_run_pipeline per-phase split), packaging-neutral recursive COPY (Dockerfile:45, verified in the coder review), and terminal-criterion (allowlist files: map EMPTY, verified) prose are all correct. This is the documenter half the coder correctly deferred as a restricted path. Pure documentation, no behaviour change.", + "ack_version": 1, + "attestation": { + "files_reviewed": [ + "orchestrator/CLAUDE.md" + ], + "issues_found": 0, + "checks_run": [ + "submodule count 46+barrel=47", + "16 @pipelines_bp.route decorators", + "spot-check 7 cited line counts exact", + "20 cited submodule filenames exist" + ], + "verdict": "accurate_documentation" + } + }, + "version": 1, + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:56:38.768738+00:00", + "phase": "implement" + }, + { + "id": "ac41488d-2525-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=104060 util=0.10 cache_hit=0.98 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:56:43.872687+00:00", + "phase": "implement" + }, + { + "id": "8e0d0ab4-913a-46", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_contract", + "to_role": "documenter", + "message_type": "CONSENSUS_ACK", + "subject": "ACK from reviewer_contract for documenter", + "body": "Contract verification PASS for documenter slice-4 (92beb1769) \u2014 completes task-4-5 doc-half. The new ### routes/pipelines/ seam section accurately documents the landed decomposition: (a) line counts EXACT vs the package (barrel __init__.py 1466, _run_implement.py 1496, _run_pipeline.py 1483, _populate.py 1460, _prompt_phase.py 1407); (b) '46 submodules' correct (47 .py files = 46 submodules + 1 barrel); (c) decision-8 route convention (16 @pipelines_bp.route decorators in barrel on thin wrappers) matches; (d) _run_pipeline per-phase split (non-negotiable #7) names the real modules _run_pipeline/_run_pipeline_setup/_run_phase/_run_phase_blocks/_run_implement/_run_concurrent/_run_hitl_gate; (e) packaging-neutral recursive COPY orchestrator/routes/ ./routes/ (Dockerfile:45) correctly stated as NO Dockerfile change; (f) terminal criterion \u2014 pipelines.py was the LAST allowlist entry, files: map now EMPTY, #3312 complete \u2014 matches the verified allowlist state. All four CLAUDE.md seam files (orchestrator/gateway/sandbox/shared) present and the orchestrator roll-up now lists routes/pipelines/ and declares the allowlist empty. Pure-documentation, no behaviour change. task-4-5 AC 'orchestrator/CLAUDE.md carries the concrete pipelines/ + _run_pipeline submodule layout' satisfied.", + "metadata": { + "payload": { + "artifact_references": [ + "orchestrator/CLAUDE.md" + ], + "reason": "Contract verification PASS for documenter slice-4 (92beb1769) \u2014 completes task-4-5 doc-half. The new ### routes/pipelines/ seam section accurately documents the landed decomposition: (a) line counts EXACT vs the package (barrel __init__.py 1466, _run_implement.py 1496, _run_pipeline.py 1483, _populate.py 1460, _prompt_phase.py 1407); (b) '46 submodules' correct (47 .py files = 46 submodules + 1 barrel); (c) decision-8 route convention (16 @pipelines_bp.route decorators in barrel on thin wrappers) matches; (d) _run_pipeline per-phase split (non-negotiable #7) names the real modules _run_pipeline/_run_pipeline_setup/_run_phase/_run_phase_blocks/_run_implement/_run_concurrent/_run_hitl_gate; (e) packaging-neutral recursive COPY orchestrator/routes/ ./routes/ (Dockerfile:45) correctly stated as NO Dockerfile change; (f) terminal criterion \u2014 pipelines.py was the LAST allowlist entry, files: map now EMPTY, #3312 complete \u2014 matches the verified allowlist state. All four CLAUDE.md seam files (orchestrator/gateway/sandbox/shared) present and the orchestrator roll-up now lists routes/pipelines/ and declares the allowlist empty. Pure-documentation, no behaviour change. task-4-5 AC 'orchestrator/CLAUDE.md carries the concrete pipelines/ + _run_pipeline submodule layout' satisfied.", + "ack_version": 1, + "attestation": { + "tasks_verified": [ + "task-4-5" + ], + "notes": "Documenter doc-half of task-4-5: orchestrator/CLAUDE.md routes/pipelines/ decomposition seam section. Factual claims verified against the landed package at 92beb1769." + } + }, + "version": 1, + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:56:47.518045+00:00", + "phase": "implement" + }, + { + "id": "3e72e33b-34e9-48", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_contract", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=121347 util=0.12 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:57:11.615467+00:00", + "phase": "implement" + }, + { + "id": "e4a9a87d-0fc8-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "tester", + "to_role": "all", + "message_type": "CONSENSUS_PROPOSE", + "subject": "Proposal from tester", + "body": "Slice-4 tester verification \u2014 orchestrator/routes/pipelines.py decomposition (coder tip aa6d2b619) verified as a clean pure refactor. (1) File-size gate: check-file-sizes.py exit 0; scripts/file-size-allowlist.yaml files: map is EMPTY \u2014 the terminal acceptance criterion of the whole #3312/#3450/#3447 program (no source file over the 1,500-line/100KB hard cap; barrel __init__.py 1,466L, all 46 _*.py submodules under cap). (2) Import/re-export sanity: routes.pipelines barrel loads and re-exports resolve (e.g. _build_brc_preamble); routes.event_prompt intact. (3) Coder-fixed source-introspection seam tests pass: test_advance_phase_thread.py + test_ble001_narrowing_audit.py = 19 passed. (4) Regression net: targeted route/pipeline/phase/signal/consensus/run_ subset = 3894 passed, 10 skipped. (5) Lint: ruff check + ruff format --check clean on the pipelines package (47 files). (6) Security: bandit -r on the pipelines package = 0 High, 0 Medium, 67 Low; bodies are AST-identical to the original file so those Lows are pre-existing \u2014 zero new findings introduced by the split. The 30 failures + 4 errors in the broad run are ALL pre-existing sandbox-environmental, NONE in the slice-4 diff and NONE touching the decomposed package \u2014 proven by failure signature: `git init` returns exit 1 (gateway-blocked; agent_salvage/commit_statefiles/consensus_wrapper/contract_preserved/slice_diff_summary/slice_phase_restart clusters set up temp git repos), urllib3 Timeout>60s (network-blocked; deployment tests), and 403->200 (auth-secret env unset). Every failing test file confirmed NOT-in-diff vs merge-base; all executed (no ImportError), so barrel re-exports are intact. Two collection errors (test_brc_preamble_collapsed/test_compose_event_prompt) are an invocation artifact of running pytest from orchestrator/ with the `orchestrator.` absolute-prefix import \u2014 their symbols resolve via the routes. prefix; not a regression. No new tests authored: pure structural refactor with AST-identical bodies, so the existing suite IS the regression net; the only test edits needed (3 source-introspection seams) were landed by the coder. Matches coder attestation exactly.", + "metadata": { + "payload": { + "summary": "Slice-4 tester verification \u2014 orchestrator/routes/pipelines.py decomposition (coder tip aa6d2b619) verified as a clean pure refactor. (1) File-size gate: check-file-sizes.py exit 0; scripts/file-size-allowlist.yaml files: map is EMPTY \u2014 the terminal acceptance criterion of the whole #3312/#3450/#3447 program (no source file over the 1,500-line/100KB hard cap; barrel __init__.py 1,466L, all 46 _*.py submodules under cap). (2) Import/re-export sanity: routes.pipelines barrel loads and re-exports resolve (e.g. _build_brc_preamble); routes.event_prompt intact. (3) Coder-fixed source-introspection seam tests pass: test_advance_phase_thread.py + test_ble001_narrowing_audit.py = 19 passed. (4) Regression net: targeted route/pipeline/phase/signal/consensus/run_ subset = 3894 passed, 10 skipped. (5) Lint: ruff check + ruff format --check clean on the pipelines package (47 files). (6) Security: bandit -r on the pipelines package = 0 High, 0 Medium, 67 Low; bodies are AST-identical to the original file so those Lows are pre-existing \u2014 zero new findings introduced by the split. The 30 failures + 4 errors in the broad run are ALL pre-existing sandbox-environmental, NONE in the slice-4 diff and NONE touching the decomposed package \u2014 proven by failure signature: `git init` returns exit 1 (gateway-blocked; agent_salvage/commit_statefiles/consensus_wrapper/contract_preserved/slice_diff_summary/slice_phase_restart clusters set up temp git repos), urllib3 Timeout>60s (network-blocked; deployment tests), and 403->200 (auth-secret env unset). Every failing test file confirmed NOT-in-diff vs merge-base; all executed (no ImportError), so barrel re-exports are intact. Two collection errors (test_brc_preamble_collapsed/test_compose_event_prompt) are an invocation artifact of running pytest from orchestrator/ with the `orchestrator.` absolute-prefix import \u2014 their symbols resolve via the routes. prefix; not a regression. No new tests authored: pure structural refactor with AST-identical bodies, so the existing suite IS the regression net; the only test edits needed (3 source-introspection seams) were landed by the coder. Matches coder attestation exactly.", + "attestation": { + "tests_run": 3913, + "checks_passed": [ + "lint", + "test", + "security", + "file-size-gate" + ], + "tests_execution_blocked": false, + "security_scan": "bandit -r orchestrator/routes/pipelines/: 0 High, 0 Medium, 67 Low (all pre-existing, AST-identical bodies)", + "environmental_failures_classified": true, + "environmental_failure_reason": "30 failed + 4 errored are all pre-existing sandbox-environmental (git-init gateway-blocked, network-blocked, auth-secret env unset); all outside slice-4 diff, none touch routes/pipelines/, all executed (no ImportError)", + "pure_refactor": true, + "allowlist_empty": true + }, + "artifacts": [ + "aa6d2b619", + "orchestrator/routes/pipelines/__init__.py", + "orchestrator/routes/pipelines/_run_pipeline.py", + "scripts/file-size-allowlist.yaml", + "orchestrator/tests/test_advance_phase_thread.py", + "orchestrator/tests/test_ble001_narrowing_audit.py" + ], + "risk_considered": "Pure structural refactor of a load-bearing back-compat surface (routes/pipelines.py -> routes/pipelines/ barrel + 46 submodules). De-risked: barrel re-exports every external symbol (verified _build_brc_preamble + module load); all patch-seam / source-introspection tests pass (19 seam + 3894 targeted); file-size hard-cap gate exit 0 with allowlist now EMPTY; bandit clean (0 High/Medium, Lows pre-existing). Distinguished environmental failures (git-init/network/auth-env, all NOT-in-diff, all executed) from regressions by failure signature and diff-membership \u2014 zero regressions attributable to the decomposition.", + "commit_sha": "aa6d2b619", + "files_changed": [], + "tests_run": [], + "tasks_satisfied": [ + "task-4-1", + "task-4-2", + "task-4-3", + "task-4-4", + "task-4-5", + "task-4-6" + ], + "no_changes_needed": false, + "no_changes_reason": "" + }, + "version": 1, + "commit_sha": "aa6d2b619", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:12.367129+00:00", + "phase": "implement" + }, + { + "id": "b65a3dfb-c38c-49", + "pipeline_id": "issue-3312-v2", + "from_role": "orchestrator", + "to_role": "documenter", + "message_type": "STATUS", + "subject": "Ready to confirm \u2014 all confirm preconditions satisfied", + "body": "Your proposal (version 1) is ready to confirm \u2014 all blocking reviews are clear and global confirm preconditions are met. Run `egg-orch consensus confirmed` to confirm.", + "metadata": { + "ready_to_confirm": true, + "version": 1, + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:12.369175+00:00", + "phase": "implement" + }, + { + "id": "57b53e22-557c-47", + "pipeline_id": "issue-3312-v2", + "from_role": "tester", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:12.803826+00:00", + "phase": "implement" + }, + { + "id": "74c733e2-7920-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "CONSENSUS_CONFIRMED", + "subject": "Consensus confirmed by documenter", + "body": "orchestrator-side confirm (#3064 event loop)", + "metadata": { + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:18.392191+00:00", + "phase": "implement" + }, + { + "id": "570b22d8-2b29-48", + "pipeline_id": "issue-3312-v2", + "from_role": "tester", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=ack (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:19.872497+00:00", + "phase": "implement" + }, + { + "id": "b9f1773f-8210-46", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=ack (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:19.987557+00:00", + "phase": "implement" + }, + { + "id": "f5b10347-ec3b-45", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code_holistic", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=ack (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:20.635669+00:00", + "phase": "implement" + }, + { + "id": "65f57036-2394-44", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_contract", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=ack (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:21.292010+00:00", + "phase": "implement" + }, + { + "id": "1847193d-ec8e-40", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_security", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=ack (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:21.791833+00:00", + "phase": "implement" + }, + { + "id": "1e8a297b-3f8b-46", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_concurrency", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "one-shot event handler action=ack (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:22.519243+00:00", + "phase": "implement" + }, + { + "id": "effea01a-ce70-41", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code_holistic", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:32.977902+00:00", + "phase": "implement" + }, + { + "id": "8d3386e1-7843-45", + "pipeline_id": "issue-3312-v2", + "from_role": "tester", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:34.162279+00:00", + "phase": "implement" + }, + { + "id": "7f2f22bd-28db-42", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_concurrency", + "to_role": "tester", + "message_type": "CONSENSUS_ACK", + "subject": "ACK from reviewer_concurrency for tester", + "body": "Concurrency review clean (same terminal commit aa6d2b619). Tester's deliverable is the test-mechanical surface: the source-introspection retargeting in test_advance_phase_thread.py and the test_ble001 reformat. Verified the thread-safety guards still validate real structure, not vacuously: stripping the `_pkg.` decomposition prefix RESTORES the pre-split structural regexes (moved _run_pipeline body references barrel globals as `_pkg.<name>`), and repointing to `_start_pipeline_body` (decision-8 thin @route wrapper) resolves the real body carrying `TEST_MARKER: recover_advance_clear`; the auto-advance marker resolves in `_run_pipeline`. Concurrency invariants guarded and green: 15/15 advance-phase-thread (auto-advance thread respawn, recover-pipeline concurrent-state clearing, post-BRC error swallowing keeping the driver thread alive), 82 consensus-polling + slice-run-loop, 37/37 concurrent-status in isolation. No concurrency regression from the test changes.", + "metadata": { + "payload": { + "artifact_references": [ + "orchestrator/tests/test_advance_phase_thread.py", + "orchestrator/tests/test_ble001_narrowing_audit.py" + ], + "reason": "Concurrency review clean (same terminal commit aa6d2b619). Tester's deliverable is the test-mechanical surface: the source-introspection retargeting in test_advance_phase_thread.py and the test_ble001 reformat. Verified the thread-safety guards still validate real structure, not vacuously: stripping the `_pkg.` decomposition prefix RESTORES the pre-split structural regexes (moved _run_pipeline body references barrel globals as `_pkg.<name>`), and repointing to `_start_pipeline_body` (decision-8 thin @route wrapper) resolves the real body carrying `TEST_MARKER: recover_advance_clear`; the auto-advance marker resolves in `_run_pipeline`. Concurrency invariants guarded and green: 15/15 advance-phase-thread (auto-advance thread respawn, recover-pipeline concurrent-state clearing, post-BRC error swallowing keeping the driver thread alive), 82 consensus-polling + slice-run-loop, 37/37 concurrent-status in isolation. No concurrency regression from the test changes.", + "ack_version": 1 + }, + "version": 1, + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:40.652943+00:00", + "phase": "implement" + }, + { + "id": "d9c792c9-6472-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code", + "to_role": "tester", + "message_type": "CONSENSUS_ACK", + "subject": "ACK from reviewer_code for tester", + "body": "Tester's terminal proposal at aa6d2b619 verified this session (same SHA, unchanged tree). The tester deliverable \u2014 the source-introspection seam fixes in test_advance_phase_thread.py (strip the _pkg. decomposition prefix on _run_pipeline; introspect the real _start_pipeline_body vs the thin @route wrapper for the recover_advance_clear marker) plus the cosmetic ruff-format line in test_ble001_narrowing_audit.py \u2014 is correct: both files pass 19/19 under pytest. The fixes accurately track the _run_pipeline move into its own submodule and the decision-8 route-wrapper split. check-file-sizes.py exits 0 with the allowlist files: map EMPTY, and all pipelines/ modules resolve their seams. Pure test-mechanical follow-through on a behavior-preserving refactor.", + "metadata": { + "payload": { + "artifact_references": [ + "orchestrator/tests/test_advance_phase_thread.py", + "orchestrator/tests/test_ble001_narrowing_audit.py", + "scripts/file-size-allowlist.yaml", + "orchestrator/routes/pipelines/__init__.py", + "orchestrator/routes/pipelines/_run_pipeline.py" + ], + "reason": "Tester's terminal proposal at aa6d2b619 verified this session (same SHA, unchanged tree). The tester deliverable \u2014 the source-introspection seam fixes in test_advance_phase_thread.py (strip the _pkg. decomposition prefix on _run_pipeline; introspect the real _start_pipeline_body vs the thin @route wrapper for the recover_advance_clear marker) plus the cosmetic ruff-format line in test_ble001_narrowing_audit.py \u2014 is correct: both files pass 19/19 under pytest. The fixes accurately track the _run_pipeline move into its own submodule and the decision-8 route-wrapper split. check-file-sizes.py exits 0 with the allowlist files: map EMPTY, and all pipelines/ modules resolve their seams. Pure test-mechanical follow-through on a behavior-preserving refactor.", + "ack_version": 1, + "attestation": { + "files_reviewed": [ + "orchestrator/tests/test_advance_phase_thread.py", + "orchestrator/tests/test_ble001_narrowing_audit.py", + "scripts/file-size-allowlist.yaml", + "orchestrator/routes/pipelines/__init__.py", + "orchestrator/routes/pipelines/_run_pipeline.py" + ], + "issues_found": 0, + "checks_run": [ + "pytest test_advance_phase_thread + test_ble001 19/19", + "check-file-sizes.py exit 0", + "seam import resolution" + ], + "verdict": "clean_test_mechanical" + } + }, + "version": 1, + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:40.894404+00:00", + "phase": "implement" + }, + { + "id": "90844636-c36d-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_security", + "to_role": "tester", + "message_type": "CONSENSUS_ACK", + "subject": "ACK from reviewer_security for tester", + "body": "Security ACK (tester, terminal slice-4 proposal aa6d2b619 \u2014 same commit already security-reviewed for coder). Tester's contribution is test-execution attestation over the pure mechanical decomposition of routes/pipelines.py. The test-file changes (test_advance_phase_thread.py _pkg.-prefix stripping in source-introspection helpers; test_ble001_narrowing_audit.py format-line fix) are pure test mechanics with zero security impact: no secrets, network calls, subprocess/eval, auth changes, or trust-boundary shifts introduced. Security invariants from the coder review hold: @require_lifecycle_secret auth guards preserved 9\u21929, package imports cleanly, file-size gate green. Consistent with durable memory verdict.", + "metadata": { + "payload": { + "artifact_references": [ + "orchestrator/tests/test_advance_phase_thread.py", + "orchestrator/tests/test_ble001_narrowing_audit.py", + "scripts/file-size-allowlist.yaml" + ], + "reason": "Security ACK (tester, terminal slice-4 proposal aa6d2b619 \u2014 same commit already security-reviewed for coder). Tester's contribution is test-execution attestation over the pure mechanical decomposition of routes/pipelines.py. The test-file changes (test_advance_phase_thread.py _pkg.-prefix stripping in source-introspection helpers; test_ble001_narrowing_audit.py format-line fix) are pure test mechanics with zero security impact: no secrets, network calls, subprocess/eval, auth changes, or trust-boundary shifts introduced. Security invariants from the coder review hold: @require_lifecycle_secret auth guards preserved 9\u21929, package imports cleanly, file-size gate green. Consistent with durable memory verdict.", + "ack_version": 1 + }, + "version": 1, + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:41.060510+00:00", + "phase": "implement" + }, + { + "id": "ce768f59-488b-4e", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_concurrency", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:41.084289+00:00", + "phase": "implement" + }, + { + "id": "b739609d-7653-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:41.322938+00:00", + "phase": "implement" + }, + { + "id": "67dcaa1a-5b57-41", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_security", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:41.493118+00:00", + "phase": "implement" + }, + { + "id": "1a0ef6a5-0b93-42", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code", + "to_role": "all", + "message_type": "CONSENSUS_CONFIRMED", + "subject": "Consensus confirmed by reviewer_code", + "body": "orchestrator-side confirm (#3064 event loop)", + "metadata": { + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:42.282657+00:00", + "phase": "implement" + }, + { + "id": "a3644204-c66a-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_security", + "to_role": "all", + "message_type": "CONSENSUS_CONFIRMED", + "subject": "Consensus confirmed by reviewer_security", + "body": "orchestrator-side confirm (#3064 event loop)", + "metadata": { + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:42.285890+00:00", + "phase": "implement" + }, + { + "id": "fcb78b00-734b-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_concurrency", + "to_role": "all", + "message_type": "CONSENSUS_CONFIRMED", + "subject": "Consensus confirmed by reviewer_concurrency", + "body": "orchestrator-side confirm (#3064 event loop)", + "metadata": { + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:42.286402+00:00", + "phase": "implement" + }, + { + "id": "37558a00-f444-48", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_contract", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:44.320559+00:00", + "phase": "implement" + }, + { + "id": "ef900d26-9be1-43", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=107466 util=0.11 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:48.105079+00:00", + "phase": "implement" + }, + { + "id": "c8333935-d8ec-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code_holistic", + "to_role": "tester", + "message_type": "CONSENSUS_ACK", + "subject": "ACK from reviewer_code_holistic for tester", + "body": "Holistic ACK of the tester's slice-4 terminal proposal (same commit aa6d2b619 I verified for the coder; no new commits). The tester's core deliverable \u2014 the source-introspection test-seam fixes (task-4-6) \u2014 is sound: (1) test_advance_phase_thread.py adapts three helpers to the post-move structure by stripping the decomposition-only _pkg. prefix and introspecting the real _start_pipeline_body (the route body) instead of the thin @route start_pipeline wrapper; the structural markers (_BLOCK_MARKER auto_advance_block / recover_advance_clear, try/except regexes) still resolve and fire, so no assertion was weakened to mask a regression. (2) test_ble001_narrowing_audit.py change is a benign ruff-format normalization in the not-taken pre-split fallback branch. Verified independently: all 317 touched/seam tests pass (advance_phase_thread, ble001, start_pipeline, slice_loop_import_seam, pipelines_apply, origin_main, overseer_model, role_to_reviewer_mapping, handlers_brc); 7612 orchestrator tests collect with 0 import errors; barrel imports clean with all patch-target re-exports resolving; ruff clean on the pipelines package; terminal criterion met (allowlist files: {} empty, check-file-sizes.py exit 0). No holistic correctness or test-integrity concerns.", + "metadata": { + "payload": { + "artifact_references": [ + "orchestrator/tests/test_advance_phase_thread.py", + "orchestrator/tests/test_ble001_narrowing_audit.py", + "scripts/file-size-allowlist.yaml", + "orchestrator/routes/pipelines/__init__.py", + "orchestrator/routes/pipelines/_run_pipeline.py" + ], + "reason": "Holistic ACK of the tester's slice-4 terminal proposal (same commit aa6d2b619 I verified for the coder; no new commits). The tester's core deliverable \u2014 the source-introspection test-seam fixes (task-4-6) \u2014 is sound: (1) test_advance_phase_thread.py adapts three helpers to the post-move structure by stripping the decomposition-only _pkg. prefix and introspecting the real _start_pipeline_body (the route body) instead of the thin @route start_pipeline wrapper; the structural markers (_BLOCK_MARKER auto_advance_block / recover_advance_clear, try/except regexes) still resolve and fire, so no assertion was weakened to mask a regression. (2) test_ble001_narrowing_audit.py change is a benign ruff-format normalization in the not-taken pre-split fallback branch. Verified independently: all 317 touched/seam tests pass (advance_phase_thread, ble001, start_pipeline, slice_loop_import_seam, pipelines_apply, origin_main, overseer_model, role_to_reviewer_mapping, handlers_brc); 7612 orchestrator tests collect with 0 import errors; barrel imports clean with all patch-target re-exports resolving; ruff clean on the pipelines package; terminal criterion met (allowlist files: {} empty, check-file-sizes.py exit 0). No holistic correctness or test-integrity concerns.", + "ack_version": 1 + }, + "version": 1, + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:49.122661+00:00", + "phase": "implement" + }, + { + "id": "93f6c70d-e757-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_security", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=83081 util=0.08 cache_hit=0.98 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:49.808495+00:00", + "phase": "implement" + }, + { + "id": "8f30eb35-3b9f-4c", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code_holistic", + "to_role": "all", + "message_type": "CONSENSUS_CONFIRMED", + "subject": "Consensus confirmed by reviewer_code_holistic", + "body": "orchestrator-side confirm (#3064 event loop)", + "metadata": { + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:52.946098+00:00", + "phase": "implement" + }, + { + "id": "213d501c-dc88-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code_holistic", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=93341 util=0.09 cache_hit=0.98 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:56.141158+00:00", + "phase": "implement" + }, + { + "id": "2b8ad8b6-edbd-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_concurrency", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=84761 util=0.08 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:58:57.020960+00:00", + "phase": "implement" + }, + { + "id": "ea0d60aa-2e1c-42", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_contract", + "to_role": "tester", + "message_type": "CONSENSUS_ACK", + "subject": "ACK from reviewer_contract for tester", + "body": "Contract verification PASS for tester slice-4 terminal (aa6d2b619) \u2014 satisfies task-4-6 test-half. Tester-owned artifacts re-confirmed green: test_advance_phase_thread.py (15) + test_ble001_narrowing_audit.py (4) = 19 passed. The source-introspection seam repairs (strip _pkg. prefix; introspect _start_pipeline_body vs the thin @route wrapper; ruff-format the pre-split fallback branch) are legitimate mechanical fixes for the _run_pipeline module move \u2014 NOT behavior changes. Contract shows ZERO open coverage gaps on any slice-4 task (task-4-1..4-6). The pure-refactor no-behavior-change guarantee is exercised by the dense loop-seam coverage (337 tests: test_consensus_polling/test_brc_nack/test_concurrent_*/test_slice_run_loop_integration) that stays green at this SHA \u2014 verified in my coder review of the same commit. Terminal criterion (allowlist files: map EMPTY, check-file-sizes exit 0) holds. Documented non-regressions (2 orchestrator.-prefix PYTHONPATH collect quirks in slice-6 event_prompt tests) are pre-existing, not split-induced.", + "metadata": { + "payload": { + "artifact_references": [ + "orchestrator/tests/test_advance_phase_thread.py", + "orchestrator/tests/test_ble001_narrowing_audit.py", + "scripts/file-size-allowlist.yaml", + "orchestrator/routes/pipelines/__init__.py", + "orchestrator/routes/pipelines/_run_pipeline.py" + ], + "reason": "Contract verification PASS for tester slice-4 terminal (aa6d2b619) \u2014 satisfies task-4-6 test-half. Tester-owned artifacts re-confirmed green: test_advance_phase_thread.py (15) + test_ble001_narrowing_audit.py (4) = 19 passed. The source-introspection seam repairs (strip _pkg. prefix; introspect _start_pipeline_body vs the thin @route wrapper; ruff-format the pre-split fallback branch) are legitimate mechanical fixes for the _run_pipeline module move \u2014 NOT behavior changes. Contract shows ZERO open coverage gaps on any slice-4 task (task-4-1..4-6). The pure-refactor no-behavior-change guarantee is exercised by the dense loop-seam coverage (337 tests: test_consensus_polling/test_brc_nack/test_concurrent_*/test_slice_run_loop_integration) that stays green at this SHA \u2014 verified in my coder review of the same commit. Terminal criterion (allowlist files: map EMPTY, check-file-sizes exit 0) holds. Documented non-regressions (2 orchestrator.-prefix PYTHONPATH collect quirks in slice-6 event_prompt tests) are pre-existing, not split-induced.", + "ack_version": 1, + "attestation": { + "tasks_verified": [ + "task-4-6" + ], + "notes": "Tester test-half of task-4-6 at terminal SHA aa6d2b619: source-introspection seam repairs green (test_advance_phase_thread 15 + test_ble001_narrowing_audit 4 = 19 passed); zero coverage gaps on any slice-4 task; pure-refactor no-behavior-change exercised by 337 green loop-seam tests (verified in coder review of same SHA)." + } + }, + "version": 1, + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:59:02.659340+00:00", + "phase": "implement" + }, + { + "id": "68655195-2cf6-43", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_contract", + "to_role": "all", + "message_type": "CONSENSUS_CONFIRMED", + "subject": "Consensus confirmed by reviewer_contract", + "body": "orchestrator-side confirm (#3064 event loop)", + "metadata": { + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:59:03.416808+00:00", + "phase": "implement" + }, + { + "id": "f40acde0-5b12-49", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_contract", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "context-measure occ=126821 util=0.13 cache_hit=0.99 decision=below_threshold", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T07:59:12.572826+00:00", + "phase": "implement" + }, + { + "id": "359aafe6-ec89-4b", + "pipeline_id": "issue-3312-v2", + "from_role": "tester", + "to_role": "all", + "message_type": "HEARTBEAT", + "subject": "heartbeat: WORKING", + "body": "in-tool-loop liveness (slice=slice-4)", + "metadata": { + "state": "WORKING", + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T08:00:34.820143+00:00", + "phase": "implement" + }, + { + "id": "d49535e3-447e-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "tester", + "to_role": "coder", + "message_type": "CONSENSUS_ACK", + "subject": "ACK from tester for coder", + "body": "Verified the terminal proposal. (1) Both modified test files pass: test_advance_phase_thread.py 15/15 + test_ble001_narrowing_audit.py 4/4. The source-introspection seam fixes are correct \u2014 barrel re-exports _run_pipeline, _start_pipeline_body, and start_pipeline all resolve; the .replace(\"_pkg.\",\"\") + _start_pipeline_body switch make the pre-split structural assertions match the post-move source. (2) Terminal acceptance criterion met: scripts/file-size-allowlist.yaml files map is EMPTY ({}), and check-file-sizes.py exits 0 (soft-cap warnings only, no hard-cap violations). (3) 151 split-surface seam tests pass (start_pipeline, pipelines_apply, slice_loop_import_seam, origin_main_parameterization, role-mapping, overseer_model). (4) Whole orchestrator suite collects clean: 7612 tests, 0 import errors. (5) The 4 test_compose_event_prompt.py failures are pre-existing \u2014 they reproduce identically on origin/main (4 failed/100 passed) and event_prompt.py is untouched by this branch; documented non-regression, not caused by the decomposition. Clean pure-refactor + test-mechanical terminal. ACK.", + "metadata": { + "payload": { + "artifact_references": [ + "scripts/file-size-allowlist.yaml", + "orchestrator/tests/test_advance_phase_thread.py", + "orchestrator/tests/test_ble001_narrowing_audit.py", + "orchestrator/routes/pipelines/__init__.py" + ], + "reason": "Verified the terminal proposal. (1) Both modified test files pass: test_advance_phase_thread.py 15/15 + test_ble001_narrowing_audit.py 4/4. The source-introspection seam fixes are correct \u2014 barrel re-exports _run_pipeline, _start_pipeline_body, and start_pipeline all resolve; the .replace(\"_pkg.\",\"\") + _start_pipeline_body switch make the pre-split structural assertions match the post-move source. (2) Terminal acceptance criterion met: scripts/file-size-allowlist.yaml files map is EMPTY ({}), and check-file-sizes.py exits 0 (soft-cap warnings only, no hard-cap violations). (3) 151 split-surface seam tests pass (start_pipeline, pipelines_apply, slice_loop_import_seam, origin_main_parameterization, role-mapping, overseer_model). (4) Whole orchestrator suite collects clean: 7612 tests, 0 import errors. (5) The 4 test_compose_event_prompt.py failures are pre-existing \u2014 they reproduce identically on origin/main (4 failed/100 passed) and event_prompt.py is untouched by this branch; documented non-regression, not caused by the decomposition. Clean pure-refactor + test-mechanical terminal. ACK.", + "ack_version": 1, + "attestation": { + "tests_run": 170, + "checks_passed": [ + "test_advance_phase_thread(15/15)", + "test_ble001_narrowing_audit(4/4)", + "split-surface-seam(151/151)", + "check-file-sizes=exit0", + "allowlist-empty", + "orchestrator-collect(7612,0-import-errors)" + ], + "non_regressions": [ + "test_compose_event_prompt(4 fail) is PRE-EXISTING on origin/main; event_prompt.py untouched by this branch" + ] + } + }, + "version": 1, + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T08:02:16.310156+00:00", + "phase": "implement" + }, + { + "id": "2a45377a-4fe6-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "orchestrator", + "to_role": "coder", + "message_type": "STATUS", + "subject": "Ready to confirm \u2014 all confirm preconditions satisfied", + "body": "Your proposal (version 1) is ready to confirm \u2014 all blocking reviews are clear and global confirm preconditions are met. Run `egg-orch consensus confirmed` to confirm.", + "metadata": { + "ready_to_confirm": true, + "version": 1, + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T08:02:16.314954+00:00", + "phase": "implement" + }, + { + "id": "5a6ad7c3-979f-4f", + "pipeline_id": "issue-3312-v2", + "from_role": "orchestrator", + "to_role": "tester", + "message_type": "STATUS", + "subject": "Ready to confirm \u2014 all confirm preconditions satisfied", + "body": "Your proposal (version 1) is ready to confirm \u2014 all blocking reviews are clear and global confirm preconditions are met. Run `egg-orch consensus confirmed` to confirm.", + "metadata": { + "ready_to_confirm": true, + "version": 1, + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T08:02:16.315420+00:00", + "phase": "implement" + }, + { + "id": "55bafc2a-75a9-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "coder", + "to_role": "all", + "message_type": "CONSENSUS_CONFIRMED", + "subject": "Consensus confirmed by coder", + "body": "orchestrator-side confirm (#3064 event loop)", + "metadata": { + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T08:02:21.884132+00:00", + "phase": "implement" + }, + { + "id": "3d497b50-c88c-40", + "pipeline_id": "issue-3312-v2", + "from_role": "tester", + "to_role": "all", + "message_type": "CONSENSUS_CONFIRMED", + "subject": "Consensus confirmed by tester", + "body": "orchestrator-side confirm (#3064 event loop)", + "metadata": { + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T08:02:21.887487+00:00", + "phase": "implement" + }, + { + "id": "dc9491ab-243e-4d", + "pipeline_id": "issue-3312-v2", + "from_role": "documenter", + "to_role": "all", + "message_type": "CONSENSUS_CONFIRMED", + "subject": "Consensus confirmed by documenter", + "body": "orchestrator-side confirm (#3064 event loop)", + "metadata": { + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T08:02:21.889647+00:00", + "phase": "implement" + }, + { + "id": "f3f19659-ab4f-47", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code", + "to_role": "all", + "message_type": "CONSENSUS_CONFIRMED", + "subject": "Consensus confirmed by reviewer_code", + "body": "orchestrator-side confirm (#3064 event loop)", + "metadata": { + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T08:02:21.891534+00:00", + "phase": "implement" + }, + { + "id": "6909cc0e-cf97-48", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_code_holistic", + "to_role": "all", + "message_type": "CONSENSUS_CONFIRMED", + "subject": "Consensus confirmed by reviewer_code_holistic", + "body": "orchestrator-side confirm (#3064 event loop)", + "metadata": { + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T08:02:21.893848+00:00", + "phase": "implement" + }, + { + "id": "62de0939-3b7b-47", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_contract", + "to_role": "all", + "message_type": "CONSENSUS_CONFIRMED", + "subject": "Consensus confirmed by reviewer_contract", + "body": "orchestrator-side confirm (#3064 event loop)", + "metadata": { + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T08:02:21.895861+00:00", + "phase": "implement" + }, + { + "id": "cc08f9dd-3b6a-4a", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_security", + "to_role": "all", + "message_type": "CONSENSUS_CONFIRMED", + "subject": "Consensus confirmed by reviewer_security", + "body": "orchestrator-side confirm (#3064 event loop)", + "metadata": { + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T08:02:21.897609+00:00", + "phase": "implement" + }, + { + "id": "863e082e-0606-47", + "pipeline_id": "issue-3312-v2", + "from_role": "reviewer_concurrency", + "to_role": "all", + "message_type": "CONSENSUS_CONFIRMED", + "subject": "Consensus confirmed by reviewer_concurrency", + "body": "orchestrator-side confirm (#3064 event loop)", + "metadata": { + "slice_id": "slice-4" + }, + "timestamp": "2026-07-06T08:02:21.899062+00:00", + "phase": "implement" + } +] \ No newline at end of file diff --git a/.egg-state/brc-history/issue-3312-v2-implement-slice-4.md b/.egg-state/brc-history/issue-3312-v2-implement-slice-4.md new file mode 100644 index 0000000000..66f6c488e3 --- /dev/null +++ b/.egg-state/brc-history/issue-3312-v2-implement-slice-4.md @@ -0,0 +1,9978 @@ +# BRC Consensus History — implement phase, slice-4 + +Generated: 2026-07-06T08:02:21Z +Pipeline: issue-3312-v2 +Slice: slice-4 + +### [2026-07-04T07:29:43Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 47f4435a-d820-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:29:43Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 362f36aa-9cc9-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:29:49Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 76aaa56a-05a4-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:29:51Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 0d5997bc-7552-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:30:57Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter deliverable is the orchestrator/CLAUDE.md routes/pipelines/ seam-table row (task-4-5), which must document the coder's CONCRETE landed submodule layout — same pattern as slices 1-3 where each seam table was authored against the landed split, not a guessed one. pipelines.py is still a single 30,520-line file on both the coder work branch and this branch (HEAD 711b0de43); coder producer_phase=WORKING, no CONSENSUS_PROPOSE for issue-3312-v2/slice-4 yet. Blocked on the coder landing the pipelines/ decomposition before I can author an accurate seam table. Not proposing speculative docs. + +````yaml +id: e268e790-7ca8-40 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-04T07:31:25Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=80604 util=0.08 cache_hit=0.94 decision=no_warm_session + +````yaml +id: c4881f52-67a3-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:31:30Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: ecd395c2-2afa-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:31:36Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 792b0126-3c64-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:32:18Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter still blocked on coder. BRC state unchanged: coder producer_phase=WORKING, no CONSENSUS_PROPOSE for issue-3312-v2/slice-4; pipelines.py not yet decomposed. Additionally the local worktree /home/egg/repos/egg is currently empty / re-provisioning (mount recreated 07:31, not a git repo this instant), so no tree to author the routes/pipelines/ seam table against even if I wanted to. Deliverable (orchestrator/CLAUDE.md pipelines/ seam row, task-4-5) must document the coder's CONCRETE landed submodule layout — same pattern as slices 1-3. Not proposing speculative docs; exiting cleanly. + +````yaml +id: 78e2e40c-4e55-42 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-04T07:32:27Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=86500 util=0.09 cache_hit=0.97 decision=below_threshold + +````yaml +id: 9fd47a32-0fb8-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:32:33Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 5a1509cf-53da-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:32:43Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: b9f0e6e9-b7b1-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:32:56Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 68c2f656-7514-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:33:24Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter blocked on coder (3rd consecutive spawn, no state change). coder producer_phase=WORKING, no CONSENSUS_PROPOSE for issue-3312-v2/slice-4; pipelines.py still undecomposed, HEAD 711b0de43 on coder+documenter branches. My deliverable (orchestrator/CLAUDE.md routes/pipelines/ seam row, task-4-5) documents the coder's CONCRETE landed submodule layout — cannot author until the split lands. The orchestrator keeps deriving documenter/propose, but the correct predecessor is coder/propose; documentation is strictly downstream of the split. Exiting cleanly, no speculative proposal. + +````yaml +id: 12f6135b-7f50-49 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-04T07:33:30Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=91649 util=0.09 cache_hit=0.97 decision=below_threshold + +````yaml +id: 5e453853-da82-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:35:11Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 88ea2526-a7e0-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:37:19Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 02dd03ab-cd91-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:39:27Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: e3e01da3-ce65-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:41:48Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: adb1d89c-8ad2-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:46:26Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 496dd8de-c214-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:46:44Z] coder (HEARTBEAT): heartbeat: WORKING + +slice-4 pipelines.py decomposition: task-4-2 pure-move baseline committed (ce433c299) + verified (imports clean, ruff clean, ratchet exit 0, 142/142 targeted tests; 4 test_compose_event_prompt failures proven pre-existing/environmental via stash). Baseline is local-only (direct push gateway-blocked; branch persists across events). Remaining task-4-3/4/5/6 (extraction + _run_pipeline split + drop last allowlist entry) planned in durable BRC memory for continuation; propose only when the whole slice is green. Not proposing this event (slice incomplete). + +````yaml +id: e6cc7a28-e9f6-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:47:07Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=175936 util=0.18 cache_hit=0.99 decision=no_warm_session + +````yaml +id: 54ea6acd-810a-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:47:16Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 289b8c1b-8121-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:47:25Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: e37d8bb0-f2d7-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:49:18Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=189601 util=0.19 cache_hit=0.96 decision=below_threshold + +````yaml +id: 2d65fa63-5299-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:49:25Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 83067c54-5c8f-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:49:36Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: e032f0d4-9b40-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:49:54Z] coder (HEARTBEAT): heartbeat: WORKING + +slice-4 coder still BLOCKED on the same unmounted/read-only worktree infra fault (see standing OVERSEER_ALERT d6ef622d): /home/egg/repos/egg has 0 files, writes return Permission denied, HEAD still 711b0de43 (verified baseline ce433c299 still orphaned off-branch). No code/state write is possible; no coder action can clear this. Awaiting infra repair: remount a writable tree + `git update-ref refs/heads/egg/issue-3312-v2-slice-4-coder/work ce433c299`. Not re-broadcasting full alert to avoid noise. + +````yaml +id: b0098ca8-139c-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T07:50:02Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=193252 util=0.19 cache_hit=0.99 decision=below_threshold + +````yaml +id: 466c68cc-ddbd-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T08:03:39Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 864658c8-e445-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T08:03:44Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 5a78d9ed-805f-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T08:04:25Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter blocked on coder (4th consecutive spawn, no state change). coder WORKING, no proposal; pipelines.py undecomposed at HEAD 711b0de43. Raised a medium overseer alert this round flagging the stall + empty-worktree infra symptom. Not proposing speculative docs; exiting cleanly. + +````yaml +id: 9d93d54e-79c7-44 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-04T08:04:33Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=98451 util=0.10 cache_hit=0.95 decision=below_threshold + +````yaml +id: 1573068c-9600-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T08:20:14Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 5b738d82-dd25-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T08:20:20Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 1adb48a8-99f6-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T08:20:32Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=194608 util=0.19 cache_hit=1.00 decision=below_threshold + +````yaml +id: 82efa9bf-68dd-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T08:33:42Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 77f5abc8-05ea-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T08:33:47Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 3fe94bb8-00a1-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T08:34:02Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter blocked on coder (5th spawn, no change since prior). coder WORKING, no CONSENSUS_PROPOSE; pipelines.py undecomposed at HEAD 711b0de43. Overseer alert already raised at 08:04 (stall + empty-worktree infra symptom) — not re-alerting to avoid noise. Deliverable (orchestrator/CLAUDE.md routes/pipelines/ seam row) is downstream of the coder's split; not proposing speculative docs. Exiting cleanly. + +````yaml +id: 0a37f6e8-b276-4c +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-04T08:34:08Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=101650 util=0.10 cache_hit=0.99 decision=below_threshold + +````yaml +id: ea99ae3b-36c0-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T08:50:16Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 73c3fcae-3c79-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T08:50:34Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 5abf4733-f614-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T08:50:49Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=196019 util=0.20 cache_hit=1.00 decision=below_threshold + +````yaml +id: d8ea461f-767c-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T09:03:46Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: b4d94758-8486-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T09:03:50Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: da03a679-df84-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T09:04:06Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter blocked on coder (6th spawn, no change). coder WORKING, no CONSENSUS_PROPOSE; pipelines.py undecomposed at HEAD 711b0de43. Overseer alerted at 08:04 (stall + empty-worktree infra symptom); holding on re-alert. Deliverable is downstream of the coder's split — no speculative docs. Exiting cleanly. + +````yaml +id: e3c6eff7-7ec0-41 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-04T09:04:11Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=104793 util=0.10 cache_hit=0.99 decision=below_threshold + +````yaml +id: 8f8903cb-1b79-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T09:20:21Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 49bedeca-9274-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T09:20:31Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 58255e35-c1e2-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T09:20:37Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=197538 util=0.20 cache_hit=1.00 decision=below_threshold + +````yaml +id: 43ab27cd-5c78-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T09:33:50Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 7969d8f1-3d1a-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T09:33:54Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 3462b592-a07c-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T09:34:22Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter blocked on coder (7th spawn, ~1.5h stalled, no change). coder WORKING, no proposal; pipelines.py undecomposed at HEAD 711b0de43. Escalated overseer alert to HIGH this round (persistent stall + empty-worktree infra symptom unaddressed since 08:04 medium). No speculative docs; exiting cleanly. + +````yaml +id: 5ac49061-f5a2-45 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-04T09:34:29Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=110495 util=0.11 cache_hit=0.96 decision=below_threshold + +````yaml +id: fe47d917-5f73-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T09:50:21Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 03065faf-bb5c-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T09:50:29Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: abe3db33-cb1a-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T09:51:11Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=200079 util=0.20 cache_hit=0.99 decision=below_threshold + +````yaml +id: 7387738f-ada6-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T10:03:51Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 9ced9c23-339b-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T10:03:55Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: e5aee041-8ea2-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T10:04:03Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter blocked on coder (8th spawn, no change). coder WORKING, no proposal; pipelines.py undecomposed at HEAD 711b0de43. Escalation ladder complete (medium 08:04, high 09:34) — holding on further alerts. No speculative docs; exiting cleanly. + +````yaml +id: 1774666d-f362-4d +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-04T10:04:09Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=113151 util=0.11 cache_hit=0.99 decision=below_threshold + +````yaml +id: 36c735c1-0cf1-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T10:20:25Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 9ea5823b-5172-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T10:20:34Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 333d9c17-1ff2-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T10:20:40Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=202120 util=0.20 cache_hit=1.00 decision=below_threshold + +````yaml +id: 46b3ce05-38c1-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T10:33:53Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: f963d9d7-a433-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T10:33:57Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 5bda66ea-1d25-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T10:34:04Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter blocked on coder (9th spawn, no change). coder WORKING, no proposal; pipelines.py undecomposed at HEAD 711b0de43. Escalation ladder complete (medium 08:04, high 09:34). No speculative docs; exiting cleanly. + +````yaml +id: 881787e5-6f53-47 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-04T10:34:09Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=115762 util=0.12 cache_hit=0.99 decision=below_threshold + +````yaml +id: 7d5855ce-35b7-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T10:50:27Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: c7148438-9ea9-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T10:50:33Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 1864cdec-7c7a-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T10:50:39Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=203072 util=0.20 cache_hit=1.00 decision=below_threshold + +````yaml +id: 5e0ccd9f-9572-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T22:57:30Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 06c094d4-4b17-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T22:57:30Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: c9fea317-2b54-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T22:57:37Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: a2726b3f-53d5-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T22:57:38Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 9c2d41e2-128f-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T22:59:44Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: a31a6c6c-281d-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T22:59:55Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 55b9afda-5960-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:00:02Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter cannot propose yet — my only deliverable (task-4-5) is the concrete orchestrator/CLAUDE.md `pipelines/` submodule-layout seam row, which is strictly downstream of the coder's decomposition. Grounded on the live tree (HEAD=711b0de43): orchestrator/routes/pipelines.py is STILL the 1.44MB monolith on-branch, no orchestrator/routes/pipelines/ package exists, and the coder's pure-move baseline ce433c299 is orphaned off-branch (coder tip=711b0de43). Writing a seam row now would fabricate a layout that does not exist (correctly NACKable); no_changes_needed is also false (real deliverable). Waiting on coder to land the pipelines/ decomposition on-branch; then I read the concrete layout from the tree, author the seam row, and propose. Documenter worktree is writable and healthy; blocker is upstream (coder infra faults, OVERSEER_ALERTs d6ef622d / b694dadc). Verdict recorded in durable BRC memory. + +````yaml +id: 005d0100-f267-4a +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-04T23:00:14Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=97209 util=0.10 cache_hit=0.99 decision=no_warm_session + +````yaml +id: d71d0a1c-9c43-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:00:19Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 2245e123-3396-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:00:45Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 018658be-b823-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:01:42Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: NO legal move this event. Worktree is now empty + read-only (0 files; git checkout HEAD -- . restores 0; writes to worktree AND .egg-state denied) — same unmounted-worktree infra fault the coder flagged (d6ef622d/b694dadc), now hitting documenter at base 64fa30773. Cannot write orchestrator/CLAUDE.md seam row (task-4-5), cannot update durable memory, cannot report_impasse (all write .egg-state). Independently, my deliverable is downstream of the coder's pipelines.py decomposition, which is not on-branch (coder tip=711b0de43, baseline ce433c299 orphaned). Raised OVERSEER_ALERT 72762213 with the infra-repair recipe. Alive; blocked on infra repair + coder. Exiting. + +````yaml +id: 226205ec-4a21-47 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-04T23:01:57Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=105755 util=0.11 cache_hit=0.99 decision=below_threshold + +````yaml +id: 5f5ca10d-37b5-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:02:05Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: d7bf9a4a-bc9a-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:02:17Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 33d04a28-0330-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:02:20Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 33656349-4a6c-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:02:28Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter still BLOCKED, state unchanged: worktree empty + read-only (0 files, HEAD=64fa30773, writes to worktree/.egg-state denied); pipelines.py decomposition not on-branch (coder tip=711b0de43). No legal FS move — cannot write orchestrator/CLAUDE.md seam row, update durable memory, or report_impasse. Standing OVERSEER_ALERT 72762213 holds the full infra-repair recipe (remount writable trees @64fa30773 → restore ce433c299 → coder lands decomposition → documenter writes seam row). Not re-broadcasting the full alert to avoid noise. Alive; awaiting infra repair + coder. Exiting. + +````yaml +id: f3d16428-291a-4f +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-04T23:02:36Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=109169 util=0.11 cache_hit=0.99 decision=below_threshold + +````yaml +id: 5df55ab1-e356-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:02:51Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=117069 util=0.12 cache_hit=0.99 decision=no_warm_session + +````yaml +id: c81d1b43-7171-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:02:58Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 75ffe6f8-a917-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:03:06Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 117061bc-4422-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:03:46Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=120064 util=0.12 cache_hit=0.98 decision=below_threshold + +````yaml +id: a272a689-80d8-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:03:54Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 2e0a8852-7551-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:04:00Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 7a7db4c7-c2b8-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:04:07Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=122363 util=0.12 cache_hit=0.99 decision=below_threshold + +````yaml +id: 58976c8b-7928-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:32:44Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 0ed7ae06-18fb-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:32:53Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: d947fb72-cb8d-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:33:47Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: INFRA REPAIRED — worktree is populated + writable again (durable memory refreshed). Still cannot propose: the pipelines/ decomposition is NOT on-branch (orchestrator/routes/pipelines.py is still the 1.44MB monolith; no pipelines/ package; allowlist still holds the single pipelines.py entry). My task-4-5 seam row is strictly downstream of that decomposition, so writing it now would fabricate a non-existent layout (NACKable); no_changes_needed is false (real deliverable). NOTE for coder/operator: coder branch tip is now 64fa30773 (slice-4 base) — the earlier pure-move baseline ce433c299 was NOT restored via update-ref and remains orphaned off-branch (still in object store); coder must re-land/cherry-pick it and complete the extraction + _run_pipeline split + drop the LAST allowlist entry. Infra is no longer my blocker; sole remaining dependency = coder landing the decomposition. Alive; will author the seam row and propose once it lands. Exiting. + +````yaml +id: 8dff876d-b851-40 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-04T23:33:56Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=117397 util=0.12 cache_hit=0.99 decision=below_threshold + +````yaml +id: b7c82127-58f3-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:34:16Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 2a2efc14-edb7-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:34:23Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 74c74218-65ab-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:34:29Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=125885 util=0.13 cache_hit=0.99 decision=below_threshold + +````yaml +id: 2923b950-de12-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:36:23Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: bad922f2-f14c-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:36:23Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 0598923d-bdcd-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:36:27Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: a0c0ffe0-59c8-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:36:31Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 03122658-18e4-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:36:40Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change since last event. Infra healthy (worktree writable), but the pipelines/ decomposition is still NOT on-branch — orchestrator/routes/pipelines.py remains the 1.44MB monolith, no pipelines/ package, allowlist still holds the single pipelines.py entry, coder tip=64fa30773. My task-4-5 seam row is downstream of that decomposition; cannot propose without fabricating a non-existent layout, and no_changes_needed is false. Sole dependency = coder landing the decomposition (must re-land/cherry-pick orphaned baseline ce433c299 first). Durable memory current. Alive; will author the seam row and propose once the code lands. Exiting. + +````yaml +id: 88d8b9ba-a556-43 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-04T23:36:45Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=120724 util=0.12 cache_hit=0.99 decision=below_threshold + +````yaml +id: fc6a52ae-8525-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:38:36Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: aef698ce-ff39-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:40:50Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 1813ded1-911f-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:43:17Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: bb757af9-d9d6-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:45:32Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: e2086c3e-762f-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-04T23:45:54Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=215534 util=0.22 cache_hit=1.00 decision=below_threshold + +````yaml +id: 8b263ded-70e6-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T00:06:58Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: f3219c5b-60a9-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T00:07:20Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: babdeaa6-c688-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T00:08:25Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: CODER PROGRESS observed. Coder branch tip advanced 64fa30773 → 0228f4a9f = "Move pipelines.py to pipelines/__init__.py (pure move baseline)" (task-4-2): pipelines.py is gone, pipelines/__init__.py exists. BUT this is only the pure-move baseline — pipelines/ holds ONLY __init__.py; the EXTRACTION (task-4-3 _run_pipeline split / task-4-4 cluster submodules) has NOT landed yet, and the coder hasn't proposed a complete slice. My task-4-5 seam row documents the concrete submodule layout, which appears only after extraction — writing it now (whole file = single __init__.py) would be premature/inaccurate and NACKable; no_changes_needed is false. Still blocked on coder, but now on the EXTRACTION step, not the move. Durable memory refreshed (worktree re-provisioning wiped the prior copy). Will author the seam row + propose once the extraction lands. Exiting. + +````yaml +id: 64b3f123-992b-4d +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T00:08:34Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=133525 util=0.13 cache_hit=0.97 decision=below_threshold + +````yaml +id: c7df0fcd-a195-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T00:16:03Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 2111b8b1-fc84-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T00:16:13Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: be6d0abb-826e-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T00:18:53Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 71976105-a8e6-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T00:20:46Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: effe37fd-31e3-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T00:20:55Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 62219cfb-7cb9-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T00:21:07Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 079e615c-b1cf-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T00:21:20Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=248824 util=0.25 cache_hit=0.99 decision=below_threshold + +````yaml +id: 54d25411-e6d6-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T00:21:28Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: a9719fb9-5faf-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T00:21:39Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: f1a64283-9a86-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T00:21:49Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=252628 util=0.25 cache_hit=0.99 decision=below_threshold + +````yaml +id: 8efa4951-a954-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T00:22:25Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: ROOT CAUSE isolated from the coder branch reflog and reported (OVERSEER_ALERT b4ca0796). The coder's pure-move baseline is committed then ORPHANED every spawn by `git reset --hard origin/egg/issue-3312-v2/slice-4` at container setup (0228f4a9f and earlier ce433c299 both orphaned; origin still @64fa30773). Coder can't push (gated behind slice-complete propose) so it can never build the extraction on its baseline → infinite restart, zero integration. This is the day-long-stall wedge; the earlier read-only-worktree fault is now REPAIRED. My task-4-5 seam row is strictly downstream — cannot propose (would fabricate a non-existent layout; no_changes_needed is false), and I cannot fix the branch reset (role/push-gated). Awaiting operator branch-persistence fix → coder extraction → then I author the orchestrator/CLAUDE.md pipelines/ seam row and propose. Durable memory refreshed. Exiting. + +````yaml +id: e0eb37f8-a59e-46 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T00:22:38Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=149631 util=0.15 cache_hit=0.99 decision=below_threshold + +````yaml +id: 6557f9dc-7f5f-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T00:51:57Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 94c38dc7-5cfd-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T00:52:02Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 33fc0f0d-fb03-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T00:52:15Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=257729 util=0.26 cache_hit=0.99 decision=below_threshold + +````yaml +id: 6d2625e0-80df-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T00:52:47Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: f9384fa8-a0c4-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T00:52:58Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 590a600d-127f-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T00:53:15Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change. coder_tip=64fa30773, origin/egg/issue-3312-v2/slice-4=64fa30773, no new coder commits, pipelines/ extraction still not integrated. The root-cause branch-persistence wedge (coder baseline orphaned by per-spawn `git reset --hard origin`) is captured in standing OVERSEER_ALERT b4ca0796 with the fix recipe; awaiting operator action. My task-4-5 seam row is downstream — cannot propose (would fabricate a non-existent layout; no_changes_needed false) and cannot fix the branch reset (role/push-gated). Not re-raising the alert to avoid noise. Alive; will author the seam row + propose once the extraction integrates. Exiting. + +````yaml +id: 502d7ef0-385c-42 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T00:53:23Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=153894 util=0.15 cache_hit=0.99 decision=below_threshold + +````yaml +id: f1ea83a4-42d1-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T01:22:00Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: abe6c581-0f3f-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T01:22:05Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 7a022e9f-2608-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T01:22:10Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=261056 util=0.26 cache_hit=0.99 decision=below_threshold + +````yaml +id: 93207bc1-72a3-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T01:22:51Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 3a752618-98e8-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T01:23:00Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 18aedb7b-53e1-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T01:23:15Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (HEAD=coder_tip=origin/slice-4=64fa30773; no pipelines/ extraction integrated). Stable deadlock on the branch-persistence wedge — coder's baseline orphaned by per-spawn `git reset --hard origin`; fix recipe in standing OVERSEER_ALERT b4ca0796, awaiting operator. My task-4-5 seam row is downstream; no legal producer move (can't fabricate a non-existent layout; no_changes_needed false; can't fix the reset — role/push-gated). Alive; will author the seam row + propose once the coder extraction integrates. Exiting. + +````yaml +id: 7bf937b9-5525-41 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T01:23:21Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=157457 util=0.16 cache_hit=0.99 decision=below_threshold + +````yaml +id: 4df99b0e-3088-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T01:52:01Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: dee3c345-e1ec-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T01:52:06Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 4fb20364-716c-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T01:52:13Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=264428 util=0.26 cache_hit=0.99 decision=below_threshold + +````yaml +id: 72cc682f-0e1d-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T01:52:53Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 128a19be-bd93-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T01:53:02Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: df536d0d-db89-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T01:53:15Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERT b4ca0796, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose the seam row once the coder extraction integrates. Exiting. + +````yaml +id: 49d3aa11-b9ec-44 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T01:53:21Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=160565 util=0.16 cache_hit=0.99 decision=below_threshold + +````yaml +id: bfc4d455-de29-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T02:22:04Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 8e6d0f12-b01b-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T02:22:09Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 35cf7fd9-bb4f-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T02:22:19Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=267651 util=0.27 cache_hit=0.99 decision=below_threshold + +````yaml +id: d827efe9-e20f-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T02:22:56Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: f1e02f60-834c-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T02:23:05Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 5a884e3e-d871-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T02:23:13Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERT b4ca0796, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: fa06b444-551a-4f +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T02:23:18Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=163444 util=0.16 cache_hit=0.99 decision=below_threshold + +````yaml +id: 2aacd2c4-72af-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T02:52:07Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 9919a32d-36b9-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T02:52:12Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 3799040a-efdd-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T02:52:16Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=271174 util=0.27 cache_hit=0.99 decision=below_threshold + +````yaml +id: 2b7a081c-c70f-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T02:52:58Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 5e41ac4e-b027-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T02:53:05Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 78494e01-2326-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T02:53:17Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERT b4ca0796, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: a5841912-bb9b-4a +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T02:53:22Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=166243 util=0.17 cache_hit=0.99 decision=below_threshold + +````yaml +id: 0a6d85b5-8634-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T03:22:10Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 755dbd7d-9bd3-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T03:22:14Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: badf6ea3-ecb9-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T03:22:21Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=274364 util=0.27 cache_hit=0.99 decision=below_threshold + +````yaml +id: b1f5b008-4ea0-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T03:23:01Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: cac669e3-8162-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T03:23:08Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 9a0838dc-870d-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T03:23:14Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERT b4ca0796, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: cad764ac-d245-4b +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T03:23:19Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=168762 util=0.17 cache_hit=0.99 decision=below_threshold + +````yaml +id: a3f0382c-2954-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T03:52:13Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 60bb90fe-67b2-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T03:52:22Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: acfcb3ed-2246-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T03:52:26Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=277688 util=0.28 cache_hit=0.99 decision=below_threshold + +````yaml +id: d1f2ff87-a01e-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T03:53:06Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 1f99ce83-9ac8-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T03:53:13Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 4dc0781d-1efa-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T03:53:20Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERT b4ca0796, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: fbfc6246-e90d-43 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T03:53:24Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=171182 util=0.17 cache_hit=1.00 decision=below_threshold + +````yaml +id: ca09c002-52ff-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T04:22:16Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: a4e1d36f-c7fb-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T04:22:21Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 33f8740c-a4da-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T04:22:43Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=280978 util=0.28 cache_hit=0.99 decision=below_threshold + +````yaml +id: f8566b65-66fe-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T04:23:09Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 34c96d54-fba8-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T04:23:18Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 68396e64-a3d7-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T04:23:25Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERT b4ca0796, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: d34c36be-5b0a-42 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T04:23:30Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=173784 util=0.17 cache_hit=0.99 decision=below_threshold + +````yaml +id: 5d57d0a8-b00c-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T04:52:19Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 8e437634-decd-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T04:52:24Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 535493b4-6044-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T04:52:28Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=285326 util=0.29 cache_hit=0.99 decision=below_threshold + +````yaml +id: 98990f84-8234-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T04:53:11Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 0afd3df0-7877-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T04:53:17Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 87281758-b865-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T04:53:23Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERT b4ca0796, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: 2bf879f7-5734-44 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T04:53:28Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=176165 util=0.18 cache_hit=1.00 decision=below_threshold + +````yaml +id: 97399cf9-88ba-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T05:22:22Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 86451b3e-1923-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T05:22:27Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 7255e804-263a-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T05:23:01Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=288486 util=0.29 cache_hit=0.99 decision=below_threshold + +````yaml +id: 5951300d-c2fc-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T05:23:13Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: e5c57edd-33dd-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T05:23:20Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 592b4d56-b3d2-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T05:23:26Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERT b4ca0796, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: c9428aa8-451a-4f +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T05:23:32Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=178541 util=0.18 cache_hit=1.00 decision=below_threshold + +````yaml +id: b210d5cc-7735-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T05:52:24Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: f7bef729-a06c-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T05:52:29Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 09d84989-4143-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T05:53:07Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=293493 util=0.29 cache_hit=0.99 decision=below_threshold + +````yaml +id: 9f8783db-9642-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T05:53:15Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 7c45f450-6482-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T05:53:22Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 0daf6351-ab23-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T05:53:28Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERT b4ca0796, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: b7f460d6-1f34-41 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T05:53:33Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=181052 util=0.18 cache_hit=1.00 decision=below_threshold + +````yaml +id: ee8ce83e-a765-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T06:22:27Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: f668bc5e-98d0-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T06:22:32Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 0cae3ded-3c6a-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T06:22:58Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=298874 util=0.30 cache_hit=0.99 decision=below_threshold + +````yaml +id: 9a11c724-291e-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T06:23:19Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 1f9b7fa8-01c0-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T06:23:33Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 189d1dca-edf8-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T06:24:04Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction). Deadlock ~5.5h unaddressed; re-surfaced the root-cause branch-persistence alert (a3b4d28a, ref b4ca0796) with the operator fix recipe. Task-4-5 seam row downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: 269066c4-5544-47 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T06:24:11Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=187198 util=0.19 cache_hit=1.00 decision=below_threshold + +````yaml +id: 429294e6-7f31-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T06:52:27Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 014f76de-6d34-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T06:52:33Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 61ba6cfe-57a8-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T06:52:53Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=303452 util=0.30 cache_hit=0.99 decision=below_threshold + +````yaml +id: 856ce034-b04c-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T06:53:19Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: c82af57d-6f0b-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T06:53:26Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 7757bbf5-c28b-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T06:53:32Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: cdf0c1f7-72ea-4e +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T06:53:39Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=189883 util=0.19 cache_hit=1.00 decision=below_threshold + +````yaml +id: 269838c6-8610-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T07:22:31Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 8f268a96-49a3-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T07:22:40Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 72f17fc2-5133-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T07:23:04Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=307619 util=0.31 cache_hit=0.99 decision=below_threshold + +````yaml +id: 7deee7a6-d946-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T07:23:22Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 093caa45-2da7-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T07:23:28Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 1c587993-94c6-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T07:23:39Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: 9c7a5c16-ea3d-4e +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T07:23:43Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=192299 util=0.19 cache_hit=1.00 decision=below_threshold + +````yaml +id: 41237b8c-2e54-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T07:52:34Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 76fe9b23-fea3-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T07:52:40Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 0b8cfc9f-133f-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T07:53:01Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=312014 util=0.31 cache_hit=0.99 decision=below_threshold + +````yaml +id: 329a777c-365f-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T07:53:26Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: effbd66e-6245-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T07:53:34Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 8606be4a-302d-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T07:53:42Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: a41ce9e1-69cd-4a +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T07:53:46Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=194697 util=0.19 cache_hit=1.00 decision=below_threshold + +````yaml +id: 8f04f605-0c72-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T08:22:38Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: c69a88ef-e6c2-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T08:22:42Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 90669dc7-048d-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T08:22:53Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=316283 util=0.32 cache_hit=0.99 decision=below_threshold + +````yaml +id: 552204bc-11f6-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T08:23:28Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: b25645cc-640d-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T08:23:37Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: b8900d70-e052-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T08:23:43Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: a73d5e4d-9711-47 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T08:23:49Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=197222 util=0.20 cache_hit=1.00 decision=below_threshold + +````yaml +id: ec590891-6196-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T08:52:39Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 07f73b25-7c19-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T08:52:46Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 2a2a4945-889c-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T08:53:17Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=319444 util=0.32 cache_hit=0.99 decision=below_threshold + +````yaml +id: b3e5f5ec-c374-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T08:53:32Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 46104acd-b821-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T08:53:39Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 434fdab0-3c65-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T08:53:45Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: ec35c25c-1900-4b +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T08:53:50Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=199618 util=0.20 cache_hit=1.00 decision=below_threshold + +````yaml +id: 761dd245-1c60-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T09:22:44Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: b61e7f7c-3906-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T09:22:50Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 91d70c3e-ae35-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T09:22:53Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=324327 util=0.32 cache_hit=0.99 decision=below_threshold + +````yaml +id: 4813a4f0-8c5b-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T09:23:36Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: fddfde26-581f-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T09:23:42Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: c6d3cd1d-3fec-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T09:23:48Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: 0e4e165e-cd33-4f +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T09:23:54Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=202144 util=0.20 cache_hit=1.00 decision=below_threshold + +````yaml +id: c9d1bc20-8687-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T09:52:47Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 8b085687-0843-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T09:52:52Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 22a0a3e0-f8d4-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T09:52:56Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=327489 util=0.33 cache_hit=0.99 decision=below_threshold + +````yaml +id: 10918e42-5fd4-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T09:53:38Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 6669f69e-8081-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T09:53:45Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 9da77389-23ac-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T09:53:53Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: 34366636-fec9-43 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T09:53:58Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=204729 util=0.20 cache_hit=1.00 decision=below_threshold + +````yaml +id: 3ec5557b-d42f-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T10:22:48Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: dd381638-b70e-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T10:22:54Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 3bdba7d5-acf9-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T10:23:12Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=330636 util=0.33 cache_hit=0.99 decision=below_threshold + +````yaml +id: db87639c-cbee-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T10:23:41Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 5328b94a-70b8-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T10:23:48Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 5ff24b50-e754-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T10:23:56Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: 187a6597-c847-4e +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T10:24:01Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=207285 util=0.21 cache_hit=1.00 decision=below_threshold + +````yaml +id: 7214b210-37f6-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T10:52:52Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: de36b07f-6b6e-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T10:52:57Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 5a76126d-5651-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T10:53:01Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=334570 util=0.33 cache_hit=0.99 decision=below_threshold + +````yaml +id: e81c9a01-6d54-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T10:53:43Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 3b6eec9a-2997-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T10:53:51Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: edb41444-3377-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T10:54:00Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: 889ed6a2-d2b2-47 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T10:54:05Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=209911 util=0.21 cache_hit=1.00 decision=below_threshold + +````yaml +id: 5b510d44-f137-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T11:22:53Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: ee31aa30-9c85-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T11:22:58Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 07a921a5-2ad2-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T11:23:38Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=337717 util=0.34 cache_hit=0.99 decision=below_threshold + +````yaml +id: 86759aba-7e4b-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T11:23:46Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: b6e2e307-6671-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T11:23:54Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: c0b4d225-41ba-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T11:24:00Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: 962bb3dc-5153-40 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T11:24:06Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=212313 util=0.21 cache_hit=1.00 decision=below_threshold + +````yaml +id: 47650646-9b6f-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T11:52:57Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: e9707a7d-1fc9-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T11:53:02Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 67c78d44-51e7-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T11:53:24Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=343141 util=0.34 cache_hit=0.99 decision=below_threshold + +````yaml +id: 1897eb42-333f-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T11:53:48Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 2d5fb923-66fa-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T11:53:57Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: e64f2083-e091-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T11:54:09Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: ad14db31-b673-46 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T11:54:14Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=215184 util=0.22 cache_hit=0.99 decision=below_threshold + +````yaml +id: a7ddac3d-af94-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T12:23:00Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 441a1e22-d31f-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T12:23:07Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 70e1c52e-1864-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T12:23:10Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=347194 util=0.35 cache_hit=0.99 decision=below_threshold + +````yaml +id: 2745b73f-9284-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T12:23:50Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 466799b3-33ed-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T12:23:58Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 06f4e022-06bb-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T12:24:22Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (~12h deadlock; all refs @64fa30773; no pipelines/ extraction). Re-surfaced the root-cause branch-persistence alert (b8f628cf, ref b4ca0796/a3b4d28a) with the operator fix recipe. Task-4-5 seam row downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: d1ea9e5d-37fe-47 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T12:24:29Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=220689 util=0.22 cache_hit=1.00 decision=below_threshold + +````yaml +id: ba61a204-d088-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T12:53:04Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 7bdb2314-9cd8-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T12:53:11Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 9d5e15a7-931f-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T12:53:15Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=350408 util=0.35 cache_hit=0.99 decision=below_threshold + +````yaml +id: 8485a9b2-5f07-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T12:53:55Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: e54c1c76-1ca2-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T12:54:03Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 94c80c4c-c9e8-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T12:54:12Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a / b8f628cf, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: b6b3b69f-eadb-4c +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T12:54:17Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=223209 util=0.22 cache_hit=1.00 decision=below_threshold + +````yaml +id: 6628d766-fa73-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T13:23:05Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: bfef8fd1-9654-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T13:23:10Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: bb2f00d2-60c7-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T13:23:14Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=353555 util=0.35 cache_hit=0.99 decision=below_threshold + +````yaml +id: b3738189-ee0d-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T13:23:57Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 6d636c40-02db-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T13:24:10Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 015bef5b-e275-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T13:24:19Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a / b8f628cf, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: 5fb27226-53b3-48 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T13:24:24Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=226027 util=0.23 cache_hit=1.00 decision=below_threshold + +````yaml +id: 6275736c-6990-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T13:53:08Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: c76c444a-d51d-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T13:53:18Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: ecd4d467-cfa1-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T13:53:23Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=356811 util=0.36 cache_hit=0.99 decision=below_threshold + +````yaml +id: d664e5b4-bb6d-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T13:54:00Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 573ffb40-635b-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T13:54:07Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: ba68c9e9-e1b8-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T13:54:15Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a / b8f628cf, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: f80c0f8a-575c-43 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T13:54:22Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=228462 util=0.23 cache_hit=1.00 decision=below_threshold + +````yaml +id: f5187a99-9804-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T14:23:11Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: e9541d04-03c9-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T14:23:19Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 39f1c429-8a0f-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T14:23:24Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=360087 util=0.36 cache_hit=0.99 decision=below_threshold + +````yaml +id: 2fd19059-714f-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T14:24:02Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 949cdfeb-c181-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T14:24:09Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 7b3847b2-c214-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T14:24:15Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a / b8f628cf, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: 09dfe6fb-7b02-43 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T14:24:22Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=231027 util=0.23 cache_hit=1.00 decision=below_threshold + +````yaml +id: b34f4ac8-18a3-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T14:53:12Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 31991124-06be-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T14:53:20Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: e7790621-fab4-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T14:53:27Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=363234 util=0.36 cache_hit=0.99 decision=below_threshold + +````yaml +id: f52b3f63-b51f-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T14:54:05Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: bda4a821-2651-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T14:54:14Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: abad561c-dd10-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T14:54:22Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a / b8f628cf, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: 46f853ea-9908-4b +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T14:54:26Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=233461 util=0.23 cache_hit=1.00 decision=below_threshold + +````yaml +id: 0fd56fc7-e682-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T15:23:16Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: fa49ff75-3bd7-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T15:23:21Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 4cf4a7f9-6b6c-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T15:23:27Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=366381 util=0.37 cache_hit=0.99 decision=below_threshold + +````yaml +id: b7b08204-5c79-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T15:24:07Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: af7e634a-ba5f-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T15:24:16Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 06577c31-c0b3-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T15:24:24Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a / b8f628cf, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: cc0e34e4-6899-4d +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T15:24:30Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=236088 util=0.24 cache_hit=1.00 decision=below_threshold + +````yaml +id: e377a126-d9ac-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T15:53:18Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 577cbdd2-e634-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T15:53:26Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 3fe2c1b3-a1ad-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T15:53:30Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=369708 util=0.37 cache_hit=0.99 decision=below_threshold + +````yaml +id: bcaeed43-0bb3-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T15:54:10Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 7cde76a7-9f56-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T15:54:18Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 25ddb449-a32b-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T15:54:25Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a / b8f628cf, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: bd15da5f-5f83-4f +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T15:54:31Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=238524 util=0.24 cache_hit=1.00 decision=below_threshold + +````yaml +id: 75fef3cd-aef6-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T16:23:21Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: bc385a62-1c1e-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T16:23:32Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: d83b2bbd-1a2a-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T16:23:36Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=372855 util=0.37 cache_hit=0.99 decision=below_threshold + +````yaml +id: 3560702c-ac29-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T16:24:12Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: a75327bb-a51e-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T16:24:18Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 6e5b7172-4734-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T16:24:26Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a / b8f628cf, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: 361b084b-381d-45 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T16:24:31Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=241026 util=0.24 cache_hit=1.00 decision=below_threshold + +````yaml +id: 0705ecce-6a78-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T16:53:23Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: eb4c9f21-4f58-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T16:53:32Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 4df48a2b-d1d2-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T16:53:39Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=376131 util=0.38 cache_hit=0.99 decision=below_threshold + +````yaml +id: 09e4bfff-b155-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T16:54:14Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 006c3cbb-03e9-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T16:54:21Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 29c31897-2ae5-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T16:54:38Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a / b8f628cf, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: b7829587-6495-4a +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T16:54:45Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=243588 util=0.24 cache_hit=1.00 decision=below_threshold + +````yaml +id: 9e51aaf2-a5e0-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:23:25Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 9707ec9e-51a6-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:23:35Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 26157777-91ab-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:23:39Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=379440 util=0.38 cache_hit=0.99 decision=below_threshold + +````yaml +id: 6914ec04-7e6d-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:24:18Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: c2f7d430-8264-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:24:25Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 4d6970ff-56f4-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:24:31Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a / b8f628cf, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: 2a2404e1-35ba-4e +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T17:24:38Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=246024 util=0.25 cache_hit=1.00 decision=below_threshold + +````yaml +id: 6b92cede-fbc3-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:35:23Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: a79d12dc-c794-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:35:24Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: ff06da5d-182f-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:35:29Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 33de4de1-28f6-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:35:32Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: fa491e5a-7331-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:35:34Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=382587 util=0.38 cache_hit=0.99 decision=below_threshold + +````yaml +id: b2cdfce4-b8ed-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:35:41Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 14a72194-f644-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:35:42Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / a3b4d28a / b8f628cf, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: dd748bcb-987e-4f +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T17:35:48Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: be094132-f3ac-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:35:52Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=248761 util=0.25 cache_hit=1.00 decision=below_threshold + +````yaml +id: 45317d19-8438-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:35:53Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=385810 util=0.39 cache_hit=0.99 decision=below_threshold + +````yaml +id: 8987bc0b-ec03-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:36:02Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 257268e8-37a1-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:36:03Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 4b61bb02-9f97-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:36:06Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 69d51ddf-c5be-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:36:11Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: c98db4af-9f06-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:36:11Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=388957 util=0.39 cache_hit=0.99 decision=below_threshold + +````yaml +id: 067e3fac-bf26-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:36:33Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (~18h deadlock; all refs @64fa30773; no pipelines/ extraction). Re-surfaced the root-cause branch-persistence alert (dddae924, ref b4ca0796) with the operator fix recipe. Task-4-5 seam row downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: a0c25e1d-581e-42 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T17:36:39Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=254167 util=0.25 cache_hit=1.00 decision=below_threshold + +````yaml +id: 1857b93d-c086-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:36:49Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: c9fd3b6f-ae74-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:36:58Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 682ae240-c054-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:37:05Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / dddae924, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: ac2c63bc-2165-49 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T17:37:12Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=256639 util=0.26 cache_hit=1.00 decision=below_threshold + +````yaml +id: 56883515-2e15-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:37:27Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: f2aed5e1-8c15-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:37:28Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 69961b6a-07c4-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:37:33Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 931565a2-3480-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:37:36Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: bc1aa2ef-d7ad-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:37:44Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / dddae924, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: e07acb4d-544c-4a +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T17:37:50Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=259167 util=0.26 cache_hit=1.00 decision=below_threshold + +````yaml +id: 361340cf-2b87-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:39:43Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 5711583d-c2be-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:42:18Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 48581051-0fb9-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:44:27Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 2d4013a3-cc64-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T17:45:24Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=466064 util=0.47 cache_hit=0.99 decision=below_threshold + +````yaml +id: 7ff74740-b4bc-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T18:07:31Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 0a9d5834-f3d6-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T18:07:32Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: d3274261-4a9c-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T18:07:40Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: b19b332f-6471-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T18:07:45Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: bcadc00e-aae4-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T18:07:47Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / dddae924, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: e47e5730-e60d-42 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T18:07:54Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=261557 util=0.26 cache_hit=1.00 decision=below_threshold + +````yaml +id: 124ff2e1-ce90-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T18:09:55Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: b44f39d4-fa93-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T18:12:17Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: f597b35a-c3a8-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T18:14:28Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: c531bc71-2f6b-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T18:16:35Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 10bce558-483d-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T18:18:37Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 778c2844-2e8c-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T18:19:03Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=560036 util=0.56 cache_hit=0.99 decision=below_threshold + +````yaml +id: 0367d353-3065-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T18:37:36Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 965acc0e-5ee0-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T18:37:36Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: fba31fc1-18db-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T18:37:43Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 637860cc-8356-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T18:37:52Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / dddae924, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: 5dae9347-1931-4a +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T18:37:56Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=264076 util=0.26 cache_hit=1.00 decision=below_threshold + +````yaml +id: a0732625-823f-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T18:38:17Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: c34d9b8a-caff-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T18:40:22Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: dfcc545d-e6bb-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T18:42:32Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=625098 util=0.63 cache_hit=0.99 decision=below_threshold + +````yaml +id: 2140f133-d296-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T19:07:36Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: dccf21f4-d09a-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T19:07:36Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 683c7c40-866c-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T19:07:43Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 3bccaac7-7391-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T19:07:50Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / dddae924, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: 8e24c785-2639-46 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T19:07:55Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=266465 util=0.27 cache_hit=1.00 decision=below_threshold + +````yaml +id: 2d3c7fcc-2c50-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T19:07:55Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: e002de07-15e7-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T19:10:03Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 65e30ed4-81dd-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T19:12:05Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: ca303506-4c06-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T19:14:12Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 502a0a1c-5ade-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T19:16:12Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=704858 util=0.70 cache_hit=1.00 decision=below_threshold + +````yaml +id: 40f30a9b-b1ca-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T19:37:40Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: ca0f7bd6-58e8-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T19:37:40Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 2fff5b1b-2854-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T19:37:48Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 5afc2762-bd74-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T19:37:54Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / dddae924, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: eae7e717-13d9-4f +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T19:37:55Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 4fe04f81-f21c-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T19:37:58Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=268856 util=0.27 cache_hit=1.00 decision=below_threshold + +````yaml +id: d8a8598d-75a6-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T19:39:59Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: ce511560-47c5-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T19:42:22Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 844d85eb-d673-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T19:43:04Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=772422 util=0.77 cache_hit=1.00 decision=below_threshold + +````yaml +id: f9e1b182-f12e-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T20:07:41Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 0a01d460-e692-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T20:07:41Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 610fac13-87e7-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T20:07:48Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 2a9c52a7-804b-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T20:07:54Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / dddae924, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: a2f7a920-e457-4a +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T20:07:59Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=271376 util=0.27 cache_hit=1.00 decision=below_threshold + +````yaml +id: 81ca3088-d978-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T20:08:03Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 29f7bff2-42bc-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T20:10:04Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: c65e7a17-5ca6-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T20:11:22Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=832192 util=0.83 cache_hit=1.00 decision=below_threshold + +````yaml +id: e89b176a-3819-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T20:37:44Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 772bd119-1878-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T20:37:45Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 81012cb7-67f6-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T20:37:51Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 32535f67-21ad-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T20:37:51Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 9dd6efd0-5dd2-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T20:37:58Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (all refs @64fa30773; no pipelines/ extraction integrated). Deadlock persists on the branch-persistence wedge — fix recipe in standing OVERSEER_ALERTs b4ca0796 / dddae924, awaiting operator. Task-4-5 seam row is downstream; no legal producer move. Alive; will propose once the coder extraction integrates. Exiting. + +````yaml +id: 34c4f1c0-b0a1-41 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T20:38:04Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=273834 util=0.27 cache_hit=1.00 decision=below_threshold + +````yaml +id: d30bf925-50fb-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T20:39:54Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 93e189b4-27d4-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T20:42:15Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 561ef392-a7a5-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T20:44:28Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: ee4a3c04-469a-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T20:47:16Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: c4d552e8-aff9-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T20:50:04Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 1fdfb015-ecbb-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T20:52:15Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: c0812fb1-95c4-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T20:54:35Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: bbb5fc11-41fe-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T20:56:40Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: f9f1815f-6742-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T20:58:44Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 79425114-940d-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:00:49Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: e88cae6f-dfea-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:02:52Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 71de496e-56d1-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:04:56Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: a66eda14-dcbf-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:07:06Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 1cf95067-3a05-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:07:47Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 2469ff13-dde4-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:07:54Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 9e7cb2c8-a546-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:08:56Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: PROGRESS — branch-persistence wedge RESOLVED. Coder commits now persist across spawns; coder branch tip advanced 64fa30773 → c7f676c27 with 12 commits (pure-move baseline + __file__-path fix + 10 extraction commits creating pipelines/ with __init__.py + 11 submodules: _criteria/_drafts/_reviews/_context_pr/_brc_history/_statefiles/_worktree_sync/_alerts/_overseer/_slice_state/_drivers). BUT extraction is still IN PROGRESS and NOT integrated: work is on the coder branch only (my HEAD + origin/slice-4 still 64fa30773), and the allowlist was re-keyed to pipelines/__init__.py grandfathering the still-oversized barrel (dropped in the FINAL commit once under cap). My task-4-5 seam row documents the FINAL layout — authoring now (barrel still over cap, more commits expected) would be inaccurate/NACKable. Still blocked on coder to finish the extraction + drop the LAST allowlist entry; then I author the seam row and propose. Durable memory updated. Alive. Exiting. + +````yaml +id: 901db6f1-aa2f-48 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T21:09:07Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=285794 util=0.29 cache_hit=0.98 decision=below_threshold + +````yaml +id: 3ad35c65-7559-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:09:10Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: fe111071-4926-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:11:26Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 0c5ac636-cc0c-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:13:36Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 1dc4cd09-7385-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:15:49Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: bac4c874-b53e-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:16:30Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=296514 util=0.30 cache_hit=0.14 decision=at_or_above_threshold + +````yaml +id: 1c2621b6-3fbd-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:16:40Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: a1595d9d-2537-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:16:49Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: a73f43bc-f97f-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:19:07Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: bef64360-c07d-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:21:10Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 453c8674-69c3-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:23:11Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 3249b522-4929-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:25:14Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: e81de50c-41c1-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:27:28Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 2178e718-e644-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:29:51Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 1127c1d4-0b70-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:31:54Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 0c43c41a-513e-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:33:57Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=421714 util=0.42 cache_hit=1.00 decision=below_threshold + +````yaml +id: ac6d6e0a-d83e-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:37:51Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: f4f46eab-796e-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:38:01Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: fa8cf3ac-dff7-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:38:18Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: coder extraction ACTIVELY PROGRESSING (wedge stays resolved). Coder tip 0e7510ae4; barrel pipelines/__init__.py now 14,167 lines (from 30,520) — still well over the 1,500-line cap, allowlist entry still grandfathering it. More submodules landed (_decisions/_pod_liveness/_ledger/_populate/_prompt_review/_prompt_agent/_prompt_phase/_prompt_reviewer). NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed a complete slice. My task-4-5 seam row documents the FINAL layout — still premature. Blocked on coder to finish extraction (barrel under cap + drop the LAST allowlist entry); then I author the seam row and propose. Alive. Exiting. + +````yaml +id: 3af564d6-ec86-4b +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T21:38:27Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=290571 util=0.29 cache_hit=0.99 decision=below_threshold + +````yaml +id: 1b0d5edd-7d64-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:46:44Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 296b1a76-1e8c-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:46:54Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 6c8b3682-4552-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:49:08Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 6ef44646-50c4-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:51:33Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: f6ac7414-cd91-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:53:34Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: a9aa87c3-318e-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:56:16Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 6b088822-9fb3-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T21:58:19Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 73d6a388-7ce5-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:00:32Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: b7e779ca-02a1-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:02:33Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 9147a9c6-958d-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:02:59Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=540407 util=0.54 cache_hit=1.00 decision=below_threshold + +````yaml +id: b7d3ba57-d788-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:07:54Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: ff456deb-c276-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:08:03Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 5e8ce1aa-6c07-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:08:11Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: coder extraction still progressing well. Tip 99ca6a72b; barrel pipelines/__init__.py down to 11,853 lines (from 30,520), 29 submodules landed, coder notes "only route-bodies + 3 giants remain." Still over the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout — still premature. Blocked on coder to finish extraction (barrel under cap + drop the LAST allowlist entry); then I author the seam row and propose. Alive. Exiting. + +````yaml +id: 78a378c0-3587-48 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T22:08:18Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=293841 util=0.29 cache_hit=1.00 decision=below_threshold + +````yaml +id: 47233e30-7850-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:16:47Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: b7d3b666-fb19-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:16:58Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: a9bf5815-7ef4-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:19:14Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 29a39786-2bea-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:21:22Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: e651d39b-8236-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:23:23Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: a0ddde9d-1161-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:26:41Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: f2d34dce-f7d3-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:31:40Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 313cbbc5-360f-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:33:40Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=604444 util=0.60 cache_hit=1.00 decision=below_threshold + +````yaml +id: 80bd9f3e-271b-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:37:56Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 9f907422-27cc-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:38:04Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: fd86d4db-2516-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:38:19Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: coder extraction nearing completion. Tip 6d0e3f942; barrel pipelines/__init__.py down to 8,364 lines (from 30,520); all route bodies extracted (decision-8: @route decorators stay in barrel); coder notes "only 3 giants remain." Still over the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout — still premature until the barrel is under cap. Blocked on coder to finish + drop the LAST allowlist entry; then I author the seam row and propose. Alive. Exiting. + +````yaml +id: 947bb8a1-c6e8-44 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T22:38:25Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=296834 util=0.30 cache_hit=1.00 decision=below_threshold + +````yaml +id: fecbbabe-1060-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:46:50Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 2316b5b9-3037-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:47:02Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 37e29a85-39c0-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:49:03Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 55402420-557e-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:51:15Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: fdee66de-0c2b-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:54:10Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 3f467cd1-a182-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T22:56:14Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=693145 util=0.69 cache_hit=1.00 decision=below_threshold + +````yaml +id: 464325ed-2bde-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T23:08:00Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: dde476c9-ce73-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T23:08:08Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 8849c498-43be-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T23:08:17Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: coder extraction near-final. Tip dbee33571 (last commit a BRC-memory note: giant-split recipe ready for mechanical execution next invocation). Barrel pipelines/__init__.py still 8,364 lines — 3 giant helpers remain to split under the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout — premature until the barrel is under cap. Blocked on coder to finish the giant-split + drop the LAST allowlist entry; then I author the seam row and propose. Alive. Exiting. + +````yaml +id: 49b5ce8f-ebfb-4f +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T23:08:23Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=299981 util=0.30 cache_hit=1.00 decision=below_threshold + +````yaml +id: 1e1c95bc-ffc0-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T23:16:53Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 6f960fe3-615a-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T23:17:07Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 81d1aa66-2bdc-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T23:19:09Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 1888dad3-f563-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T23:21:38Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: abfc770b-2565-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T23:23:48Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: e5832998-e530-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T23:26:01Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 1c57ac4c-d117-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T23:28:17Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 1a10c8fc-397b-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T23:28:30Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=788405 util=0.79 cache_hit=1.00 decision=below_threshold + +````yaml +id: 2bf85b75-1bef-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T23:38:01Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 875f38c6-f40b-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T23:38:10Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 144bd546-b8a9-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T23:38:18Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: coder giant-split progressing. Tip 0d2a78a48; barrel pipelines/__init__.py down to 6,694 lines (from 30,520); giant #1 split done, giants #2/#3 remain. Still over the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout — premature until the barrel is under cap. Blocked on coder to finish giants #2/#3 + drop the LAST allowlist entry; then I author the seam row and propose. Alive. Exiting. + +````yaml +id: 154cc3d9-1bfc-45 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-05T23:38:23Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=302872 util=0.30 cache_hit=1.00 decision=below_threshold + +````yaml +id: ea1c4e5d-3342-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T23:46:55Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 01ad47a9-84ff-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T23:47:10Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 206ff74e-a045-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T23:49:48Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: bc0a5412-dd16-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T23:51:56Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: fe06a59e-a288-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-05T23:52:33Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=859804 util=0.86 cache_hit=1.00 decision=below_threshold + +````yaml +id: 1d936646-1728-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:08:04Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 932b1b10-dee2-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:08:16Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: e8d04227-8974-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:08:33Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: coder working giant #2 split. Tip 331ad4e7f (last commit a BRC-memory planning note refining the giant #2 recipe); barrel pipelines/__init__.py still 6,694 lines. Giants #2/#3 remain to bring the barrel under the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout — premature until under cap. Blocked on coder to finish + drop the LAST allowlist entry; then I author the seam row and propose. Alive. Exiting. + +````yaml +id: 3caa8ebc-3778-47 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-06T00:08:40Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=305779 util=0.31 cache_hit=1.00 decision=below_threshold + +````yaml +id: c865f5a2-ab0b-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:16:57Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 5d74061b-f34a-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:17:07Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 635513f8-54d5-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:19:18Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: c89a7c6e-ca33-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:22:13Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: fd35d1a5-235d-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:24:38Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 728dfdc9-fdfb-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:26:55Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 1472e2cb-36e9-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:28:58Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 3843e939-b898-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:31:00Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: c8d8f397-2ba2-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:33:28Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 9f93e484-2829-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:35:58Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: ba1aa72d-3716-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:38:05Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: f2a0d188-0ac3-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:38:14Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: af4129e1-8525-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:38:23Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change since last event (coder tip 331ad4e7f; barrel pipelines/__init__.py still 6,694 lines; coder mid-work on giant #2 split). Still over the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout — premature until under cap. Blocked on coder to finish giants #2/#3 + drop the LAST allowlist entry; then I author the seam row and propose. Alive. Exiting. + +````yaml +id: aa1f5429-1833-49 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-06T00:38:28Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=308749 util=0.31 cache_hit=1.00 decision=below_threshold + +````yaml +id: f67f3eaa-3507-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:38:33Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 2d749f68-707c-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:40:36Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: d0ab8998-078c-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:42:42Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=241543 util=0.24 cache_hit=0.99 decision=at_or_above_threshold + +````yaml +id: 2bba8484-7a22-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:47:00Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 4d0d2710-5c3f-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:47:14Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 70fb78f8-442d-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:49:16Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 89db5438-b559-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:51:21Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: ae07ce5a-f1df-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:54:05Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 9261cd0e-ba54-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:56:38Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 533d016d-dfdf-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T00:58:48Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: f2014278-7b66-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T01:00:50Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=344432 util=0.34 cache_hit=1.00 decision=below_threshold + +````yaml +id: 7e48a573-c114-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T01:08:10Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 1682ffaf-8891-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T01:08:19Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 63c7d46e-3300-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T01:08:27Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: coder on giant #3 (_run_pipeline split, task-4-3). Tip 6641929ab; barrel pipelines/__init__.py down to 4,529 lines (from 30,520). Still over the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout — premature until under cap. Blocked on coder to finish the _run_pipeline split + drop the LAST allowlist entry; then I author the seam row and propose. Alive. Exiting. + +````yaml +id: cc69b1dc-66fa-42 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-06T01:08:32Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=311634 util=0.31 cache_hit=1.00 decision=below_threshold + +````yaml +id: 6e3553bf-1170-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T01:17:04Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: f5979677-456f-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T01:17:17Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 724310bb-d4ee-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T01:19:19Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: b147374a-3599-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T01:21:26Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 84f94859-1efc-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T01:23:47Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 46948ac7-6b71-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T01:25:21Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=422843 util=0.42 cache_hit=1.00 decision=below_threshold + +````yaml +id: 98ea1ff9-246b-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T01:38:12Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: a4c9c5f9-6ec9-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T01:38:22Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: d43a836b-35c4-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T01:38:31Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: coder still splitting giant #3 (_run_pipeline, task-4-3). Tip adeb2f2ef; barrel pipelines/__init__.py down to 4,299 lines; _run_pipeline now 2,853L. Still over the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout — premature until under cap. Blocked on coder to finish the _run_pipeline split + drop the LAST allowlist entry; then I author the seam row and propose. Alive. Exiting. + +````yaml +id: 03dd2c1b-dd5b-4a +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-06T01:38:38Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=314686 util=0.31 cache_hit=1.00 decision=below_threshold + +````yaml +id: 6a119770-6e08-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T01:47:05Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 0dbdaa7c-6ddf-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T01:47:17Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 5bc9695c-81e6-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T01:49:22Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 17a7a21d-794f-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T01:51:20Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=482865 util=0.48 cache_hit=1.00 decision=below_threshold + +````yaml +id: 2d5f95ff-a035-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T02:08:13Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 48538e00-3008-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T02:08:22Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 4015c5a2-578d-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T02:08:33Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: coder still splitting giant #3 (_run_pipeline). Tip 596472736; barrel pipelines/__init__.py 4,244 lines; _run_pipeline now 2,797L. Still over the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout — premature until under cap. Blocked on coder to finish the _run_pipeline split + drop the LAST allowlist entry; then I author the seam row and propose. Alive. Exiting. + +````yaml +id: dee55669-fefe-48 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-06T02:08:40Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=317658 util=0.32 cache_hit=1.00 decision=below_threshold + +````yaml +id: 89d097b5-71a6-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T02:17:08Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: c5945b92-ad13-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T02:17:21Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 3ae355e8-7a5f-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T02:19:23Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 0b3cf6dd-ada5-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T02:20:11Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=535928 util=0.54 cache_hit=1.00 decision=below_threshold + +````yaml +id: c6512c28-d167-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T02:38:16Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: e0b8731b-7b28-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T02:38:25Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 335dceef-7b91-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T02:38:33Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: coder still on giant #3 (_run_pipeline). Tip 233533d86; barrel pipelines/__init__.py 4,159 lines; _run_pipeline 2,711L (all 5 setup blocks extracted; while-loop split next). Still over the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout — premature until under cap. Blocked on coder to finish the _run_pipeline split + drop the LAST allowlist entry; then I author the seam row and propose. Alive. Exiting. + +````yaml +id: a346a10a-d63f-4b +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-06T02:38:40Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=320602 util=0.32 cache_hit=1.00 decision=below_threshold + +````yaml +id: 628d0e63-9cca-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T02:47:09Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 9481fe4e-d621-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T02:47:21Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 4a5968a4-5090-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T02:50:04Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 37bb51df-81b3-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T02:51:53Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=596600 util=0.60 cache_hit=1.00 decision=below_threshold + +````yaml +id: a23f6523-78cf-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T03:08:21Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: d91f62c4-7163-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T03:08:30Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 8cbf2520-3aa3-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T03:08:39Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: coder near end of giant #3 (_run_pipeline). Tip d3c0c182c; barrel pipelines/__init__.py 4,136 lines; _run_pipeline 2,684L; coder notes only the while-loop split remains. Still over the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout — premature until under cap. Blocked on coder to finish + drop the LAST allowlist entry; then I author the seam row and propose. Alive. Exiting. + +````yaml +id: 34cefba5-caeb-4a +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-06T03:08:45Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=323647 util=0.32 cache_hit=1.00 decision=below_threshold + +````yaml +id: a0903ff7-c26c-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T03:17:10Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 9ac856c5-52b0-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T03:17:24Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: eeaf1063-5be1-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T03:19:28Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: e563e4e3-c355-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T03:21:41Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 76403aed-3b0a-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T03:23:42Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 7587fa8c-a4eb-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T03:25:20Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=669065 util=0.67 cache_hit=1.00 decision=below_threshold + +````yaml +id: 15f1b7fb-89da-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T03:38:25Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: b7bec6b1-22ca-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T03:38:36Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 621de6bc-bba8-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T03:38:45Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: coder closing on giant #3 (_run_pipeline). Tip c62d83064; barrel pipelines/__init__.py 3,445 lines; _run_pipeline 1,988L (down from 2,684; 708L HITL-gate block extracted). Still over the 1,500-line cap; allowlist entry still grandfathering; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout — premature until under cap. Blocked on coder to finish + drop the LAST allowlist entry; then I author the seam row and propose. Alive. Exiting. + +````yaml +id: 3466b765-b118-4a +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-06T03:38:51Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=326574 util=0.33 cache_hit=1.00 decision=below_threshold + +````yaml +id: d43b7a1e-acf7-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T03:47:14Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 1b6c8e72-1729-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T03:47:35Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 4f08080f-e591-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T03:49:55Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: c32e460d-e7a9-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T03:52:23Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 854f47dc-e09e-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T03:54:47Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: e4f8b667-55a4-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T03:56:59Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=747536 util=0.75 cache_hit=1.00 decision=below_threshold + +````yaml +id: a10b14a7-2fb3-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T04:08:30Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 378dc786-807c-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T04:08:41Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 445c77c0-21bf-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T04:08:50Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: coder nearly done with giant #3. Tip a5506d41a; barrel pipelines/__init__.py 3,172 lines; _run_pipeline 1,712L (~212L over cap); coder notes ONE block left then the final move + terminal allowlist drop. Still over the 1,500-line cap; allowlist entry still present; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout — premature until under cap. Blocked on coder to finish + drop the LAST allowlist entry (files: → EMPTY); then I author the seam row and propose. Alive. Exiting. + +````yaml +id: 4117c4cd-31bd-47 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-06T04:08:58Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=329707 util=0.33 cache_hit=1.00 decision=below_threshold + +````yaml +id: 91ebcdee-6f29-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T04:17:14Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 11c9f36d-e5d5-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T04:17:28Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 14663c18-a1e5-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T04:20:09Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 251eb429-1a8b-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T04:22:17Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: e63d7c4f-28bf-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T04:24:20Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 6e28314f-7a63-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T04:24:59Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=817250 util=0.82 cache_hit=1.00 decision=below_threshold + +````yaml +id: 4441e30a-3bd4-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T04:38:29Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 6edd4521-146a-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T04:38:39Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: dce23de2-9fbf-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T04:38:54Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: milestone — _run_pipeline now UNDER cap (1,467L). Tip e953a9525; but the barrel pipelines/__init__.py is still 2,932 lines (over the 1,500-line cap); coder notes only the final move+terminal+propose bite remains. Allowlist entry still present; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout — premature until the barrel is under cap. Blocked on coder to finish the final move + drop the LAST allowlist entry (files: → EMPTY); then I author the seam row and propose. Alive. Exiting. + +````yaml +id: deab6f64-b4e3-47 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-06T04:39:01Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=333073 util=0.33 cache_hit=1.00 decision=below_threshold + +````yaml +id: 32203083-ad46-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T04:47:18Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 800384c6-50bf-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T04:47:26Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: b28d20d8-12d7-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T04:49:29Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 73584fae-90a6-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T04:51:42Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: b1776cf2-af46-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T04:53:43Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 256eb04e-0ebd-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T04:56:17Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: ed5148f7-980f-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T04:58:24Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: c77b410a-63cf-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T05:00:34Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: df661424-62b9-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T05:02:50Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: fcaf3a9b-32c0-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T05:08:35Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: bd58bde3-8178-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T05:08:43Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 53fb6dcf-7ca0-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T05:08:54Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change since last event (coder tip e953a9525; barrel pipelines/__init__.py still 2,932 lines, over the 1,500-line cap; coder on the final move+terminal+propose bite). _run_pipeline is under cap (1,467L) but the barrel isn't yet. Allowlist entry still present; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout — premature until the barrel is under cap. Blocked on coder to finish + drop the LAST allowlist entry (files: → EMPTY); then I author the seam row and propose. Alive. Exiting. + +````yaml +id: 7a909b66-1b5a-41 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-06T05:08:59Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=336037 util=0.34 cache_hit=1.00 decision=below_threshold + +````yaml +id: 1a0b450e-cc56-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T05:38:36Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 5df00ad0-9eca-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T05:38:47Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 4b780632-180a-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T05:38:54Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (coder tip e953a9525; barrel pipelines/__init__.py still 2,932 lines, over cap; coder on final move+terminal+propose bite; _run_pipeline under cap at 1,467L). Allowlist entry still present; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout — premature until the barrel is under cap. Blocked on coder to finish + drop the LAST allowlist entry (files: → EMPTY); then I author the seam row and propose. Alive. Exiting. + +````yaml +id: f61b241d-809a-4d +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-06T05:38:59Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=338963 util=0.34 cache_hit=1.00 decision=below_threshold + +````yaml +id: bef4fc30-ac16-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T06:08:38Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 32358fa5-96d1-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T06:08:49Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 67b991c7-6574-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T06:08:57Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: no change (coder tip e953a9525; barrel pipelines/__init__.py still 2,932 lines, over cap; coder on final move+terminal+propose bite; _run_pipeline under cap 1,467L). Allowlist entry still present; NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row documents the FINAL layout — premature until the barrel is under cap. Blocked on coder to finish + drop the LAST allowlist entry (files: → EMPTY); then I author the seam row and propose. Alive. Exiting. + +````yaml +id: 7ff127e8-b41d-48 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-06T06:09:02Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=341806 util=0.34 cache_hit=1.00 decision=below_threshold + +````yaml +id: 60d74919-3b9a-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T06:38:40Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: c745486a-6fea-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T06:38:50Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 2dd54a7e-4105-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T06:39:57Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: extraction still progressing but INTERMITTENT reset-churn observed. Coder tip e953a9525; reflog shows a reset→64fa30773 then fast-forward back to e953a9525 that recovered most commits but ORPHANED 923a2b351 ("Split _run_pipeline while-loop → _run_phase_blocks.py; giant under cap", task-4-3/non-negotiable #7) — it's a sibling of the tip (both children of a5506d41a), reachable in the object store but off-branch. NOTE: even at 923a2b351 the barrel pipelines/__init__.py is still 2,932 lines (over the 1,500-line cap) and the allowlist entry is still present, so it is NOT the terminal commit — the final barrel-shrink + allowlist-drop still remain regardless. Fast-forward recovery is mostly working (barrel 30,520→2,932 has persisted), so this is a milder/recovering recurrence of the earlier branch-persistence wedge (b4ca0796), not a full stall — watching, not re-alerting yet. Still NOT integrated (my HEAD + origin/slice-4 still 64fa30773); coder has not proposed. My task-4-5 seam row is premature until the barrel is under cap. Blocked on coder to finish; then I author the seam row and propose. Alive. Exiting. + +````yaml +id: bd7d3399-61ed-42 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-06T06:40:08Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=353340 util=0.35 cache_hit=0.98 decision=below_threshold + +````yaml +id: de6baaec-ea9b-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:08:44Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: d11bd035-7565-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:08:57Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: d1ab8100-4a29-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:10:18Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: RECURRENCE escalated (OVERSEER_ALERT 8642ccd3, ref b4ca0796). Coder tip stuck at e953a9525 / barrel 2,932L (over cap) for ~2.5h; reflog shows per-spawn reset-to-origin bouncing the coder to base 64fa30773 (origin/slice-4 never advanced past base → coder re-does early work each spawn instead of committing the final barrel-shrink + allowlist-drop). Durable fix = push coder progress to origin/slice-4 (or fast-forward origin+branch to e953a9525/923a2b351). No agent-side move can clear it; my task-4-5 seam row is downstream. Alive; will author the seam row + propose once the extraction completes and integrates. Exiting. + +````yaml +id: 76f0797f-7d75-4e +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-06T07:10:28Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=368612 util=0.37 cache_hit=1.00 decision=below_threshold + +````yaml +id: b89cdafa-3249-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:15:12Z] coder (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 65d2ca73-3fd6-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:15:13Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 29c20049-d501-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:15:17Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 6a14435c-0b17-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:15:25Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: d7ba5591-ca9d-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:15:35Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: still stalled at coder tip e953a9525 / barrel 2,932L (over cap); no advance since last event. Root cause = per-spawn reset-to-origin bouncing the coder to base (origin/slice-4 still 64fa30773); escalated in standing OVERSEER_ALERT 8642ccd3 (ref b4ca0796) with the push-to-origin fix. Allowlist entry still present; NOT integrated; coder has not proposed. My task-4-5 seam row is downstream. Alive; will author the seam row + propose once the extraction completes and integrates. Exiting. + +````yaml +id: a686f628-5331-45 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-06T07:15:42Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=371754 util=0.37 cache_hit=1.00 decision=below_threshold + +````yaml +id: 61c97220-5c0e-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:15:52Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 42e802da-debb-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:16:02Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: c5f0cb68-a3f3-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:16:12Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: still stalled at coder tip e953a9525 / barrel 2,932L (over cap); no advance. Root cause = per-spawn reset-to-origin bouncing the coder to base (origin/slice-4 still 64fa30773); escalated in standing OVERSEER_ALERT 8642ccd3 (ref b4ca0796), awaiting operator push-to-origin fix. Allowlist entry still present; NOT integrated; coder has not proposed. My task-4-5 seam row is downstream. Alive; will author the seam row + propose once the extraction completes and integrates. Exiting. + +````yaml +id: 0c97ed41-cbb2-4a +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-06T07:16:18Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=374699 util=0.37 cache_hit=1.00 decision=below_threshold + +````yaml +id: 692fafa3-a5b1-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:16:24Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 34518009-12a5-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:16:35Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 34d91d44-4fd3-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:16:42Z] documenter (HEARTBEAT): heartbeat: WAITING_ON_ROLE + +slice-4 documenter: still stalled at coder tip e953a9525 / barrel 2,932L (over cap); no advance. Root cause = per-spawn reset-to-origin (origin/slice-4 still 64fa30773); escalated in standing OVERSEER_ALERT 8642ccd3 (ref b4ca0796), awaiting operator push-to-origin fix. Allowlist entry still present; NOT integrated; coder has not proposed. My task-4-5 seam row is downstream. Alive; will author the seam row + propose once the extraction completes and integrates. Exiting. + +````yaml +id: 7bc79cde-1f88-43 +phase: implement +metadata: + state: WAITING_ON_ROLE + waiting_on: coder + slice_id: slice-4 +```` + +### [2026-07-06T07:16:49Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=377478 util=0.38 cache_hit=1.00 decision=below_threshold + +````yaml +id: 7cd8aa46-6b3a-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:17:28Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 49e7eebb-2853-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:19:34Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 66dc83ec-d40b-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:21:36Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: b930386c-9369-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:23:54Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 8be2ea35-43ca-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:26:11Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: bf091234-16bb-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:28:34Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 9d099c8f-4d91-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:30:47Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: ec966853-636b-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:36:10Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 00841cec-3cf0-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:38:23Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 90515dc7-ab2d-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:43:30Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: c26596a5-f284-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:45:52Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 44307c28-668b-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:46:59Z] documenter (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: 3fadcc1f-6df9-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:47:09Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 43df70fd-f8ba-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:48:01Z] coder (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: b469a23b-f96a-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:48:48Z] coder (CONSENSUS_PROPOSE): Proposal from coder + +Slice-4 COMPLETE — orchestrator/routes/pipelines.py (30,520L structural outlier, over both caps) is fully decomposed into routes/pipelines/ (barrel + 46 underscore-prefixed submodules, ALL under the 1,500-line / 100KB cap). This invocation landed the final bite: moved the last giant _run_pipeline (the per-phase driver, non-negotiable #7) out of the barrel into _run_pipeline.py, dropping the barrel to 1,466L (under cap). With that, scripts/file-size-allowlist.yaml's files: map is now EMPTY — the terminal acceptance criterion of the whole #3312/#3450/#3447 file-size decomposition program: no orchestrator source file remains over the global cap. Non-negotiable #7 addressed head-on: _run_pipeline's phase-transition state machine was split across prior bites into _run_pipeline_setup, _run_pipeline_support (health closures via functools.partial), _run_phase/_run_phase_blocks (per-phase execution + plan/PENDING/IMPLEMENT advance via (pipeline, signal) returns), and _run_hitl_gate — transition ordering preserved exactly. Pure refactor: bodies byte-verbatim modulo _pkg. barrel-ref prefixing + docstring re-indent. All 16 orphaned barrel imports verified as genuine _pkg. seam refs (each >=2 submodule uses) -> retained with noqa F401, none deleted; patch("routes.pipelines._run_pipeline") + _pkg._run_pipeline resolve via barrel re-export. Fixed 3 source-introspection seam tests broken by the move (strip _pkg. prefix; introspect _start_pipeline_body vs the thin @route wrapper) + a stale ruff-format line. ROLE-BOUNDARY NOTE: the routes/pipelines/ seam subsection for orchestrator/CLAUDE.md (task-4-5's doc half) is a documenter-owned restricted path, so it is intentionally NOT in this coder proposal — the documenter authors that seam row. Coder deliverables (code split, EMPTY allowlist, test fixes) are complete. + +````yaml +id: 946ff580-846a-41 +phase: implement +metadata: + payload: + summary: "Slice-4 COMPLETE \u2014 orchestrator/routes/pipelines.py (30,520L structural\ + \ outlier, over both caps) is fully decomposed into routes/pipelines/ (barrel\ + \ + 46 underscore-prefixed submodules, ALL under the 1,500-line / 100KB cap).\ + \ This invocation landed the final bite: moved the last giant _run_pipeline\ + \ (the per-phase driver, non-negotiable #7) out of the barrel into _run_pipeline.py,\ + \ dropping the barrel to 1,466L (under cap). With that, scripts/file-size-allowlist.yaml's\ + \ files: map is now EMPTY \u2014 the terminal acceptance criterion of the whole\ + \ #3312/#3450/#3447 file-size decomposition program: no orchestrator source\ + \ file remains over the global cap. Non-negotiable #7 addressed head-on: _run_pipeline's\ + \ phase-transition state machine was split across prior bites into _run_pipeline_setup,\ + \ _run_pipeline_support (health closures via functools.partial), _run_phase/_run_phase_blocks\ + \ (per-phase execution + plan/PENDING/IMPLEMENT advance via (pipeline, signal)\ + \ returns), and _run_hitl_gate \u2014 transition ordering preserved exactly.\ + \ Pure refactor: bodies byte-verbatim modulo _pkg. barrel-ref prefixing + docstring\ + \ re-indent. All 16 orphaned barrel imports verified as genuine _pkg. seam refs\ + \ (each >=2 submodule uses) -> retained with noqa F401, none deleted; patch(\"\ + routes.pipelines._run_pipeline\") + _pkg._run_pipeline resolve via barrel re-export.\ + \ Fixed 3 source-introspection seam tests broken by the move (strip _pkg. prefix;\ + \ introspect _start_pipeline_body vs the thin @route wrapper) + a stale ruff-format\ + \ line. ROLE-BOUNDARY NOTE: the routes/pipelines/ seam subsection for orchestrator/CLAUDE.md\ + \ (task-4-5's doc half) is a documenter-owned restricted path, so it is intentionally\ + \ NOT in this coder proposal \u2014 the documenter authors that seam row. Coder\ + \ deliverables (code split, EMPTY allowlist, test fixes) are complete." + attestation: {} + artifacts: + - orchestrator/routes/pipelines/__init__.py + - orchestrator/routes/pipelines/_run_pipeline.py + - scripts/file-size-allowlist.yaml + - orchestrator/tests/test_advance_phase_thread.py + - orchestrator/tests/test_ble001_narrowing_audit.py + - aa6d2b619 + risk_considered: 'Load-bearing back-compat surface (~137 referencing files, ~64 + patch seams). De-risked: whole-repo collect-only = 16,757 tests / 0 import errors; + barrel re-exports every external symbol; patch targets + _pkg. seams verified + resolving. Pure refactor (byte-verbatim). Documented pre-existing NON-regressions + unaffected: sandbox git-init env failures (test setup) and test_concurrent_status + message-store pollution flakes (pass in isolation). CLAUDE.md doc row deferred + to documenter (restricted path); no commit in the pushed range touches it.' + commit_sha: aa6d2b619 + files_changed: + - orchestrator/routes/pipelines/__init__.py + - orchestrator/routes/pipelines/_run_pipeline.py + - scripts/file-size-allowlist.yaml + - orchestrator/tests/test_advance_phase_thread.py + - orchestrator/tests/test_ble001_narrowing_audit.py + tests_run: [] + tasks_satisfied: + - task-4-1 + - task-4-2 + - task-4-3 + - task-4-4 + - task-4-5 + - task-4-6 + no_changes_needed: false + no_changes_reason: '' + version: 1 + commit_sha: aa6d2b619 + slice_id: slice-4 +```` + +### [2026-07-06T07:48:54Z] tester (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=propose (slice=slice-4) + +````yaml +id: def64281-75cb-4a +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:48:54Z] reviewer_code (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=ack (slice=slice-4) + +````yaml +id: f025d1cc-07d6-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:48:55Z] reviewer_code_holistic (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=ack (slice=slice-4) + +````yaml +id: b2dd4417-91cd-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:48:59Z] reviewer_contract (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=ack (slice=slice-4) + +````yaml +id: 426f760a-2917-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:49:00Z] reviewer_security (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=ack (slice=slice-4) + +````yaml +id: 95f0efd2-8fee-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:49:01Z] reviewer_concurrency (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=ack (slice=slice-4) + +````yaml +id: b6f16eb7-b519-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:49:05Z] tester (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 40cdcabb-08c0-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:49:07Z] reviewer_code (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 947bee0a-81f9-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:49:09Z] reviewer_contract (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 725bdc13-2c3c-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:49:09Z] reviewer_code_holistic (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 61f84bfa-3c54-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:49:13Z] reviewer_security (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 2b8dd57d-2103-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:49:15Z] reviewer_concurrency (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 028694cf-c3dd-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:49:22Z] coder (HEARTBEAT): heartbeat: WORKING + +context-measure occ=229524 util=0.23 cache_hit=1.00 decision=at_or_above_threshold + +````yaml +id: f85ca296-0a4a-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:49:40Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: f9a60613-e57b-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:50:32Z] reviewer_code → coder (CONSENSUS_ACK): ACK from reviewer_code for coder + +Terminal slice-4 decomposition verified clean against live tree. check-file-sizes.py exits 0 with the allowlist files: map EMPTY (terminal acceptance criterion); every pipelines/ barrel+submodule is under the 1500-line/100KB hard cap (largest _run_implement.py 1496L). All seams resolve (_run_pipeline, _start_pipeline_body, start_pipeline, 4 extracted helpers). Both modified seam-test files pass 19/19 — the source-introspection fixes are correct: strip the _pkg. decomposition prefix and introspect the real _start_pipeline_body vs the thin @route wrapper. R3 Dockerfile parity holds (routes/ ships via recursive COPY line 45; no Dockerfile change needed). test_ble001 change is cosmetic ruff-format churn. Deferred orchestrator/CLAUDE.md seam subsection is a documenter-owned restricted path, correctly out of coder scope. Pure refactor, behavior-preserving. + +````yaml +id: 9b956f53-76e7-4e +phase: implement +metadata: + payload: + artifact_references: + - scripts/file-size-allowlist.yaml + - orchestrator/routes/pipelines/__init__.py + - orchestrator/routes/pipelines/_run_pipeline.py + - orchestrator/tests/test_advance_phase_thread.py + - orchestrator/tests/test_ble001_narrowing_audit.py + - orchestrator/Dockerfile + reason: "Terminal slice-4 decomposition verified clean against live tree. check-file-sizes.py\ + \ exits 0 with the allowlist files: map EMPTY (terminal acceptance criterion);\ + \ every pipelines/ barrel+submodule is under the 1500-line/100KB hard cap (largest\ + \ _run_implement.py 1496L). All seams resolve (_run_pipeline, _start_pipeline_body,\ + \ start_pipeline, 4 extracted helpers). Both modified seam-test files pass 19/19\ + \ \u2014 the source-introspection fixes are correct: strip the _pkg. decomposition\ + \ prefix and introspect the real _start_pipeline_body vs the thin @route wrapper.\ + \ R3 Dockerfile parity holds (routes/ ships via recursive COPY line 45; no Dockerfile\ + \ change needed). test_ble001 change is cosmetic ruff-format churn. Deferred\ + \ orchestrator/CLAUDE.md seam subsection is a documenter-owned restricted path,\ + \ correctly out of coder scope. Pure refactor, behavior-preserving." + ack_version: 1 + attestation: + files_reviewed: + - scripts/file-size-allowlist.yaml + - orchestrator/routes/pipelines/__init__.py + - orchestrator/routes/pipelines/_run_pipeline.py + - orchestrator/tests/test_advance_phase_thread.py + - orchestrator/tests/test_ble001_narrowing_audit.py + - orchestrator/Dockerfile + issues_found: 0 + checks_run: + - check-file-sizes.py exit 0 + - pipelines pkg all under hard cap + - seam import resolution + - pytest 19/19 + - Dockerfile recursive COPY parity + verdict: clean_pure_refactor + version: 1 + slice_id: slice-4 +```` + +### [2026-07-06T07:50:42Z] reviewer_code (HEARTBEAT): heartbeat: WORKING + +context-measure occ=86817 util=0.09 cache_hit=0.95 decision=no_warm_session + +````yaml +id: b278a1c1-9707-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:51:03Z] reviewer_concurrency → coder (CONSENSUS_ACK): ACK from reviewer_concurrency for coder + +Concurrency review clean. Terminal commit aa6d2b619 touches no production concurrency code (tests + empty allowlist + scratch delete only). The test_advance_phase_thread.py retargeting is correct: stripping the `_pkg.` decomposition prefix RESTORES the pre-split structural regexes (moved bodies reference barrel globals as `_pkg.<name>`, 166x in _run_pipeline), and repointing to `_start_pipeline_body` (decision-8 thin @route wrapper) resolves the real body carrying `TEST_MARKER: recover_advance_clear`. Verified the introspection guards do NOT pass vacuously — markers present, invariants still asserted. Whole-slice concurrency surface green: 15/15 advance-phase-thread (auto-advance thread respawn, recover-pipeline concurrent-state clearing, post-BRC error swallowing), 82 consensus-polling + slice-run-loop, 37/37 concurrent-status in isolation. _run_pipeline decomposition preserves thread semantics (verbatim bodies; continue/break->return-signal threading covered by passing thread tests). + +````yaml +id: 96ca518d-a3d7-44 +phase: implement +metadata: + payload: + artifact_references: + - orchestrator/tests/test_advance_phase_thread.py + - orchestrator/tests/test_ble001_narrowing_audit.py + - scripts/file-size-allowlist.yaml + - orchestrator/routes/pipelines/__init__.py + - orchestrator/routes/pipelines/_run_pipeline.py + - orchestrator/routes/pipelines/_routes_lifecycle.py + reason: "Concurrency review clean. Terminal commit aa6d2b619 touches no production\ + \ concurrency code (tests + empty allowlist + scratch delete only). The test_advance_phase_thread.py\ + \ retargeting is correct: stripping the `_pkg.` decomposition prefix RESTORES\ + \ the pre-split structural regexes (moved bodies reference barrel globals as\ + \ `_pkg.<name>`, 166x in _run_pipeline), and repointing to `_start_pipeline_body`\ + \ (decision-8 thin @route wrapper) resolves the real body carrying `TEST_MARKER:\ + \ recover_advance_clear`. Verified the introspection guards do NOT pass vacuously\ + \ \u2014 markers present, invariants still asserted. Whole-slice concurrency\ + \ surface green: 15/15 advance-phase-thread (auto-advance thread respawn, recover-pipeline\ + \ concurrent-state clearing, post-BRC error swallowing), 82 consensus-polling\ + \ + slice-run-loop, 37/37 concurrent-status in isolation. _run_pipeline decomposition\ + \ preserves thread semantics (verbatim bodies; continue/break->return-signal\ + \ threading covered by passing thread tests)." + ack_version: 1 + version: 1 + slice_id: slice-4 +```` + +### [2026-07-06T07:51:09Z] tester (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 04007146-5748-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:51:12Z] reviewer_concurrency (HEARTBEAT): heartbeat: WORKING + +context-measure occ=81576 util=0.08 cache_hit=0.99 decision=no_warm_session + +````yaml +id: eb3741e5-64a0-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:51:14Z] reviewer_security → coder (CONSENSUS_ACK): ACK from reviewer_security for coder + +Security ACK (terminal slice-4 proposal aa6d2b619). Pure mechanical decomposition of routes/pipelines.py giant — no behavioral change. Security-critical checks all pass: (1) @require_lifecycle_secret auth guards preserved 9→9 (origin/main had 9, decomposed package has 9: 8 barrel wrappers + 1 in _criteria.py); none dropped, decorators still gate every originally-gated route (verified barrel imports cleanly, routes resolve). (2) No new secrets/eval/exec/subprocess/shell/verify=False/http introduced — grep of +diff found only pre-existing security infra moved verbatim. (3) Trust boundary intact. (4) Terminal changes (empty allowlist = stricter policy, test source-introspection fixes, scratch-tool delete) have zero security impact. (5) check-file-sizes.py green with empty allowlist. + +````yaml +id: f2785071-c4f9-44 +phase: implement +metadata: + payload: + artifact_references: + - orchestrator/routes/pipelines/__init__.py + - orchestrator/routes/pipelines/_run_pipeline.py + - orchestrator/routes/pipelines/_criteria.py + - scripts/file-size-allowlist.yaml + - orchestrator/tests/test_advance_phase_thread.py + - orchestrator/tests/test_ble001_narrowing_audit.py + reason: "Security ACK (terminal slice-4 proposal aa6d2b619). Pure mechanical decomposition\ + \ of routes/pipelines.py giant \u2014 no behavioral change. Security-critical\ + \ checks all pass: (1) @require_lifecycle_secret auth guards preserved 9\u2192\ + 9 (origin/main had 9, decomposed package has 9: 8 barrel wrappers + 1 in _criteria.py);\ + \ none dropped, decorators still gate every originally-gated route (verified\ + \ barrel imports cleanly, routes resolve). (2) No new secrets/eval/exec/subprocess/shell/verify=False/http\ + \ introduced \u2014 grep of +diff found only pre-existing security infra moved\ + \ verbatim. (3) Trust boundary intact. (4) Terminal changes (empty allowlist\ + \ = stricter policy, test source-introspection fixes, scratch-tool delete) have\ + \ zero security impact. (5) check-file-sizes.py green with empty allowlist." + ack_version: 1 + version: 1 + slice_id: slice-4 +```` + +### [2026-07-06T07:51:14Z] reviewer_security (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: a427c50a-004a-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:51:16Z] reviewer_code_holistic → coder (CONSENSUS_ACK): ACK from reviewer_code_holistic for coder + +Holistic ACK of the slice-4 terminal (routes/pipelines.py giant decomposed into ~45 under-cap submodules + program terminal). Verified: (1) Terminal acceptance criterion met — file-size-allowlist.yaml files: map is EMPTY and check-file-sizes.py exits 0 (barrel __init__.py 1466L < 1500 hard cap; remaining entries are soft-cap warnings only). (2) Behavior-preserving: barrel imports cleanly and all patch-target re-exports resolve (_run_pipeline, _start_pipeline_body, start_pipeline, _run_concurrent_phase, _run_implement_phase_slices) so test patches on routes.pipelines.<name> keep intercepting. (3) Test-seam fixes are principled — source-introspection helpers strip the decomposition-only _pkg. prefix and introspect the real _start_pipeline_body rather than the thin @route wrapper; test_ble001 change is a benign ruff-format normalization in the not-taken pre-split fallback branch. (4) R3 Dockerfile parity holds: COPY orchestrator/routes/ ./routes/ ships the new pipelines/ subpackage recursively (no Dockerfile change needed — claim confirmed); models/ and event_loop/ have explicit COPY lines from prior slices. (5) 7612 orchestrator tests collect with 0 import errors; all 317 touched/seam tests pass (advance_phase_thread, ble001, start_pipeline, slice_loop_import_seam, pipelines_apply, origin_main, overseer_model, role_to_reviewer_mapping, handlers_brc); ruff clean on the pipelines package. (6) Scratch extraction tool deleted. The orchestrator/CLAUDE.md routes/pipelines/ seam subsection is correctly deferred to the documenter (restricted path) — a legitimate role boundary, not a code defect. No holistic correctness or architecture concerns. + +````yaml +id: baa4b67a-794c-4e +phase: implement +metadata: + payload: + artifact_references: + - scripts/file-size-allowlist.yaml + - orchestrator/routes/pipelines/__init__.py + - orchestrator/routes/pipelines/_run_pipeline.py + - orchestrator/tests/test_advance_phase_thread.py + - orchestrator/tests/test_ble001_narrowing_audit.py + - orchestrator/Dockerfile + reason: "Holistic ACK of the slice-4 terminal (routes/pipelines.py giant decomposed\ + \ into ~45 under-cap submodules + program terminal). Verified: (1) Terminal\ + \ acceptance criterion met \u2014 file-size-allowlist.yaml files: map is EMPTY\ + \ and check-file-sizes.py exits 0 (barrel __init__.py 1466L < 1500 hard cap;\ + \ remaining entries are soft-cap warnings only). (2) Behavior-preserving: barrel\ + \ imports cleanly and all patch-target re-exports resolve (_run_pipeline, _start_pipeline_body,\ + \ start_pipeline, _run_concurrent_phase, _run_implement_phase_slices) so test\ + \ patches on routes.pipelines.<name> keep intercepting. (3) Test-seam fixes\ + \ are principled \u2014 source-introspection helpers strip the decomposition-only\ + \ _pkg. prefix and introspect the real _start_pipeline_body rather than the\ + \ thin @route wrapper; test_ble001 change is a benign ruff-format normalization\ + \ in the not-taken pre-split fallback branch. (4) R3 Dockerfile parity holds:\ + \ COPY orchestrator/routes/ ./routes/ ships the new pipelines/ subpackage recursively\ + \ (no Dockerfile change needed \u2014 claim confirmed); models/ and event_loop/\ + \ have explicit COPY lines from prior slices. (5) 7612 orchestrator tests collect\ + \ with 0 import errors; all 317 touched/seam tests pass (advance_phase_thread,\ + \ ble001, start_pipeline, slice_loop_import_seam, pipelines_apply, origin_main,\ + \ overseer_model, role_to_reviewer_mapping, handlers_brc); ruff clean on the\ + \ pipelines package. (6) Scratch extraction tool deleted. The orchestrator/CLAUDE.md\ + \ routes/pipelines/ seam subsection is correctly deferred to the documenter\ + \ (restricted path) \u2014 a legitimate role boundary, not a code defect. No\ + \ holistic correctness or architecture concerns." + ack_version: 1 + version: 1 + slice_id: slice-4 +```` + +### [2026-07-06T07:51:16Z] reviewer_code_holistic (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: e2b05fe5-276f-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:51:24Z] reviewer_code_holistic (HEARTBEAT): heartbeat: WORKING + +context-measure occ=89035 util=0.09 cache_hit=0.95 decision=no_warm_session + +````yaml +id: 3541a777-6fc4-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:51:25Z] reviewer_security (HEARTBEAT): heartbeat: WORKING + +context-measure occ=79512 util=0.08 cache_hit=0.99 decision=no_warm_session + +````yaml +id: e3f5cc2a-4cf7-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:51:27Z] reviewer_contract (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 0cba55fa-5c22-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:52:26Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 2d74ffe4-3e08-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:52:59Z] reviewer_contract → coder (CONSENSUS_ACK): ACK from reviewer_contract for coder + +Contract verification PASS for coder slice-4 terminal (aa6d2b619). TERMINAL CRITERION MET: scripts/file-size-allowlist.yaml files: map is EMPTY ({}) and check-file-sizes.py exits 0 — every previously-allowlisted giant decomposed, all pipelines/ submodules under the hard cap (barrel __init__.py 1466L, _run_pipeline.py 1483L, _run_implement.py 1496L — all <=1500L/<=100KB). task-4-3: _run_pipeline decomposed into _run_pipeline.py + _run_hitl_gate/_run_phase/_run_phase_blocks/_run_pipeline_setup blocks; 337 loop-seam tests green (test_consensus_polling, test_brc_nack, test_concurrent_*, test_slice_run_loop_integration) — no transition-ordering change. task-4-1/4-4: barrel re-exports resolve (import routes.pipelines OK; patch('routes.pipelines._run_pipeline') and _start_pipeline_body intercept via barrel). R3 Dockerfile parity: COPY orchestrator/routes/ ./routes/ recursively ships pipelines/, no Dockerfile change needed. task-4-6: 144 touched/seam + 337 loop tests pass; terminal test-mechanical fixes in test_advance_phase_thread.py (strip _pkg. prefix; introspect _start_pipeline_body) and test_ble001 format line are legitimate seam repairs. The 2 collect-only errors (test_compose_event_prompt, test_brc_preamble_collapsed) are a pre-existing orchestrator.-prefix PYTHONPATH quirk in slice-6 event_prompt tests, NOT split-induced. task-4-5 doc-half (CLAUDE.md routes/pipelines/ seam section) remains pending on the documenter (restricted path, correctly deferred by coder) — verified separately against documenter's proposal, not a coder-side blocker. + +````yaml +id: 90b8046f-1fc9-46 +phase: implement +metadata: + payload: + artifact_references: + - scripts/file-size-allowlist.yaml + - orchestrator/routes/pipelines/__init__.py + - orchestrator/routes/pipelines/_run_pipeline.py + - orchestrator/tests/test_advance_phase_thread.py + - orchestrator/tests/test_ble001_narrowing_audit.py + - orchestrator/Dockerfile + reason: "Contract verification PASS for coder slice-4 terminal (aa6d2b619). TERMINAL\ + \ CRITERION MET: scripts/file-size-allowlist.yaml files: map is EMPTY ({}) and\ + \ check-file-sizes.py exits 0 \u2014 every previously-allowlisted giant decomposed,\ + \ all pipelines/ submodules under the hard cap (barrel __init__.py 1466L, _run_pipeline.py\ + \ 1483L, _run_implement.py 1496L \u2014 all <=1500L/<=100KB). task-4-3: _run_pipeline\ + \ decomposed into _run_pipeline.py + _run_hitl_gate/_run_phase/_run_phase_blocks/_run_pipeline_setup\ + \ blocks; 337 loop-seam tests green (test_consensus_polling, test_brc_nack,\ + \ test_concurrent_*, test_slice_run_loop_integration) \u2014 no transition-ordering\ + \ change. task-4-1/4-4: barrel re-exports resolve (import routes.pipelines OK;\ + \ patch('routes.pipelines._run_pipeline') and _start_pipeline_body intercept\ + \ via barrel). R3 Dockerfile parity: COPY orchestrator/routes/ ./routes/ recursively\ + \ ships pipelines/, no Dockerfile change needed. task-4-6: 144 touched/seam\ + \ + 337 loop tests pass; terminal test-mechanical fixes in test_advance_phase_thread.py\ + \ (strip _pkg. prefix; introspect _start_pipeline_body) and test_ble001 format\ + \ line are legitimate seam repairs. The 2 collect-only errors (test_compose_event_prompt,\ + \ test_brc_preamble_collapsed) are a pre-existing orchestrator.-prefix PYTHONPATH\ + \ quirk in slice-6 event_prompt tests, NOT split-induced. task-4-5 doc-half\ + \ (CLAUDE.md routes/pipelines/ seam section) remains pending on the documenter\ + \ (restricted path, correctly deferred by coder) \u2014 verified separately\ + \ against documenter's proposal, not a coder-side blocker." + ack_version: 1 + attestation: + tasks_verified: + - task-4-1 + - task-4-2 + - task-4-3 + - task-4-4 + - task-4-5 + - task-4-6 + notes: task-4-5 coder-half (allowlist EMPTY) verified; doc-half (orchestrator/CLAUDE.md + routes/pipelines/ seam) is a documenter-owned restricted path, verified against + documenter proposal not coder. + version: 1 + slice_id: slice-4 +```` + +### [2026-07-06T07:53:35Z] reviewer_contract (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 4ba0c4bd-858a-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:53:49Z] reviewer_contract (HEARTBEAT): heartbeat: WORKING + +context-measure occ=101554 util=0.10 cache_hit=0.98 decision=no_warm_session + +````yaml +id: 8229c3d9-2e60-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:53:54Z] tester (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 338d665a-3148-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:54:51Z] documenter (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: bc60573c-6b7d-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:55:47Z] documenter (CONSENSUS_PROPOSE): Proposal from documenter + +Slice-4 documenter deliverable (task-4-5): authored the concrete `routes/pipelines/` decomposition seam row in orchestrator/CLAUDE.md, rebased onto the origin slice-4 tip (aa6d2b619, the coder's terminal commit). The row documents the 46-submodule layout, the decision-8 routes-in-barrel convention (16 @pipelines_bp.route thin wrappers stay in the barrel delegating to _routes_* bodies), the _run_pipeline per-phase split (non-negotiable #7 / task-4-3: thin loop → _run_pipeline_setup/_run_phase/_run_phase_blocks/_run_implement/_run_concurrent/_run_hitl_gate), packaging-neutral recursive `COPY orchestrator/routes/`, and the terminal criterion. This is the FINAL slice of the #3312 program: pipelines.py (30,520L/1.44MB, the last allowlisted file) is decomposed into routes/pipelines/ with the barrel __init__.py at 1,466 lines (under cap) and every submodule under both caps; scripts/file-size-allowlist.yaml's files: map is now EMPTY (terminal acceptance criterion), closing #3312. check-file-sizes.py ratchet exit 0 with the empty allowlist; this documenter commit (92beb1769) changes ONLY orchestrator/CLAUDE.md, stacked directly on the origin slice-4 tip. Verified all four CLAUDE.md seam tables (orchestrator/gateway/sandbox/shared) present and current. + +````yaml +id: eba83283-38f0-4d +phase: implement +metadata: + payload: + summary: "Slice-4 documenter deliverable (task-4-5): authored the concrete `routes/pipelines/`\ + \ decomposition seam row in orchestrator/CLAUDE.md, rebased onto the origin\ + \ slice-4 tip (aa6d2b619, the coder's terminal commit). The row documents the\ + \ 46-submodule layout, the decision-8 routes-in-barrel convention (16 @pipelines_bp.route\ + \ thin wrappers stay in the barrel delegating to _routes_* bodies), the _run_pipeline\ + \ per-phase split (non-negotiable #7 / task-4-3: thin loop \u2192 _run_pipeline_setup/_run_phase/_run_phase_blocks/_run_implement/_run_concurrent/_run_hitl_gate),\ + \ packaging-neutral recursive `COPY orchestrator/routes/`, and the terminal\ + \ criterion. This is the FINAL slice of the #3312 program: pipelines.py (30,520L/1.44MB,\ + \ the last allowlisted file) is decomposed into routes/pipelines/ with the barrel\ + \ __init__.py at 1,466 lines (under cap) and every submodule under both caps;\ + \ scripts/file-size-allowlist.yaml's files: map is now EMPTY (terminal acceptance\ + \ criterion), closing #3312. check-file-sizes.py ratchet exit 0 with the empty\ + \ allowlist; this documenter commit (92beb1769) changes ONLY orchestrator/CLAUDE.md,\ + \ stacked directly on the origin slice-4 tip. Verified all four CLAUDE.md seam\ + \ tables (orchestrator/gateway/sandbox/shared) present and current." + attestation: {} + artifacts: + - orchestrator/CLAUDE.md + risk_considered: Docs-only change on documenter-owned orchestrator/CLAUDE.md (check_file_restriction + can_write=true). Rebased onto origin/egg/issue-3312-v2/slice-4 (aa6d2b619) so + the push is a clean fast-forward; diff vs origin tip is exactly one file (orchestrator/CLAUDE.md). + Seam row verified against the landed pipelines/ package (46 submodules, exact + line counts, 16 route decorators in barrel, barrel 1,466L under cap). Ratchet + exit 0 with empty allowlist; remaining check-file-sizes output is pre-existing + soft-cap advisories on unrelated shared/ files, not hard-cap failures. + commit_sha: 92beb1769 + files_changed: + - orchestrator/CLAUDE.md + tests_run: [] + tasks_satisfied: + - task-4-5 + no_changes_needed: false + no_changes_reason: '' + version: 1 + commit_sha: 92beb1769 + slice_id: slice-4 +```` + +### [2026-07-06T07:55:54Z] reviewer_code (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=ack (slice=slice-4) + +````yaml +id: 073dc69f-59bb-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:55:55Z] reviewer_contract (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=ack (slice=slice-4) + +````yaml +id: 259d20ae-e091-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:56:01Z] reviewer_code (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: dc07ff70-87aa-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:56:03Z] reviewer_contract (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 490f4e0c-cc4f-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:56:08Z] tester (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 02f67d4b-5c73-42 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:56:23Z] documenter (HEARTBEAT): heartbeat: PROPOSED + +slice-4 documenter: PROPOSED ✅ (task-4-5 delivered). Authored the orchestrator/CLAUDE.md routes/pipelines/ decomposition seam row (46-submodule layout, decision-8 routes-in-barrel, _run_pipeline per-phase split, packaging-neutral recursive COPY, terminal criterion) in commit 92beb1769 — changes ONLY orchestrator/CLAUDE.md, rebased onto the origin slice-4 terminal tip aa6d2b619 so the push fast-forwarded. mcp__brc__propose: status=proposed, version=1, reviewers=[reviewer_contract, reviewer_code]. The branch-persistence wedge (b4ca0796/8642ccd3) is resolved — the coder's completed extraction reached origin (barrel 1,466L under cap; allowlist files: EMPTY = terminal criterion, #3312 complete). Awaiting reviewer ACKs; will address any NACK on orchestrator/CLAUDE.md and re-propose, else confirm once all ACK. Exiting. + +````yaml +id: 483cae4f-69fe-40 +phase: implement +metadata: + state: PROPOSED + slice_id: slice-4 +```` + +### [2026-07-06T07:56:37Z] documenter (HEARTBEAT): heartbeat: WORKING + +context-measure occ=443673 util=0.44 cache_hit=1.00 decision=below_threshold + +````yaml +id: 20c579d8-6d38-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:56:38Z] reviewer_code → documenter (CONSENSUS_ACK): ACK from reviewer_code for documenter + +orchestrator/CLAUDE.md routes/pipelines/ seam subsection verified accurate against the landed code. Every quantitative claim checks out: "46 submodules" (47 .py files minus the barrel), exactly 16 @pipelines_bp.route decorators in the barrel (decision-8), and all spot-checked cited line counts exact (_run_implement.py 1,496 largest, _run_pipeline.py 1,483, barrel 1,466, _run_concurrent.py 1,439, _populate.py 1,460, _routes_status.py 572, _run_pipeline_setup.py 727). All 20 spot-checked submodule filenames exist. The decision-8 (route decorators in barrel), non-negotiable #7 (_run_pipeline per-phase split), packaging-neutral recursive COPY (Dockerfile:45, verified in the coder review), and terminal-criterion (allowlist files: map EMPTY, verified) prose are all correct. This is the documenter half the coder correctly deferred as a restricted path. Pure documentation, no behaviour change. + +````yaml +id: 4a17d2e4-aa05-47 +phase: implement +metadata: + payload: + artifact_references: + - orchestrator/CLAUDE.md + reason: 'orchestrator/CLAUDE.md routes/pipelines/ seam subsection verified accurate + against the landed code. Every quantitative claim checks out: "46 submodules" + (47 .py files minus the barrel), exactly 16 @pipelines_bp.route decorators in + the barrel (decision-8), and all spot-checked cited line counts exact (_run_implement.py + 1,496 largest, _run_pipeline.py 1,483, barrel 1,466, _run_concurrent.py 1,439, + _populate.py 1,460, _routes_status.py 572, _run_pipeline_setup.py 727). All + 20 spot-checked submodule filenames exist. The decision-8 (route decorators + in barrel), non-negotiable #7 (_run_pipeline per-phase split), packaging-neutral + recursive COPY (Dockerfile:45, verified in the coder review), and terminal-criterion + (allowlist files: map EMPTY, verified) prose are all correct. This is the documenter + half the coder correctly deferred as a restricted path. Pure documentation, + no behaviour change.' + ack_version: 1 + attestation: + files_reviewed: + - orchestrator/CLAUDE.md + issues_found: 0 + checks_run: + - submodule count 46+barrel=47 + - 16 @pipelines_bp.route decorators + - spot-check 7 cited line counts exact + - 20 cited submodule filenames exist + verdict: accurate_documentation + version: 1 + slice_id: slice-4 +```` + +### [2026-07-06T07:56:43Z] reviewer_code (HEARTBEAT): heartbeat: WORKING + +context-measure occ=104060 util=0.10 cache_hit=0.98 decision=below_threshold + +````yaml +id: ac41488d-2525-4d +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:56:47Z] reviewer_contract → documenter (CONSENSUS_ACK): ACK from reviewer_contract for documenter + +Contract verification PASS for documenter slice-4 (92beb1769) — completes task-4-5 doc-half. The new ### routes/pipelines/ seam section accurately documents the landed decomposition: (a) line counts EXACT vs the package (barrel __init__.py 1466, _run_implement.py 1496, _run_pipeline.py 1483, _populate.py 1460, _prompt_phase.py 1407); (b) '46 submodules' correct (47 .py files = 46 submodules + 1 barrel); (c) decision-8 route convention (16 @pipelines_bp.route decorators in barrel on thin wrappers) matches; (d) _run_pipeline per-phase split (non-negotiable #7) names the real modules _run_pipeline/_run_pipeline_setup/_run_phase/_run_phase_blocks/_run_implement/_run_concurrent/_run_hitl_gate; (e) packaging-neutral recursive COPY orchestrator/routes/ ./routes/ (Dockerfile:45) correctly stated as NO Dockerfile change; (f) terminal criterion — pipelines.py was the LAST allowlist entry, files: map now EMPTY, #3312 complete — matches the verified allowlist state. All four CLAUDE.md seam files (orchestrator/gateway/sandbox/shared) present and the orchestrator roll-up now lists routes/pipelines/ and declares the allowlist empty. Pure-documentation, no behaviour change. task-4-5 AC 'orchestrator/CLAUDE.md carries the concrete pipelines/ + _run_pipeline submodule layout' satisfied. + +````yaml +id: 8e0d0ab4-913a-46 +phase: implement +metadata: + payload: + artifact_references: + - orchestrator/CLAUDE.md + reason: "Contract verification PASS for documenter slice-4 (92beb1769) \u2014\ + \ completes task-4-5 doc-half. The new ### routes/pipelines/ seam section accurately\ + \ documents the landed decomposition: (a) line counts EXACT vs the package (barrel\ + \ __init__.py 1466, _run_implement.py 1496, _run_pipeline.py 1483, _populate.py\ + \ 1460, _prompt_phase.py 1407); (b) '46 submodules' correct (47 .py files =\ + \ 46 submodules + 1 barrel); (c) decision-8 route convention (16 @pipelines_bp.route\ + \ decorators in barrel on thin wrappers) matches; (d) _run_pipeline per-phase\ + \ split (non-negotiable #7) names the real modules _run_pipeline/_run_pipeline_setup/_run_phase/_run_phase_blocks/_run_implement/_run_concurrent/_run_hitl_gate;\ + \ (e) packaging-neutral recursive COPY orchestrator/routes/ ./routes/ (Dockerfile:45)\ + \ correctly stated as NO Dockerfile change; (f) terminal criterion \u2014 pipelines.py\ + \ was the LAST allowlist entry, files: map now EMPTY, #3312 complete \u2014\ + \ matches the verified allowlist state. All four CLAUDE.md seam files (orchestrator/gateway/sandbox/shared)\ + \ present and the orchestrator roll-up now lists routes/pipelines/ and declares\ + \ the allowlist empty. Pure-documentation, no behaviour change. task-4-5 AC\ + \ 'orchestrator/CLAUDE.md carries the concrete pipelines/ + _run_pipeline submodule\ + \ layout' satisfied." + ack_version: 1 + attestation: + tasks_verified: + - task-4-5 + notes: 'Documenter doc-half of task-4-5: orchestrator/CLAUDE.md routes/pipelines/ + decomposition seam section. Factual claims verified against the landed package + at 92beb1769.' + version: 1 + slice_id: slice-4 +```` + +### [2026-07-06T07:57:11Z] reviewer_contract (HEARTBEAT): heartbeat: WORKING + +context-measure occ=121347 util=0.12 cache_hit=0.99 decision=below_threshold + +````yaml +id: 3e72e33b-34e9-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:58:12Z] tester (CONSENSUS_PROPOSE): Proposal from tester + +Slice-4 tester verification — orchestrator/routes/pipelines.py decomposition (coder tip aa6d2b619) verified as a clean pure refactor. (1) File-size gate: check-file-sizes.py exit 0; scripts/file-size-allowlist.yaml files: map is EMPTY — the terminal acceptance criterion of the whole #3312/#3450/#3447 program (no source file over the 1,500-line/100KB hard cap; barrel __init__.py 1,466L, all 46 _*.py submodules under cap). (2) Import/re-export sanity: routes.pipelines barrel loads and re-exports resolve (e.g. _build_brc_preamble); routes.event_prompt intact. (3) Coder-fixed source-introspection seam tests pass: test_advance_phase_thread.py + test_ble001_narrowing_audit.py = 19 passed. (4) Regression net: targeted route/pipeline/phase/signal/consensus/run_ subset = 3894 passed, 10 skipped. (5) Lint: ruff check + ruff format --check clean on the pipelines package (47 files). (6) Security: bandit -r on the pipelines package = 0 High, 0 Medium, 67 Low; bodies are AST-identical to the original file so those Lows are pre-existing — zero new findings introduced by the split. The 30 failures + 4 errors in the broad run are ALL pre-existing sandbox-environmental, NONE in the slice-4 diff and NONE touching the decomposed package — proven by failure signature: `git init` returns exit 1 (gateway-blocked; agent_salvage/commit_statefiles/consensus_wrapper/contract_preserved/slice_diff_summary/slice_phase_restart clusters set up temp git repos), urllib3 Timeout>60s (network-blocked; deployment tests), and 403->200 (auth-secret env unset). Every failing test file confirmed NOT-in-diff vs merge-base; all executed (no ImportError), so barrel re-exports are intact. Two collection errors (test_brc_preamble_collapsed/test_compose_event_prompt) are an invocation artifact of running pytest from orchestrator/ with the `orchestrator.` absolute-prefix import — their symbols resolve via the routes. prefix; not a regression. No new tests authored: pure structural refactor with AST-identical bodies, so the existing suite IS the regression net; the only test edits needed (3 source-introspection seams) were landed by the coder. Matches coder attestation exactly. + +````yaml +id: e4a9a87d-0fc8-4c +phase: implement +metadata: + payload: + summary: "Slice-4 tester verification \u2014 orchestrator/routes/pipelines.py\ + \ decomposition (coder tip aa6d2b619) verified as a clean pure refactor. (1)\ + \ File-size gate: check-file-sizes.py exit 0; scripts/file-size-allowlist.yaml\ + \ files: map is EMPTY \u2014 the terminal acceptance criterion of the whole\ + \ #3312/#3450/#3447 program (no source file over the 1,500-line/100KB hard cap;\ + \ barrel __init__.py 1,466L, all 46 _*.py submodules under cap). (2) Import/re-export\ + \ sanity: routes.pipelines barrel loads and re-exports resolve (e.g. _build_brc_preamble);\ + \ routes.event_prompt intact. (3) Coder-fixed source-introspection seam tests\ + \ pass: test_advance_phase_thread.py + test_ble001_narrowing_audit.py = 19 passed.\ + \ (4) Regression net: targeted route/pipeline/phase/signal/consensus/run_ subset\ + \ = 3894 passed, 10 skipped. (5) Lint: ruff check + ruff format --check clean\ + \ on the pipelines package (47 files). (6) Security: bandit -r on the pipelines\ + \ package = 0 High, 0 Medium, 67 Low; bodies are AST-identical to the original\ + \ file so those Lows are pre-existing \u2014 zero new findings introduced by\ + \ the split. The 30 failures + 4 errors in the broad run are ALL pre-existing\ + \ sandbox-environmental, NONE in the slice-4 diff and NONE touching the decomposed\ + \ package \u2014 proven by failure signature: `git init` returns exit 1 (gateway-blocked;\ + \ agent_salvage/commit_statefiles/consensus_wrapper/contract_preserved/slice_diff_summary/slice_phase_restart\ + \ clusters set up temp git repos), urllib3 Timeout>60s (network-blocked; deployment\ + \ tests), and 403->200 (auth-secret env unset). Every failing test file confirmed\ + \ NOT-in-diff vs merge-base; all executed (no ImportError), so barrel re-exports\ + \ are intact. Two collection errors (test_brc_preamble_collapsed/test_compose_event_prompt)\ + \ are an invocation artifact of running pytest from orchestrator/ with the `orchestrator.`\ + \ absolute-prefix import \u2014 their symbols resolve via the routes. prefix;\ + \ not a regression. No new tests authored: pure structural refactor with AST-identical\ + \ bodies, so the existing suite IS the regression net; the only test edits needed\ + \ (3 source-introspection seams) were landed by the coder. Matches coder attestation\ + \ exactly." + attestation: + tests_run: 3913 + checks_passed: + - lint + - test + - security + - file-size-gate + tests_execution_blocked: false + security_scan: 'bandit -r orchestrator/routes/pipelines/: 0 High, 0 Medium, + 67 Low (all pre-existing, AST-identical bodies)' + environmental_failures_classified: true + environmental_failure_reason: 30 failed + 4 errored are all pre-existing sandbox-environmental + (git-init gateway-blocked, network-blocked, auth-secret env unset); all outside + slice-4 diff, none touch routes/pipelines/, all executed (no ImportError) + pure_refactor: true + allowlist_empty: true + artifacts: + - aa6d2b619 + - orchestrator/routes/pipelines/__init__.py + - orchestrator/routes/pipelines/_run_pipeline.py + - scripts/file-size-allowlist.yaml + - orchestrator/tests/test_advance_phase_thread.py + - orchestrator/tests/test_ble001_narrowing_audit.py + risk_considered: "Pure structural refactor of a load-bearing back-compat surface\ + \ (routes/pipelines.py -> routes/pipelines/ barrel + 46 submodules). De-risked:\ + \ barrel re-exports every external symbol (verified _build_brc_preamble + module\ + \ load); all patch-seam / source-introspection tests pass (19 seam + 3894 targeted);\ + \ file-size hard-cap gate exit 0 with allowlist now EMPTY; bandit clean (0 High/Medium,\ + \ Lows pre-existing). Distinguished environmental failures (git-init/network/auth-env,\ + \ all NOT-in-diff, all executed) from regressions by failure signature and diff-membership\ + \ \u2014 zero regressions attributable to the decomposition." + commit_sha: aa6d2b619 + files_changed: [] + tests_run: [] + tasks_satisfied: + - task-4-1 + - task-4-2 + - task-4-3 + - task-4-4 + - task-4-5 + - task-4-6 + no_changes_needed: false + no_changes_reason: '' + version: 1 + commit_sha: aa6d2b619 + slice_id: slice-4 +```` + +### [2026-07-06T07:58:12Z] orchestrator → documenter (STATUS): Ready to confirm — all confirm preconditions satisfied + +Your proposal (version 1) is ready to confirm — all blocking reviews are clear and global confirm preconditions are met. Run `egg-orch consensus confirmed` to confirm. + +````yaml +id: b65a3dfb-c38c-49 +phase: implement +metadata: + ready_to_confirm: true + version: 1 + slice_id: slice-4 +```` + +### [2026-07-06T07:58:12Z] tester (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 57b53e22-557c-47 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:58:18Z] documenter (CONSENSUS_CONFIRMED): Consensus confirmed by documenter + +orchestrator-side confirm (#3064 event loop) + +````yaml +id: 74c733e2-7920-4c +phase: implement +metadata: + slice_id: slice-4 +```` + +### [2026-07-06T07:58:19Z] tester (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=ack (slice=slice-4) + +````yaml +id: 570b22d8-2b29-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:58:19Z] reviewer_code (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=ack (slice=slice-4) + +````yaml +id: b9f1773f-8210-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:58:20Z] reviewer_code_holistic (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=ack (slice=slice-4) + +````yaml +id: f5b10347-ec3b-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:58:21Z] reviewer_contract (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=ack (slice=slice-4) + +````yaml +id: 65f57036-2394-44 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:58:21Z] reviewer_security (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=ack (slice=slice-4) + +````yaml +id: 1847193d-ec8e-40 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:58:22Z] reviewer_concurrency (HEARTBEAT): heartbeat: WORKING + +one-shot event handler action=ack (slice=slice-4) + +````yaml +id: 1e8a297b-3f8b-46 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:58:32Z] reviewer_code_holistic (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: effea01a-ce70-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:58:34Z] tester (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 8d3386e1-7843-45 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:58:40Z] reviewer_concurrency → tester (CONSENSUS_ACK): ACK from reviewer_concurrency for tester + +Concurrency review clean (same terminal commit aa6d2b619). Tester's deliverable is the test-mechanical surface: the source-introspection retargeting in test_advance_phase_thread.py and the test_ble001 reformat. Verified the thread-safety guards still validate real structure, not vacuously: stripping the `_pkg.` decomposition prefix RESTORES the pre-split structural regexes (moved _run_pipeline body references barrel globals as `_pkg.<name>`), and repointing to `_start_pipeline_body` (decision-8 thin @route wrapper) resolves the real body carrying `TEST_MARKER: recover_advance_clear`; the auto-advance marker resolves in `_run_pipeline`. Concurrency invariants guarded and green: 15/15 advance-phase-thread (auto-advance thread respawn, recover-pipeline concurrent-state clearing, post-BRC error swallowing keeping the driver thread alive), 82 consensus-polling + slice-run-loop, 37/37 concurrent-status in isolation. No concurrency regression from the test changes. + +````yaml +id: 7f2f22bd-28db-42 +phase: implement +metadata: + payload: + artifact_references: + - orchestrator/tests/test_advance_phase_thread.py + - orchestrator/tests/test_ble001_narrowing_audit.py + reason: 'Concurrency review clean (same terminal commit aa6d2b619). Tester''s + deliverable is the test-mechanical surface: the source-introspection retargeting + in test_advance_phase_thread.py and the test_ble001 reformat. Verified the thread-safety + guards still validate real structure, not vacuously: stripping the `_pkg.` decomposition + prefix RESTORES the pre-split structural regexes (moved _run_pipeline body references + barrel globals as `_pkg.<name>`), and repointing to `_start_pipeline_body` (decision-8 + thin @route wrapper) resolves the real body carrying `TEST_MARKER: recover_advance_clear`; + the auto-advance marker resolves in `_run_pipeline`. Concurrency invariants + guarded and green: 15/15 advance-phase-thread (auto-advance thread respawn, + recover-pipeline concurrent-state clearing, post-BRC error swallowing keeping + the driver thread alive), 82 consensus-polling + slice-run-loop, 37/37 concurrent-status + in isolation. No concurrency regression from the test changes.' + ack_version: 1 + version: 1 + slice_id: slice-4 +```` + +### [2026-07-06T07:58:40Z] reviewer_code → tester (CONSENSUS_ACK): ACK from reviewer_code for tester + +Tester's terminal proposal at aa6d2b619 verified this session (same SHA, unchanged tree). The tester deliverable — the source-introspection seam fixes in test_advance_phase_thread.py (strip the _pkg. decomposition prefix on _run_pipeline; introspect the real _start_pipeline_body vs the thin @route wrapper for the recover_advance_clear marker) plus the cosmetic ruff-format line in test_ble001_narrowing_audit.py — is correct: both files pass 19/19 under pytest. The fixes accurately track the _run_pipeline move into its own submodule and the decision-8 route-wrapper split. check-file-sizes.py exits 0 with the allowlist files: map EMPTY, and all pipelines/ modules resolve their seams. Pure test-mechanical follow-through on a behavior-preserving refactor. + +````yaml +id: d9c792c9-6472-4a +phase: implement +metadata: + payload: + artifact_references: + - orchestrator/tests/test_advance_phase_thread.py + - orchestrator/tests/test_ble001_narrowing_audit.py + - scripts/file-size-allowlist.yaml + - orchestrator/routes/pipelines/__init__.py + - orchestrator/routes/pipelines/_run_pipeline.py + reason: "Tester's terminal proposal at aa6d2b619 verified this session (same SHA,\ + \ unchanged tree). The tester deliverable \u2014 the source-introspection seam\ + \ fixes in test_advance_phase_thread.py (strip the _pkg. decomposition prefix\ + \ on _run_pipeline; introspect the real _start_pipeline_body vs the thin @route\ + \ wrapper for the recover_advance_clear marker) plus the cosmetic ruff-format\ + \ line in test_ble001_narrowing_audit.py \u2014 is correct: both files pass\ + \ 19/19 under pytest. The fixes accurately track the _run_pipeline move into\ + \ its own submodule and the decision-8 route-wrapper split. check-file-sizes.py\ + \ exits 0 with the allowlist files: map EMPTY, and all pipelines/ modules resolve\ + \ their seams. Pure test-mechanical follow-through on a behavior-preserving\ + \ refactor." + ack_version: 1 + attestation: + files_reviewed: + - orchestrator/tests/test_advance_phase_thread.py + - orchestrator/tests/test_ble001_narrowing_audit.py + - scripts/file-size-allowlist.yaml + - orchestrator/routes/pipelines/__init__.py + - orchestrator/routes/pipelines/_run_pipeline.py + issues_found: 0 + checks_run: + - pytest test_advance_phase_thread + test_ble001 19/19 + - check-file-sizes.py exit 0 + - seam import resolution + verdict: clean_test_mechanical + version: 1 + slice_id: slice-4 +```` + +### [2026-07-06T07:58:41Z] reviewer_security → tester (CONSENSUS_ACK): ACK from reviewer_security for tester + +Security ACK (tester, terminal slice-4 proposal aa6d2b619 — same commit already security-reviewed for coder). Tester's contribution is test-execution attestation over the pure mechanical decomposition of routes/pipelines.py. The test-file changes (test_advance_phase_thread.py _pkg.-prefix stripping in source-introspection helpers; test_ble001_narrowing_audit.py format-line fix) are pure test mechanics with zero security impact: no secrets, network calls, subprocess/eval, auth changes, or trust-boundary shifts introduced. Security invariants from the coder review hold: @require_lifecycle_secret auth guards preserved 9→9, package imports cleanly, file-size gate green. Consistent with durable memory verdict. + +````yaml +id: 90844636-c36d-4c +phase: implement +metadata: + payload: + artifact_references: + - orchestrator/tests/test_advance_phase_thread.py + - orchestrator/tests/test_ble001_narrowing_audit.py + - scripts/file-size-allowlist.yaml + reason: "Security ACK (tester, terminal slice-4 proposal aa6d2b619 \u2014 same\ + \ commit already security-reviewed for coder). Tester's contribution is test-execution\ + \ attestation over the pure mechanical decomposition of routes/pipelines.py.\ + \ The test-file changes (test_advance_phase_thread.py _pkg.-prefix stripping\ + \ in source-introspection helpers; test_ble001_narrowing_audit.py format-line\ + \ fix) are pure test mechanics with zero security impact: no secrets, network\ + \ calls, subprocess/eval, auth changes, or trust-boundary shifts introduced.\ + \ Security invariants from the coder review hold: @require_lifecycle_secret\ + \ auth guards preserved 9\u21929, package imports cleanly, file-size gate green.\ + \ Consistent with durable memory verdict." + ack_version: 1 + version: 1 + slice_id: slice-4 +```` + +### [2026-07-06T07:58:41Z] reviewer_concurrency (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: ce768f59-488b-4e +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:58:41Z] reviewer_code (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: b739609d-7653-4f +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:58:41Z] reviewer_security (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 67dcaa1a-5b57-41 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:58:42Z] reviewer_code (CONSENSUS_CONFIRMED): Consensus confirmed by reviewer_code + +orchestrator-side confirm (#3064 event loop) + +````yaml +id: 1a0ef6a5-0b93-42 +phase: implement +metadata: + slice_id: slice-4 +```` + +### [2026-07-06T07:58:42Z] reviewer_security (CONSENSUS_CONFIRMED): Consensus confirmed by reviewer_security + +orchestrator-side confirm (#3064 event loop) + +````yaml +id: a3644204-c66a-4d +phase: implement +metadata: + slice_id: slice-4 +```` + +### [2026-07-06T07:58:42Z] reviewer_concurrency (CONSENSUS_CONFIRMED): Consensus confirmed by reviewer_concurrency + +orchestrator-side confirm (#3064 event loop) + +````yaml +id: fcb78b00-734b-4b +phase: implement +metadata: + slice_id: slice-4 +```` + +### [2026-07-06T07:58:44Z] reviewer_contract (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 37558a00-f444-48 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:58:48Z] reviewer_code (HEARTBEAT): heartbeat: WORKING + +context-measure occ=107466 util=0.11 cache_hit=0.99 decision=below_threshold + +````yaml +id: ef900d26-9be1-43 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:58:49Z] reviewer_code_holistic → tester (CONSENSUS_ACK): ACK from reviewer_code_holistic for tester + +Holistic ACK of the tester's slice-4 terminal proposal (same commit aa6d2b619 I verified for the coder; no new commits). The tester's core deliverable — the source-introspection test-seam fixes (task-4-6) — is sound: (1) test_advance_phase_thread.py adapts three helpers to the post-move structure by stripping the decomposition-only _pkg. prefix and introspecting the real _start_pipeline_body (the route body) instead of the thin @route start_pipeline wrapper; the structural markers (_BLOCK_MARKER auto_advance_block / recover_advance_clear, try/except regexes) still resolve and fire, so no assertion was weakened to mask a regression. (2) test_ble001_narrowing_audit.py change is a benign ruff-format normalization in the not-taken pre-split fallback branch. Verified independently: all 317 touched/seam tests pass (advance_phase_thread, ble001, start_pipeline, slice_loop_import_seam, pipelines_apply, origin_main, overseer_model, role_to_reviewer_mapping, handlers_brc); 7612 orchestrator tests collect with 0 import errors; barrel imports clean with all patch-target re-exports resolving; ruff clean on the pipelines package; terminal criterion met (allowlist files: {} empty, check-file-sizes.py exit 0). No holistic correctness or test-integrity concerns. + +````yaml +id: c8333935-d8ec-4d +phase: implement +metadata: + payload: + artifact_references: + - orchestrator/tests/test_advance_phase_thread.py + - orchestrator/tests/test_ble001_narrowing_audit.py + - scripts/file-size-allowlist.yaml + - orchestrator/routes/pipelines/__init__.py + - orchestrator/routes/pipelines/_run_pipeline.py + reason: "Holistic ACK of the tester's slice-4 terminal proposal (same commit aa6d2b619\ + \ I verified for the coder; no new commits). The tester's core deliverable \u2014\ + \ the source-introspection test-seam fixes (task-4-6) \u2014 is sound: (1) test_advance_phase_thread.py\ + \ adapts three helpers to the post-move structure by stripping the decomposition-only\ + \ _pkg. prefix and introspecting the real _start_pipeline_body (the route body)\ + \ instead of the thin @route start_pipeline wrapper; the structural markers\ + \ (_BLOCK_MARKER auto_advance_block / recover_advance_clear, try/except regexes)\ + \ still resolve and fire, so no assertion was weakened to mask a regression.\ + \ (2) test_ble001_narrowing_audit.py change is a benign ruff-format normalization\ + \ in the not-taken pre-split fallback branch. Verified independently: all 317\ + \ touched/seam tests pass (advance_phase_thread, ble001, start_pipeline, slice_loop_import_seam,\ + \ pipelines_apply, origin_main, overseer_model, role_to_reviewer_mapping, handlers_brc);\ + \ 7612 orchestrator tests collect with 0 import errors; barrel imports clean\ + \ with all patch-target re-exports resolving; ruff clean on the pipelines package;\ + \ terminal criterion met (allowlist files: {} empty, check-file-sizes.py exit\ + \ 0). No holistic correctness or test-integrity concerns." + ack_version: 1 + version: 1 + slice_id: slice-4 +```` + +### [2026-07-06T07:58:49Z] reviewer_security (HEARTBEAT): heartbeat: WORKING + +context-measure occ=83081 util=0.08 cache_hit=0.98 decision=below_threshold + +````yaml +id: 93f6c70d-e757-4c +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:58:52Z] reviewer_code_holistic (CONSENSUS_CONFIRMED): Consensus confirmed by reviewer_code_holistic + +orchestrator-side confirm (#3064 event loop) + +````yaml +id: 8f30eb35-3b9f-4c +phase: implement +metadata: + slice_id: slice-4 +```` + +### [2026-07-06T07:58:56Z] reviewer_code_holistic (HEARTBEAT): heartbeat: WORKING + +context-measure occ=93341 util=0.09 cache_hit=0.98 decision=below_threshold + +````yaml +id: 213d501c-dc88-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:58:57Z] reviewer_concurrency (HEARTBEAT): heartbeat: WORKING + +context-measure occ=84761 util=0.08 cache_hit=0.99 decision=below_threshold + +````yaml +id: 2b8ad8b6-edbd-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T07:59:02Z] reviewer_contract → tester (CONSENSUS_ACK): ACK from reviewer_contract for tester + +Contract verification PASS for tester slice-4 terminal (aa6d2b619) — satisfies task-4-6 test-half. Tester-owned artifacts re-confirmed green: test_advance_phase_thread.py (15) + test_ble001_narrowing_audit.py (4) = 19 passed. The source-introspection seam repairs (strip _pkg. prefix; introspect _start_pipeline_body vs the thin @route wrapper; ruff-format the pre-split fallback branch) are legitimate mechanical fixes for the _run_pipeline module move — NOT behavior changes. Contract shows ZERO open coverage gaps on any slice-4 task (task-4-1..4-6). The pure-refactor no-behavior-change guarantee is exercised by the dense loop-seam coverage (337 tests: test_consensus_polling/test_brc_nack/test_concurrent_*/test_slice_run_loop_integration) that stays green at this SHA — verified in my coder review of the same commit. Terminal criterion (allowlist files: map EMPTY, check-file-sizes exit 0) holds. Documented non-regressions (2 orchestrator.-prefix PYTHONPATH collect quirks in slice-6 event_prompt tests) are pre-existing, not split-induced. + +````yaml +id: ea0d60aa-2e1c-42 +phase: implement +metadata: + payload: + artifact_references: + - orchestrator/tests/test_advance_phase_thread.py + - orchestrator/tests/test_ble001_narrowing_audit.py + - scripts/file-size-allowlist.yaml + - orchestrator/routes/pipelines/__init__.py + - orchestrator/routes/pipelines/_run_pipeline.py + reason: "Contract verification PASS for tester slice-4 terminal (aa6d2b619) \u2014\ + \ satisfies task-4-6 test-half. Tester-owned artifacts re-confirmed green: test_advance_phase_thread.py\ + \ (15) + test_ble001_narrowing_audit.py (4) = 19 passed. The source-introspection\ + \ seam repairs (strip _pkg. prefix; introspect _start_pipeline_body vs the thin\ + \ @route wrapper; ruff-format the pre-split fallback branch) are legitimate\ + \ mechanical fixes for the _run_pipeline module move \u2014 NOT behavior changes.\ + \ Contract shows ZERO open coverage gaps on any slice-4 task (task-4-1..4-6).\ + \ The pure-refactor no-behavior-change guarantee is exercised by the dense loop-seam\ + \ coverage (337 tests: test_consensus_polling/test_brc_nack/test_concurrent_*/test_slice_run_loop_integration)\ + \ that stays green at this SHA \u2014 verified in my coder review of the same\ + \ commit. Terminal criterion (allowlist files: map EMPTY, check-file-sizes exit\ + \ 0) holds. Documented non-regressions (2 orchestrator.-prefix PYTHONPATH collect\ + \ quirks in slice-6 event_prompt tests) are pre-existing, not split-induced." + ack_version: 1 + attestation: + tasks_verified: + - task-4-6 + notes: 'Tester test-half of task-4-6 at terminal SHA aa6d2b619: source-introspection + seam repairs green (test_advance_phase_thread 15 + test_ble001_narrowing_audit + 4 = 19 passed); zero coverage gaps on any slice-4 task; pure-refactor no-behavior-change + exercised by 337 green loop-seam tests (verified in coder review of same SHA).' + version: 1 + slice_id: slice-4 +```` + +### [2026-07-06T07:59:03Z] reviewer_contract (CONSENSUS_CONFIRMED): Consensus confirmed by reviewer_contract + +orchestrator-side confirm (#3064 event loop) + +````yaml +id: 68655195-2cf6-43 +phase: implement +metadata: + slice_id: slice-4 +```` + +### [2026-07-06T07:59:12Z] reviewer_contract (HEARTBEAT): heartbeat: WORKING + +context-measure occ=126821 util=0.13 cache_hit=0.99 decision=below_threshold + +````yaml +id: f40acde0-5b12-49 +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T08:00:34Z] tester (HEARTBEAT): heartbeat: WORKING + +in-tool-loop liveness (slice=slice-4) + +````yaml +id: 359aafe6-ec89-4b +phase: implement +metadata: + state: WORKING + slice_id: slice-4 +```` + +### [2026-07-06T08:02:16Z] tester → coder (CONSENSUS_ACK): ACK from tester for coder + +Verified the terminal proposal. (1) Both modified test files pass: test_advance_phase_thread.py 15/15 + test_ble001_narrowing_audit.py 4/4. The source-introspection seam fixes are correct — barrel re-exports _run_pipeline, _start_pipeline_body, and start_pipeline all resolve; the .replace("_pkg.","") + _start_pipeline_body switch make the pre-split structural assertions match the post-move source. (2) Terminal acceptance criterion met: scripts/file-size-allowlist.yaml files map is EMPTY ({}), and check-file-sizes.py exits 0 (soft-cap warnings only, no hard-cap violations). (3) 151 split-surface seam tests pass (start_pipeline, pipelines_apply, slice_loop_import_seam, origin_main_parameterization, role-mapping, overseer_model). (4) Whole orchestrator suite collects clean: 7612 tests, 0 import errors. (5) The 4 test_compose_event_prompt.py failures are pre-existing — they reproduce identically on origin/main (4 failed/100 passed) and event_prompt.py is untouched by this branch; documented non-regression, not caused by the decomposition. Clean pure-refactor + test-mechanical terminal. ACK. + +````yaml +id: d49535e3-447e-4a +phase: implement +metadata: + payload: + artifact_references: + - scripts/file-size-allowlist.yaml + - orchestrator/tests/test_advance_phase_thread.py + - orchestrator/tests/test_ble001_narrowing_audit.py + - orchestrator/routes/pipelines/__init__.py + reason: "Verified the terminal proposal. (1) Both modified test files pass: test_advance_phase_thread.py\ + \ 15/15 + test_ble001_narrowing_audit.py 4/4. The source-introspection seam\ + \ fixes are correct \u2014 barrel re-exports _run_pipeline, _start_pipeline_body,\ + \ and start_pipeline all resolve; the .replace(\"_pkg.\",\"\") + _start_pipeline_body\ + \ switch make the pre-split structural assertions match the post-move source.\ + \ (2) Terminal acceptance criterion met: scripts/file-size-allowlist.yaml files\ + \ map is EMPTY ({}), and check-file-sizes.py exits 0 (soft-cap warnings only,\ + \ no hard-cap violations). (3) 151 split-surface seam tests pass (start_pipeline,\ + \ pipelines_apply, slice_loop_import_seam, origin_main_parameterization, role-mapping,\ + \ overseer_model). (4) Whole orchestrator suite collects clean: 7612 tests,\ + \ 0 import errors. (5) The 4 test_compose_event_prompt.py failures are pre-existing\ + \ \u2014 they reproduce identically on origin/main (4 failed/100 passed) and\ + \ event_prompt.py is untouched by this branch; documented non-regression, not\ + \ caused by the decomposition. Clean pure-refactor + test-mechanical terminal.\ + \ ACK." + ack_version: 1 + attestation: + tests_run: 170 + checks_passed: + - test_advance_phase_thread(15/15) + - test_ble001_narrowing_audit(4/4) + - split-surface-seam(151/151) + - check-file-sizes=exit0 + - allowlist-empty + - orchestrator-collect(7612,0-import-errors) + non_regressions: + - test_compose_event_prompt(4 fail) is PRE-EXISTING on origin/main; event_prompt.py + untouched by this branch + version: 1 + slice_id: slice-4 +```` + +### [2026-07-06T08:02:16Z] orchestrator → coder (STATUS): Ready to confirm — all confirm preconditions satisfied + +Your proposal (version 1) is ready to confirm — all blocking reviews are clear and global confirm preconditions are met. Run `egg-orch consensus confirmed` to confirm. + +````yaml +id: 2a45377a-4fe6-4a +phase: implement +metadata: + ready_to_confirm: true + version: 1 + slice_id: slice-4 +```` + +### [2026-07-06T08:02:16Z] orchestrator → tester (STATUS): Ready to confirm — all confirm preconditions satisfied + +Your proposal (version 1) is ready to confirm — all blocking reviews are clear and global confirm preconditions are met. Run `egg-orch consensus confirmed` to confirm. + +````yaml +id: 5a6ad7c3-979f-4f +phase: implement +metadata: + ready_to_confirm: true + version: 1 + slice_id: slice-4 +```` + +### [2026-07-06T08:02:21Z] coder (CONSENSUS_CONFIRMED): Consensus confirmed by coder + +orchestrator-side confirm (#3064 event loop) + +````yaml +id: 55bafc2a-75a9-4a +phase: implement +metadata: + slice_id: slice-4 +```` + +### [2026-07-06T08:02:21Z] tester (CONSENSUS_CONFIRMED): Consensus confirmed by tester + +orchestrator-side confirm (#3064 event loop) + +````yaml +id: 3d497b50-c88c-40 +phase: implement +metadata: + slice_id: slice-4 +```` + +### [2026-07-06T08:02:21Z] documenter (CONSENSUS_CONFIRMED): Consensus confirmed by documenter + +orchestrator-side confirm (#3064 event loop) + +````yaml +id: dc9491ab-243e-4d +phase: implement +metadata: + slice_id: slice-4 +```` + +### [2026-07-06T08:02:21Z] reviewer_code (CONSENSUS_CONFIRMED): Consensus confirmed by reviewer_code + +orchestrator-side confirm (#3064 event loop) + +````yaml +id: f3f19659-ab4f-47 +phase: implement +metadata: + slice_id: slice-4 +```` + +### [2026-07-06T08:02:21Z] reviewer_code_holistic (CONSENSUS_CONFIRMED): Consensus confirmed by reviewer_code_holistic + +orchestrator-side confirm (#3064 event loop) + +````yaml +id: 6909cc0e-cf97-48 +phase: implement +metadata: + slice_id: slice-4 +```` + +### [2026-07-06T08:02:21Z] reviewer_contract (CONSENSUS_CONFIRMED): Consensus confirmed by reviewer_contract + +orchestrator-side confirm (#3064 event loop) + +````yaml +id: 62de0939-3b7b-47 +phase: implement +metadata: + slice_id: slice-4 +```` + +### [2026-07-06T08:02:21Z] reviewer_security (CONSENSUS_CONFIRMED): Consensus confirmed by reviewer_security + +orchestrator-side confirm (#3064 event loop) + +````yaml +id: cc08f9dd-3b6a-4a +phase: implement +metadata: + slice_id: slice-4 +```` + +### [2026-07-06T08:02:21Z] reviewer_concurrency (CONSENSUS_CONFIRMED): Consensus confirmed by reviewer_concurrency + +orchestrator-side confirm (#3064 event loop) + +````yaml +id: 863e082e-0606-47 +phase: implement +metadata: + slice_id: slice-4 +```` diff --git a/gateway/gateway/_confluence.py b/gateway/gateway/_confluence.py index 516f8bf6fb..63508ae7d0 100644 --- a/gateway/gateway/_confluence.py +++ b/gateway/gateway/_confluence.py @@ -35,10 +35,10 @@ from confluence_client import ( # type: ignore[no-redef, import-untyped] DEFAULT_LIMIT as CONFLUENCE_DEFAULT_LIMIT, ) - from confluence_client import ( # type: ignore[no-redef, import-untyped] + from confluence_client import ( # type: ignore[no-redef] HARD_MAX_LIMIT as CONFLUENCE_HARD_MAX_LIMIT, ) - from confluence_client import ( # type: ignore[no-redef, import-untyped] + from confluence_client import ( # type: ignore[no-redef] ConfluenceCredentialsUnavailable, ConfluenceResponseTooLarge, ConfluenceUpstreamError, @@ -277,7 +277,8 @@ def _resolve_space_key_for_payload(payload: Any) -> str | None: if space_id is None: return None client = _b().get_confluence_client() - return client.space_cache.key_for_id(str(space_id)) + key: str | None = client.space_cache.key_for_id(str(space_id)) + return key def _resolve_space_key_via_list(allowed: frozenset[str], space_id: str | None) -> str | None: @@ -295,7 +296,7 @@ def _resolve_space_key_via_list(allowed: frozenset[str], space_id: str | None) - if not space_id: return None client = _b().get_confluence_client() - cached = client.space_cache.key_for_id(str(space_id)) + cached: str | None = client.space_cache.key_for_id(str(space_id)) if cached is not None: return cached # Walk paginated /wiki/api/v2/spaces so a target space on page 2+ still @@ -312,7 +313,8 @@ def _resolve_space_key_via_list(allowed: frozenset[str], space_id: str | None) - # so the outer post-fetch check fail-closes through # confluence_space_denied rather than leaking a Flask 500. return None - return client.space_cache.key_for_id(str(space_id)) + warmed: str | None = client.space_cache.key_for_id(str(space_id)) + return warmed def _confluence_clamp_limit(value: Any) -> int | None: diff --git a/gateway/gateway/_git_ops.py b/gateway/gateway/_git_ops.py index b591c843fa..196490e26c 100644 --- a/gateway/gateway/_git_ops.py +++ b/gateway/gateway/_git_ops.py @@ -662,7 +662,7 @@ def git_push() -> tuple[Response, int] | Response: _partition_fn: Any = getattr(_ar_mod, "partition_files_by_role", None) if _ar_mod else None if _partition_fn is None: try: - from agent_restrictions import ( + from agent_restrictions import ( # type: ignore[import-untyped] partition_files_by_role as _imported_partition, ) @@ -680,7 +680,7 @@ def git_push() -> tuple[Response, int] | Response: ) if _get_attributed_fn is None: try: - from git_client import ( # type: ignore[no-redef, import-untyped] + from git_client import ( get_attributed_changed_files_in_push as _imported_attr, ) diff --git a/gateway/gateway/_jira.py b/gateway/gateway/_jira.py index 4ca9677786..f8c5d3e42d 100644 --- a/gateway/gateway/_jira.py +++ b/gateway/gateway/_jira.py @@ -35,7 +35,7 @@ JiraUpstreamError, validate_jira_api_path, ) - from jira_client import ( # type: ignore[no-redef, import-untyped] + from jira_client import ( # type: ignore[no-redef] validate_fields as validate_jira_fields, ) from jira_policy import ( # type: ignore[no-redef, import-untyped] diff --git a/gateway/gateway/_jira_writes.py b/gateway/gateway/_jira_writes.py index 2f88cc9932..54ea73a0e7 100644 --- a/gateway/gateway/_jira_writes.py +++ b/gateway/gateway/_jira_writes.py @@ -251,7 +251,7 @@ def _validate_jira_text_field( try: from ..jira_adf import is_adf_dict except ImportError: - from jira_adf import is_adf_dict # type: ignore[no-redef] + from jira_adf import is_adf_dict # type: ignore[no-redef, import-untyped] if not is_adf_dict(value): return None, make_error( f"{field} must be a string or a valid ADF document", diff --git a/orchestrator/CLAUDE.md b/orchestrator/CLAUDE.md index 9087e3560b..39afa233a7 100644 --- a/orchestrator/CLAUDE.md +++ b/orchestrator/CLAUDE.md @@ -266,4 +266,21 @@ Pure refactor, no behaviour change: every model / enum / helper is AST-identical Pure refactor, no behaviour change: the 27 method bodies are AST-identical to the pre-split file (modulo docstring re-indentation and the intentional `_pkg.`-prefixing of the monkeypatched module-global seams). Patch seams preserved: the class definitions + their method bindings resolve on the barrel, so `patch.object(JobSupervisor, …)` / `patch.object(OrchestratorEventLoop, "start")` and instance-method calls keep working; the five monkeypatched module-global function seams (`_derive_next_action`, `event_identity`, `compute_dedupe_key`, `get_idle_budget_minutes`, `_idle_budget_anomaly_name`) are reached from `_loop.py` via `import event_loop as _pkg`, so `setattr(event_loop, "_derive_next_action", …)` keeps intercepting the loop's call; `time` stays a plain `import time` so `setattr(event_loop.time, "time", …)` is seen. `event_loop.py`'s allowlist entry is dropped (the file is gone) and `orchestrator/Dockerfile` carries the explicit `COPY orchestrator/event_loop/ ./event_loop/` line (R3 parity). -`routes/decisions/`, `state_store/`, `routes/phases/`, `routes/deployment/`, `routes/event_prompt/`, `overseer/monitor/`, `peer_consensus/`, `mcp_tools/`, `kubernetes_spawner/`, `routes/signals/`, `gateway_client/`, `models/`, and `event_loop/` are the landed `orchestrator/` decompositions; later orchestrator slices append their own subsections here. +### `routes/pipelines/` — pipeline REST API + the `_run_pipeline` phase state machine ([#3312](https://github.com/jwbron/egg/issues/3312) continuation, slice 4; **closes #3312 — terminal slice**) + +`routes/pipelines.py` (30,520 lines / 1.44 MB — the largest file in the repo, a structural outlier over the byte cap) → `routes/pipelines/` (46 submodules; largest `_run_implement.py`, 1,496 lines; every module under BOTH the 1,500-line and 100 KB caps). Two shapes combined: the **routes-handling convention** (non-negotiable #8 / decision-8) — the `pipelines_bp` `Blueprint` and all **16 `@pipelines_bp.route` decorators stay in the barrel** on thin wrapper functions that delegate to bodies in the responsibility-grouped `_routes_*` submodules, so the 16 URL rules register identically — plus the **`_run_pipeline` split** (issue #3312 non-negotiable #7 / task-4-3): the phase-transition state machine is split into per-phase handlers (`_run_pipeline` thin loop → `_run_pipeline_setup` / `_run_phase` / `_run_phase_blocks` / `_run_implement` / `_run_concurrent` / `_run_hitl_gate`), preserving transition ordering exactly. **Packaging-neutral (NOT a Dockerfile change):** `pipelines/` sits under the already-recursive `COPY orchestrator/routes/ ./routes/` (orchestrator/Dockerfile:45), so the new sub-package ships with no Dockerfile edit — contrast the top-level `models/` / `event_loop/` slices. **Terminal criterion:** `pipelines.py` was the **LAST** entry in `scripts/file-size-allowlist.yaml`; dropping it makes the `files:` map **EMPTY**, completing the file-size decomposition program and closing #3312. + +| Submodule(s) | Responsibility | Key symbols | +|--------------|----------------|-------------| +| `__init__.py` (barrel, 1,466 lines) | Stable public API: the `pipelines_bp` `Blueprint` + all 16 `@pipelines_bp.route` thin wrappers (decision-8); retains the test-patched module globals (`subprocess`, `json`, `threading`, `time`, …) so their `_pkg` re-export / patch seams resolve; explicit per-symbol re-exports of every symbol from all 46 submodules below | `pipelines_bp` (+ re-exports of every externally-referenced symbol below) | +| Route bodies (decision-8): `_routes_crud.py` (990), `_routes_lifecycle.py` (822), `_routes_restart.py` (1,070), `_routes_status.py` (572), `_routes_stream.py` (123), `_routes_read.py` (120) | Implementation bodies for the 16 route wrappers: CRUD, start/visualization lifecycle, agent/phase restart, status, SSE stream, local-commits read | `create_pipeline`, `update_pipeline`, `delete_pipeline`, `start_pipeline`, `restart_agent`, `restart_phase`, `get_status`, `stream_pipeline`, … | +| `_run_pipeline` state machine (non-negotiable #7): `_run_pipeline.py` (1,483), `_run_pipeline_setup.py` (727), `_run_pipeline_support.py` (62), `_run_phase.py` (331), `_run_phase_blocks.py` (338), `_run_implement.py` (largest, 1,496), `_run_implement_support.py` (242), `_run_concurrent.py` (1,439), `_run_concurrent_support.py` (422), `_run_concurrent_retry.py` (215), `_run_hitl_gate.py` (722), `_run_support.py` (376) | The phase-transition driver split into a thin orchestration loop + per-phase handlers (refine/plan/implement/PR), the concurrent-slice DAG runner, and the HITL-gate block — ordering/transition semantics preserved verbatim | `_run_pipeline`, `_run_implement_phase_slices`, `_run_concurrent_slices`, `_run_hitl_gate`, `_run_phase_blocks` | +| Prompt building: `_prompt_phase.py` (1,407), `_prompt_agent.py` (1,346), `_prompt_review.py` (765), `_prompt_reviewer.py` (593) | Phase-prompt, agent-prompt, review-prompt and reviewer-prep prompt assembly | `build_phase_prompt`, `build_agent_prompt`, `build_review_prompt`, `build_reviewer_prompt` | +| Readers / synthesis: `_criteria.py` (961), `_drafts.py` (758), `_reviews.py` (186), `_context_pr.py` (1,220), `_brc_history.py` (961), `_populate.py` (1,460) | Review-criteria builders, draft-path + source-branch-artifact readers, review-verdict readers, context-PR composition, BRC-history readers, plan-draft synthesis + contract population | `build_review_criteria`, `read_draft_path`, `read_review_verdicts`, `compose_context_pr`, `populate_contract` | +| State / lifecycle: `_slice_state.py` (1,094), `_statefiles.py` (613), `_worktree_sync.py` (1,393), `_slice_completion.py` (135), `_lifecycle_helpers.py` (339), `_status_view.py` (391), `_status_wait.py` (147) | Slice-DAG state helpers, statefile read/write, worktree-sync, slice-completion, lifecycle helpers, status view + long-poll wait | `slice_state`, `sync_worktree`, `complete_slice`, `status_view`, `wait_for_status` | +| Decisions / overseer: `_ledger.py` (1,364), `_decisions.py` (259), `_resolve.py` (196), `_hitl_rerun.py` (337), `_overseer.py` (740), `_alerts.py` (1,277), `_pod_liveness.py` (228), `_first_principles.py` (247) | Decision-ledger + gap-gate + apply-handoff, HITL + divergence-reconcile decisions, decision resolution, HITL rerun, overseer detection-plane, divergence/alert/timeout emission, live-pod guarding, first-principles review seed | `register_decision`, `resolve_decision`, `rerun_hitl`, `detect_divergence`, `guard_live_pods` | +| PR / drivers / salvage: `_drivers.py` (263), `_stacked_pr.py` (251), `_salvage.py` (71) | Pipeline-driver lifecycle helpers, stacked-PR assembly, agent-output salvage | `pipeline_drivers`, `build_stacked_pr`, `salvage_agent_output` | + +Pure refactor, no behaviour change: every route handler body, helper, and constant is AST-identical to the pre-split file (modulo the sanctioned `_pkg.`-prefixing and the decorator relocation onto thin wrappers). Patch seams preserved: the 16 `@pipelines_bp.route` decorators stay in the barrel so the URL rule → handler map registers identically; the private submodules reach the test-patched module globals via `import routes.pipelines as _pkg`, and the barrel re-exports every externally-referenced symbol across the dominant back-compat import surface (~137 referencing files repo-wide; the audited ~57 distinct `patch("routes.pipelines.<name>")` targets), so both `from routes.pipelines import X` and `patch("routes.pipelines.X")` resolve unchanged — the existing dense seam coverage (`test_consensus_polling`, `test_brc_nack_iteration`, `test_concurrent_*`, `test_advance_phase_*`) stays green. `_run_pipeline` becomes a thin loop delegating to per-phase handlers with no transition-ordering change. **Packaging-neutral:** `orchestrator/routes/` is already shipped by the recursive `COPY orchestrator/routes/ ./routes/` (Dockerfile:45), so the new submodules are auto-included — no Dockerfile change. `pipelines.py`'s allowlist entry — the **LAST** in the program — is dropped, so `scripts/file-size-allowlist.yaml`'s `files:` map is now **EMPTY**: the terminal acceptance criterion of the file-size decomposition program, closing #3312. + +`routes/decisions/`, `state_store/`, `routes/phases/`, `routes/deployment/`, `routes/event_prompt/`, `overseer/monitor/`, `peer_consensus/`, `mcp_tools/`, `kubernetes_spawner/`, `routes/signals/`, `gateway_client/`, `models/`, `event_loop/`, and `routes/pipelines/` are the landed `orchestrator/` decompositions; with `routes/pipelines/` the file-size allowlist is empty and the program (#3312) is complete. diff --git a/orchestrator/event_loop/_supervisor.py b/orchestrator/event_loop/_supervisor.py index f613b0e631..5ebc1a79fa 100644 --- a/orchestrator/event_loop/_supervisor.py +++ b/orchestrator/event_loop/_supervisor.py @@ -368,8 +368,7 @@ def reset_exhausted(self) -> list[str]: self.retire(key) if cleared: logger.info( - "JobSupervisor: operator reset cleared %d exhausted key(s) — " - "fresh spawn budgets: %s", + "JobSupervisor: operator reset cleared %d exhausted key(s) — fresh spawn budgets: %s", len(cleared), ", ".join(cleared), ) diff --git a/orchestrator/routes/pipelines.py b/orchestrator/routes/pipelines.py deleted file mode 100644 index 852d49aa1f..0000000000 --- a/orchestrator/routes/pipelines.py +++ /dev/null @@ -1,30663 +0,0 @@ -""" -Pipeline CRUD endpoints for egg-orchestrator. -""" - -import concurrent.futures -import glob -import json -import os -import re -import subprocess -import sys -import threading -import time -from collections.abc import Callable -from datetime import UTC, datetime -from enum import StrEnum -from pathlib import Path -from typing import TYPE_CHECKING, Any, Literal, NamedTuple -from uuid import uuid4 - -import yaml - -try: - from docker.errors import DockerException -except ImportError: - - class DockerException(Exception): # type: ignore[no-redef] - pass - - -from flask import Blueprint, Response, jsonify, request, stream_with_context - -# Re-export the slice-3 per-event prompt composer so callers can still -# import it via ``orchestrator.routes.pipelines.compose_event_prompt`` -# (the contract assigns this file in TASK-3-1) even though the body -# lives in a sibling module to keep this file under the orchestrator -# decomposition cap (#2261). The slice-3 plan acceptance is satisfied -# by either import path; tests bind on -# ``orchestrator.routes.event_prompt`` directly. -from .event_prompt import compose_event_prompt # noqa: F401 - - -# Closed enumeration of ``ContextPrCreationError.reason`` values -# (#2777). Producer and downstream tests (TASK-3-8) bind on these -# strings so a single source of truth avoids the synthetic-key -# divergence reviewer_code_holistic flagged. New reasons MUST be -# added here AND to ``ContextPrCreationReason`` so the type narrows. -class ContextPrCreationReason(StrEnum): - """Closed set of typed reasons for :class:`ContextPrCreationError` (#2777).""" - - UNKNOWN = "unknown" - # Lookup of the pipeline / store / spawner failed before any - # gateway call could be attempted. - PIPELINE_LOAD_FAILED = "pipeline_load_failed" - ROUTES_UNAVAILABLE = "routes_unavailable" - LOADER_UNAVAILABLE = "loader_unavailable" - # Pipeline misconfiguration. ``base_branch`` left unset alongside a - # ``repo`` is NOT a misconfiguration — it is the normal "auto-detect - # the repo's default branch" state (#3031), so the opener resolves it - # rather than raising. The remaining genuine misconfigurations are a - # ``base_branch`` with no ``repo`` to open a PR against - # (``missing_repo``) and a remote pipeline with no work branch - # (``missing_branch``). - MISSING_BRANCH = "missing_branch" - MISSING_REPO = "missing_repo" - # Contract / PR-metadata failures encountered after the pipeline - # passed the misconfiguration check. - CONTRACT_LOAD_FAILED = "contract_load_failed" - MISSING_PR_METADATA = "missing_pr_metadata" - SAVE_FAILED = "save_failed" - # Gateway-layer failures wrapping ``lookup_open_pr`` / - # ``create_pr`` outcomes. - LOOKUP_FAILED = "lookup_failed" - GATEWAY_ERROR = "gateway_error" - GATEWAY_NO_URL = "gateway_no_url" - GATEWAY_BAD_URL = "gateway_bad_url" - - -class ContextPrCreationError(Exception): - """Raised by :func:`_open_context_pr_at_implement_start` when the - hard-required up-front context PR cannot be opened (#2777, cq-4). - - Replaces the soft-fail ``return None`` swallow path that the legacy - ``_maybe_open_base_pr_for_plan_to_implement`` wrapper used before - slice-2 deleted it. Under cq-4 the context PR is hard-required at - the plan→implement boundary; a gateway failure here must surface to - the BRC NACK / 422 surface rather than silently strand the slice - stack on ``/work``. - - Attributes: - reason: Machine-readable reason drawn from - :class:`ContextPrCreationReason`. Tests assert on these - constants so producer and tests share one source of - truth; passing an unknown string is a programming error - caught here. The instance attribute is exposed as the - underlying ``str`` value (matching ``.value`` of the - enum) so existing JSON-serialization callers continue to - work without change. - cause: The original exception, if any, that triggered the - error. Preserved so logs and the BRC NACK body show the - gateway/contract failure rather than only this wrapper's - text. - """ - - def __init__( - self, - message: str, - *, - reason: str | ContextPrCreationReason = ContextPrCreationReason.UNKNOWN, - cause: BaseException | None = None, - ) -> None: - super().__init__(message) - # Coerce-and-validate the reason against the closed - # enumeration. Passing a string that is not a known reason - # would normally raise ``ValueError`` from the ``StrEnum`` - # constructor — but the four ``except ContextPrCreationError`` - # handlers at every call site would not match that - # ``ValueError``, so a typo would surface as a 500 instead of - # the typed 422 the handlers contract on - # (egg-reviewer non-blocking #4). Catch and coerce to - # ``UNKNOWN`` so the typed-exception contract holds, and log - # the bad reason loudly so the typo is still visible in the - # operator's logs and CI grep — silent coercion would hide - # the programming error. - try: - self.reason: str = ContextPrCreationReason(reason).value - except ValueError: - logger.warning( - "ContextPrCreationError received unknown reason; " - "coercing to UNKNOWN (#2777, egg-reviewer non-blocking #4)", - bad_reason=repr(reason), - error_message=message, - ) - self.reason = ContextPrCreationReason.UNKNOWN.value - self.cause: BaseException | None = cause - - -class ForestValidationError(Exception): - """Raised by ``_populate_contract_from_plan`` on slice-DAG structural rejection. - - Added in #2137 (TASK-2-2) for the forest-shape violation (a slice - with >1 DAG parent). Generalised in #3046 to also signal the - file-overlap-ordering violation (two slices touching the same file - with no dependency edge between them — see - ``egg_contracts.validate_slice_file_overlap``). Both are slice-DAG - structural defects surfaced at plan ingestion with identical - handling: the slices are NOT written to the contract, the structured - errors are stashed on ``plan_review_feedback`` so the plan reviewer - NACKs the architect, and the exception is raised so HTTP callers can - return a 422. - - The ``reason`` discriminator (``"forest_violation"`` or - ``"slice_overlap_violation"``) selects the operator-facing prose and - the :class:`PopulateOutcome` the safe wrapper maps to. Any future - Flask route that ingests a plan in-band can catch this and - ``body, status = err.to_response(); return jsonify(body), status`` - to surface the structured rejection. Internal callers - (``_populate_contract_from_plan_safe`` and the pipeline run-loop - helpers) catch it and log a warning — the ``plan_review_feedback`` - stash is the durable NACK signal either way. - """ - - def __init__( - self, message: str, *, errors: list[str], reason: str = "forest_violation" - ) -> None: - super().__init__(message) - self.errors: list[str] = list(errors) - self.reason: str = reason - self.status_code: int = 422 - - def to_response(self) -> tuple[dict[str, object], int]: - """Serialise into a Flask-compatible (body, status) tuple.""" - return ({"error": self.reason, "errors": self.errors}, 422) - - -class SliceCompletionInvariantError(RuntimeError): - """Raised when a slice would be persisted ``COMPLETE`` without a valid - completion basis (#3214). - - The #3214 wedge traced to an interior forest node (``slice-3`` on - pipeline ``issue-3200``) persisted as ``SliceStatus.COMPLETE`` while - its only task was still ``pending``, it had no integration branch, and - it carried its *parent's* commit SHA. ``_persist_slice_status_complete`` - wrote that contradictory state with no validation, so the slice-DAG - driver skipped real work and the chain wedged with no successor — and - it hung ~9h silently because nothing failed loud at the moment of the - bad write. - - A slice has a valid completion basis when ANY of these execution - signals is present: - - * a slice PR is recorded / supplied (``pr_number``); or - * the caller declares a verified ``basis`` — ``"merged"`` (the - integration branch was ancestry-verified merged into its parent) or - ``"consensus_complete"`` (BRC consensus reached, PR not yet opened - or its URL unparseable); or - * the slice forked an integration branch (``integration_base_sha`` is - set — #2871); or - * every task is ``TaskStatus.COMPLETE``. - - The predicate accepts any one signal so it can only flag the slice-3 - state where *all* are absent — a slice marked COMPLETE with zero - evidence it ran. We raise here so that corrupt write fails loud at its - source instead of wedging the forest a phase later. - - #3253 refinement: ``basis="merged"`` is no longer an unconditional - pass. A merged slice went through a PR and left commits its producers - recorded; a ``basis="merged"`` write with **no PR and no produced task - commit** is an empty / never-implemented branch that origin ancestry - mis-detected as merged (the slice-10 case — producers exhausted before - committing, so the integration branch's tip is still its fork base and - is trivially an ancestor of the advanced parent). Such a write is - rejected so the slice is re-run rather than false-completed. - """ - - -# Completion bases a caller may declare when it has positive, verified -# evidence a slice finished even though not every task is marked COMPLETE -# on the contract (the crash-recovery / merged-skip paths). See -# :class:`SliceCompletionInvariantError`. -_VERIFIED_SLICE_COMPLETION_BASES = frozenset({"merged", "consensus_complete"}) - - -def _slice_produced_commits(slice_obj: Any) -> bool: - """Return True iff any of the slice's tasks recorded a commit SHA. - - This is the base-SHA-independent "a producer actually committed work" - signal (#3253). It reads *task* commits only — a slice whose producers - all failed before committing has every ``task.commit`` ``None`` (the - AC-4 measurement in the issue-3200 slice-10 incident). It deliberately - ignores ``Slice.commit``: that field can carry the *parent's* SHA on a - false-complete (the #3214 slice-3 carryover), so it is not trustworthy - evidence the slice itself produced anything. - - An empty integration branch (tip still at its fork base, so trivially - an ancestor of an advanced parent) is indistinguishable from a merged - one by origin ancestry alone once the recorded fork base is missing or - stale (#3245). The contract's task-commit record is the durable signal - that survives that ambiguity: no task commit + no slice PR ⇒ the slice - never ran and must be re-run, not completed. - - A slice with *no tasks* returns ``False`` here (``any([])``). Paired with - "origin-detected merged, no PR" that would force such a slice to re-run - indefinitely — but a zero-task slice is unreachable in practice: - plan-derived slices always carry at least one task. The safe direction is - re-run over silently-dropped work, so the edge needs no special-casing - (#3253). - """ - tasks = getattr(slice_obj, "tasks", None) or [] - return any(getattr(t, "commit", None) for t in tasks) - - -def _validate_slice_completion_basis( - slice_obj: Any, - *, - pr_number: int | None = None, - basis: str | None = None, -) -> str | None: - """Return ``None`` when ``slice_obj`` may legitimately be marked - ``SliceStatus.COMPLETE``, else a human-readable reason it may not. - - Shared by the write chokepoint (``_persist_slice_status_complete``, - which raises :class:`SliceCompletionInvariantError` on a reason) and - the Layer-A bootstrap read-trust point (which alerts and declines to - trust a contradictory contract-recorded COMPLETE rather than - propagating it into the scheduler). See - :class:`SliceCompletionInvariantError` for the basis rules (#3214). - """ - has_pr = pr_number is not None or getattr(slice_obj, "pr_number", None) is not None - # #3253 — a ``basis="merged"`` slice with no PR and no produced task - # commits is not a merged slice; it is an empty / never-implemented - # integration branch (tip still at its fork base) that origin ancestry - # mis-detected as merged. A genuine merge went through a PR and left - # commits the producers recorded. Reject so the restart re-runs the - # slice instead of false-completing the pipeline with its work missing. - # This guard fires *before* the verified-basis / forked free-passes - # below so a recorded (possibly stale) fork base cannot rescue it. - if basis == "merged" and not has_pr and not _slice_produced_commits(slice_obj): - return ( - f"slice {getattr(slice_obj, 'id', '?')} would be marked COMPLETE " - f"basis='merged' with no slice PR and no produced task commits — an " - f"empty / never-implemented integration branch is not a merged one " - f"(#3253)" - ) - verified_basis = basis in _VERIFIED_SLICE_COMPLETION_BASES - # A slice that actually forked its integration branch recorded a base - # SHA (#2871). Its absence — together with no PR, no verified basis, - # and no completed tasks — is the slice-3 false-complete signature: a - # slice marked COMPLETE with zero evidence it ever ran. The predicate - # accepts ANY single execution signal so it can only flag that - # genuinely-contradictory state, never a legitimately-completed slice - # whose other signals happen to be absent (e.g. an unparseable PR URL - # leaves ``pr_number`` None but the slice still forked and reached - # consensus). ``tasks_all_complete`` is the canonical model-side - # predicate so this can't drift from the contract's own notion of - # "work finished". - forked = getattr(slice_obj, "integration_base_sha", None) is not None - if has_pr or verified_basis or forked or slice_obj.tasks_all_complete: - return None - return ( - f"slice {getattr(slice_obj, 'id', '?')} would be marked COMPLETE with no " - f"evidence it ran: no slice PR, no verified merge/consensus basis " - f"(basis={basis!r}), no integration-branch fork base, and tasks not all " - f"complete" - ) - - -# Add shared directory to path for egg_logging -_shared_path = Path(__file__).parent.parent.parent / "shared" -if _shared_path.exists() and str(_shared_path) not in sys.path: - sys.path.insert(0, str(_shared_path)) - -# Add config directory to path for repo_config module -_config_path = Path(__file__).parent.parent.parent / "config" -if _config_path.exists() and str(_config_path) not in sys.path: - sys.path.insert(0, str(_config_path)) - -try: - from egg_logging import get_logger -except ImportError: - import logging - - def get_logger(name: str, **kwargs) -> logging.Logger: # type: ignore[misc] - return logging.getLogger(name) - - -try: - from repo_config import get_repo_checks -except ImportError: - - def get_repo_checks(repo: str) -> list[dict[str, str]]: # type: ignore[misc] - return [] - - -# Import orchestrator modules - try relative import first -try: - from .. import agent_salvage - from ..container_spawner import ContainerSpawnError, SpawnFailureError, get_container_spawner - from ..decision_queue import get_decision_queue - from ..docker_client import ContainerNotFoundError, ContainerOperationError, DockerClientError - from ..gateway_client import ( - GatewayError, - _rebase_with_agent_output_autoresolve, - ) - from ..kubernetes_client import ( - LABEL_AGENT_ROLE, - LABEL_PIPELINE_ID, - LABEL_SLICE_ID, - JobOperationError, - KubernetesClientError, - PodNotFoundError, - ) - from ..kubernetes_spawner import KubernetesSpawnError, get_kubernetes_spawner - from ..models import ( - LIVE_POD_STATUSES, - AgentExecutionStatus, - AgentExitInfo, - AgentRole, - AggregatedReviewResult, - ContainerInfo, - ContainerStatus, - CycleTiming, - DecisionStatus, - HITLDecision, - IterationSummary, - OperatorDirective, - PhaseExecution, - Pipeline, - PipelineMode, - PipelinePhase, - PipelineStatus, - RepoSpec, - ReviewVerdict, - ) - from ..slice_id_validation import SLICE_ID_PATTERN, extract_slice_id - from ..state_store import ( - InvalidPipelineIdError, - PipelineNotFoundError, - StateStore, - StateStoreError, - StateValidationError, - get_pipeline_state_lock, - get_state_store, - ) -except ImportError: - import agent_salvage # type: ignore[no-redef] - from container_spawner import ( # type: ignore - ContainerSpawnError, - SpawnFailureError, - get_container_spawner, - ) - from decision_queue import get_decision_queue # type: ignore - from docker_client import ( # type: ignore - ContainerNotFoundError, - ContainerOperationError, - DockerClientError, - ) - from gateway_client import ( # type: ignore - GatewayError, - _rebase_with_agent_output_autoresolve, - ) - from kubernetes_client import ( # type: ignore - LABEL_AGENT_ROLE, - LABEL_PIPELINE_ID, - LABEL_SLICE_ID, - JobOperationError, - KubernetesClientError, - PodNotFoundError, - ) - from kubernetes_spawner import ( # type: ignore - KubernetesSpawnError, - get_kubernetes_spawner, - ) - from models import ( # type: ignore - LIVE_POD_STATUSES, - AgentExecutionStatus, - AgentExitInfo, - AgentRole, - AggregatedReviewResult, - ContainerInfo, - ContainerStatus, - CycleTiming, - DecisionStatus, - HITLDecision, - IterationSummary, - OperatorDirective, - PhaseExecution, - Pipeline, - PipelineMode, - PipelinePhase, - PipelineStatus, - RepoSpec, - ReviewVerdict, - ) - from slice_id_validation import SLICE_ID_PATTERN, extract_slice_id # type: ignore - from state_store import ( # type: ignore - InvalidPipelineIdError, - PipelineNotFoundError, - StateStore, - StateStoreError, - StateValidationError, - get_pipeline_state_lock, - get_state_store, - ) - -from egg_contracts.markdown import unwrap_soft_breaks -from egg_contracts.orchestrator import load_agent_output, save_agent_output -from egg_git.default_branch import get_default_branch -from lifecycle_auth import require_lifecycle_secret - -if TYPE_CHECKING: - from egg_container import MountSpec - from egg_contracts.agent_roles import AgentRole as ContractAgentRole - from egg_contracts.models import Slice as ContractSlice - from overseer.corrective import CorrectiveExecutor - from overseer.decision_maker import AdjudicationVerdict - - try: - from ..container_spawner import ContainerSpawner - except ImportError: - from container_spawner import ContainerSpawner # type: ignore - - try: - from ..kubernetes_spawner import SpawnedContainer - except ImportError: - from kubernetes_spawner import SpawnedContainer # type: ignore - -logger = get_logger("orchestrator.pipelines") - - -# ----------------------------------------------------------------- -# egg_inflight_host_waits metric (issue #1932 TASK-1-3). -# -# Gauge counting in-flight ``/status/wait`` route calls. Paired with -# ``egg_inflight_long_polls`` from ``routes/messages.py`` — both draw -# against the same Waitress thread pool so operators alert on the -# sum when it approaches ``EGG_ORCH_WAITRESS_THREADS``. The -# lame-duck daemon thread that keeps running after the route returns -# (up to ``wait`` seconds of ``message_store.get_messages``) is -# deliberately NOT counted against this gauge — the metric represents -# in-flight *route* calls, not in-flight store waits. -# -# Best-effort registration so missing-metrics-backend deployments -# degrade gracefully (matches the pattern at routes/messages.py:80-85). -# ----------------------------------------------------------------- -try: - from metrics import get_metrics_registry as _get_metrics_registry_for_host_wait - - _inflight_host_waits = _get_metrics_registry_for_host_wait().gauge( - "egg_inflight_host_waits", - labels={"endpoint": "pipelines.status_wait"}, - ) -except Exception: # pragma: no cover - metrics best-effort - _inflight_host_waits = None - - -def _track_host_wait_start() -> None: - if _inflight_host_waits is not None: - try: - _inflight_host_waits.inc() - except Exception: # pragma: no cover - pass - - -def _track_host_wait_end() -> None: - if _inflight_host_waits is not None: - try: - _inflight_host_waits.dec() - except Exception: # pragma: no cover - pass - - -# ----------------------------------------------------------------- -# Cursor protocol for /status/wait (issue #1932 TASK-1-2). -# -# Opaque compound cursor "msg:<redis_stream_id>|evt:<sequence>": -# * ``msg:<id>`` is the message-store tip ID from the prior call. -# Either half may be empty when the corresponding source has not -# emitted yet (e.g. ``msg:|evt:5`` = "no message seen, EventBus -# tip at seq 5"). -# * ``evt:<sequence>`` is the EventBus per-bus monotonic sequence -# (see ``Event.sequence`` added in TASK-1-1). The sequence is -# signed purely so malformed inputs with leading ``-`` are -# accepted by the regex and handled gracefully by the parser. -# -# The regex is intentionally permissive — unknown halves degrade to -# ``None`` which the route maps to "snap to tip" (``from_tip`` on -# the message bus, ``current_sequence`` on the EventBus) so -# first-call semantics are race-free. -# ----------------------------------------------------------------- -_STATUS_WAIT_CURSOR_RE = re.compile(r"^msg:([^|]*)\|evt:(-?\d*)$") - -# Slice-or-phase id shape used when reading the parent edge from the -# contract for the restart route's ``base_branch`` derivation (#2439). -# ``Slice.id`` permits either ``slice-<N>`` (canonical) or ``phase-<N>`` -# (legacy, pre-#2137) and the contract migration shim only normalises -# the typical case where the input has a top-level ``phases`` key. A -# directly-loaded ``slices`` field with legacy ids is rare but allowed -# by the model — accept either shape here so the gate doesn't false- -# reject a legitimate restart on a long-lived contract. -_SLICE_OR_PHASE_ID_PATTERN = re.compile(r"^(?:slice|phase)-[0-9]+$") - -# Event allowlist for ``/status/wait`` (issue #1932 locked in -# refine HITL decision 2). The route returns early when an event -# matching any of these types is published. ``DECISION_RESOLVED`` -# is intentionally excluded — it is the post-``provide_input`` -# event and would cause the host to self-wake on an action it -# initiated. Agent-lifecycle events are excluded because the host -# does not drive on them. See -# docs/reference/agent-wait-patterns.md §7. -_STATUS_WAIT_EVENT_TYPES = frozenset( - { - "phase.started", - "phase.completed", - "decision.created", - "pipeline.completed", - "pipeline.failed", - "pipeline.cancelled", - } -) - -# Message-type allowlist for ``/status/wait`` (same HITL decision). -# Wired to ``message_store.get_messages(wait_for_types=...)`` so a -# message of a non-matching type does NOT unblock the waiter. -_STATUS_WAIT_MESSAGE_TYPES = ( - "OVERSEER_ALERT", - "CONSENSUS_CONFIRMED", - "CONSENSUS_NACK", - "CONSENSUS_RE_REVIEW", -) - - -def _parse_status_wait_cursor( - raw: str | None, -) -> tuple[bool, str | None, int | None]: - """Parse a ``/status/wait`` cursor. - - Returns ``(ok, msg_since_id, event_since_seq)`` where either half - may be ``None`` (meaning "snap to tip on this source"). ``ok`` - is False only for a syntactically malformed cursor — the route - returns 400 in that case. An empty / missing cursor is treated - as "snap to tip on both sources" (``ok=True, None, None``). - """ - if raw is None or raw == "": - return True, None, None - match = _STATUS_WAIT_CURSOR_RE.match(raw) - if not match: - return False, None, None - msg_part = match.group(1) - evt_part = match.group(2) - msg_since_id = msg_part if msg_part else None - event_since_seq: int | None = None - if evt_part: - try: - event_since_seq = int(evt_part) - except ValueError: # pragma: no cover — the regex guarantees digits/- - event_since_seq = None - return True, msg_since_id, event_since_seq - - -def _build_status_wait_cursor( - msg_tip_id: str | None, - event_tip_seq: int, -) -> str: - """Format a cursor for a ``/status/wait`` response. - - Both halves are emitted — the consumer treats empty halves as - "snap to tip" on the next call, matching ``_parse_status_wait_cursor``. - """ - msg_part = msg_tip_id or "" - return f"msg:{msg_part}|evt:{event_tip_seq}" - - -def _message_store_tip_id(pipeline_id: str) -> str | None: - """Best-effort read of the message-store tip ID for a pipeline. - - Used to build the initial / terminal cursor when the route - returns without matching a message. Returns ``None`` when the - store has no messages yet — the caller formats this as the - empty ``msg:`` half of the compound cursor. - - Three distinct conditions all collapse to ``None`` here and - callers cannot distinguish between them: - - 1. **Store import failure** — the message-store module is not - loadable in this process (test harness without Redis, - packaging skew). Pre-PR / post-#2464: same behavior. - 2. **Transient ``get_latest_id`` failure** — e.g., - :class:`redis.RedisError` from ``XREVRANGE`` on a connection - blip. ``RedisMessageStore.get_latest_id`` already catches - this and returns ``None``, so we see "no tip". This conflates - a transient error with a genuinely empty store; #2464's fix - at the call site (``_message_store_tip_id() or msg_since_id`` - removal) drops the consumer's cursor on this transient as - well, which is a small behavioral regression vs. pre-PR - graceful-degradation behavior. Acceptable in practice - because transient Redis errors degrade many other paths - simultaneously, but worth knowing. - 3. **Empty store** — the ``/status/wait`` post-clear case the - PR is fixing. Returning ``None`` lets the route emit an - empty ``msg:`` half so the consumer doesn't re-feed the - dead cursor. - """ - try: - store = _get_message_store()() - except Exception: # pragma: no cover — store may not be importable - return None - try: - return store.get_latest_id(pipeline_id) - except Exception: - return None - - -def _build_minimal_status_envelope( - pipeline: Pipeline, - cursor: str, -) -> dict[str, Any]: - """Compute the small envelope used on both wait paths. - - Ships ``current_phase`` / ``status`` / ``phase_elapsed_seconds`` - so dashboards can refresh cheaply on a timeout without paying - for a second round-trip. ``concurrent.consensus`` is also - included (R5 mitigation from the refine phase) so the host - does not miss a BRC state change during a quiet interval. - """ - phase_key = pipeline.current_phase.value if pipeline.current_phase else "" - phase_data = pipeline.phases.get(phase_key, None) - envelope: dict[str, Any] = { - "current_phase": phase_key, - "status": pipeline.status.value if pipeline.status else "", - "cursor": cursor, - } - if phase_data is not None: - started_at = getattr(phase_data, "started_at", None) - if started_at: - try: - if isinstance(started_at, str): - started_dt = datetime.fromisoformat(started_at) - else: - started_dt = started_at - if started_dt.tzinfo is None: - started_dt = started_dt.replace(tzinfo=UTC) - elapsed = int((datetime.now(UTC) - started_dt).total_seconds()) - envelope["phase_elapsed_seconds"] = max(0, elapsed) - except ValueError, TypeError, AttributeError: - pass - - concurrent_data = _get_concurrent_status(pipeline) - if concurrent_data and "consensus" in concurrent_data: - envelope["concurrent"] = {"consensus": concurrent_data["consensus"]} - return envelope - - -def _spawn_overseer_agent( - *, - spawner: "ContainerSpawner", # noqa: UP037 - pipeline_id: str, - issue_number: int | None, - gateway_mode: str, - pipeline_repos: list | None, - max_turns: int, - decision_model: str = "sonnet", - prompt_override: str | None = None, -) -> "SpawnedContainer": # noqa: UP037 - """Spawn the overseer as a normal agent (#2270 §1.5). - - The overseer is just a particular agent — it goes through the generic - :meth:`ContainerSpawner.spawn_agent_job` path with a command built by - ``build_agent_command`` exactly like every other role. There is no bespoke - spawn method, no ``EGG_OVERSEER_*`` env, and no baked-in - ``overseer_monitor.py`` bootstrap (that trust-and-run script was the direct - cause of the §1 self-injection loop). Monitoring arrives via the agent's - normal MCP tools / the ``egg-orch`` CLI. - - The overseer's model tier resolves through ``resolve_overseer_model`` (Opus - by default, #2270 §1 / folds #2813); the deprecated - ``overseer_decision_maker_model`` (passed as ``decision_model``) is inert and - only warns. A resolver regression degrades to the built-in opus/anthropic - default rather than crashing spawn. - """ - from agent_model_resolution import ( - DEFAULT_AGENT_MODEL, - UPSTREAM_ANTHROPIC, - classify_model, - resolve_overseer_model, - ) - from egg_agent import build_agent_command - - try: - from ..models import AgentRole - except ImportError: - from models import AgentRole # type: ignore[no-redef] - - # The overseer resolves its model from the pipeline's PRIMARY repo. - # ``pipeline_repos`` is canonically primary-first (#3393 slices 1-2), so - # take the first (primary) entry via ``next(iter(...))`` rather than a - # positional ``[0]`` collapse (#3393 slice-3). - overseer_repo = next(iter(pipeline_repos or []), None) - try: - overseer_decision = resolve_overseer_model( - "adversarial", - pipeline_config=None, - repo=overseer_repo, - ) - except Exception as resolve_err: # noqa: BLE001 — degrade, don't crash - logger.warning( - "Failed to resolve overseer model decision for spawn; " - "falling back to built-in opus / anthropic default", - error=str(resolve_err), - ) - overseer_decision = classify_model(DEFAULT_AGENT_MODEL) - - # The bespoke ``overseer_decision_maker_model`` no longer drives the spawn. - # Warn if an operator still sets it to a non-default value (#2270 §1 / #2813). - if decision_model and decision_model != "sonnet": - logger.warning( - "overseer_decision_maker_model=%r is deprecated and no longer " - "drives the overseer spawn; the base model now resolves via " - "resolve_agent_model(OVERSEER) -> %s. Set agent_models['overseer'] " - "to override. See #2270 §1 / #2813.", - decision_model, - overseer_decision.claude_code_alias, - ) - - # #2270 §1.5: no bespoke ``EGG_OVERSEER_*`` env — only the generic - # ``BASH_COMMAND_TIMEOUT`` (long-poll CLI calls) and the resolved-decision - # model env (custom-model registration + context guardrails, #2832/#3175). - extra_env = { - "BASH_COMMAND_TIMEOUT": "0", - **overseer_decision.env_vars(), - } - - # Monitoring arrives via MCP tools / the ``egg-orch`` CLI, not a baked-in - # script the agent is told to trust and run. The prompt describes the - # observe→classify→alert loop and leaves the mechanics to the agent's tools. - # When ``prompt_override`` is set (the #2270 slice-4 on-demand adjudicator), - # use it verbatim — a single-shot adjudication of one finding, not the - # continuous monitoring loop. - default_overseer_prompt = ( - f"You are the overseer agent for pipeline {pipeline_id}. You are a " - "normal egg agent with read-only monitoring permissions: there is no " - "baked-in script to run and no pre-built monitoring loop to trust. " - "Observe pipeline health using your MCP tools and the `egg-orch` CLI, " - "and surface only genuine anomalies.\n\n" - "Loop until the pipeline reaches a terminal state (complete, failed, or " - "cancelled):\n" - "1. Read the live pipeline state (`mcp__progress__query_status` or " - "`egg-orch pipeline status`), the BRC consensus matrix " - "(`mcp__brc__get_state`), and recent agent messages.\n" - "2. Classify what you see. The overwhelming majority of observations " - "are normal — only a wedged phase transition, a real consensus " - "deadlock, repeated agent crashes, or similar genuine failures warrant " - "action.\n" - "3. When (and only when) you find a real problem, broadcast a single " - "OVERSEER_ALERT with `mcp__progress__overseer_alert`, setting priority " - "by severity and naming the anomaly, the evidence, and a recommended " - "operator action.\n" - "4. Otherwise wait briefly and repeat.\n\n" - "Be conservative: a false alarm trains operators to ignore you, so " - "prefer silence over a low-confidence alert. When the pipeline ends, " - "emit a final health summary." - ) - overseer_prompt = prompt_override or default_overseer_prompt - command = build_agent_command( - prompt=overseer_prompt, - model=overseer_decision.claude_code_alias, - max_turns=max_turns, - effort=overseer_decision.effort, - ) - - spawn_kwargs: dict[str, Any] = { - "pipeline_id": pipeline_id, - "agent_role": AgentRole.OVERSEER, - "issue_number": issue_number, - "repo_volumes": None, - "mode": gateway_mode, - "extra_env": extra_env, - "repos": pipeline_repos if pipeline_repos else None, - "command": command, - } - # Forward per-agent upstream routing only when it would change behavior, so - # the default Anthropic overseer keeps the pre-#2769 call signature (mirrors - # ``concurrent_executor._spawn_agent``). - if ( - overseer_decision.upstream != UPSTREAM_ANTHROPIC - or overseer_decision.upstream_model is not None - ): - spawn_kwargs["upstream"] = overseer_decision.upstream - spawn_kwargs["upstream_model"] = overseer_decision.upstream_model - - return spawner.spawn_agent_job(**spawn_kwargs) - - -def _consume_adjudicator_verdict(spawned: Any, finding: Any) -> "AdjudicationVerdict": # noqa: UP037 - """Consume the structured verdict an on-demand adjudicator produced. - - Best-effort and defensive (#2270 slice-4). The adjudicator is a NORMAL - spawned agent; its structured verdict reaches the orchestrator either inline - on the spawn result (when a synchronous runner surfaces it as - ``adjudication_verdict`` / ``result_text``) or out-of-band. When no verdict - is available yet, we degrade to a conservative *defer-to-operator* verdict so - a genuine deadlock is never silently dropped — the slice-6 authority plane - executes on whatever this returns. - """ - from overseer.decision_maker import parse_adjudication_verdict - - raw: Any = None - for attr in ("adjudication_verdict", "result_text", "stdout"): - value = getattr(spawned, attr, None) - if value: - raw = value - break - return parse_adjudication_verdict(raw, finding=finding) - - -def _overseer_should_be_present( - *, running_agent_count: int, pipeline_status: PipelineStatus -) -> bool: - """Gate overseer presence on agents actually running (#2270 slice-5, §3). - - Decisive rules (the tester contract pins these exactly): - - * ``running_agent_count <= 0`` ⇒ ``False`` regardless of status — the §3 - guarantee that a multi-hour *zero-agent* HITL park spawns no overseer. - * a terminal pipeline status (``COMPLETE`` / ``FAILED`` / ``CANCELLED``) ⇒ - ``False`` regardless of the count — nothing left to monitor. - * otherwise (agents in flight, non-terminal) ⇒ ``True``. - - The overseer is only useful while a phase is actively executing agents, so - presence tracks "are there agents to watch", not the phase calendar. - """ - if running_agent_count <= 0: - return False - if pipeline_status in ( - PipelineStatus.COMPLETE, - PipelineStatus.FAILED, - PipelineStatus.CANCELLED, - ): - return False - return True - - -def _count_phase_agents(pipeline: Pipeline, phase: PipelinePhase) -> int: - """Count the agents a phase is about to run (#2270 slice-5 roster source). - - Prefers the runtime roster cached on the phase execution (populated once - the phase has spawned); falls back to the deterministic - ``get_roles_for_phase`` source the concurrent executor itself consults, so - a not-yet-spawned phase still reports its imminent cohort. A derivation - failure returns 0 — conservatively *no* overseer rather than guessing, - which keeps the §3 "no overseer with zero agents" invariant safe. - """ - phase_exec = pipeline.phases.get(phase) - if phase_exec is not None and getattr(phase_exec, "agents", None): - return len(phase_exec.agents) - try: - from egg_contracts.agent_roles import get_roles_for_phase - - roles = get_roles_for_phase( - phase.value, - include_reviewers=True, - repo=pipeline.repo, - has_contract=getattr(pipeline, "has_contract", True), - ) - return len(list(roles)) - except Exception as exc: # noqa: BLE001 - roster derivation is best-effort - logger.debug( - "Could not derive phase roster for overseer presence gate", - pipeline_id=getattr(pipeline, "id", None), - phase=getattr(phase, "value", str(phase)), - error=str(exc), - ) - return 0 - - -def _escalate_finding_to_adjudicator( - finding: Any, - *, - spawner: "ContainerSpawner", # noqa: UP037 - pipeline_id: str, - issue_number: int | None, - gateway_mode: str, - pipeline_repos: list | None, - max_turns: int = 3, - spawn_overseer: Any = None, - consume_verdict: Any = None, -) -> "AdjudicationVerdict | None": # noqa: UP037 - """Escalate a finding to an on-demand OVERSEER adjudicator (#2270 slice-4). - - The escalation→adjudicator path is the ONLY thing the orchestrator-side - overseership spends an agent on. The gate is strict: - - * a finding **without** ``requires_adjudication`` returns ``None`` and NEVER - spawns an adjudicator — the routine majority is handled deterministically; - * a finding **with** ``requires_adjudication`` spawns a NORMAL on-demand - OVERSEER agent (the slice-3 normalized spawn, Opus via the slice-2 - resolver) with a one-shot adjudication prompt, and the orchestrator - consumes its structured verdict in-process. - - ``spawn_overseer`` / ``consume_verdict`` are injectable seams so the path is - unit-testable without a live container; they default to - :func:`_spawn_overseer_agent` and :func:`_consume_adjudicator_verdict`. - """ - if not getattr(finding, "requires_adjudication", False): - return None # routine finding — deterministic handling, no agent spend - - from overseer.decision_maker import build_adjudication_prompt - - spawn = spawn_overseer or _spawn_overseer_agent - consume = consume_verdict or _consume_adjudicator_verdict - - prompt = build_adjudication_prompt(finding) - spawned = spawn( - spawner=spawner, - pipeline_id=pipeline_id, - issue_number=issue_number, - gateway_mode=gateway_mode, - pipeline_repos=pipeline_repos if pipeline_repos else None, - max_turns=max_turns, - prompt_override=prompt, - ) - verdict = consume(spawned, finding) - logger.info( - "Overseer adjudicated finding", - pipeline_id=pipeline_id, - finding_class=getattr(finding, "finding_class", "?"), - confirmed=getattr(verdict, "confirmed", None), - recommended_action=getattr(verdict, "recommended_action", None), - ) - return verdict - - -def _run_overseer_detection_plane( - snapshot: Any, - *, - spawner: "ContainerSpawner", # noqa: UP037 - pipeline_id: str, - issue_number: int | None, - gateway_mode: str, - pipeline_repos: list | None, - plane: Any = None, - max_turns: int = 3, -) -> "list[tuple[Any, AdjudicationVerdict | None]]": # noqa: UP037 - """Evaluate the detection plane and escalate only findings that need it. - - The orchestrator-side overseership spine (#2270 Option C, slice-4): run the - deterministic detectors over ``snapshot`` (no LLM), then escalate ONLY the - findings carrying ``requires_adjudication`` to the on-demand adjudicator. - Returns ``(finding, verdict)`` pairs — ``verdict`` is ``None`` for routine - findings that were handled deterministically without an agent. - - The default plane already carries the slice-8 §5 coverage-gap detectors - (registered in :meth:`DetectionPlane.default`), so production runs the full - detector set without any wiring here. - """ - from health_checks.detection_plane import default_detection_plane, escalate_findings - - active_plane = plane or default_detection_plane() - findings = active_plane.evaluate(snapshot) - - results: list[tuple[Any, Any]] = [] - - def _spawn_adjudicator(finding: Any) -> Any: - verdict = _escalate_finding_to_adjudicator( - finding, - spawner=spawner, - pipeline_id=pipeline_id, - issue_number=issue_number, - gateway_mode=gateway_mode, - pipeline_repos=pipeline_repos, - max_turns=max_turns, - ) - results.append((finding, verdict)) - return verdict - - # The canonical gate (health_checks.detection_plane.escalate_findings) calls - # the spawn callback exactly once per requires_adjudication finding and never - # for routine ones — a single source of truth shared with the tester contract. - escalate_findings(findings, spawn_adjudicator=_spawn_adjudicator) - return results - - -def _send_brc_confirmation_nudge( - escalation: dict[str, Any], - pipeline_id: str, - phase: str | None, -) -> bool: - """Wake a producer stuck post-ACK with a directed OVERSEER_ALERT (#2079). - - Wired as an escalation callback for HealthMonitor's - ``brc_confirmation_timeout`` alert. The deterministic detector in - ``check_brc_progress`` knows the exact remediation, so we deliver - it directly to the stuck producer rather than relying on the - overseer agent's discretion. - - Uses ``OVERSEER_ALERT`` (not ``STATUS`` or ``NUDGE``) because it - appears in **both** the producer's pre-confirm wait_loop filter - (``CONSENSUS_ACK,CONSENSUS_NACK,CONSENSUS_RE_REVIEW,STATUS,OVERSEER_ALERT``, - post-#2531) and post-confirm wait_loop filter - (``CONSENSUS_CONFIRMED,CONSENSUS_RE_REVIEW,OVERSEER_ALERT``) and has - no protocol-specific semantics that would conflict with a producer - nudge — ``CONSENSUS_RE_REVIEW`` is also in both filters but means - "a peer re-proposed; re-review their artifact," not "you are - wedged; confirm." ``STATUS`` is in the pre-confirm filter (it - carries the orchestrator's *Ready to confirm* nudge) but not the - post-confirm filter, so it wouldn't reach a producer wedged after - a successful confirm. A wedged producer is in the - ``fully_acked but not confirmed`` set, which means they are most - likely blocked on the pre-confirm wait. The subject calls out - that the alert originated from the orchestrator's deterministic - detector rather than the overseer agent. - - Returns True when a message was posted, False otherwise (wrong - alert type, missing fields, message store unavailable, send error). - """ - if escalation.get("alert_type") != "brc_confirmation_timeout": - return False - - producer = escalation.get("agent_id") - if not producer: - return False - - elapsed = escalation.get("elapsed_seconds") - # check_brc_progress always populates elapsed_seconds; treat - # missing or non-positive values as a malformed escalation rather - # than rendering "have not confirmed in 0s" in the body. - if elapsed is None or elapsed <= 0: - return False - - store_fn = _get_message_store() - if store_fn is None: - return False - - # _get_message_store already verified the package is importable; - # Message/MessageType live in the same module so a defensive - # try/except here would only add per-call import overhead. - from message_store import Message, MessageType - - body = ( - f"You are PROPOSED and fully ACKed but have not confirmed in " - f"{elapsed}s. Call `mcp__brc__confirm` now. If it returns " - "`status='pending_acks'`, read `message` for the guard reason and " - "wait on the prerequisite events instead: `CONSENSUS_PROPOSE` if a " - "producer hasn't proposed (`zero_proposal_producers`), " - "`CONSENSUS_ACK` / `CONSENSUS_RE_REVIEW` if a reviewer's ACK is " - "stale or unresolved. Then retry confirm." - ) - - try: - msg_store = store_fn() - # Bypass the POST /messages/send route on purpose: this is an - # orchestrator-internal nudge, and we do not want HealthMonitor's - # MESSAGE_SENT handler (rate-limit + HEARTBEAT tracking) to see it. - # Future audit/observability subscribers should be aware this path - # does not emit EventType.MESSAGE_SENT. - msg_store.add_message( - Message( - pipeline_id=pipeline_id, - from_role="orchestrator", - to_role=producer, - message_type=MessageType.OVERSEER_ALERT, - subject="BRC confirmation timeout — call mcp__brc__confirm", - body=body, - phase=phase, - metadata={ - "alert_type": "brc_confirmation_timeout", - "elapsed_seconds": elapsed, - "source": "health_monitor", - }, - ) - ) - logger.info( - "Sent BRC confirmation-timeout nudge", - pipeline_id=pipeline_id, - producer=producer, - elapsed_seconds=elapsed, - ) - return True - except Exception as send_err: - logger.warning( - "Failed to send BRC confirmation-timeout nudge (non-fatal)", - pipeline_id=pipeline_id, - producer=producer, - error=str(send_err), - ) - return False - - -# --------------------------------------------------------------------------- -# Overseer authority plane (#2270 slice-6, §4) — the orchestrator-side seams the -# CorrectiveExecutor dispatches to. The overseer ADVISES (returns a verdict); the -# control plane EXECUTES exactly three bounded actions. Agents — including the -# overseer — cannot reach these directly: the gateway file patterns deny agents -# from contract writes (the "403"), and the executor only runs control-plane-side. -# The seams are invoked by CorrectiveExecutor with keyword arguments. See -# orchestrator/overseer/corrective.py and gateway/agent_restrictions.py. -# --------------------------------------------------------------------------- - - -def _corrective_open_operator_hitl( - *, - pipeline_id: str, - issue_number: int | None = None, - repo_path: Any = None, - question: str | None = None, - options: Any = None, - finding: Any = None, - phase: str | None = None, - **_: Any, -) -> str: - """``open_operator_hitl`` seam: open a HITL contract decision (orchestrator id). - - The decision is written via :func:`apply_mutation` under ``Role.IMPLEMENTER`` - — the same ``decisions.*`` owner the ``register_open_question`` MCP tool and - the impasse router use — with an orchestrator-side actor so the audit trail - stays distinct from agent-authored decisions. This is the REAL enforcement - point: the contract write runs as the control plane (which has no gateway - agent pattern), while agents — incl. the overseer — stay blocked from - ``.egg-state/contracts/``. Returns the new decision id. - """ - from egg_contracts.decisions import next_cq_id - from egg_contracts.loader import load_contract, save_contract - from egg_contracts.models import Decision, DecisionOption, DecisionType - from egg_contracts.roles import Role - from egg_contracts.validator import apply_mutation - - identifier = _pipeline_identifier(issue_number, pipeline_id) - resolved_repo = repo_path or get_repo_path() - contract = load_contract(identifier, resolved_repo) - existing = contract.decisions or [] - next_idx = len(existing) - decision_id = next_cq_id(existing) - - finding_class = str(getattr(finding, "finding_class", "") or "") - severity = str(getattr(finding, "severity", "") or "medium") - - if question: - question_text = question - else: - lines = [ - f"The overseer detection plane flagged ``{finding_class or 'an anomaly'}`` " - f"(severity ``{severity}``) in pipeline ``{pipeline_id}`` and the on-demand " - "adjudicator escalated it for operator judgement.", - ] - evidence = getattr(finding, "evidence", None) - if evidence: - lines.append(f"**Evidence**: {evidence}") - question_text = "\n".join(lines) - - if options: - decision_options = [ - DecisionOption(id=f"opt-{i + 1}", label=str(label)) for i, label in enumerate(options) - ] - else: - decision_options = [ - DecisionOption(id="opt-1", label="Intervene now (operator will act manually)"), - DecisionOption(id="opt-2", label="Dismiss — detector over-fired (calibration data)"), - DecisionOption(id="opt-3", label="Other (explain in reply)"), - ] - - decision = Decision( - id=decision_id, - question=question_text, - type=DecisionType.HITL, - phase=contract.current_phase, - options=decision_options, - ) - result = apply_mutation( - contract, - role=Role.IMPLEMENTER, - actor="orchestrator-overseer-corrective", - field_path=f"decisions.{next_idx}", - new_value=decision, - reason=f"Overseer corrective: open operator HITL for {finding_class or 'finding'}", - ) - if not result.success: - raise RuntimeError(f"failed to open operator HITL decision: {result.message}") - save_contract(contract, resolved_repo) - # NOTE(#3427): like ``route_impasses``, this overseer-corrective writer - # lands the ``cq-N`` decision with a bare ``save_contract`` and no - # write-time ``persist_contract_statefiles`` — so a HITL opened between - # checkpoints shares the same phase-restart volatility window (the - # ``git reset --hard origin/<work>`` can revert it). The append-only - # guard protects it from id reuse, but not from reversion. Not persisted - # here because the corrective seam runs against ``get_repo_path()`` (the - # base repo), not a pushable pipeline worktree — wiring a worktree-scoped - # persist through the CorrectiveExecutor is the residual follow-up. - return decision_id - - -def _corrective_nudge_agent( - *, - pipeline_id: str, - target_role: str | None = None, - phase: str | None = None, - finding: Any = None, - escalation: dict[str, Any] | None = None, - **_: Any, -) -> bool: - """``nudge_agent`` seam: deliver the deterministic BRC-confirmation nudge. - - Wires to :func:`_send_brc_confirmation_nudge` (the #2079 directed wake), which - posts an ``OVERSEER_ALERT`` the stuck producer's wait-loop filters admit. An - explicit ``escalation`` dict is used when present, otherwise synthesized in - the ``brc_confirmation_timeout`` shape that helper requires. Returns whether - the nudge was delivered. - """ - payload = dict(escalation or {}) - payload.setdefault("alert_type", "brc_confirmation_timeout") - payload.setdefault("agent_id", target_role) - elapsed = payload.get("elapsed_seconds") - payload["elapsed_seconds"] = elapsed if (elapsed and elapsed > 0) else 1 - return _send_brc_confirmation_nudge(payload, pipeline_id, phase) - - -def _corrective_respawn_cohort( - *, - pipeline_id: str, - target_role: str | None = None, - reason: str | None = None, - **_: Any, -) -> bool: - """``respawn_cohort`` seam: restart the target role(s) via the general path. - - Delegates to the orchestrator's public restart endpoint - (``POST /agents/<role>/restart``) — the same general-restart machinery the - overseer monitor's ``_execute_restart_agent`` uses — so restart-budget - enforcement, consensus reset, and one-shot Job teardown all happen - server-side, with no bespoke respawn plumbing. ``target_role`` may be a single - role or a comma-separated cohort. Returns whether every role restarted. - """ - import urllib.request - from urllib.parse import quote - - roles = [r.strip() for r in str(target_role or "").split(",") if r.strip()] - if not roles: - raise RuntimeError("respawn_cohort: empty target cohort") - - orchestrator_url = os.environ.get("EGG_ORCHESTRATOR_URL", "http://localhost:9849") - restart_reason = (reason or "overseer corrective respawn")[:500] - for role in roles: - restart_url = ( - f"{orchestrator_url}/api/v1/pipelines/" - f"{quote(pipeline_id, safe='')}/agents/{quote(role, safe='')}/restart" - ) - req = urllib.request.Request( - restart_url, - data=json.dumps({"reason": restart_reason}).encode(), - headers={"Content-Type": "application/json"}, - method="POST", - ) - opener = urllib.request.build_opener(urllib.request.ProxyHandler({})) - with opener.open(req, timeout=60) as resp: - result = json.loads(resp.read().decode()) - if not result.get("success"): - raise RuntimeError( - f"restart of {role!r} failed: {result.get('message', 'unknown error')}" - ) - return True - - -def _build_overseer_corrective_executor( - *, - issue_number: int | None = None, - repo_path: Any = None, - config: Any = None, - audit_sink: Any = None, - open_operator_hitl: Any = None, - nudge_agent: Any = None, - respawn_cohort: Any = None, -) -> "CorrectiveExecutor": # noqa: UP037 - """Construct the §4 :class:`CorrectiveExecutor` wired to the production seams. - - Seams are injectable so the path stays unit-testable without a live - orchestrator. The default ``open_operator_hitl`` seam is bound to the - pipeline's ``issue_number`` / ``repo_path`` so it can resolve the contract. - The rate-limit window derives from the overseer config when present, falling - back to the executor default. - """ - from overseer.corrective import CorrectiveExecutor - - def _default_open_hitl(**kwargs: Any) -> str: - kwargs.setdefault("issue_number", issue_number) - kwargs.setdefault("repo_path", repo_path) - return _corrective_open_operator_hitl(**kwargs) - - kwargs: dict[str, Any] = {} - window = getattr(config, "overseer_infra_error_dedup_window_seconds", None) - if isinstance(window, int) and window > 0: - kwargs["window_seconds"] = float(window) - - return CorrectiveExecutor( - open_operator_hitl=open_operator_hitl or _default_open_hitl, - nudge_agent=nudge_agent or _corrective_nudge_agent, - respawn_cohort=respawn_cohort or _corrective_respawn_cohort, - audit_sink=audit_sink, - **kwargs, - ) - - -def _execute_overseer_verdicts( - results: list[tuple[Any, Any]], - *, - pipeline_id: str, - issue_number: int | None, - running_agent_count: int, - phase: str | None = None, - executor: Any = None, -) -> list[Any]: - """Run the §4 authority plane over adjudicated ``(finding, verdict)`` pairs. - - For each pair carrying a verdict, dispatch the recommended action through the - :class:`CorrectiveExecutor`. The executor enforces the closed vocabulary (a - ``none`` recommendation is skipped here as the non-executable no-op), the - zero-agent-park bar, rate-limiting, idempotency, and audit logging. Returns - the per-verdict :class:`CorrectiveOutcome` list (empty when nothing was - adjudicated or actioned). - """ - active = executor or _build_overseer_corrective_executor(issue_number=issue_number) - outcomes: list[Any] = [] - for finding, verdict in results: - if verdict is None: - continue # routine finding — handled deterministically, no action - action = str(getattr(verdict, "recommended_action", "") or "").strip() - if action in ("", "none"): - continue # adjudicator advised no action — nothing to execute - evidence = getattr(finding, "evidence", None) or {} - target_role = str(getattr(verdict, "target", "") or "") or str( - evidence.get("agent_role") or evidence.get("agent_id") or "" - ) - finding_class = str(getattr(finding, "finding_class", "") or "") - outcomes.append( - active.execute( - action, - pipeline_id=pipeline_id, - running_agent_count=running_agent_count, - phase=phase, - target_role=target_role, - finding=finding, - idempotency_key=f"{finding_class}:{target_role}" if finding_class else None, - ) - ) - return outcomes - - -def _teardown_phase_overseer( - spawner: "ContainerSpawner", # noqa: UP037 - container_id: str, - pipeline_id: str, - phase_label: str, - reason: str, -) -> None: - """Stop the phase-scoped overseer container. - - Caller is responsible for holding ``overseer_lock`` and setting - ``phase_overseer_active = False`` before this call. - """ - try: - spawner.stop_agent_container( - container_id, - cleanup_session=True, - timeout=10, - ) - logger.info( - f"Overseer container stopped ({reason})", - pipeline_id=pipeline_id, - phase=phase_label, - container_id=container_id[:12], - ) - except Exception as overseer_err: - logger.debug( - f"Failed to stop overseer container ({reason})", - pipeline_id=pipeline_id, - error=str(overseer_err), - ) - - -# Base directory where the gateway creates per-pipeline worktrees. -# Must match the gateway's WORKTREE_BASE_DIR and docker-compose volume mounts. -WORKTREE_BASE_DIR = Path("/home/egg/.egg-worktrees") - -# Sentinel header used in tester gap summaries. Checked in prompt-building -# functions to adapt language when tester findings are present. -TESTER_FINDINGS_HEADER = "### tester findings" - - -def _ensure_pipeline_work_ref(branch: str | None) -> str | None: - """Return the actual remote ref for an orchestrator-managed pipeline branch. - - The orchestrator pushes the pipeline tip to ``<branch>/work`` so the - ``<branch>/`` namespace can hold slice integration branches as - siblings (``<branch>/slice-N``) without git's ``directory file - conflict`` rejection — see #2399. A leaf ref at ``<branch>`` and a - child at ``<branch>/slice-N`` cannot coexist on origin, so the - pipeline tip is moved one level deeper into the namespace. - - Idempotent and bounded to ``egg/<id>``-shaped branches: - - * ``None`` → ``None`` (prompt-driven; the caller generates a - ``/work``-shaped branch later). - * ``egg/<id>`` → ``egg/<id>/work`` (issue submissions). - * ``egg/<id>/work`` → unchanged (resubmission, internal callers). - * non-``egg/`` (passed unchanged) — a pipeline pointed at a foreign - branch (e.g. ``feature/foo``). Slices on a non-``egg/`` branch are - not a guaranteed-safe shape and are intentionally not normalised - here — the conflict would resurface at the slice push and is - tracked separately. - - The trailing-``/work`` check is structural rather than a plain - suffix match (``branch.count("/") >= 2 and branch.rsplit("/", 1)[1] - == "work"``) so a degenerate input like ``egg/work`` — a single - segment that *happens* to end in ``/work`` — gets normalised to - ``egg/work/work`` (siblings ``egg/work/slice-N``) rather than - treated as already-normalised. Trailing slashes are stripped first - so ``egg/`` does not collapse to a double-slash ``egg//work``. - """ - if branch is None: - return None - branch = branch.rstrip("/") - if not branch.startswith("egg/"): - return branch - # Structural check: only treat ``egg/<id>/work`` (≥2 slashes, last - # segment is ``work``) as already-normalised. ``egg/work`` looks - # like a suffix match but is a single-segment id and still needs the - # ``/work`` namespace deepening. - if branch.count("/") >= 2 and branch.rsplit("/", 1)[1] == "work": - return branch - return f"{branch}/work" - - -def _slice_namespace_root(pipeline_branch: str) -> str: - """Return the slice-integration-branch namespace root for a pipeline branch. - - Slice integration branches live as siblings of the pipeline tip - under ``egg/<id>/`` (see :func:`_ensure_pipeline_work_ref`). The - namespace root is the pipeline branch with the trailing ``/work`` - stripped — that's the prefix slice paths (``<root>/slice-N``) are - built from. For legacy / non-normalised branches that do not end in - ``/work``, the branch itself is the root. - - The trailing-``/work`` check mirrors the structural check in - :func:`_ensure_pipeline_work_ref` (≥2 slashes, last segment is - ``work``) so a degenerate single-segment input like ``egg/work`` - is treated as the root itself rather than collapsing to ``egg``. - """ - if pipeline_branch.count("/") >= 2 and pipeline_branch.rsplit("/", 1)[1] == "work": - return pipeline_branch.rsplit("/", 1)[0] - return pipeline_branch - - -def _pipeline_identifier( - issue_number: int | None, - pipeline_id: str, -) -> int | str: - """Derive the pipeline identifier used for namespaced .egg-state filenames. - - Prefers ``issue_number`` when available, falling back to ``pipeline_id``. - - A pipeline whose id carries a qualifier beyond the bare ``issue-<N>`` - form (e.g. ``issue-1557-v2`` for a versioned re-run) keys by - ``pipeline_id`` instead, so concurrent pipelines on the same issue - don't collide on ``.egg-state/drafts/<N>-analysis.md``. - """ - if pipeline_id and issue_number is not None: - expected_issue_prefix = f"issue-{issue_number}" - if pipeline_id.startswith(expected_issue_prefix + "-"): - # A qualifier is present beyond the bare ``issue-<N>`` form; - # key by pipeline_id so concurrent runs on the same issue do - # not collide on draft files. - return pipeline_id - return issue_number if issue_number is not None else pipeline_id - - -def _brc_history_identifier(pipeline) -> int | str: - """Return the identifier used to namespace BRC-history artifacts. - - Mirrors :func:`_pipeline_identifier` (favouring the issue number). - """ - return _pipeline_identifier( - getattr(pipeline, "issue_number", None), - getattr(pipeline, "id", "") or "", - ) - - -# Network constants for sandbox container URLs -try: - from egg_config import ( - ORCHESTRATOR_EXTERNAL_IP, - ORCHESTRATOR_ISOLATED_IP, - ORCHESTRATOR_PORT, - ) -except ImportError: - ORCHESTRATOR_ISOLATED_IP = "172.32.0.3" - ORCHESTRATOR_EXTERNAL_IP = "172.33.0.3" - ORCHESTRATOR_PORT = 9849 - -try: - from egg_config.validators import validate_checks -except ImportError: - - def validate_checks(checks: list) -> list[dict[str, str]]: # type: ignore[misc] - if not isinstance(checks, list): - return [] - return [ - {"name": str(c["name"]), "command": str(c["command"])} - for c in checks - if isinstance(c, dict) and "name" in c and "command" in c - ] - - -pipelines_bp = Blueprint("pipelines", __name__, url_prefix="/api/v1/pipelines") - - -# Runtime detection: use Kubernetes spawner when EGG_RUNTIME=kubernetes -_RUNTIME = os.environ.get("EGG_RUNTIME", "docker") - - -def _get_spawner(): - """Get the appropriate spawner for the current runtime. - - Returns KubernetesSpawner when EGG_RUNTIME=kubernetes, otherwise - ContainerSpawner (Docker). - """ - if _RUNTIME == "kubernetes": - return get_kubernetes_spawner() - return get_container_spawner() - - -# Live-pod status filter (#2420). Hoisted to ``models.LIVE_POD_STATUSES`` -# in #2650 so ``startup_reconciliation`` and this module can't drift; -# this alias preserves the historical underscore-prefixed name used by -# existing tests and prose references. -_LIVE_POD_STATUSES = LIVE_POD_STATUSES - - -def _count_live_pods_for_pipeline(pipeline_id: str, *, quiet: bool = False) -> int | None: - """Count live pods labeled to this pipeline (#2420). - - Live = ``ContainerStatus`` in :data:`_LIVE_POD_STATUSES` (Pending / - Creating / Running). Pods in terminal phases (``Failed`` / ``Succeeded`` - → ``ContainerStatus.FAILED`` / ``EXITED``) are excluded — they have - already exited and the start_pipeline reset orphans no work tied to - them. - - Returns the number of live pods, or ``None`` if the label query failed — - callers must distinguish "verified zero" from "unknown" because the - start_pipeline reset would orphan any pods we couldn't see. - - ``quiet=True`` suppresses the helper-level warning when the label query - fails. The guard's ``force=true`` branch passes this flag because it - emits its own structured audit log on the ``live is None`` path; the - helper's warning would just duplicate it. - """ - try: - spawner = _get_spawner() - pods = spawner.backend.list_containers( - labels={LABEL_PIPELINE_ID: pipeline_id}, - ) - return sum(1 for p in pods if p.status in _LIVE_POD_STATUSES) - except Exception as e: - if not quiet: - logger.warning( - "start_pipeline live-pod check failed", - pipeline_id=pipeline_id, - error=str(e), - ) - return None - - -def _live_event_agents(pipeline_id: str, slice_id: str | None) -> list[dict[str, Any]]: - """Running-agent view reconstructed from live Job labels (#3230). - - Under the orchestrator-owned BRC event loop (#3164, now unconditional) - each role's pod is an on-demand one-shot the loop deliberately does NOT - persist into ``phase_exec.agents`` — ``event_loop.py`` treats the - consensus tracker plus live-Job labels as the only sources of truth. So - the persisted agent list is empty even while role pods are ``Running``, - which the dashboard (``get_status.running_agents``) and the overseer - (``concurrent.agents`` stall-duration math) both read as "0 running - agents" — a blind dashboard and false ``phase stalled`` alerts. - - This reconstructs the running-pod cohort from the labels that ARE - authoritative. Live = ``status`` in :data:`_LIVE_POD_STATUSES` - (Pending / Creating / Running); terminal pods lingering in the - ``ttlSecondsAfterFinished`` window are excluded so between-spawn - quiescence reads as "no running agents" (the normal idle state, not a - stall). Scoped to ``slice_id`` when supplied so a slice-DAG implement - phase reports its own slice's pods rather than a cross-slice union; - refine/plan phases are unsliced and query by pipeline label alone. - - Entry shape mirrors the persisted ``agents`` entries (``role`` / - ``status`` / ``started_at`` / ``elapsed_seconds`` / ``container_id``) - so consumers need no special-casing. ``status`` is reported as - ``"running"`` for every live pod — Pending/Creating pods are agents - spinning up, and the dashboard's running-agent filter keys on that - literal. - - Best-effort: an absent/failed label query yields ``[]`` (callers treat - that identically to "no persisted agents", so there is no regression - versus the pre-fix behavior). - """ - try: - spawner = _get_spawner() - labels = {LABEL_PIPELINE_ID: pipeline_id} - if slice_id: - labels[LABEL_SLICE_ID] = slice_id - pods = spawner.backend.list_containers(labels=labels) - except Exception as e: # noqa: BLE001 — observability backfill is best-effort - logger.debug( - "Live event-agent backfill query failed (#3230)", - pipeline_id=pipeline_id, - slice_id=slice_id, - error=str(e), - ) - return [] - - now = datetime.now(UTC) - entries: list[dict[str, Any]] = [] - for pod in pods: - if pod.status not in _LIVE_POD_STATUSES: - continue - role = pod.agent_role.value if pod.agent_role is not None else None - if not role: - continue - entry: dict[str, Any] = {"role": role, "status": "running"} - if isinstance(pod.container_id, str) and pod.container_id: - entry["container_id"] = pod.container_id - started_at = pod.started_at - if isinstance(started_at, datetime): - started_dt = started_at if started_at.tzinfo else started_at.replace(tzinfo=UTC) - entry["started_at"] = started_dt.isoformat() - entry["elapsed_seconds"] = max(0, int((now - started_dt).total_seconds())) - entries.append(entry) - return entries - - -def _slice_agents_alive(spawner: Any, pipeline_id: str, slice_id: str) -> bool: - """Check if any live agents exist for a slice (#2914). - - Returns ``True`` if at least one pod labeled with the pipeline and - slice IDs is in a live state (Pending/Creating/Running). Returns - ``False`` if zero live pods or if the label query fails — the - conservative default forces re-spawn rather than risking a wedge. - - Caller contract: callers must have already torn down stale cohorts - with foreground propagation (e.g. ``restart_phase`` step 4 calls - ``remove_agent_container(force=True)``). A pod whose Job is being - deleted but is still in its termination grace period still reports - ``phase=Running`` (``kubernetes_client.py`` maps Running → RUNNING - without a Terminating-specific status), so without foreground - teardown the helper can false-positive against terminating pods - and wedge again. The ``spawner`` is taken as a parameter (rather - than fetched via ``_get_spawner``) so tests can inject a stub - directly, paralleling how ``_classify_non_complete_slice`` - receives ``gateway``. - """ - try: - pods = spawner.backend.list_containers( - labels={ - LABEL_PIPELINE_ID: pipeline_id, - LABEL_SLICE_ID: slice_id, - }, - ) - live_count = sum(1 for p in pods if p.status in _LIVE_POD_STATUSES) - return live_count > 0 - except Exception as e: # noqa: BLE001 - logger.warning( - "Slice liveness check failed; treating as not-alive to force re-spawn (#2914)", - pipeline_id=pipeline_id, - slice_id=slice_id, - error=str(e), - ) - return False - - -def _guard_live_pods_or_force( - pipeline_id: str, - force: bool, - force_reason: str | None, -) -> tuple[Response, int] | None: - """Refuse a phase reset that would orphan live pods (#2420). - - Returns ``None`` when the reset is safe to proceed (zero live pods, or - ``force=true``). Returns a 409 ``(response, status)`` when live pods are - present (or the label query failed) and the caller did not pass - ``force=true``. - """ - if force: - # ``quiet=True`` because the ``live is None`` branch below emits - # its own structured audit log; the helper-level warning would - # just duplicate it on the override path. - live = _count_live_pods_for_pipeline(pipeline_id, quiet=True) - # Template the audit log so the static message reflects what the - # override actually did. ``live == 0`` means the override was a - # no-op — log at ``info`` so it doesn't read like a near-miss. - if live is None: - logger.warning( - "start_pipeline force=true override; live-pod check failed, " - "phase reset will proceed regardless", - pipeline_id=pipeline_id, - live_pod_count=None, - force_reason=force_reason, - ) - elif live > 0: - logger.warning( - "start_pipeline force=true override; phase reset will proceed " - "and orphan live pods labeled to the pipeline", - pipeline_id=pipeline_id, - live_pod_count=live, - force_reason=force_reason, - ) - else: - logger.info( - "start_pipeline force=true override applied (no live pods present)", - pipeline_id=pipeline_id, - live_pod_count=0, - force_reason=force_reason, - ) - return None - - live = _count_live_pods_for_pipeline(pipeline_id) - if live is None: - return make_error_response( - f"Could not verify live pod count for pipeline {pipeline_id}; " - "the start_pipeline reset would orphan any pods labeled to it. " - "Cancel them first via cancel_task(cleanup=true) or pass " - "force=true to override.", - status_code=409, - reason="live_pod_check_failed", - ) - if live > 0: - return make_error_response( - f"Pipeline {pipeline_id} has {live} live pod(s); the " - "start_pipeline reset would orphan them. Cancel them first via " - "cancel_task(cleanup=true) or pass force=true to override.", - status_code=409, - details={"live_pod_count": live}, - reason="live_pods_present", - ) - return None - - -from routes import get_repo_path # noqa: E402 — shared helpers - -try: - from gateway_client import get_gateway_client -except ImportError: - from orchestrator.gateway_client import get_gateway_client # type: ignore - -# Import status reporter for real-time updates -try: - from status_reporter import get_status_reporter, report_pipeline_status -except ImportError: - # Fallback if status_reporter not available - def get_status_reporter(): # type: ignore[misc] - return None - - def report_pipeline_status(pipeline, event_type=None, message=None): # type: ignore[misc] - pass - - -# Import event bus for SSE streaming. -# report_pipeline_status dispatches to StatusReporter handlers, but the -# SSE stream subscribes to the EventBus — a separate system. We need to -# emit events to both so SSE clients see live updates. -try: - from events import EventType - from events import emit_event as _emit_event -except ImportError: - _emit_event = None # type: ignore[assignment] - -# Map report_pipeline_status event_type strings to EventType enum values -_EVENT_TYPE_MAP: dict[str, EventType] = {} -if _emit_event is not None: - _EVENT_TYPE_MAP = { - "phase.started": EventType.PHASE_STARTED, - "phase.completed": EventType.PHASE_COMPLETED, - "phase.revision_requested": EventType.PHASE_STARTED, # re-entering phase - "pipeline.completed": EventType.PIPELINE_COMPLETED, - "pipeline.failed": EventType.PIPELINE_FAILED, - "pipeline.cancelled": EventType.PIPELINE_CANCELLED, - "decision.created": EventType.DECISION_CREATED, - } - - -def _emit_pipeline_event( - pipeline: Pipeline, - event_type_str: str, -) -> None: - """Emit a pipeline event to the EventBus for SSE streaming.""" - if _emit_event is None: - return - mapped = _EVENT_TYPE_MAP.get(event_type_str) - if mapped is None: - return - _emit_event( - mapped, - pipeline.id, - data={ - "status": pipeline.status.value, - "phase": pipeline.current_phase.value, - }, - ) - - -# Import visualization modules for DAG endpoint -try: - from dag_visualizer import ( - generate_status_report, - render_compact_status, - render_pipeline_dag, - render_progress_bar, - ) - - _DAG_VISUALIZER_AVAILABLE = True -except ImportError: - _DAG_VISUALIZER_AVAILABLE = False - -# Import SSE streaming support -try: - from sse import create_sse_stream - - _SSE_AVAILABLE = True -except ImportError: - _SSE_AVAILABLE = False - -# Import unified SSE streaming support -try: - from unified_sse import create_unified_sse_stream - - _UNIFIED_SSE_AVAILABLE = True -except ImportError: - _UNIFIED_SSE_AVAILABLE = False - - -def make_error_response( - message: str, - status_code: int = 400, - details: dict[str, Any] | None = None, - reason: str | None = None, -) -> tuple[Response, int]: - """Create an error response. - - ``reason`` is a stable, machine-readable enum-like code that disambiguates - responses sharing the same HTTP status (especially 409, where distinct - gates would otherwise collapse into one signal). Callers should switch on - ``reason`` rather than parsing ``message``. See #1939. - """ - response: dict[str, Any] = {"success": False, "message": message} - if reason is not None: - response["reason"] = reason - if details: - response["details"] = details - return jsonify(response), status_code - - -def make_success_response( - message: str, - data: dict[str, Any] | None = None, -) -> tuple[Response, int]: - """Create a success response.""" - response: dict[str, Any] = {"success": True, "message": message} - if data: - response["data"] = data - return jsonify(response), 200 - - -def _resolve_pipeline(pipeline_id: str, base_path: Path) -> tuple[StateStore, Pipeline]: - """Load a pipeline, resolving the correct repo subdirectory. - - Each repo has its own state store and worktree. This function - searches all repos under ``base_path`` to find the pipeline. - - Returns: - (store, pipeline) tuple - - Raises: - PipelineNotFoundError: if the pipeline cannot be found anywhere - InvalidPipelineIdError: if the ID format is invalid - GitOperationError: if the state-store worktree cannot be loaded - (e.g. ``git worktree add`` contention). Callers should - surface this as 500, not 404 — it is recoverable - infrastructure failure, not a missing pipeline. - """ - from state_store import discover_repo_paths - - for repo_path in discover_repo_paths(base_path): - try: - store = get_state_store(repo_path) - pipeline = store.load_pipeline(pipeline_id) - return store, pipeline - except PipelineNotFoundError: - continue - # NOTE: do NOT broaden this to ``StateStoreError``. Swallowing - # ``GitOperationError`` here re-raised every state-store wedge - # as ``Pipeline not found`` and surfaced to operators as 404, - # masking a recoverable git contention as a missing pipeline - # (#2167). Let infrastructure failures propagate so the route - # can return 500 with the actual error. - - raise PipelineNotFoundError(f"Pipeline {pipeline_id} not found") from None - - -def _collect_all_pipelines(base_path: Path) -> list: - """Collect pipelines from all git repos under base_path. - - Each repo has its own state store and worktree. Pipelines are - deduplicated by ID in case of overlapping stores. - """ - from state_store import discover_repo_paths - - seen: set[str] = set() - pipelines = [] - - def _add_from_store(store): - for pid in store.list_pipelines(): - if pid in seen: - continue - try: - pipelines.append(store.load_pipeline(pid)) - seen.add(pid) - except StateStoreError: - continue - - for repo_path in discover_repo_paths(base_path): - try: - _add_from_store(get_state_store(repo_path)) - except StateStoreError: - continue - - return pipelines - - -@pipelines_bp.route("", methods=["GET"]) -def list_pipelines() -> tuple[Response, int]: - """ - List all pipelines. - - Query params: - repo_path: Path to repository (optional) - active_only: Only return active pipelines (default: false) - - Response: - { - "success": true, - "data": { - "pipelines": [ - {"id": "issue-123", "status": "running", ...}, - ... - ] - } - } - """ - repo_path = get_repo_path() - active_only = request.args.get("active_only", "false").lower() == "true" - - try: - all_pipelines = _collect_all_pipelines(repo_path) - - if active_only: - pipelines = [ - p - for p in all_pipelines - if p.status - not in ( - PipelineStatus.COMPLETE, - PipelineStatus.FAILED, - PipelineStatus.CANCELLED, - ) - ] - else: - pipelines = all_pipelines - - # Convert to response format - pipeline_data = [ - { - "id": p.id, - "issue_number": p.issue_number, - "repo": p.repo, - "branch": p.branch, - "status": p.status.value, - "current_phase": p.current_phase.value, - "created_at": p.created_at.isoformat(), - "updated_at": p.updated_at.isoformat(), - } - for p in pipelines - ] - - return make_success_response( - f"Found {len(pipelines)} pipeline(s)", - data={"pipelines": pipeline_data}, - ) - - except StateStoreError as e: - logger.error("Failed to list pipelines", error=str(e)) - return make_error_response(f"Failed to list pipelines: {e}", status_code=500) - - -@pipelines_bp.route("/<pipeline_id>", methods=["GET"]) -def get_pipeline(pipeline_id: str) -> tuple[Response, int]: - """ - Get a pipeline by ID. - - URL params: - pipeline_id: Pipeline ID (e.g., "issue-123") - - Query params: - repo_path: Path to repository (optional) - - Response: - { - "success": true, - "data": { - "pipeline": {...} - } - } - """ - repo_path = get_repo_path() - - try: - _store, pipeline = _resolve_pipeline(pipeline_id, repo_path) - - return make_success_response( - "Pipeline retrieved", - data={"pipeline": pipeline.model_dump(mode="json")}, - ) - - except InvalidPipelineIdError: - return make_error_response( - f"Invalid pipeline ID format: {pipeline_id}", - status_code=400, - ) - except PipelineNotFoundError: - return make_error_response( - f"Pipeline {pipeline_id} not found", - status_code=404, - ) - except StateValidationError as e: - logger.error("Pipeline validation failed", pipeline_id=pipeline_id, error=str(e)) - return make_error_response( - f"Pipeline state is invalid: {e}", - status_code=500, - ) - - -def _normalize_submission_repos( - repos_arg: Any, -) -> tuple[str | None, list[dict[str, str | None]], str | None, str | None]: - """Validate + normalize a multi-repo submission list (#3393). - - Accepts the ``repos`` payload from ``POST /api/v1/pipelines`` — a list of - ``{repo, base_branch?, primary?}`` entries (a bare ``"owner/name"`` string - is tolerated as ``{repo: ...}``). Returns - ``(error, entries, primary_repo, primary_base_branch)``: - - * ``error`` — a human-readable message when validation fails (the other - fields are meaningless in that case), else ``None``. - * ``entries`` — normalized ``{"repo", "base_branch"}`` dicts, reordered so - the primary is ``entries[0]`` (the ``Pipeline`` validator mirrors - ``repos[0]`` onto the legacy singleton and ``primary_repo``). - - Per-entry repo/base_branch formats are validated with the same regexes the - single-repo path uses. Same-name repos under different owners are NOT - rejected here — they are distinct full ``owner/name`` slugs (operator - ruling #6; the owner/repo re-key lands in slice 3). - """ - if not isinstance(repos_arg, list) or not repos_arg: - return ("repos must be a non-empty list of {repo, base_branch} entries", [], None, None) - entries: list[dict[str, str | None]] = [] - primary_index = 0 - seen_primary = False - for idx, raw in enumerate(repos_arg): - entry = {"repo": raw} if isinstance(raw, str) else raw - if not isinstance(entry, dict) or not entry.get("repo"): - return (f"repos[{idx}] must be an object with a 'repo' field", [], None, None) - repo_val = entry["repo"] - if not re.match(r"^[a-zA-Z0-9._-]+/[a-zA-Z0-9._-]+$", repo_val): - return ( - f"Invalid repo format in repos[{idx}]: {repo_val!r} (expected owner/name)", - [], - None, - None, - ) - base_val = entry.get("base_branch") - if base_val is not None and ( - not re.match(r"^[a-zA-Z0-9_./-]+$", base_val) or ".." in base_val - ): - return (f"Invalid base_branch in repos[{idx}]: {base_val!r}", [], None, None) - entries.append({"repo": repo_val, "base_branch": base_val}) - if entry.get("primary"): - if seen_primary: - return ("At most one repos entry may set 'primary'", [], None, None) - seen_primary = True - primary_index = idx - # Reorder so the primary is first: the Pipeline model mirrors repos[0] - # onto the legacy repo/base_branch singleton and exposes it as - # ``primary_repo``. - if primary_index != 0: - entries.insert(0, entries.pop(primary_index)) - primary = entries[0] - return (None, entries, primary["repo"], primary["base_branch"]) - - -def _assert_repo_set_uniform(repos: list[str]) -> str | None: - """Reject mixed-visibility / mixed-auth repo sets at submission (#3393, task-2-2). - - A pipeline-wide private-mode posture (context filtering, egress rules) - requires every repo in one run to be uniformly private or uniformly public, - and — for v1 — to share a single auth mode. Returns an actionable, - repo-naming error string when the set diverges on either dimension, or - ``None`` when it is uniform. A single repo (after de-duplication) is - trivially uniform and short-circuits before any lookup, so N=1 pipelines - pay no cost and make no gateway round-trip. - - Runtime note (container boundary): the orchestrator image bundles - ``config/repo_config.py`` but NOT ``gateway/``, so the per-repo lookups are - reached the way the orchestrator already reaches them — auth via - ``repo_config.assert_uniform_auth`` (imported directly, the same callable the - gateway's ``validate_auth_mode_uniformity`` delegates to) and visibility via - ``GatewayClient.get_repo_visibility`` over HTTP (the gateway holds the - tokens; mirrors ``_compute_gateway_mode``). ``internal`` counts as private. - The visibility comparison below is the HTTP-boundary twin of - ``gateway.repo_visibility.validate_visibility_uniformity`` (which the - orchestrator cannot import); keep the two in step. - """ - unique = list(dict.fromkeys(repos)) - if len(unique) <= 1: - return None - - # Auth-mode uniformity — repo_config is bundled into the orchestrator image. - try: - from repo_config import assert_uniform_auth - - assert_uniform_auth(unique) - except ValueError as exc: - return str(exc) - except Exception as exc: # pragma: no cover - defensive (config read failure) - # Fail CLOSED for consistency with the visibility boundary below - # (reviewer_security v1): a config-read failure means we cannot prove a - # uniform auth mode, so we must not admit the set. repo_config is a - # local, bundled read — this path is genuinely exceptional, not a - # transient network hiccup. - logger.warning("Auth-mode uniformity check errored; failing closed", error=str(exc)) - return ( - "Could not determine the auth mode for the pipeline's repos, so a " - "uniform bot/user auth mode cannot be verified. Resubmit once repo " - "configuration is resolvable." - ) - - # Visibility uniformity — resolved via the gateway (the orchestrator's only - # visibility source). FAIL CLOSED on an indeterminate lookup (reviewer_security - # v1): for a multi-repo set (we only reach here when len(unique) > 1) a repo - # whose visibility cannot be resolved to a known bucket means the uniform - # private/public posture cannot be PROVEN — and this is a confidentiality - # boundary (a mixed set that slips through would let private-repo content - # flow through shared plan/contract/PR surfaces into a public repo, with no - # downstream re-check: _compute_gateway_mode derives the network mode from - # the PRIMARY repo only). N=1 short-circuits above, so the common case pays - # nothing. This mirrors gateway.repo_visibility.validate_visibility_uniformity; - # keep the two in step. Unrecognized (non-None) labels are treated as - # indeterminate too — only the known {public|private|internal} contract admits. - gw = get_gateway_client() - posture: dict[str, list[str]] = {} - for repo in unique: - vis = gw.get_repo_visibility(repo) - if vis in ("private", "internal"): - bucket = "private" - elif vis == "public": - bucket = "public" - else: - return ( - f"Could not determine repository visibility for {repo!r}; cannot " - "verify a uniform private/public posture across the pipeline's " - "repos (a run must be uniformly private or uniformly public so " - "private-repo content cannot leak through shared plan/contract/PR " - "surfaces). Resubmit once the repo's visibility is resolvable." - ) - posture.setdefault(bucket, []).append(repo) - if len(posture) > 1: - groups = "; ".join(f"{b}: {', '.join(sorted(rs))}" for b, rs in sorted(posture.items())) - return ( - "Mixed repository visibility across the pipeline's repos is not allowed " - "(a run must be uniformly private or uniformly public, so private-repo " - f"content cannot leak through shared plan/PR surfaces). Diverging repos — {groups}." - ) - return None - - -@pipelines_bp.route("", methods=["POST"]) -@require_lifecycle_secret -def create_pipeline() -> tuple[Response, int]: - """ - Create a new pipeline. - - Request body: - { - "issue_number": 123, - "repo": "owner/name", - "branch": "egg/issue-123", - "config": {...} // optional - } - - Response: - { - "success": true, - "message": "Pipeline created", - "data": { - "pipeline": {...} - } - } - """ - data = request.get_json() - if data is None: - return make_error_response("Missing request body") - if not isinstance(data, dict): - return make_error_response("Request body must be a JSON object") - - network_mode = data.get("network_mode") - if network_mode is not None and network_mode not in ("public", "private"): - return make_error_response( - f"Invalid network_mode: {network_mode!r} (must be 'public' or 'private')" - ) - - issue_number = data.get("issue_number") - repo = data.get("repo") - branch = data.get("branch") - base_branch = data.get("base_branch") - prompt = data.get("prompt") - - # #3393 (multi-repo): a submission may carry a ``repos`` list instead of - # (or in addition to) the single ``repo``. Normalize it up front and derive - # the primary onto the legacy ``repo``/``base_branch`` scalars so the - # single-repo plumbing below (naming, base-branch detection, branch checks) - # keeps working and a direct HTTP submission — one that bypasses the - # submit_task MCP tool that would otherwise mirror the primary — is - # supported. ``repos_entries`` is None for a single-repo submission. - repos_entries: list[dict[str, str | None]] | None = None - repos_arg = data.get("repos") - if repos_arg is not None: - _repos_err, repos_entries, _primary_repo, _primary_base = _normalize_submission_repos( - repos_arg - ) - if _repos_err: - return make_error_response( - _repos_err, status_code=400, details={"reason": "invalid_repos"} - ) - if repo and _primary_repo and repo != _primary_repo: - return make_error_response( - f"Conflicting repo {repo!r} and repos primary {_primary_repo!r}; " - "pass one or the other.", - status_code=400, - details={"reason": "repo_repos_conflict"}, - ) - if not repo: - repo = _primary_repo - if not base_branch: - base_branch = _primary_base - mode = data.get("mode", "issue") - analysis = data.get("analysis") - plan = data.get("plan") - source_branch = data.get("source_branch") - if source_branch is not None: - if not re.match(r"^[a-zA-Z0-9_./-]+$", source_branch) or ".." in source_branch: - return make_error_response( - f"Invalid source_branch: {source_branch!r}", - status_code=400, - ) - source_artifact_prefix = data.get("source_artifact_prefix") - if source_artifact_prefix is not None: - if not re.match(r"^[a-zA-Z0-9_.-]+$", source_artifact_prefix): - return make_error_response( - f"Invalid source_artifact_prefix: {source_artifact_prefix!r}", - status_code=400, - ) - - # Issue #1557: Jira-epic SDLC parameters. ``jira_ticket`` is the - # Atlassian key; ``epic_mode`` is the operator's override - # (``'auto' | 'fresh' | 'reassess'``). The MCP submit_task tool - # normalises ``jira_ticket`` to upper-case before forwarding. - jira_ticket_arg = data.get("jira_ticket") - epic_mode_arg = data.get("epic_mode") - if jira_ticket_arg is not None: - if not isinstance(jira_ticket_arg, str) or not re.fullmatch( - r"[A-Z][A-Z0-9_]*-\d+", jira_ticket_arg - ): - return make_error_response( - f"Invalid jira_ticket: {jira_ticket_arg!r} (expected <PROJECT>-<number>)", - status_code=400, - details={"reason": "invalid_jira_ticket"}, - ) - if epic_mode_arg is not None: - if epic_mode_arg not in ("auto", "fresh", "reassess"): - return make_error_response( - f"Invalid epic_mode: {epic_mode_arg!r} (must be 'auto' / 'fresh' / 'reassess')", - status_code=400, - details={"reason": "invalid_epic_mode"}, - ) - if not jira_ticket_arg: - return make_error_response( - "epic_mode requires jira_ticket", - status_code=400, - details={"reason": "epic_mode_without_ticket"}, - ) - - # Validate mode - valid_modes = {m.value for m in PipelineMode} - if mode not in valid_modes: - return make_error_response(f"Invalid mode: {mode!r} (must be one of {sorted(valid_modes)})") - - if not repo: - return make_error_response("Missing repo") - - # Repo format sanity check — a lightweight shell-metacharacter guard. - # The repo_config allowlist (repositories.yaml) is enforced gateway-side. - if not re.match(r"^[a-zA-Z0-9._-]+/[a-zA-Z0-9._-]+$", repo): - return make_error_response( - f"Invalid repo format: {repo!r} (expected owner/name)", - status_code=400, - details={"reason": "repo_not_allowed"}, - ) - - # Validate branch and base_branch — reject values that could be - # interpreted as git flags (e.g. "--upload-pack=...") or contain - # path-traversal sequences. Same regex used for source_branch above. - for _ref_name, _ref_val in [("branch", branch), ("base_branch", base_branch)]: - if _ref_val is not None: - if not re.match(r"^[a-zA-Z0-9_./-]+$", _ref_val) or ".." in _ref_val: - return make_error_response( - f"Invalid {_ref_name}: {_ref_val!r}", - status_code=400, - ) - - # Issue-driven or explicitly-named pipelines require a branch; - # prompt-driven ones do not. - pipeline_id = data.get("pipeline_id") - - if (issue_number or pipeline_id) and not branch: - return make_error_response("Missing branch") - - # #2399 — push the pipeline tip to ``<branch>/work`` so slice - # integration branches at ``<branch>/slice-N`` can coexist as - # siblings under the same namespace (git rejects a leaf ref and - # children of that ref's path with ``directory file conflict``). - branch = _ensure_pipeline_work_ref(branch) - - # Wait for the gateway to be ready before any gateway-dependent work. - # On fresh deploys / pod restarts the orchestrator can accept requests - # while the gateway HTTP listener is still coming up; without this gate - # the first submission proceeds, hits the gateway during pipeline-level - # worktree creation or per-agent fan-out, and surfaces as a cascade of - # generic per-agent ConnectionRefused / "Remote end closed connection" - # errors that operators have to reverse-engineer. See #1851. - try: - _ready_timeout = int(os.environ.get("EGG_GATEWAY_READY_TIMEOUT_SECONDS", "60")) - except ValueError: - _ready_timeout = 60 - _ready_timeout = max(0, _ready_timeout) - if _ready_timeout > 0: - _gw_ready = get_gateway_client() - if not _gw_ready.wait_for_healthy(timeout_seconds=_ready_timeout): - _last = _gw_ready.check_health() - _resp, _status = make_error_response( - f"Gateway not ready after {_ready_timeout}s " - f"(status={_last.status}): {_last.error or 'unhealthy'}. " - "Retry once the gateway has finished starting up.", - status_code=503, - details={ - "reason": "gateway_not_ready", - "gateway_status": _last.status, - "gateway_error": _last.error, - "timeout_seconds": _ready_timeout, - }, - ) - _resp.headers["Retry-After"] = str(_ready_timeout) - return _resp, _status - - repo_path = get_repo_path() - - # #3038: resolve the repo's default branch ONCE at submit time and - # persist it on the pipeline record, so every downstream consumer - # (the context-PR opener, the restart/spawn paths, the gateway - # ``register_session`` base, the spawner ``EGG_BASE_BRANCH`` export) - # reads a concrete base off the record instead of re-deriving it on - # every invocation. Re-deriving each time opened a narrow race the - # #3035 reviewer flagged: a single flaky ``git symbolic-ref - # origin/HEAD`` read drops the opener into the ``origin/main → - # origin/master → "main"`` fallback chain, which can pick the wrong - # default on a ``master`` repo and 422 a second ``create_pr``. - # Persisting closes the race because the consumers' ``base_branch or - # _detect_default_branch(...)`` short-circuits on the stored value and - # never reaches the subprocess. ``_detect_default_branch`` is the - # local/fast helper (``git symbolic-ref``) and is the same resolution - # the stale-branch reuse check below already performs. - # - # An explicit ``base_branch`` (validated above) is passed through - # untouched; ``repo`` is already guaranteed non-empty by the early - # ``Missing repo`` guard, so only the ``base_branch`` side needs a - # check here. - if not base_branch: - base_branch = _detect_default_branch(repo_path) - - # Check that the target branch does not already exist on the remote. - # This catches conflicts early (before spawning agents). However, - # allow branch reuse when the pipeline is in a terminal state - # (CANCELLED/FAILED/COMPLETE) or doesn't exist at all — this lets - # callers resubmit against the same branch after a prior run ended. - if branch: - try: - gw = get_gateway_client() - if gw.ls_remote_branch( - pipeline_id=pipeline_id or f"branch-check-{uuid4().hex[:8]}", - repo_path=str(repo_path), - ref=f"refs/heads/{branch}", - ): - # Branch exists — only block if there is an active pipeline - _branch_store = get_state_store(repo_path) - _has_active_pipeline = False - # When pipeline_id is None (auto-generated later), we skip - # the existence check — we can't look up a pipeline that - # hasn't been assigned an ID yet. This is acceptable because - # auto-generated IDs are unique and won't collide. - if pipeline_id and _branch_store.pipeline_exists(pipeline_id): - try: - _existing = _branch_store.load_pipeline(pipeline_id) - _terminal = { - PipelineStatus.CANCELLED, - PipelineStatus.FAILED, - PipelineStatus.COMPLETE, - } - _has_active_pipeline = _existing.status not in _terminal - except Exception: - # If we can't load the pipeline, treat as no active pipeline - pass - - if _has_active_pipeline: - hint = "" - if pipeline_id: - hint = ( - f" Use a qualifier to create a separate pipeline" - f" (e.g. '{pipeline_id}-<qualifier>')." - ) - return make_error_response( - f"Branch '{branch}' already exists on remote.{hint}", - status_code=409, - details={"reason": "branch_exists", "branch": branch}, - ) - else: - # No active pipeline, but the branch may carry commits - # from a prior failed/cancelled run. Inheriting that - # state was the precondition for #2222 (stale - # pipeline-branch tip + advanced main → contaminated - # PR via the push-reconcile fallback). Compare the - # branch tip to the configured base; only a fresh - # branch (tip == base) is safe to silently reuse. - # - # Resolve the default branch via ``_detect_default_branch`` - # rather than hardcoding ``"main"`` so repos whose default - # is ``master`` / ``develop`` still get the stale-branch - # check (otherwise the ``origin/main`` lookup returns - # ``None``, the guard falls through, and the precondition - # check is silently disabled). - _resolved_base = base_branch or _detect_default_branch(repo_path) - _branch_sha = gw.get_remote_branch_sha( - pipeline_id=pipeline_id or f"branch-check-{uuid4().hex[:8]}", - repo_path=str(repo_path), - ref=f"refs/heads/{branch}", - ) - _base_sha = gw.get_remote_branch_sha( - pipeline_id=pipeline_id or f"branch-check-{uuid4().hex[:8]}", - repo_path=str(repo_path), - ref=f"refs/heads/{_resolved_base}", - ) - # When either lookup returns ``None`` the stale-branch - # check is bypassed. ``get_remote_branch_sha`` swallows - # transient gateway errors and returns ``None`` (same - # value it returns when the ref legitimately doesn't - # exist), so we surface a warning here to make the - # silent skip visible to operators investigating a - # post-merge contamination — rather than letting the - # precondition fix vanish behind a transient hiccup. - if _branch_sha is None or _base_sha is None: - logger.warning( - "Stale-branch check skipped: SHA lookup returned None " - "(transient gateway error or ref missing — see #2222)", - branch=branch, - base_branch=_resolved_base, - branch_sha=_branch_sha, - base_sha=_base_sha, - ) - if _branch_sha and _base_sha and _branch_sha != _base_sha: - logger.warning( - "Branch exists with prior-pipeline commits — refusing reuse (#2222)", - branch=branch, - base_branch=_resolved_base, - branch_sha=_branch_sha, - base_sha=_base_sha, - ) - cleanup_hint = ( - f" Run cancel_task(task_id='{pipeline_id}', cleanup=true) " - "to delete the stale branch and pipeline state, then " - "resubmit." - if pipeline_id - else ( - " Delete the stale branch and any associated " - "pipeline state, then resubmit." - ) - ) - return make_error_response( - f"Branch '{branch}' exists with commits from a prior " - f"pipeline run (tip {_branch_sha[:8]} != " - f"origin/{_resolved_base} {_base_sha[:8]}). Starting a " - "new pipeline on top of it would inherit that history.", - status_code=409, - details={ - "reason": "stale_branch", - "branch": branch, - "branch_sha": _branch_sha, - "base_sha": _base_sha, - "hint": cleanup_hint.strip(), - }, - ) - logger.info( - "Branch exists but no active pipeline — allowing reuse", - branch=branch, - pipeline_id=pipeline_id, - branch_sha=_branch_sha, - base_sha=_base_sha, - ) - except Exception as e: - # Non-fatal — if we can't reach the gateway, let creation proceed - # and fail later on push. - logger.warning( - "Branch existence check failed, proceeding anyway", - branch=branch, - error=str(e), - ) - - # Validate config before creating the pipeline so invalid config - # returns a 400 instead of bubbling up as a 500. - config = data.get("config") - if config is not None: - if isinstance(config, str): - try: - config = json.loads(config) - except json.JSONDecodeError as e: - return make_error_response(f"Invalid config JSON: {e}") - try: - from models import PipelineConfig - from pydantic import ValidationError - - PipelineConfig.model_validate(config) - except ValidationError as e: - errors = [ - {"field": ".".join(str(loc) for loc in err["loc"]), "message": err["msg"]} - for err in e.errors() - ] - return make_error_response( - f"Invalid pipeline config: {errors}", - details={"validation_errors": errors}, - ) - - # Validate analysis/plan size before creating the pipeline. - _MAX_DRAFT_LEN = 200_000 - for field_name in ("analysis", "plan"): - value = data.get(field_name) - if isinstance(value, str) and len(value) > _MAX_DRAFT_LEN: - return make_error_response( - f"{field_name} exceeds maximum length ({len(value)} > {_MAX_DRAFT_LEN})" - ) - - # Issue #1557: epic detection. Before persisting, resolve - # is_epic + pipeline_mode against the gateway when a jira_ticket - # was supplied. Failures are non-fatal (the helper fails open) — - # we surface them as warnings in the API response but always - # proceed with the pipeline creation. - epic_warnings: list[str] = [] - is_epic_resolved = False - pipeline_mode_resolved: str | None = None - if jira_ticket_arg: - try: - from jira_epic import resolve_epic_mode - except ImportError: # pragma: no cover - defensive - try: - from orchestrator.jira_epic import resolve_epic_mode # type: ignore[no-redef] - except ImportError: - resolve_epic_mode = None # type: ignore[assignment] - if resolve_epic_mode is not None: - try: - is_epic_resolved, pipeline_mode_resolved, epic_warnings = resolve_epic_mode( - ticket=jira_ticket_arg, - epic_mode_arg=epic_mode_arg, - ) - except Exception as exc: # pragma: no cover - defensive - logger.warning( - "Epic detection raised; treating as non-epic", - pipeline_id=pipeline_id, - ticket=jira_ticket_arg, - error=str(exc), - ) - # Both explicit overrides (``reassess`` and ``fresh``) against - # a non-epic ticket are operator errors: the operator - # specifically asked for epic-mode treatment but the ticket - # doesn't qualify. Surface as HTTP 400 rather than the - # silent demotion ``resolve_epic_mode`` returns - # (is_epic=False with a warning). ``mode='auto'`` continues - # to demote silently to standard ticket mode — that's the - # whole point of auto. - if epic_mode_arg in {"reassess", "fresh"} and not is_epic_resolved: - return make_error_response( - f"epic_mode={epic_mode_arg!r} but Jira ticket {jira_ticket_arg!r} is not an Epic", - status_code=400, - details={ - "reason": f"{epic_mode_arg}_not_epic", - "warnings": epic_warnings, - }, - ) - - # #3393 (multi-repo): enforce uniform visibility + auth across the run's - # repos before creating the pipeline. Single-repo submissions are trivially - # uniform and short-circuit without a gateway round-trip. Runs after the - # gateway-ready gate above so the visibility lookup can reach the gateway. - _uniform_repos = ( - [e["repo"] for e in repos_entries] if repos_entries else ([repo] if repo else []) - ) - _uniformity_err = _assert_repo_set_uniform([r for r in _uniform_repos if r]) - if _uniformity_err: - return make_error_response( - _uniformity_err, - status_code=400, - details={"reason": "non_uniform_repo_set"}, - ) - - # Assemble the full list-shaped repo set persisted onto the Pipeline. The - # primary (entries[0]) carries the resolved ``base_branch`` (detected above - # when absent); secondary repos keep their submitted base_branch (None ⇒ - # auto-detected downstream). For a single-repo submission we leave - # ``repos_specs`` as None and let the Pipeline validator synthesize a - # one-element list from the legacy singleton (N=1 back-compat). - repos_specs: list[RepoSpec] | None = None - if repos_entries is not None: - repos_specs = [ - RepoSpec( - repo=entry["repo"], - base_branch=(base_branch if idx == 0 else entry["base_branch"]), - ) - for idx, entry in enumerate(repos_entries) - ] - - try: - store = get_state_store(repo_path) - pipeline = store.create_pipeline( - issue_number=issue_number, - repo=repo, - branch=branch, - base_branch=base_branch, - repos=repos_specs, - config=config, - prompt=prompt, - network_mode=network_mode, - pipeline_id=pipeline_id, - analysis=analysis, - plan=plan, - source_branch=source_branch, - source_artifact_prefix=source_artifact_prefix, - has_contract=True, - jira_ticket=jira_ticket_arg, - is_epic=is_epic_resolved, - pipeline_mode=pipeline_mode_resolved, - ) - - # Contract creation is deferred to _run_pipeline so it writes - # into the per-pipeline worktree instead of the main repo. - - # When state_store replaces a terminal pipeline with the same id - # (state_store.create_pipeline:850), the in-memory consensus - # tracker / message-store entries for the prior run survive. Same - # for Redis-backed message-store entries across orchestrator - # restarts. Clear here so the new run starts with empty consensus - # state regardless of how the prior run ended (#2053). - # - # This is the *primary* eviction site for auto-FAILED prior runs, - # not just a defensive backstop: paths like restart_agent spawn - # failure call store.update_pipeline / store.save_pipeline directly - # (bypassing PATCH), so the PATCH-site clear never fires for them. - # Without this POST-site clear, those auto-FAILED pipelines would - # leak consensus + message-store state into the next run that - # reuses the id. - _clear_pipeline_runtime_state(pipeline.id, reason="pipeline_create") - - logger.info( - "Pipeline created", - pipeline_id=pipeline.id, - issue_number=issue_number, - ) - - return make_success_response( - "Pipeline created", - data={"pipeline": pipeline.model_dump(mode="json")}, - ) - - except StateStoreError as e: - if "already exists" in str(e): - # Include existing pipeline details so callers can decide - # whether to cancel+resubmit or resume monitoring. - details: dict[str, Any] = {} - try: - # Derive pipeline ID using the same logic as state_store - pid = pipeline_id or (f"issue-{issue_number}" if issue_number else None) - if pid: - existing = store.load_pipeline(pid) - details = { - "existing_pipeline_id": existing.id, - "existing_status": existing.status.value, - "existing_phase": existing.current_phase.value, - } - except Exception: - pass # Best-effort enrichment - return make_error_response(str(e), status_code=409, details=details) - logger.error("Failed to create pipeline", error=str(e)) - return make_error_response(f"Failed to create pipeline: {e}", status_code=500) - except Exception as e: - # Catch non-StateStoreError exceptions (e.g., ValidationError, - # OSError) that would otherwise produce a generic 500 from the - # Flask error handler with no detail (#1396). - logger.error( - "Unexpected error creating pipeline", - error=str(e), - error_type=type(e).__name__, - exc_info=True, - ) - msg = f"{type(e).__name__}: {e}" - return make_error_response( - f"Failed to create pipeline: {msg[:500]}", - status_code=500, - ) - - -def _clear_pipeline_runtime_state(pipeline_id: str, *, reason: str) -> None: - """Evict per-pipeline runtime state that is keyed by pipeline_id alone. - - The peer-consensus tracker, the legacy consensus evaluator, and the - inter-agent message store are all keyed by pipeline_id. Without a - matching ``run_epoch`` namespace, a fresh pipeline that reuses an id - from a prior terminal run (same branch, e.g. ``issue-1965``) will - inherit the prior run's CONFIRMED consensus and message history. The - leak surfaces in the ``/status/wait`` route's Path-B envelope, which - would report ``concurrent.consensus.is_complete: true`` for a - pipeline that has not spawned any agents yet (#2053). - - Called when a pipeline transitions to a terminal status, when its - state file is deleted, and immediately after a fresh pipeline is - created (covers paths that bypass PATCH/DELETE — auto-FAILED, and - Redis-backed message-store entries that survived an orchestrator - restart between cancel and resubmit). - """ - try: - try: - from peer_consensus import remove_peer_consensus_tracker - except ImportError: - from ..peer_consensus import ( # type: ignore[no-redef] - remove_peer_consensus_tracker, - ) - remove_peer_consensus_tracker(pipeline_id) - except ImportError: - pass - except Exception as e: - logger.warning( - "Failed to clear peer consensus tracker", - pipeline_id=pipeline_id, - reason=reason, - error=str(e), - ) - - # Reconstruct-from-messages would otherwise replay the prior run's - # CONSENSUS_* messages and rebuild a CONFIRMED tracker, defeating the - # tracker eviction above. - try: - try: - from message_store import get_message_store - except ImportError: - from ..message_store import get_message_store # type: ignore[no-redef] - get_message_store().clear(pipeline_id) - except ImportError: - pass - except Exception as e: - logger.warning( - "Failed to clear message store", - pipeline_id=pipeline_id, - reason=reason, - error=str(e), - ) - - -def _mark_pipeline_records_terminated( - store: StateStore, - pipeline_id: str, -) -> Pipeline: - """Mark all running containers and agents as stopped after pipeline termination. - - Called when a pipeline transitions to a terminal state (cancelled or failed). - After Docker containers are force-removed, the pipeline state still shows - them as "running". This reloads the latest state from the store (to avoid - overwriting updates made between the status change and container - cleanup), marks running records as stopped, and saves. - - Returns the updated pipeline so the caller can use it in the response. - """ - pipeline = store.load_pipeline(pipeline_id) - now = datetime.now(UTC) - changed = False - - for phase_exec in pipeline.phases.values(): - for container in phase_exec.containers: - if container.status in ( - ContainerStatus.PENDING, - ContainerStatus.CREATING, - ContainerStatus.RUNNING, - ): - container.status = ContainerStatus.REMOVED - container.exited_at = now - changed = True - - for agent in phase_exec.agents: - if agent.status in ( - AgentExecutionStatus.PENDING, - AgentExecutionStatus.RUNNING, - ): - agent.status = AgentExecutionStatus.FAILED - agent.completed_at = now - agent.error = f"Pipeline {pipeline.status.value}" - changed = True - - if changed: - store.save_pipeline(pipeline) - logger.info( - "Synced pipeline state after termination", - pipeline_id=pipeline_id, - ) - - return pipeline - - -@pipelines_bp.route("/<pipeline_id>", methods=["PATCH"]) -@require_lifecycle_secret -def update_pipeline(pipeline_id: str) -> tuple[Response, int]: - """ - Update a pipeline. - - URL params: - pipeline_id: Pipeline ID - - Request body: - { - "status": "running", - "current_phase": "plan", - ... - } - - Response: - { - "success": true, - "data": { - "pipeline": {...} - } - } - """ - data = request.get_json() - if data is None: - return make_error_response("Missing request body") - if not isinstance(data, dict): - return make_error_response("Request body must be a JSON object") - - repo_path = get_repo_path() - - try: - store, _pipeline = _resolve_pipeline(pipeline_id, repo_path) - prev_status = _pipeline.status - pipeline = store.update_pipeline(pipeline_id, data) - - # Emit the terminal event before kicking off cleanup so /status/wait - # long-pollers wake immediately on cancellation rather than waiting - # for the late-subscriber synth path on their next poll (#2663). The - # run loop emits pipeline.completed / pipeline.failed from its own - # terminal transitions; the PATCH path is the only place the - # CANCELLED transition originates, so we emit it here. Gate on the - # status *transition* (not equality) so idempotent retries against an - # already-cancelled pipeline don't re-wake long-pollers. - if pipeline.status == PipelineStatus.CANCELLED and prev_status != PipelineStatus.CANCELLED: - _emit_pipeline_event(pipeline, "pipeline.cancelled") - - # If pipeline is being cancelled or failed, clean up containers - # and cancel any pending decisions so wait_for_decision() unblocks. - if pipeline.status in (PipelineStatus.CANCELLED, PipelineStatus.FAILED): - try: - dq = get_decision_queue(pipeline_id, repo_path) - pending = dq.get_pending_decisions() - for decision in pending: - dq.cancel_decision(decision.id) - if pending: - logger.info( - "Cancelled pending decisions after pipeline status change", - pipeline_id=pipeline_id, - decisions_cancelled=len(pending), - ) - except Exception as e: - logger.warning( - "Failed to cancel pending decisions", - pipeline_id=pipeline_id, - error=str(e), - ) - - # Sync pipeline state: reload latest state (agents may have - # written updates between status change and container cleanup), - # mark all running records as stopped, and re-save. - try: - pipeline = _mark_pipeline_records_terminated(store, pipeline_id) - except Exception as e: - logger.warning( - "Failed to sync pipeline state after termination", - pipeline_id=pipeline_id, - error=str(e), - ) - # Reload pipeline so the response reflects current state - # rather than the stale pre-cleanup object. - try: - pipeline = store.load_pipeline(pipeline_id) - except Exception: - pass # Use stale pipeline if reload also fails - - # Move container/worktree cleanup to a background daemon thread - # so the PATCH response returns immediately. The DELETE handler - # already re-runs cleanup_pipeline() as a safety net, so it will - # catch anything the background thread hasn't finished. - # - # Compute the salvage mode + base branch up front (in the - # request thread, where ``pipeline`` is still in scope) so the - # background thread can pass them to ``cleanup_pipeline`` - # without re-loading state. Using the wrong mode here would - # mismatch the policy the rest of the pipeline ran under and - # the launcher-auth push could be rejected — see #2429 - # review. - _bg_salvage_mode, _ = _compute_gateway_mode(pipeline) - _bg_salvage_base_branch = pipeline.base_branch - - def _background_cleanup(pid: str, status_value: str) -> None: - try: - spawner = _get_spawner() - # Preserve worktrees for CANCELLED pipelines so that - # restart_phase/restart_agent can resume with local - # committed work intact (see #1725). - removed = spawner.cleanup_pipeline( - pid, - force=True, - preserve_worktrees=(status_value == "cancelled"), - salvage_mode=_bg_salvage_mode, - salvage_base_branch=_bg_salvage_base_branch, - ) - if removed > 0: - logger.info( - "Cleaned up pipeline containers after status change", - pipeline_id=pid, - status=status_value, - containers_removed=removed, - ) - except (DockerClientError, DockerException, KubernetesClientError) as e: - logger.warning( - "Failed to clean up pipeline containers", - pipeline_id=pid, - error=str(e), - ) - except Exception as e: - logger.error( - "Unexpected error during pipeline container cleanup", - pipeline_id=pid, - error=str(e), - exc_info=True, - ) - - cleanup_thread = threading.Thread( - target=_background_cleanup, - args=(pipeline_id, pipeline.status.value), - daemon=True, - name=f"cleanup-{pipeline_id}", - ) - cleanup_thread.start() - - # Evict per-pipeline runtime state (consensus tracker, legacy - # consensus evaluator, message store) so a future pipeline - # that reuses this id (same branch) does not inherit this - # run's CONFIRMED consensus or message history (#2053). - _clear_pipeline_runtime_state(pipeline_id, reason=f"pipeline_{pipeline.status.value}") - - logger.info("Pipeline updated", pipeline_id=pipeline_id) - - response_data = {"pipeline": pipeline.model_dump(mode="json")} - if pipeline.status in (PipelineStatus.CANCELLED, PipelineStatus.FAILED): - response_data["cleanup_pending"] = True - - return make_success_response( - "Pipeline updated", - data=response_data, - ) - - except InvalidPipelineIdError: - return make_error_response( - f"Invalid pipeline ID format: {pipeline_id}", - status_code=400, - ) - except PipelineNotFoundError: - return make_error_response( - f"Pipeline {pipeline_id} not found", - status_code=404, - ) - except StateValidationError as e: - return make_error_response( - f"Invalid update: {e}", - status_code=400, - ) - - -# Config keys the live config-update route accepts. Deliberately a tight -# allowlist (#3174): most of PipelineConfig is consumed at submit time or -# mid-phase in ways a partial update could corrupt. Two families qualify: -# -# * ``agent_models`` is re-resolved from a fresh store load before every -# spawn (the run loop reloads the pipeline at the top of each cycle, and -# the restart_agent / restart_phase paths load fresh state), so mutating -# it on a live pipeline is honored by construction. -# * ``consensus_timeout_minutes*`` is re-resolved from a fresh store load -# by the phase poll loop right before the consensus wall fires (#3490), -# so a widened window takes effect on a running slice without a restart. -# -# Widen only after verifying the same fresh-reload guarantee holds for the -# new key. -_CONSENSUS_TIMEOUT_CONFIG_KEYS = ( - "consensus_timeout_minutes", - "consensus_timeout_minutes_refine", - "consensus_timeout_minutes_plan", - "consensus_timeout_minutes_implement", -) -_MUTABLE_CONFIG_KEYS = frozenset({"agent_models", *_CONSENSUS_TIMEOUT_CONFIG_KEYS}) - - -@pipelines_bp.route("/<pipeline_id>/config", methods=["PATCH"]) -@require_lifecycle_secret -def update_pipeline_config(pipeline_id: str) -> tuple[Response, int]: - """Update the safely-mutable subset of a live pipeline's config (#3174). - - That subset is ``agent_models`` plus the ``consensus_timeout_minutes*`` - family (#3490). - - ``agent_models`` semantics are a per-role merge with the pipeline's - existing override map: roles absent from the request keep their - current value, a string value sets that role's override, and an - explicit ``null`` clears it (the role falls back to the repository - default / built-in tiers). The updated map takes effect at the next - agent spawn; currently running agents keep the model they were - started with. Pair with ``restart_phase`` / ``restart_agent`` to - apply the change to a running phase. Model *values* are not - validated against a registry here (any non-Claude string routes to - LiteLLM, mirroring submit-time behavior); a typo surfaces as a - model-not-found error at spawn. - - ``consensus_timeout_minutes`` / ``consensus_timeout_minutes_refine`` - / ``_plan`` / ``_implement`` set the corresponding override to an - integer number of minutes (>= 1); ``null`` clears the override so - the phase falls back to the resolution chain (per-phase override, - then legacy global, then the phase-aware default). The phase poll - loop re-resolves the budget from fresh config right before the - consensus wall fires, so a widened window takes effect on a running - slice without a restart (#3490). - - URL params: - pipeline_id: Pipeline ID - - Request body (any non-empty subset of the mutable keys): - { - "agent_models": { - "coder": "deepseek-v4-pro", - "tester": null - }, - "consensus_timeout_minutes_implement": 480 - } - - Response: - { - "success": true, - "data": { - "pipeline_id": "issue-123", - "agent_models": {...}, # effective map after the merge - "updated_roles": {...}, # roles set by this request - "cleared_roles": [...], # roles cleared by this request - "consensus_timeouts": {...},# effective timeout overrides - "updated_timeouts": {...} # timeout keys set/cleared here - } - } - """ - data = request.get_json() - if data is None: - return make_error_response("Missing request body") - if not isinstance(data, dict): - return make_error_response("Request body must be a JSON object") - - unsupported = sorted(set(data) - _MUTABLE_CONFIG_KEYS) - if unsupported: - return make_error_response( - f"Unsupported config keys: {unsupported}. This endpoint updates " - f"only the safely-mutable config subset: {sorted(_MUTABLE_CONFIG_KEYS)}", - status_code=400, - ) - if not data: - return make_error_response( - f"Request body must set at least one mutable config key: " - f"{sorted(_MUTABLE_CONFIG_KEYS)}", - status_code=400, - ) - - agent_models = data.get("agent_models") - if "agent_models" in data: - if not isinstance(agent_models, dict) or not agent_models: - return make_error_response( - "agent_models must be a non-empty object mapping role -> model " - "(use null as the model to clear a role's override)", - status_code=400, - ) - - # Pre-validate role keys against MODEL_OVERRIDE_ROLES so the operator - # gets the same actionable message as PipelineConfig's field validator - # instead of a wrapped pydantic StateValidationError. Lazy import - # mirrors models._validate_agent_models_roles. - from egg_contracts.agent_roles import MODEL_OVERRIDE_ROLES - - valid_roles = {role.value for role in MODEL_OVERRIDE_ROLES} - invalid_roles = sorted(role for role in agent_models if role not in valid_roles) - if invalid_roles: - return make_error_response( - f"Invalid agent_models role keys: {invalid_roles}. agent_models " - f"is honored only for SDLC phase producer and reviewer roles: " - f"{sorted(valid_roles)}", - status_code=400, - ) - invalid_values = sorted( - role - for role, model in agent_models.items() - if model is not None and (not isinstance(model, str) or not model.strip()) - ) - if invalid_values: - return make_error_response( - f"Invalid agent_models values for roles {invalid_values}: each " - f"value must be a non-empty model string, or null to clear the " - f"role's override", - status_code=400, - ) - - timeout_updates: dict[str, int | None] = {} - invalid_timeout_keys: list[str] = [] - for timeout_key in _CONSENSUS_TIMEOUT_CONFIG_KEYS: - if timeout_key not in data: - continue - timeout_value = data[timeout_key] - if timeout_value is None or ( - isinstance(timeout_value, int) - and not isinstance(timeout_value, bool) - and timeout_value >= 1 - ): - timeout_updates[timeout_key] = timeout_value - else: - invalid_timeout_keys.append(timeout_key) - if invalid_timeout_keys: - return make_error_response( - f"Invalid values for {invalid_timeout_keys}: each consensus " - f"timeout must be an integer number of minutes >= 1, or null to " - f"clear the override (the phase falls back to the resolution " - f"chain: per-phase override, legacy global, phase-aware default)", - status_code=400, - ) - - repo_path = get_repo_path() - - try: - store, _pipeline = _resolve_pipeline(pipeline_id, repo_path) - - # Merge under the pipeline state lock so a concurrent writer - # (another config update, the run loop persisting state) can't - # interleave between our load and the store's load-modify-save. - # The per-pipeline lock is an RLock, so update_pipeline's own - # acquisition nests cleanly. - with get_pipeline_state_lock(pipeline_id): - current = store.load_pipeline(pipeline_id) - - # Reject mutations on terminal pipelines (#3174 review). Nothing - # consumes config once a pipeline is COMPLETE / FAILED / - # CANCELLED, so the merge would be a silent no-op; a 409 gives - # the operator a clear signal and matches restart_phase's - # terminal-state precondition style. Checked under the lock - # against freshly-loaded state so a concurrent terminal - # transition can't slip a mutation through. - if current.status in PipelineStatus.terminal(): - return make_error_response( - f"Pipeline {pipeline_id} is in terminal state " - f"{current.status.value}; config cannot be updated " - "(nothing would consume the change).", - status_code=409, - ) - - updates: dict[str, Any] = {} - updated_roles: dict[str, str] = {} - cleared_roles: list[str] = [] - if isinstance(agent_models, dict): - merged = dict(current.config.agent_models) - for role_key, model in agent_models.items(): - if model is None: - if merged.pop(role_key, None) is not None: - cleared_roles.append(role_key) - else: - merged[role_key] = model.strip() - updated_roles[role_key] = model.strip() - updates["config.agent_models"] = merged - for timeout_key, timeout_value in timeout_updates.items(): - updates[f"config.{timeout_key}"] = timeout_value - pipeline = store.update_pipeline(pipeline_id, updates) - - logger.info( - "Pipeline config updated", - pipeline_id=pipeline_id, - updated_roles=updated_roles, - cleared_roles=cleared_roles, - updated_timeouts=timeout_updates, - ) - - return make_success_response( - "Pipeline config updated", - data={ - "pipeline_id": pipeline.id, - "agent_models": pipeline.config.agent_models, - "updated_roles": updated_roles, - "cleared_roles": cleared_roles, - "consensus_timeouts": { - timeout_key: getattr(pipeline.config, timeout_key) - for timeout_key in _CONSENSUS_TIMEOUT_CONFIG_KEYS - }, - "updated_timeouts": timeout_updates, - }, - ) - - except InvalidPipelineIdError: - return make_error_response( - f"Invalid pipeline ID format: {pipeline_id}", - status_code=400, - ) - except PipelineNotFoundError: - return make_error_response( - f"Pipeline {pipeline_id} not found", - status_code=404, - ) - except StateValidationError as e: - return make_error_response( - f"Invalid update: {e}", - status_code=400, - ) - - -def _compute_gateway_mode( - pipeline: Pipeline, -) -> tuple[Literal["public", "private"], str | None]: - """Compute gateway session mode from pipeline config and repo visibility. - - Uses the explicit ``network_mode`` if set, otherwise auto-detects from - repository visibility via the gateway. Defaults to ``"public"``. - - Returns: - A ``(mode, visibility)`` tuple. ``visibility`` is ``None`` when - ``network_mode`` is explicit, the pipeline has no repo, or the - gateway query failed. - """ - if pipeline.network_mode: - return pipeline.network_mode, None - if pipeline.repo: - vis = get_gateway_client().get_repo_visibility(pipeline.repo) - if vis in ("private", "internal"): - return "private", vis - return "public", vis - return "public", None - - -def _cleanup_remote_branches( - pipeline_id: str, - pipeline: Pipeline, - repo_path: Path, -) -> None: - """Best-effort cleanup of remote branches for a pipeline. - - Deletes the pipeline's shared branch (``pipeline.branch``, typically - ``egg/{pipeline_id}/work`` since #2399) and every per-container - worktree branch (``egg/{container_id}/work``). Slice integration - branches at ``egg/{pipeline_id}/slice-N`` are siblings of the - pipeline tip and are NOT deleted here — see follow-up tracking on - #2399 for full namespace cleanup. Failures are logged as warnings - and do not block pipeline deletion. - """ - branches: set[str] = set() - if pipeline.branch: - branches.add(pipeline.branch) - for phase_exec in pipeline.phases.values(): - for container in phase_exec.containers: - branches.add(f"egg/{container.container_id}/work") - - if not branches: - return - - gateway_client = get_gateway_client() - repo_path_str = str(repo_path) - mode, _vis = _compute_gateway_mode(pipeline) - - deleted = 0 - for branch in sorted(branches): - result = gateway_client.delete_remote_branch(pipeline_id, repo_path_str, branch, mode=mode) - # ``already_deleted`` means the desired state (branch absent on - # remote) is satisfied — count it as success rather than churning a - # warning every time a pipeline is cleaned up before any branch was - # ever pushed. - if result or result.category == "already_deleted": - deleted += 1 - else: - logger.warning( - "Remote branch deletion failed during pipeline cleanup", - pipeline_id=pipeline_id, - branch=branch, - category=result.category, - detail=result.detail, - ) - - if deleted: - logger.info( - "Cleaned up remote branches", - pipeline_id=pipeline_id, - branches_deleted=deleted, - branches_total=len(branches), - ) - - -@pipelines_bp.route("/<pipeline_id>", methods=["DELETE"]) -@require_lifecycle_secret -def delete_pipeline(pipeline_id: str) -> tuple[Response, int]: - """ - Delete a pipeline. - - URL params: - pipeline_id: Pipeline ID - - Response: - { - "success": true, - "message": "Pipeline deleted" - } - """ - repo_path = get_repo_path() - - try: - store, _pipeline = _resolve_pipeline(pipeline_id, repo_path) - - # Clean up any running containers for this pipeline - try: - spawner = _get_spawner() - # Pass the running pipeline's gateway mode + base branch so the - # auto-salvage hook in cleanup_pipeline pushes recovery refs - # under the same policy the pipeline ran under (#2429 review). - _delete_salvage_mode, _ = _compute_gateway_mode(_pipeline) - removed = spawner.cleanup_pipeline( - pipeline_id, - force=True, - salvage_mode=_delete_salvage_mode, - salvage_base_branch=_pipeline.base_branch, - ) - if removed > 0: - logger.info( - "Cleaned up pipeline containers", - pipeline_id=pipeline_id, - containers_removed=removed, - ) - except (DockerClientError, DockerException, KubernetesClientError) as e: - logger.warning( - "Failed to clean up pipeline containers", - pipeline_id=pipeline_id, - error=str(e), - ) - except Exception as e: - logger.error( - "Unexpected error during pipeline container cleanup", - pipeline_id=pipeline_id, - error=str(e), - exc_info=True, - ) - - # Clean up remote branches (best-effort) - try: - _cleanup_remote_branches(pipeline_id, _pipeline, repo_path) - except Exception as e: - logger.warning( - "Failed to clean up remote branches", - pipeline_id=pipeline_id, - error=str(e), - ) - - # Clean up the message store stream/counters AND the in-memory - # consensus tracker / legacy evaluator so a fresh pipeline that - # later reuses this id starts with empty consensus state (#2053). - _clear_pipeline_runtime_state(pipeline_id, reason="pipeline_delete") - - store.delete_pipeline(pipeline_id) - - logger.info("Pipeline deleted", pipeline_id=pipeline_id) - - return make_success_response("Pipeline deleted") - - except InvalidPipelineIdError: - return make_error_response( - f"Invalid pipeline ID format: {pipeline_id}", - status_code=400, - ) - except PipelineNotFoundError: - return make_error_response( - f"Pipeline {pipeline_id} not found", - status_code=404, - ) - - -@pipelines_bp.route("/<pipeline_id>/agents/<agent_role>/restart", methods=["POST"]) -@require_lifecycle_secret -def restart_agent(pipeline_id: str, agent_role: str) -> tuple[Response, int]: - """Restart a single agent in a pipeline (orchestrator-native). - - After #3164 the orchestrator unconditionally owns the BRC event - loop: agent work runs as one-shot Jobs spawned per actionable - event by the event loop, and the in-pod wait arm is gone. A - resident pod spawned here without ``EGG_EVENT_ACTION`` would - immediately log FATAL and ``exit 64``, so ``restart_agent`` no - longer spawns anything itself. Instead it: - - 0. Enforces the per-(pipeline, role, slice) restart budget - (``check_and_increment_restart_count``); a request over budget is - rejected with HTTP 429 before any state is mutated (#3244). - 1. Best-effort deletes the role's live one-shot Job(s) (to kill a - stuck pod). One-shot Jobs carry an event-discriminator suffix - in their name, so they are found by label - (``LABEL_PIPELINE_ID`` + ``LABEL_AGENT_ROLE`` [+ ``LABEL_SLICE_ID`` - when slice-scoped]), not by name. - 2. Resets the role's consensus state and health-monitor anchor. - 3. Marks the agent record RUNNING with ``container_id = None``. - - For a pipeline that is already RUNNING, the live event loop (polling - ~every 5s during the concurrent phase) spawns a fresh one-shot pod once - the role's consensus state is reset — that is the respawn. For a pipeline - that was FAILED/CANCELLED the event loop and its ``_run_pipeline`` driver - thread are already dead, so the route also relaunches a fresh driver - thread (mirroring ``restart_phase``) to restart the event loop; otherwise - the reset would leave the pipeline RUNNING-but-idle with nothing to - respawn it (#3244). The agent's per-agent worktree is preserved so - committed work is retained. - - URL params: - pipeline_id: Pipeline ID - agent_role: Agent role to restart (e.g. "coder", "tester") - - Query string (optional): - slice_id: Slice scope (``slice-<N>``). When supplied, the - slice-scoped Job and worktree are restarted, ``EGG_SLICE_ID`` - is propagated to the new Job, and consensus reset targets - the per-slice tracker. ``slice_id`` may also be supplied via - the JSON body. When omitted for a role that runs as a - per-slice agent, it is derived from the phase's agent records - (#2759): if exactly one slice has a non-complete record for - the role, that slice is used; otherwise the request is - rejected with the candidate list rather than spawning an - unscoped agent. The scan is scoped to ``pipeline.current_phase`` - only — if the pipeline has advanced past the slice's phase - (e.g. to ``pr`` or a later iteration) no current-phase records - will name the role, derivation falls through, and the operator - should supply ``slice_id`` explicitly. This is operator guidance, - not a code-enforced precondition: the fall-through branch - proceeds to a pipeline-level spawn rather than rejecting. - Genuinely pipeline-level agents (no per-slice records for the - role) omit ``slice_id``. - - Request body (optional): - { - "reason": "Human-readable reason for the restart", - "slice_id": "slice-2" - } - - Response: - { - "success": true, - "data": { - "agent_role": "coder", - "slice_id": "slice-2", - "respawn": "delegated to orchestrator event loop", - "restart_count": 1 - } - } - """ - repo_path = get_repo_path() - - try: - store, pipeline = _resolve_pipeline(pipeline_id, repo_path) - except InvalidPipelineIdError: - return make_error_response(f"Invalid pipeline ID format: {pipeline_id}", status_code=400) - except PipelineNotFoundError: - return make_error_response(f"Pipeline {pipeline_id} not found", status_code=404) - - # Validate agent role - try: - role = AgentRole(agent_role) - except ValueError: - return make_error_response(f"Invalid agent role: {agent_role}", status_code=400) - - # Validate pipeline is in a restartable state. CANCELLED is included so - # that a cancel_task(cleanup=false) pipeline can be resumed without a - # full resubmission (see #1725). - if pipeline.status not in ( - PipelineStatus.RUNNING, - PipelineStatus.AWAITING_HUMAN, - PipelineStatus.FAILED, - PipelineStatus.CANCELLED, - ): - return make_error_response( - f"Pipeline {pipeline_id} is not in a restartable state (status: {pipeline.status.value})", - status_code=409, - ) - - body = request.get_json(silent=True) or {} - reason = body.get("reason", "Manual restart via API") - - # Slice scope (#2410): query param wins over body so the URL - # form is unambiguous; both forms validate against the canonical - # ``slice-<N>`` shape via ``extract_slice_id``. - raw_slice_id = request.args.get("slice_id") - slice_payload = {"slice_id": raw_slice_id} if raw_slice_id is not None else body - try: - slice_id = extract_slice_id(slice_payload) - except ValueError as e: - return make_error_response(str(e), status_code=400) - - # Slice auto-derivation (#2759). A slice-mode restart that omits - # ``slice_id`` would otherwise spawn the agent pipeline-level: - # ``EGG_SLICE_ID`` is set by the spawner only when ``slice_id`` is - # non-None, so the respawned agent's BRC signals route to the bare - # pipeline tracker instead of the slice's tracker. The slice's own - # tracker keeps the dead agent registered while the live one ACKs - # into the wrong tracker — the slice's consensus then wedges with no - # message-bus recovery path. Since ``restart_agent`` is the - # operator's normal tool for recovering a failed container, the - # omission must not silently produce an unscoped agent. - # - # When the role runs as a per-slice agent (it has slice-scoped - # records in the current phase), derive the slice: the k8s monitor - # marks a cleanly-exited agent COMPLETE and a crashed one FAILED, so - # a single non-COMPLETE record isolates the slice that needs the - # restart. If the choice is ambiguous — multiple non-COMPLETE - # records, or none at all — reject with the candidate list so the - # operator re-issues with an explicit ``slice_id``. - if slice_id is None: - derive_phase_exec = pipeline.phases.get(pipeline.current_phase.value) - if derive_phase_exec is not None: - role_records = [ - a - for a in derive_phase_exec.agents - if hasattr(a, "role") - and (a.role == role or (hasattr(a.role, "value") and a.role.value == role.value)) - ] - sliced_records = [a for a in role_records if getattr(a, "slice_id", None)] - if sliced_records: - known_slices = sorted({a.slice_id for a in sliced_records}) - restart_candidates = sorted( - { - a.slice_id - for a in sliced_records - if a.status != AgentExecutionStatus.COMPLETE - } - ) - if len(restart_candidates) == 1: - slice_id = restart_candidates[0] - logger.info( - "restart_agent: derived slice_id from phase agent records", - pipeline_id=pipeline_id, - agent_role=agent_role, - slice_id=slice_id, - ) - else: - detail = ( - "no slice has a non-complete agent record for this role" - if not restart_candidates - else f"{len(restart_candidates)} slices have a non-complete record" - ) - return make_error_response( - f"Agent role {agent_role!r} runs as a per-slice agent in " - f"pipeline {pipeline_id}; restart_agent could not derive " - f"slice_id ({detail}). Re-issue with an explicit slice_id.", - status_code=400, - details={ - "agent_role": agent_role, - "known_slices": known_slices, - "restart_candidates": restart_candidates, - }, - reason="slice_id_required", - ) - - # Slice-existence check (#2421): a well-formed but unknown - # ``slice_id`` would otherwise spawn an orphan Job + worktree - # the rest of the system has no record of. The shape regex in - # ``extract_slice_id`` only catches malformed values; only the - # contract knows which slices the pipeline actually has. - # - # Pipelines without a contract are not - # slice-aware, so any non-``None`` ``slice_id`` targeting them is - # by definition unknown — reject outright. For contracted - # pipelines, load the contract and check membership; fall through - # silently if the contract can't be loaded (worktree pruned, - # contract not yet populated, filesystem error) so we don't - # regress legitimate restarts on the existing pipeline-level path. - # - # After #3164 ``restart_agent`` no longer spawns a worktree itself, - # so the slice's parent-edge / base-branch resolution that used to - # feed the spawn is gone. Only the existence check below remains. - if slice_id is not None: - if not pipeline.has_contract: - return make_error_response( - f"slice_id {slice_id!r} is invalid for pipeline " - f"{pipeline_id} (pipeline has no contract; not slice-aware)", - status_code=404, - details={ - "slice_id": slice_id, - "known_slices": [], - }, - ) - try: - from egg_contracts.loader import ( - ContractNotFoundError, - ContractValidationError, - load_contract, - ) - from routes import resolve_worktree_path - except ImportError: - logger.warning( - "Required modules unavailable; skipping slice_id existence check", - pipeline_id=pipeline_id, - slice_id=slice_id, - ) - else: - contract = None - try: - worktree_path = resolve_worktree_path(pipeline_id, Path(repo_path)) - contract_id = _pipeline_identifier(pipeline.issue_number, pipeline_id) - try: - contract = load_contract(contract_id, worktree_path) - except ContractNotFoundError: - # Contract not yet populated — fall through silently - # (``contract`` already initialised to ``None`` above). - pass - except (OSError, ValueError, ContractValidationError) as exc: - # Worktree pruned, filesystem failure, or corrupt/invalid - # contract JSON: log and fall through. The reviewer's #2421 - # ask was to catch the easy "wrong slice_id" case, not to - # gate restarts on contract reachability. Programmer errors - # (AttributeError, TypeError, NameError) are left to - # propagate so they surface during development. - logger.warning( - "Could not load contract for slice_id existence check; allowing restart", - pipeline_id=pipeline_id, - slice_id=slice_id, - error=str(exc), - ) - if contract is not None: - slice_obj = next((s for s in contract.slices if s.id == slice_id), None) - if slice_obj is None: - return make_error_response( - f"slice_id {slice_id!r} does not match any slice in " - f"pipeline {pipeline_id}'s contract", - status_code=404, - details={ - "slice_id": slice_id, - "known_slices": sorted(s.id for s in contract.slices), - }, - ) - - spawner = _get_spawner() - - current_phase = pipeline.current_phase.value - phase_exec = pipeline.phases.get(current_phase) - - # Enforce the per-(pipeline, role, slice) restart budget BEFORE any - # destructive action (#3244 review). Pre-#3164 this cap lived inside - # ``restart_agent_job``, which the route no longer calls — without - # re-enforcing it here an operator/overseer could call ``restart_agent`` - # without bound, each call resetting consensus and actively preventing a - # live phase from converging. ``check_and_increment_restart_count`` raises - # when the budget is exhausted; reject loudly (429) instead of flipping - # status / resetting consensus and returning a misleading success. The - # returned count is the source of truth for the ``restart_count`` - # telemetry below (the old read-only ``get_restart_count`` read always - # reported 0 on this path since nothing incremented it). - try: - new_restart_count = spawner.check_and_increment_restart_count( - pipeline_id, role, slice_id=slice_id - ) - except KubernetesSpawnError as budget_err: - logger.warning( - "restart_agent rejected: restart budget exhausted", - pipeline_id=pipeline_id, - agent_role=agent_role, - slice_id=slice_id, - error=str(budget_err), - ) - return make_error_response(str(budget_err), status_code=429) - - # Early status update: transition FAILED/CANCELLED -> RUNNING so that - # get_status returns "running" immediately. Unlike a RUNNING pipeline — - # whose live event loop picks up the consensus reset below and respawns - # within one poll — a FAILED/CANCELLED pipeline has NO live event loop: - # ``_run_concurrent_phase`` already returned and ``stop_event_loop()`` - # tore the loop down on its way out, and the ``_run_pipeline`` driver - # thread has exited. Resetting consensus alone would leave the pipeline - # RUNNING-but-idle with nothing to respawn it (#3244 review). So when we - # make this transition we record it and relaunch a fresh ``_run_pipeline`` - # driver thread at the end of the route (mirroring ``restart_phase`` step - # 7) — that restarts the event loop, which then performs the respawn. - pipeline_was_inactive = pipeline.status in ( - PipelineStatus.FAILED, - PipelineStatus.CANCELLED, - ) - if pipeline_was_inactive: - early_lock = get_pipeline_state_lock(pipeline_id) - with early_lock: - pipeline = store.load_pipeline(pipeline_id) - if pipeline.status in (PipelineStatus.FAILED, PipelineStatus.CANCELLED): - pipeline.status = PipelineStatus.RUNNING - _phase_exec = pipeline.phases.get(current_phase) - if _phase_exec is not None: - _phase_exec.status = PipelineStatus.RUNNING - # Bump run_epoch so the relaunched driver thread (below) owns a - # fresh epoch namespace and any stale thread that observes the - # transition detects itself as superseded (mirrors - # ``restart_phase`` / ``advance_phase``). - pipeline.run_epoch = datetime.now(UTC) - pipeline.updated_at = datetime.now(UTC) - store.update_pipeline(pipeline_id, pipeline.model_dump(mode="json")) - else: - # Lost the race — another writer already moved it off - # FAILED/CANCELLED, so its driver thread / event loop is - # live and will own the respawn. Don't relaunch a duplicate. - pipeline_was_inactive = False - - # #3164: ``restart_agent`` no longer spawns a resident pod. The - # orchestrator event loop owns the BRC respawn — once the role's - # consensus state is reset (below), it spawns a fresh one-shot pod - # within one ~5s poll. Here we only (1) kill any live one-shot Job - # for the role so a stuck pod is torn down, then (2) reset consensus - # + health so the event loop reschedules. - - # Delete the role's live one-shot Job(s), best-effort. One-shot - # event Jobs carry an event-discriminator SUFFIX in their name (one - # Job per actionable BRC event), so they can't be addressed by a - # deterministic name — find them by LABEL. Match on pipeline + - # role (and slice when scoped). Zero matches is fine (the role may - # have already exited cleanly); the event loop will respawn either - # way once consensus is reset. Wrap broadly so a k8s/list failure - # never fails the restart. - job_labels = { - LABEL_PIPELINE_ID: pipeline_id, - # The role label value is the underscore form (e.g. - # ``reviewer_code``), which is exactly ``agent_role`` / ``role.value``. - LABEL_AGENT_ROLE: role.value, - } - if slice_id is not None: - job_labels[LABEL_SLICE_ID] = slice_id - try: - live_jobs = spawner.k8s.list_containers(labels=job_labels) - removed_jobs = 0 - for job in live_jobs: - try: - # Mirror the cleanup call sites: prefer the explicit - # ``job_name`` (already Job-prefixed), fall back to the - # container id which ``remove_agent_job`` -> ``remove_container`` - # resolves to a Job name. - spawner.remove_agent_job(job.job_name or job.container_id, force=True) - removed_jobs += 1 - except Exception as job_err: # noqa: BLE001 - best-effort teardown - logger.warning( - "Failed to delete live one-shot Job during restart (best-effort)", - pipeline_id=pipeline_id, - agent_role=agent_role, - slice_id=slice_id, - job_name=getattr(job, "job_name", None), - error=str(job_err), - ) - logger.info( - "restart_agent: deleted live one-shot Job(s) for role", - pipeline_id=pipeline_id, - agent_role=agent_role, - slice_id=slice_id, - removed=removed_jobs, - ) - except Exception as list_err: # noqa: BLE001 - best-effort teardown - logger.warning( - "Failed to list live one-shot Jobs during restart (best-effort)", - pipeline_id=pipeline_id, - agent_role=agent_role, - slice_id=slice_id, - error=str(list_err), - ) - - # Reset consensus state for this agent so the event loop reschedules - # a fresh one-shot pod for it. If consensus reset fails, log a - # warning but don't fail the restart: the agent will re-enter - # consensus on its own. Slice-scoped restarts (#2410) target the - # per-slice tracker; the pipeline-level tracker has no record of the - # slice agent. - # Slice-scoped restarts (#2410) target the per-slice tracker; the - # pipeline-level tracker has no record of the slice agent. - # - # INVARIANT (#3200 task-7-1, mid-phase BRC record survival): this reset - # clears the *peer consensus tracker* (the ephemeral ACK/NACK/proposal - # bookkeeping the restarted agent rebuilds by re-proposing) but MUST NOT - # clear the *Redis message store* (``pipeline:{id}:messages``). That store - # is the durable BRC message record — CONSENSUS_PROPOSE/ACK/NACK and the - # conditional-ACK obligations — and a mid-phase restart deliberately - # preserves it so the reseeded/resumed session can re-pull it via - # ``GET /<pipeline_id>/brc-transcript`` + ``read_peer_artifact`` and - # re-derive the #3189 deterministic anchors. The store is cleared only at - # phase transitions (``_clear_concurrent_state``) and pipeline - # create/delete (``_clear_pipeline_runtime_state``), never here. Do NOT - # add ``get_message_store().clear()`` / ``_clear_concurrent_state`` to the - # restart path — that would lose the record across the restart boundary. - try: - try: - from peer_consensus import get_peer_consensus_tracker - except ImportError: - from ..peer_consensus import ( - get_peer_consensus_tracker, # type: ignore[import-not-found] - ) - - tracker = get_peer_consensus_tracker(pipeline_id, slice_id) - if tracker: - tracker.remove_agent(agent_role) - logger.info( - "Reset consensus state for agent", - pipeline_id=pipeline_id, - agent_role=agent_role, - slice_id=slice_id, - ) - except ImportError: - pass - except Exception as e: - logger.warning( - "Failed to reset consensus state (agent will re-enter consensus)", - pipeline_id=pipeline_id, - agent_role=agent_role, - slice_id=slice_id, - error=str(e), - ) - - # Reset health-monitor anchor so the pre-respawn _last_heartbeat does not - # generate a stale-elapsed heartbeat_timeout alert against the fresh - # container (issue #2084). - # - # #2270 slice-5 (restart hygiene): ``reset_agent`` also drops the agent's - # accumulated per-agent escalation state (escalation flags, error counts, - # active alerts). Clearing it on restart is what stops a freshly-restarted - # agent from inheriting a stale redirect/escalation history that would push - # it straight to HITL on its first post-restart stall. The Tier-2 overseer's - # own escalation-history clear + generation reset live on - # ``OverseerMonitor`` (overseer/monitor.py:reset_escalation_history / - # reset_generation), which the on-demand adjudicator constructs fresh. - try: - try: - from health_monitor import get_health_monitor - except ImportError: - from ..health_monitor import ( - get_health_monitor, # type: ignore[import-not-found] - ) - _hm = get_health_monitor() - if _hm is not None: - _hm.reset_agent(agent_role) - except Exception as e: - logger.warning( - "Failed to reset health-monitor state for restarted agent", - pipeline_id=pipeline_id, - agent_role=agent_role, - error=str(e), - ) - - # Update pipeline state. No resident container is spawned (#3164) — - # the event loop will respawn a one-shot pod within one poll once the - # consensus reset above takes effect. We mark the agent RUNNING with - # ``container_id = None`` (the live pod is set by the event loop) and - # refresh ``started_at`` so the overseer's - # phase_minimum_working_window suppression on the - # ``agent-heartbeat-stall`` trigger anchors on the restart (#2084). - lock = get_pipeline_state_lock(pipeline_id) - with lock: - pipeline = store.load_pipeline(pipeline_id) - if phase_exec is not None: - # Re-fetch from the freshly loaded pipeline (the outer check gates - # on "did the phase exist before the restart?"). - fresh_phase_exec = pipeline.phases.get(current_phase) - if fresh_phase_exec is not None: - from models import AgentExecution # type: ignore - - respawn_started_at = datetime.now(UTC) - # Match on ``(role, slice_id)`` — without the slice tiebreaker - # the first matching role wins, which on a multi-slice phase - # mutates the wrong slice's record (#2422). ``slice_id`` is - # the route-level scope already plumbed into the consensus - # tracker above. - found = False - for agent in fresh_phase_exec.agents: - if not hasattr(agent, "role"): - continue - role_match = agent.role == role or ( - hasattr(agent.role, "value") and agent.role.value == role.value - ) - if not role_match: - continue - if getattr(agent, "slice_id", None) != slice_id: - continue - agent.container_id = None - agent.status = AgentExecutionStatus.RUNNING - agent.started_at = respawn_started_at - found = True - break - if not found: - fresh_phase_exec.agents.append( - AgentExecution( - role=role, - container_id=None, - status=AgentExecutionStatus.RUNNING, - started_at=respawn_started_at, - slice_id=slice_id, - ) - ) - - pipeline.updated_at = datetime.now(UTC) - store.update_pipeline(pipeline_id, pipeline.model_dump(mode="json")) - - # ``restart_count`` is the value just incremented by - # ``check_and_increment_restart_count`` above (#3244). It is scoped to the - # same ``(pipeline_id, agent_role, slice_id)`` bucket the cap is enforced - # on, so it correctly reports the operator's "you've burned N of M - # restarts" telemetry — the pre-fix read-only ``get_restart_count`` read - # always reported 0 here because nothing on this path incremented it. - response_data: dict[str, object] = { - "agent_role": agent_role, - "slice_id": slice_id, - "respawn": "delegated to orchestrator event loop", - "restart_count": new_restart_count, - } - - # When the pipeline was FAILED/CANCELLED its event loop and driver thread - # are dead (see the early-status comment above), so the consensus reset - # alone has nothing to act on it. Relaunch a fresh ``_run_pipeline`` driver - # thread — exactly as ``restart_phase`` step 7 does — to restart the event - # loop, which then respawns the role's one-shot Job within one poll. For a - # pipeline that was already RUNNING we skip this: its live event loop owns - # the respawn and a second driver thread would race it (#3244 review). - if pipeline_was_inactive: - _spawn_pipeline_run_thread(pipeline_id, store.repo_path, pipeline.run_epoch) - logger.info( - "restart_agent: relaunched driver thread for inactive pipeline", - pipeline_id=pipeline_id, - agent_role=agent_role, - slice_id=slice_id, - run_epoch=pipeline.run_epoch.isoformat() if pipeline.run_epoch else None, - ) - - logger.info( - "Agent restart requested (respawn delegated to event loop)", - pipeline_id=pipeline_id, - agent_role=agent_role, - slice_id=slice_id, - restart_count=response_data.get("restart_count"), - reason=reason, - ) - - return make_success_response( - f"Agent {agent_role} restarted", - data=response_data, - ) - - -@pipelines_bp.route("/<pipeline_id>/phases/<phase>/restart", methods=["POST"]) -@require_lifecycle_secret -def restart_phase(pipeline_id: str, phase: str) -> tuple[Response, int]: - """Restart all agents in a pipeline phase. - - Stops and removes all containers for the phase, resets consensus and - review cycle state, and respawns all agents. Prior phase artifacts - (from earlier phases) are preserved. - - Preservation semantics (#3080): per-agent worktrees AND their local - branches are deleted, so per-role branch tips do not survive a phase - restart. Unpushed commits are salvaged to ``egg/recovered/*`` refs - on a best-effort basis (#2429) — ``auto_salvage_pipeline`` - re-enumerates worktrees with ``validate_git=True``, so worktrees - with a corrupted ``.git`` marker (the #1723 failure class) may be - skipped without salvage. The respawned agents' fresh worktrees - re-fork from the shared work branch tip (``origin/<assigned_branch>``, - base-branch fallback when unpushed — see #3068). Anything that - lived only on a per-role branch (e.g. a reviewer's merge history) - is therefore discarded from agent trees; only state pushed to the - shared work branch is re-materialised on respawn. Operators needing - per-worktree retention should use ``restart_agent`` instead. - - URL params: - pipeline_id: Pipeline ID - phase: Phase name to restart (e.g. "implement") - - Request body (optional): - { - "reason": "Human-readable reason for the restart" - } - - Response: - { - "success": true, - "data": { - "phase": "implement", - "agents_to_restart": ["coder", "tester", "documenter", ...] - } - } - """ - repo_path = get_repo_path() - - try: - store, pipeline = _resolve_pipeline(pipeline_id, repo_path) - except InvalidPipelineIdError: - return make_error_response(f"Invalid pipeline ID format: {pipeline_id}", status_code=400) - except PipelineNotFoundError: - return make_error_response(f"Pipeline {pipeline_id} not found", status_code=404) - - # Validate phase - try: - PipelinePhase(phase) - except ValueError: - return make_error_response(f"Invalid phase: {phase}", status_code=400) - - # Validate pipeline is in a restartable state. CANCELLED is included so - # that a cancel_task(cleanup=false) pipeline can be resumed without a - # full resubmission (see #1725). - if pipeline.status not in ( - PipelineStatus.RUNNING, - PipelineStatus.AWAITING_HUMAN, - PipelineStatus.FAILED, - PipelineStatus.CANCELLED, - ): - return make_error_response( - f"Pipeline {pipeline_id} is not in a restartable state (status: {pipeline.status.value})", - status_code=409, - ) - - # Only the current phase can be restarted — restarting a completed or - # future phase would corrupt pipeline state. - if phase != pipeline.current_phase.value: - return make_error_response( - f"Phase {phase} is not the current phase (current: {pipeline.current_phase.value})", - status_code=409, - ) - - phase_exec = pipeline.phases.get(phase) - if phase_exec is None: - return make_error_response( - f"Phase {phase} not found in pipeline {pipeline_id}", status_code=404 - ) - - body = request.get_json(silent=True) or {} - reason = body.get("reason", "Manual phase restart via API") - - # Compute gateway mode from pipeline config (not hardcoded "public") - gateway_mode, _ = _compute_gateway_mode(pipeline) - - spawner = _get_spawner() - - # Acquire the pipeline state lock to collect agent roles, snapshot - # container IDs, and update pipeline status to RUNNING *before* the - # slow container teardown. This ensures that ``get_status`` returns - # ``running`` immediately, even if the MCP call times out during - # container stop/remove (see #1594). - lock = get_pipeline_state_lock(pipeline_id) - with lock: - # Re-load pipeline under the lock so agent_roles reflects the - # latest state (guards against concurrent modifications). - pipeline = store.load_pipeline(pipeline_id) - - # Re-check current phase under the lock to prevent TOCTOU race: - # the pipeline could have advanced between the earlier check and - # lock acquisition. - if phase != pipeline.current_phase.value: - return make_error_response( - f"Phase {phase} is not the current phase (current: {pipeline.current_phase.value})", - status_code=409, - ) - - phase_exec = pipeline.phases.get(phase) - if phase_exec is None: - return make_error_response( - f"Phase {phase} not found in pipeline {pipeline_id}", status_code=404 - ) - - # 1. Collect agent roles for respawning. Prefer the runtime cache - # on ``phase_exec.agents`` since it reflects the roster from - # the most recent spawn, but fall back to the deterministic - # source the executor itself consults — ``get_roles_for_phase``. - # Without this fallback a restart whose clear step ran - # (``phase_exec.agents = []`` below) but whose spawn step - # failed leaves the pipeline unrecoverable: every subsequent - # ``restart_phase`` 400s on the now-empty cache, and - # ``start_pipeline`` 409s on the CANCELLED state (#2515). - agent_roles: list[AgentRole] = [] - for agent in phase_exec.agents: - if hasattr(agent, "role"): - role = agent.role if isinstance(agent.role, AgentRole) else AgentRole(agent.role) - agent_roles.append(role) - - if not agent_roles: - # Mirror ``_run_concurrent_phase`` exactly so the route's - # response (and the downstream worktree-delete / health- - # monitor reset) matches the roster the spawn will actually - # produce. - try: - from egg_contracts.agent_roles import ( - get_roles_for_phase as _get_roles_for_phase, - ) - - for r in _get_roles_for_phase( - phase, - include_reviewers=True, - repo=pipeline.repo, - has_contract=getattr(pipeline, "has_contract", True), - ): - try: - agent_roles.append(AgentRole(r.value)) - except ValueError: - continue - except Exception as exc: # noqa: BLE001 - # Catch derivation failures so the route returns 400 - # rather than 500 — deliberate divergence from - # ``_run_concurrent_phase``, which lets the same failure - # propagate up the worker thread. In a synchronous HTTP - # context an honest 400 ("No agents found") is more - # useful to the operator than a 500. - logger.warning( - "restart_phase: failed to derive default roster fallback", - pipeline_id=pipeline_id, - phase=phase, - error=str(exc), - ) - - if not agent_roles: - return make_error_response( - f"No agents found in phase {phase} to restart", status_code=400 - ) - - logger.info( - "restart_phase: phase_exec.agents empty, derived roster from pipeline config", - pipeline_id=pipeline_id, - phase=phase, - agent_roles=[r.value for r in agent_roles], - ) - - # 2. Snapshot container IDs for teardown outside the lock - old_container_ids = [c.container_id for c in phase_exec.containers] - - # 3. Fully reset phase execution state so the new _run_pipeline - # thread treats this as a fresh phase. Set pipeline status to - # RUNNING and bump run_epoch so any lingering old _run_pipeline - # thread detects the restart and exits (see #1638). - # NOTE: artifacts are intentionally preserved — they may contain - # outputs from partial work useful as context for the retry. - phase_exec.containers = [] - phase_exec.agents = [] - phase_exec.review_cycles = 0 - phase_exec.hitl_review_cycles = 0 - phase_exec.status = PipelineStatus.PENDING - phase_exec.started_at = None - phase_exec.work_started_at = None - phase_exec.completed_at = None - phase_exec.error = None - phase_exec.cycle_timings = [] - pipeline.status = PipelineStatus.RUNNING - pipeline.error = None - pipeline.run_epoch = datetime.now(UTC) - # ``updated_at`` is unconditionally set by ``StateStore.save_pipeline`` - # (which ``update_pipeline`` routes through). - store.update_pipeline(pipeline_id, pipeline.model_dump(mode="json")) - - # --- Outside the lock: slow, idempotent, best-effort operations --- - - # 3b. Persist the in-flight phase's BRC message record to disk BEFORE the - # destructive container/worktree teardown (#3200 task-7-1, mid-phase - # BRC record survival). Today ``_write_brc_history`` runs only at phase - # transitions (``_persist_phase_brc_history`` in complete/advance_phase, - # #1827); a mid-phase restart never wrote the durable on-disk - # transcript. - # - # PRIMARY mechanism is option (a), the live Redis stream: it survives a - # bare restart (the store is cleared only at phase transitions / - # pipeline create+delete, never here — see step 5), so a reseeded - # session re-pulls the in-flight record from Redis via - # ``/brc-transcript`` + ``read_peer_artifact``. The slice-scoped - # CONSENSUS_PROPOSE/ACK/NACK records of an in-flight implement slice - # rely on (a) for survival. - # - # This disk persist (option (b)) is a belt-and-suspenders ADD-ON with a - # deliberately NARROW durability scope — do not overstate it. It calls - # ``_persist_phase_brc_history`` -> ``_write_brc_history( - # write_per_slice=False)``. For a slice-aware implement phase that path - # writes ONLY the ``{id}-implement-unattributed.{md,json}`` sibling - # (non-CONSENSUS BRC types: HEARTBEAT/STATUS/HANDOFF/AGENT_FAILED/ - # NUDGE/OVERSEER_ALERT) and SKIPS the per-slice bucket loop; the - # slice's CONSENSUS_* proposals/verdicts/open-NACKs are NOT written to - # disk here (write_per_slice=False avoids the #2755 add/add conflict on - # ``work``; per-slice files are owned by the slice integration branch). - # So across a FULL Redis loss (orchestrator pod death, the cold-start - # case task-6-1 covers) the in-flight slice record does NOT survive on - # disk — only (a) preserves it. What (b) does buy: for non-slice phases - # (plan/refine/pr) and non-slice implement runs the aggregate - # ``{id}-{phase}.{md,json}`` transcript IS written, and for slice runs - # the unattributed audit sibling is captured — extending the #1827 - # persist-before-clear invariant to the restart path for everything - # except the per-slice CONSENSUS buckets. Best-effort and front-running - # teardown: a transcript-write hiccup must never block recovery of a - # wedged phase (mirrors the salvage step below). - try: - _persist_phase_brc_history(pipeline, store, phase) - except Exception as brc_persist_err: # noqa: BLE001 - logger.warning( - "Failed to persist in-flight BRC history during phase restart (continuing)", - pipeline_id=pipeline_id, - phase=phase, - error=str(brc_persist_err), - ) - - # 4. Stop and remove old containers - for container_id in old_container_ids: - try: - spawner.stop_agent_container(container_id, cleanup_session=True) - except Exception as e: - logger.warning( - "Failed to stop container during phase restart", - container_id=container_id[:12] if container_id else "?", - error=str(e), - ) - try: - spawner.remove_agent_container(container_id, force=True, cleanup_session=False) - except Exception as e: - logger.warning( - "Failed to remove container during phase restart", - container_id=container_id[:12] if container_id else "?", - error=str(e), - ) - - # 4b. Delete per-agent worktrees so respawned containers get fresh mounts. - # Without this, stale worktree directories (e.g. broken btrfs mounts) - # survive container removal and cause create_worktree to skip creation - # or fail. Mirrors cleanup_pipeline's worktree deletion. (#1723) - # - # Enumerate from disk rather than guess names: slice-scoped worktrees - # are ``{pipeline_id}-slice-{N}-{role}``, not ``{pipeline_id}-{role}``, - # so a name-guess loop misses every per-slice worktree on a slice - # pipeline and leaves them behind. (#2522) - # - # ``validate_git=False`` so that broken/corrupted worktrees (missing - # or unreadable ``.git`` marker — exactly the #1723 btrfs failure - # class) still reach ``delete_worktrees``. The default - # ``validate_git=True`` is salvage-correct (you can't salvage a - # broken worktree) but cleanup-incorrect (you must still delete it). - restart_role_values = {role.value for role in agent_roles} - try: - all_worktrees = agent_salvage.enumerate_agent_worktrees(pipeline_id, validate_git=False) - except (OSError, ImportError, RuntimeError) as e: - logger.warning( - "Failed to enumerate per-agent worktrees during phase restart", - pipeline_id=pipeline_id, - error=str(e), - ) - all_worktrees = [] - worktrees_to_delete = [wt for wt in all_worktrees if wt.agent_role in restart_role_values] - - # Salvage unpushed agent commits before deleting worktrees (#2429). - # Restart is *the* scenario where unpushed commits accumulate: an - # operator hits this endpoint precisely because agents are wedged or - # timed out — the same conditions that prevent pushes from landing on - # ``origin/<assigned_branch>``. Without this hook, restart would be - # the one orchestrator-side worktree-delete code path that bypasses - # salvage and silently destroys recoverable work. Best-effort: any - # failure logs and continues so cleanup cannot be blocked by salvage. - if worktrees_to_delete: - try: - agent_salvage.auto_salvage_pipeline( - spawner.gateway, - pipeline_id, - worktree_filter={wt.worktree_id for wt in worktrees_to_delete}, - mode=gateway_mode, - base_branch=pipeline.base_branch, - ) - except Exception as e: - logger.warning( - "Auto-salvage failed during phase restart; proceeding with worktree deletion", - pipeline_id=pipeline_id, - error=str(e), - ) - - for wt in worktrees_to_delete: - log_extras: dict[str, str] = {} - if wt.slice_id is not None: - log_extras["slice_id"] = wt.slice_id - try: - spawner.gateway.delete_worktrees(container_id=wt.worktree_id, force=True) - logger.info( - "Deleted per-agent worktree during phase restart", - agent_worktree_id=wt.worktree_id, - pipeline_id=pipeline_id, - **log_extras, - ) - except Exception as e: - logger.warning( - "Failed to delete per-agent worktree during phase restart", - agent_worktree_id=wt.worktree_id, - pipeline_id=pipeline_id, - error=str(e), - **log_extras, - ) - - # 5. Reset consensus state. - # Slice-4 TASK-4-1: mirror the slice-aware semantics of - # ``restart_agent`` (line ~2859) — clear BOTH the pipeline-level - # tracker AND every per-slice tracker keyed - # ``f"{pipeline_id}/{slice_id}"`` (see - # ``peer_consensus._tracker_key``). Phase-level restart wipes - # the entire phase, so any per-slice consensus state that - # survived the restart is stale and would deadlock the new run - # if left in place. - # - # INVARIANT (#3200 task-7-1, mid-phase BRC record survival): like - # ``restart_agent`` above, this clears the *peer consensus tracker* - # (ephemeral ACK/NACK state) but MUST NOT clear the *Redis message - # store* (``pipeline:{id}:messages``). That store is the durable BRC - # message record; a mid-phase phase restart preserves it so the - # reseeded session can re-pull it (``/brc-transcript`` + - # ``read_peer_artifact``) and re-derive the #3189 anchors. The store is - # cleared only at phase transitions / pipeline create+delete, never on - # restart. Do NOT add ``get_message_store().clear()`` here. - try: - try: - from peer_consensus import get_peer_consensus_tracker - except ImportError: - from ..peer_consensus import ( - get_peer_consensus_tracker, # type: ignore[import-not-found] - ) - - tracker = get_peer_consensus_tracker(pipeline_id) - if tracker: - tracker.clear() - logger.info("Cleared peer consensus tracker", pipeline_id=pipeline_id) - - # Per-slice trackers. Best-effort contract load: if the - # contract cannot be read (corrupt on disk, etc.), the - # pipeline-level clear above still ran, and the slice - # trackers will be reconstructed lazily on next consensus - # activity — preserving the historical pipeline-level-only - # behaviour as a fallback rather than blocking the restart. - # **Worktree-path resolution (reviewer_code v1 blocker 2)**: - # active pipelines' contracts live in the per-pipeline - # worktree at ``/home/egg/.egg-worktrees/<pipeline_id>/<repo>/`` - # — NOT under ``store.repo_path`` (the main orchestrator repo). - # Without ``resolve_worktree_path`` the ``load_contract`` call - # below silently fails with ``ContractNotFoundError`` for every - # active pipeline, the per-slice loop never iterates, and the - # whole per-slice clear becomes a no-op. Pattern mirrors - # ``routes/signals.py:709`` and ``routes/pipelines.py:10017``. - try: - from egg_contracts.loader import load_contract - except ImportError: - load_contract = None # type: ignore[assignment] - if load_contract is not None: - try: - from routes import resolve_worktree_path - except ImportError: - try: - from . import ( - resolve_worktree_path, # type: ignore[no-redef] - ) - except ImportError: - resolve_worktree_path = None # type: ignore[assignment] - try: - if resolve_worktree_path is not None: - _contract_repo_path = resolve_worktree_path(pipeline_id, Path(store.repo_path)) - else: - _contract_repo_path = Path(store.repo_path) - _contract = load_contract(pipeline_id, _contract_repo_path) - except Exception as load_err: # noqa: BLE001 — best-effort - logger.warning( - "Could not load contract to enumerate slice trackers " - "during phase restart; per-slice consensus state may " - "be left stale until lazy reconstruction", - pipeline_id=pipeline_id, - error=str(load_err), - ) - _contract = None - if _contract is not None and getattr(_contract, "slices", None): - for _s in _contract.slices: - _slice_tracker = get_peer_consensus_tracker(pipeline_id, slice_id=_s.id) - if _slice_tracker: - _slice_tracker.clear() - logger.info( - "Cleared per-slice peer consensus tracker", - pipeline_id=pipeline_id, - slice_id=_s.id, - ) - except ImportError: - pass - except Exception as e: - logger.warning( - "Failed to clear peer consensus", - pipeline_id=pipeline_id, - error=str(e), - ) - - # 6. Reset restart counts for this pipeline - spawner.reset_restart_counts(pipeline_id) - - # 6b. Drop health-monitor anchors for every respawned role so the Tier-1 - # heartbeat clock does not survive the restart and fire stale-elapsed - # alerts that the overseer would faithfully escalate (issue #2084). - try: - try: - from health_monitor import get_health_monitor - except ImportError: - from ..health_monitor import ( - get_health_monitor, # type: ignore[import-not-found] - ) - _hm = get_health_monitor() - if _hm is not None: - for role in agent_roles: - _hm.reset_agent(role.value) - except Exception as e: - logger.warning( - "Failed to reset health-monitor state during phase restart", - pipeline_id=pipeline_id, - phase=phase, - error=str(e), - ) - - # 7. Launch a new _run_pipeline thread to monitor the restarted phase. - # Container spawning is handled by _run_concurrent_phase within the - # thread, matching the recovery pattern used by start_pipeline. - # See #1638: the original polling thread died when the pipeline - # failed; without this, consensus completion is never detected. - agents_to_restart = [role.value for role in agent_roles] - repo_path_for_thread = store.repo_path - - _spawn_pipeline_run_thread(pipeline_id, repo_path_for_thread, pipeline.run_epoch) - - logger.info( - "Phase restarted", - pipeline_id=pipeline_id, - phase=phase, - agents_to_restart=agents_to_restart, - reason=reason, - ) - - return make_success_response( - f"Phase {phase} restarted with {len(agents_to_restart)} agent(s)", - data={ - "phase": phase, - "agents_to_restart": agents_to_restart, - }, - ) - - -def apply_first_principles_redirect( - pipeline_id: str, - new_task_description: str, - *, - reason: str, -) -> list[str]: - """Adopt a first-principles redirect: rewrite the seed and re-run refine. - - Called in-process from the decision-resolve hook when an operator adopts a - redirect raised by the ``first_principles_reviewer``. Two durable steps: - - 1. **Rewrite the seed** via the operator-grade - ``rewrite_task_description_as_operator`` (audited, ``Role.HUMAN``), then - commit+push the worktree to the work branch so the refine restart's - re-fork (which forks fresh worktrees from ``origin/<branch>``) sees the - rewritten ``task_description`` rather than the old one. - 2. **Re-run refine** via :func:`_restart_refine_phase`. - - Returns the role values respawned. Raises on failure; the caller logs and - leaves the decision resolved (the operator's intent is recorded regardless). - """ - from operator_actions import rewrite_task_description_as_operator - - repo_path = get_repo_path() - store, pipeline = _resolve_pipeline(pipeline_id, repo_path) - issue_number = getattr(pipeline, "issue_number", None) - gateway_mode, _ = _compute_gateway_mode(pipeline) - spawner = _get_spawner() - - rewrite = rewrite_task_description_as_operator( - pipeline_id, - new_task_description, - reason=reason, - actor="operator:first-principles-redirect", - issue_number=issue_number, - ) - - # Durably land the rewritten seed on the work branch. The refine restart - # below deletes per-agent worktrees and re-forks fresh ones from - # ``origin/<branch>``; without this push the re-fork would re-materialise - # the OLD seed and the redirect would be silently lost (#3080 re-fork - # semantics). - worktree = Path(rewrite["worktree"]) - identifier = _pipeline_identifier(issue_number, pipeline_id) - try: - committed = _commit_statefiles_to_worktree( - worktree, - f"first-principles redirect: rewrite seed — {reason}"[:200], - identifier, - pipeline_id=pipeline_id, - ) - if committed and pipeline.branch: - spawner.gateway.push_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(worktree), - branch=pipeline.branch, - mode=gateway_mode, - base_branch=pipeline.base_branch, - ) - except Exception as exc: # noqa: BLE001 — best-effort; restart still proceeds - logger.warning( - "Failed to push rewritten seed to work branch; refine restart may " - "re-fork the prior seed (first-principles redirect)", - pipeline_id=pipeline_id, - error=str(exc), - ) - - return _restart_refine_phase( - pipeline_id, store, reason=reason, spawner=spawner, gateway_mode=gateway_mode - ) - - -def _restart_refine_phase( - pipeline_id: str, - store: Any, - *, - reason: str, - spawner: Any, - gateway_mode: str, -) -> list[str]: - """Re-run the refine phase in-process (non-route sibling of ``restart_phase``). - - Mirrors ``restart_phase``'s essential steps for the refine phase so the - first-principles accept-path can re-run refine from the decision-resolve - hook (no Flask request). Refine has no slices, so the per-slice tracker - loop in ``restart_phase`` is intentionally omitted. Raises ``ValueError`` - if the pipeline is not currently parked at the refine phase. - """ - phase = PipelinePhase.REFINE.value - lock = get_pipeline_state_lock(pipeline_id) - with lock: - pipeline = store.load_pipeline(pipeline_id) - if pipeline.current_phase.value != phase: - raise ValueError( - f"_restart_refine_phase: pipeline {pipeline_id} is not at the " - f"refine phase (current: {pipeline.current_phase.value})" - ) - phase_exec = pipeline.phases.get(phase) - if phase_exec is None: - raise ValueError(f"Refine phase not found in pipeline {pipeline_id}") - - agent_roles: list[AgentRole] = [] - for agent in phase_exec.agents: - if hasattr(agent, "role"): - role = agent.role if isinstance(agent.role, AgentRole) else AgentRole(agent.role) - agent_roles.append(role) - if not agent_roles: - from egg_contracts.agent_roles import get_roles_for_phase as _grfp - - for r in _grfp( - phase, - include_reviewers=True, - repo=pipeline.repo, - has_contract=getattr(pipeline, "has_contract", True), - ): - try: - agent_roles.append(AgentRole(r.value)) - except ValueError: - continue - - old_container_ids = [c.container_id for c in phase_exec.containers] - phase_exec.containers = [] - phase_exec.agents = [] - phase_exec.review_cycles = 0 - phase_exec.hitl_review_cycles = 0 - phase_exec.status = PipelineStatus.PENDING - phase_exec.started_at = None - phase_exec.work_started_at = None - phase_exec.completed_at = None - phase_exec.error = None - phase_exec.cycle_timings = [] - pipeline.status = PipelineStatus.RUNNING - pipeline.error = None - pipeline.run_epoch = datetime.now(UTC) - store.update_pipeline(pipeline_id, pipeline.model_dump(mode="json")) - - # --- Outside the lock: slow, idempotent, best-effort teardown --- - for container_id in old_container_ids: - try: - spawner.stop_agent_container(container_id, cleanup_session=True) - except Exception as e: # noqa: BLE001 - logger.warning( - "Failed to stop container during refine redirect restart", - container_id=container_id[:12] if container_id else "?", - error=str(e), - ) - try: - spawner.remove_agent_container(container_id, force=True, cleanup_session=False) - except Exception as e: # noqa: BLE001 - logger.warning( - "Failed to remove container during refine redirect restart", - container_id=container_id[:12] if container_id else "?", - error=str(e), - ) - - restart_role_values = {role.value for role in agent_roles} - try: - all_worktrees = agent_salvage.enumerate_agent_worktrees(pipeline_id, validate_git=False) - except (OSError, ImportError, RuntimeError) as e: - logger.warning( - "Failed to enumerate per-agent worktrees during refine redirect restart", - pipeline_id=pipeline_id, - error=str(e), - ) - all_worktrees = [] - worktrees_to_delete = [wt for wt in all_worktrees if wt.agent_role in restart_role_values] - if worktrees_to_delete: - try: - agent_salvage.auto_salvage_pipeline( - spawner.gateway, - pipeline_id, - worktree_filter={wt.worktree_id for wt in worktrees_to_delete}, - mode=gateway_mode, - base_branch=pipeline.base_branch, - ) - except Exception as e: # noqa: BLE001 - logger.warning( - "Auto-salvage failed during refine redirect restart; proceeding", - pipeline_id=pipeline_id, - error=str(e), - ) - for wt in worktrees_to_delete: - try: - spawner.gateway.delete_worktrees(container_id=wt.worktree_id, force=True) - except Exception as e: # noqa: BLE001 - logger.warning( - "Failed to delete per-agent worktree during refine redirect restart", - agent_worktree_id=wt.worktree_id, - pipeline_id=pipeline_id, - error=str(e), - ) - - try: - from peer_consensus import get_peer_consensus_tracker - - tracker = get_peer_consensus_tracker(pipeline_id) - if tracker: - tracker.clear() - except Exception as e: # noqa: BLE001 - logger.warning( - "Failed to clear peer consensus during refine redirect restart", - pipeline_id=pipeline_id, - error=str(e), - ) - - spawner.reset_restart_counts(pipeline_id) - try: - from health_monitor import get_health_monitor - - _hm = get_health_monitor() - if _hm is not None: - for role in agent_roles: - _hm.reset_agent(role.value) - except Exception as e: # noqa: BLE001 - logger.warning( - "Failed to reset health-monitor state during refine redirect restart", - pipeline_id=pipeline_id, - error=str(e), - ) - - _spawn_pipeline_run_thread(pipeline_id, store.repo_path, pipeline.run_epoch) - agents_to_restart = [role.value for role in agent_roles] - logger.info( - "Refine phase re-run for first-principles redirect", - pipeline_id=pipeline_id, - reason=reason, - agents_to_restart=agents_to_restart, - ) - return agents_to_restart - - -def _filter_salvage_worktrees( - worktrees: list[Any], - *, - agent_role: str | None, - slice_id: str | None, -) -> list[Any]: - """Filter ``enumerate_agent_worktrees`` output by role / slice scope. - - ``agent_role`` and ``slice_id`` may both be ``None`` (return all) or - set together to scope down to one specific worktree. ``agent_role`` - set with ``slice_id=None`` matches non-slice per-agent worktrees. - The pipeline-level worktree (``agent_role=None`` on the worktree) - is included only when the caller did not specify ``agent_role``. - """ - out = [] - for wt in worktrees: - if agent_role is not None and wt.agent_role != agent_role: - continue - if slice_id is not None and wt.slice_id != slice_id: - continue - out.append(wt) - return out - - -def _serialize_commit_report(report: Any) -> dict[str, Any]: - """Convert a ``WorktreeCommitReport`` to a JSON-safe dict.""" - return { - "worktree_id": report.worktree.worktree_id, - "agent_role": report.worktree.agent_role, - "slice_id": report.worktree.slice_id, - "local_branch": report.worktree.local_branch, - "assigned_branch": report.assigned_branch, - "anchor_ref": report.anchor_ref, - "commits": [ - { - "sha": c.sha, - "summary": c.summary, - "author": c.author, - "authored_at": c.authored_at, - "files_changed": c.files_changed, - } - for c in report.commits - ], - "error": report.error, - } - - -def _serialize_salvage_result(result: Any) -> dict[str, Any]: - """Convert a ``SalvageResult`` to a JSON-safe dict.""" - return { - "worktree_id": result.worktree_id, - "agent_role": result.agent_role, - "slice_id": result.slice_id, - "recovery_ref": result.recovery_ref, - "head_sha": result.head_sha, - "n_commits": result.n_commits, - "ok": result.ok, - "error": result.error, - } - - -@pipelines_bp.route("/<pipeline_id>/local-commits", methods=["GET"]) -def list_pipeline_local_commits(pipeline_id: str) -> tuple[Response, int]: - """List unpushed commits across this pipeline's per-agent worktrees. - - Inspects every per-agent worktree on disk - (``{pipeline_id}``, ``{pipeline_id}-{role}``, - ``{pipeline_id}-slice-{N}-{role}``) and reports the commits on its - local ``egg/{worktree_id}/work`` branch that are not reachable from - ``origin/<assigned_branch>`` (or ``origin/<base_branch>`` as a - fallback). Read-only — no fetch, no push. - - Query string (optional): - agent_role: Filter to a single agent role (e.g. ``coder``). - slice_id: Filter to a single slice scope (e.g. ``slice-2``). - - Response: - { - "success": true, - "data": { - "pipeline_id": "issue-2261-v9", - "worktrees": [ - { - "worktree_id": "issue-2261-v9-slice-2-coder", - "agent_role": "coder", - "slice_id": "slice-2", - "local_branch": "egg/issue-2261-v9-slice-2-coder/work", - "assigned_branch": "egg/issue-2261-v9/slice-2", - "anchor_ref": "refs/remotes/origin/egg/issue-2261-v9/slice-2", - "commits": [ - {"sha": "...", "summary": "...", "author": "...", - "authored_at": "...", "files_changed": 3} - ], - "error": null - } - ] - } - } - """ - repo_path = get_repo_path() - - try: - _store, pipeline = _resolve_pipeline(pipeline_id, repo_path) - except InvalidPipelineIdError: - return make_error_response(f"Invalid pipeline ID format: {pipeline_id}", status_code=400) - except PipelineNotFoundError: - return make_error_response(f"Pipeline {pipeline_id} not found", status_code=404) - - agent_role = request.args.get("agent_role") or None - if agent_role is not None: - try: - AgentRole(agent_role) - except ValueError: - return make_error_response(f"Invalid agent role: {agent_role}", status_code=400) - - raw_slice_id = request.args.get("slice_id") - try: - slice_id = extract_slice_id({"slice_id": raw_slice_id} if raw_slice_id is not None else {}) - except ValueError as e: - return make_error_response(str(e), status_code=400) - - worktrees = _filter_salvage_worktrees( - agent_salvage.enumerate_agent_worktrees(pipeline_id), - agent_role=agent_role, - slice_id=slice_id, - ) - reports = [ - agent_salvage.list_unpushed_commits(wt, base_branch=pipeline.base_branch) - for wt in worktrees - ] - - return make_success_response( - f"Listed local commits for pipeline {pipeline_id}", - data={ - "pipeline_id": pipeline_id, - "worktrees": [_serialize_commit_report(r) for r in reports], - }, - ) - - -@pipelines_bp.route("/<pipeline_id>/salvage", methods=["POST"]) -@require_lifecycle_secret -def salvage_pipeline_local_commits(pipeline_id: str) -> tuple[Response, int]: - """Push unpushed agent commits to recovery refs (#2429). - - For every matching per-agent worktree, push its HEAD to - ``egg/recovered/<pipeline_id>/<scope>/<short_sha>`` via the gateway's - launcher-auth path. Launcher auth bypasses the agent-targeted - branch-allowlist check so this works even when the agent's own - pushes were rejected for the wrong-branch reason this verb exists - to recover from. - - Query string (optional): - agent_role: Salvage only this role's worktree. - slice_id: Salvage only this slice scope. - - Response (always ``success: true`` when the request was well-formed - — per-worktree failures are reported in ``data.results``): - { - "success": true, - "data": { - "pipeline_id": "issue-2261-v9", - "results": [ - {"worktree_id": "...", "agent_role": "coder", "slice_id": "slice-2", - "recovery_ref": "egg/recovered/issue-2261-v9/slice-2-coder/9665f37a6...", - "head_sha": "9665f37a6...", "n_commits": 14, "ok": true, "error": null} - ] - } - } - """ - repo_path = get_repo_path() - - try: - _store, pipeline = _resolve_pipeline(pipeline_id, repo_path) - except InvalidPipelineIdError: - return make_error_response(f"Invalid pipeline ID format: {pipeline_id}", status_code=400) - except PipelineNotFoundError: - return make_error_response(f"Pipeline {pipeline_id} not found", status_code=404) - - agent_role = request.args.get("agent_role") or None - if agent_role is not None: - try: - AgentRole(agent_role) - except ValueError: - return make_error_response(f"Invalid agent role: {agent_role}", status_code=400) - - raw_slice_id = request.args.get("slice_id") - try: - slice_id = extract_slice_id({"slice_id": raw_slice_id} if raw_slice_id is not None else {}) - except ValueError as e: - return make_error_response(str(e), status_code=400) - - worktrees = _filter_salvage_worktrees( - agent_salvage.enumerate_agent_worktrees(pipeline_id), - agent_role=agent_role, - slice_id=slice_id, - ) - - gateway_mode, _vis = _compute_gateway_mode(pipeline) - gateway = get_gateway_client() - - results = [] - for wt in worktrees: - try: - result = agent_salvage.salvage_worktree( - gateway, - wt, - base_branch=pipeline.base_branch, - mode=gateway_mode, - ) - except Exception as e: # noqa: BLE001 — must always return a result row - logger.warning( - "Salvage raised unexpectedly", - pipeline_id=pipeline_id, - worktree_id=wt.worktree_id, - error=str(e), - ) - result = agent_salvage.SalvageResult( - worktree_id=wt.worktree_id, - agent_role=wt.agent_role, - slice_id=wt.slice_id, - recovery_ref=None, - head_sha=None, - n_commits=0, - ok=False, - error=str(e), - ) - results.append(result) - - return make_success_response( - f"Salvaged {sum(1 for r in results if r.ok and r.recovery_ref)} of " - f"{len(results)} per-agent worktrees for pipeline {pipeline_id}", - data={ - "pipeline_id": pipeline_id, - "results": [_serialize_salvage_result(r) for r in results], - }, - ) - - -@pipelines_bp.route("/<pipeline_id>/status", methods=["GET"]) -def get_pipeline_status(pipeline_id: str) -> tuple[Response, int]: - """ - Get pipeline status summary. - - URL params: - pipeline_id: Pipeline ID - - Response: - { - "success": true, - "data": { - "id": "issue-123", - "status": "running", - "current_phase": "implement", - "pending_decisions": 0 - } - } - """ - repo_path = get_repo_path() - - # Validate ``slice_id`` BEFORE the StateStore disk read in - # ``_resolve_pipeline`` — a malformed value is going to 400 anyway, - # and the read is wasted (#2764 review). ``InvalidPipelineIdError`` - # / ``PipelineNotFoundError`` from ``_resolve_pipeline`` still - # naturally take precedence on the happy path: this validator only - # fires when a slice scope is supplied at all. - raw_slice_id = request.args.get("slice_id") - try: - status_slice_id = extract_slice_id( - {"slice_id": raw_slice_id} if raw_slice_id is not None else {} - ) - except ValueError as e: - return make_error_response(str(e), status_code=400) - - try: - _store, pipeline = _resolve_pipeline(pipeline_id, repo_path) - - pending = pipeline.get_pending_decisions() - - data = { - "id": pipeline.id, - "status": pipeline.status.value, - "current_phase": pipeline.current_phase.value, - "pending_decisions": len(pending), - "updated_at": pipeline.updated_at.isoformat(), - } - - # Include first pending decision details so the collaborator - # doesn't need a second round-trip to fetch it - if pending: - d = pending[0] - data["pending_decision"] = { - "id": d.id, - "question": d.question, - "context": d.context, - "options": d.options, - "created_at": d.created_at.isoformat(), - } - - # Include PR info once the PR phase has created a PR (#1625) so - # monitoring clients don't need to scrape `gh pr list` by title. - pr_url, pr_number = _get_pr_info(pipeline) - if pr_url: - data["pr_url"] = pr_url - if pr_number is not None: - data["pr_number"] = pr_number - - # Include concurrent execution monitoring when enabled. The - # ``?slice_id=`` query param (validated above before the - # StateStore read) scopes the consensus block to one slice's - # BRC tracker in a slice-DAG implement phase (#2761); without - # it, only pipeline-level consensus is reported. - concurrent_data = _get_concurrent_status(pipeline, slice_id=status_slice_id) - if concurrent_data: - data["concurrent"] = concurrent_data - - # Surface the orchestrator-process-wide slice-admission state - # (#2241 gap 1) so operators can see when slices are queued - # behind the global cap rather than wedged. The shape is - # {cap, admitted, admitted_keys}; ``admitted_keys`` lists - # ``"<pipeline_id>/<slice_id>"`` so the operator can tell - # which slices currently hold the budget. - try: - try: - from orchestrator import global_slice_admit - except ImportError: - import global_slice_admit # type: ignore[no-redef] - - data["slice_admit"] = global_slice_admit.snapshot() - except Exception: # noqa: BLE001 - # Defensive: never let admit-state collection crash the - # status endpoint — the cap is advisory, not load-bearing - # for the pipeline's own progress. - pass - - # Issue #1962 TASK-1-2: include the overseer-relevant config - # subset in the status payload so the sandbox-side overseer - # monitor can read PipelineConfig values (advisor model, - # threshold knobs, host-detection flag) without a separate - # endpoint. Only the new + load-bearing knobs are exposed - # here to keep the response compact; full config is available - # via the dedicated config endpoint. - try: - cfg = getattr(pipeline, "config", None) - if cfg is not None: - data["config"] = { - "overseer_advisor_model": getattr(cfg, "overseer_advisor_model", None), - "overseer_advisor_recent_log_bytes_cap": getattr( - cfg, "overseer_advisor_recent_log_bytes_cap", None - ), - "overseer_auto_file_issues_mode": getattr( - cfg, "overseer_auto_file_issues_mode", None - ), - "overseer_owns_host_detection": getattr( - cfg, "overseer_owns_host_detection", False - ), - "overseer_stuck_phase_transition_seconds": getattr( - cfg, "overseer_stuck_phase_transition_seconds", 180 - ), - "overseer_agent_stall_seconds": getattr( - cfg, "overseer_agent_stall_seconds", 180 - ), - "overseer_silent_agent_threshold_seconds": getattr( - cfg, "overseer_silent_agent_threshold_seconds", 600 - ), - "overseer_long_running_phase_seconds": getattr( - cfg, "overseer_long_running_phase_seconds", 3600 - ), - "overseer_nack_unresolved_seconds": getattr( - cfg, "overseer_nack_unresolved_seconds", 180 - ), - } - except AttributeError, TypeError: - # Defensive: never let a config-shape change crash the - # status endpoint. - pass - - return make_success_response("Status retrieved", data=data) - - except InvalidPipelineIdError: - return make_error_response( - f"Invalid pipeline ID format: {pipeline_id}", - status_code=400, - ) - except PipelineNotFoundError: - return make_error_response( - f"Pipeline {pipeline_id} not found", - status_code=404, - ) - - -# ----------------------------------------------------------------- -# GET /api/v1/pipelines/<pipeline_id>/status/wait (issue #1932) -# -# Event-driven host-side wait primitive. Blocks up to ``wait`` -# seconds until one of the allowlisted EventBus events or message -# types fires, then returns a small envelope the MCP handler -# enriches with a full status snapshot. See -# docs/reference/agent-wait-patterns.md §7 for the end-to-end -# protocol. -# ----------------------------------------------------------------- -@pipelines_bp.route("/<pipeline_id>/status/wait", methods=["GET"]) -def wait_pipeline_status(pipeline_id: str) -> tuple[Response, int]: - """Block up to ``wait`` seconds on the next pipeline-relevant event. - - Query params: - wait: seconds to block, default 25, clamped to - ``GET_STATUS_MAX_WAIT`` (25) so the caller stays - safely inside the Claude Code MCP tool-call timeout. - since: opaque cursor ``msg:<id>|evt:<seq>`` from a prior - response. An empty / missing cursor snaps to the tip - on both sources (first-call semantics). Returns 400 - if the cursor is syntactically malformed. - - Responses: - 200 — either a ``changed=true`` envelope (event or message - fired before the timeout) or a ``changed=false, - no_change=true`` envelope (timeout elapsed with no - pipeline-relevant event). Always carries ``cursor`` - so the caller can seed the next request. - 400 — malformed cursor or malformed ``wait``. - 404 — pipeline does not exist. - - Implementation: - * ``queue.Queue(maxsize=16)`` coordinates the two sources: - a wildcard EventBus handler (synchronous) and a daemon - thread running ``message_store.get_messages(wait=...)``. - * First source wins. On return the EventBus handler is - unsubscribed in ``finally``; the daemon thread is left - lame-duck for up to ``wait`` seconds (accepted per plan - risk R14 — bounded, non-blocking on shutdown). - * ``egg_inflight_host_waits`` gauge is incremented at entry - and decremented on return. - - Args: - pipeline_id: Pipeline ID from the URL. - """ - # Validate pipeline exists before doing any expensive setup. - repo_path = get_repo_path() - try: - _store, pipeline = _resolve_pipeline(pipeline_id, repo_path) - except InvalidPipelineIdError: - return make_error_response( - f"Invalid pipeline ID format: {pipeline_id}", - status_code=400, - ) - except PipelineNotFoundError: - return make_error_response( - f"Pipeline {pipeline_id} not found", - status_code=404, - ) - - # Parse + clamp ``wait``. ``GET_STATUS_MAX_WAIT`` lives in - # ``mcp_server`` — importing it here keeps the cap in one place. - try: - from mcp_server import GET_STATUS_MAX_WAIT - except ImportError: - try: - from ..mcp_server import GET_STATUS_MAX_WAIT # type: ignore[no-redef] - except ImportError: - GET_STATUS_MAX_WAIT = 25 # conservative fallback - try: - requested_wait = int(request.args.get("wait", str(GET_STATUS_MAX_WAIT))) - except ValueError, TypeError: - return make_error_response( - "Invalid 'wait' query parameter: must be an integer", - status_code=400, - ) - timeout = min(max(requested_wait, 1), GET_STATUS_MAX_WAIT) - - # Parse the opaque compound cursor. ``ok=False`` is the only - # 400 path here — unknown cursors on either source are tolerated - # and degrade to "snap to tip". - ok, msg_since_id, event_since_seq = _parse_status_wait_cursor(request.args.get("since")) - if not ok: - return make_error_response( - "Invalid 'since' cursor — expected 'msg:<id>|evt:<seq>' (either half may be empty).", - status_code=400, - ) - - # Lazy imports keep the route cheap to load at module import and - # match the pattern used elsewhere in this file. We compare events - # against ``_STATUS_WAIT_EVENT_TYPES`` by the string value of - # ``event.event_type`` — the ``EventType`` class itself is not - # needed here. - try: - from events import get_event_bus - except ImportError: # pragma: no cover - try: - from ..events import get_event_bus # type: ignore[no-redef] - except ImportError: - return make_error_response("Event bus not available", status_code=500) - - try: - from routes.messages import _apply_delphi_filter as _delphi - except ImportError: # pragma: no cover - try: - from .messages import _apply_delphi_filter as _delphi # type: ignore[no-redef] - except ImportError: - _delphi = None # type: ignore[assignment] - - import queue as _queue - - event_bus = get_event_bus() - - # Synchronous up-front cursor-staleness probe (issue #2464). The route - # used to silently keep re-emitting ``msg_since_id`` whenever the store - # tip was empty (post-phase-clear), so a polling client kept feeding - # the dead cursor back forever. Probe once at entry with ``wait=0`` so - # we can both stop re-emitting it and surface ``since_id_stale: True`` - # in the envelope, letting consumers (sandbox CLI cursor file, agent - # wait_loop) drop the stale cursor and re-snap to tip. Done before - # the terminal short-circuit below so a request that arrives after - # both a phase clear and pipeline completion still sees the flag. - since_id_stale = False - if msg_since_id is not None: - try: - store_fn = _get_message_store() - store = store_fn() - _msgs, meta = store.get_messages_with_meta( - pipeline_id, - since_id=msg_since_id, - limit=1, - wait=0, - # Suppress the "since_id not found in store" warning on - # this probe so a single ``/status/wait`` request that - # hits a stale cursor doesn't double-log: the long-poll - # daemon below makes its own ``get_messages`` call with - # the same cursor and emits the warning once. Pre-PR - # cadence was one warning per request; we preserve that. - _suppress_stale_warning=True, - ) - since_id_stale = meta.since_id_stale - except Exception as exc: # pragma: no cover - logger.debug( - "status_wait staleness probe error", - pipeline_id=pipeline_id, - error=str(exc), - ) - - # Late-subscriber short-circuit (issue #2378): if the pipeline is - # already terminal at request time, the relevant ``pipeline.*`` - # event was emitted before this call could subscribe — and the - # snap-to-tip below would cement that miss. Synthesize a Path-A - # envelope so callers don't loop until the 1-hour cap. This covers - # the common path where ``mark FAILED`` succeeds; the synthetic - # emit at ``_run_pipeline``'s mark-FAILED-failed branch covers the - # rarer case where the FAILED-mark itself raises. - _TERMINAL_EVENT_TYPES = { - PipelineStatus.COMPLETE: "pipeline.completed", - PipelineStatus.FAILED: "pipeline.failed", - PipelineStatus.CANCELLED: "pipeline.cancelled", - } - if pipeline.status in _TERMINAL_EVENT_TYPES: - # Issue #2464: don't fall back to ``msg_since_id`` when the tip - # is empty — that's exactly the post-clear state that perpetuates - # the dead cursor. - terminal_cursor = _build_status_wait_cursor( - _message_store_tip_id(pipeline_id), - event_bus.current_sequence(), - ) - terminal_envelope = _build_minimal_status_envelope(pipeline, terminal_cursor) - terminal_envelope.update( - { - "changed": True, - "trigger": "event", - "event_type": _TERMINAL_EVENT_TYPES[pipeline.status], - } - ) - if since_id_stale: - terminal_envelope["since_id_stale"] = True - return make_success_response("Pipeline already terminal", data=terminal_envelope) - - # Snap event_since_seq to the current tip on first call. This - # preserves the "events before the call are already seen" - # semantic and matches the message-bus ``from_tip`` behaviour - # used by ``/messages/wait`` (issue #1925). - if event_since_seq is None: - event_since_seq = event_bus.current_sequence() - - wake_q: _queue.Queue[tuple[str, Any]] = _queue.Queue(maxsize=16) - - def _on_event(event) -> None: # pragma: no cover - exercised via tests - if event.pipeline_id != pipeline_id: - return - if event.event_type.value not in _STATUS_WAIT_EVENT_TYPES: - return - if event.sequence <= event_since_seq: - return - try: - wake_q.put_nowait(("event", event)) - except _queue.Full: - logger.warning( - "status_wait event queue full; dropping event", - pipeline_id=pipeline_id, - event_type=event.event_type.value, - ) - - def _on_message_store_wake() -> None: # pragma: no cover - exercised via tests - try: - store_fn = _get_message_store() - store = store_fn() - messages = store.get_messages( - pipeline_id, - since_id=msg_since_id, - limit=100, - wait=timeout, - wait_for_types=list(_STATUS_WAIT_MESSAGE_TYPES), - from_tip=msg_since_id is None, - ) - except Exception as exc: # pragma: no cover - logger.debug( - "status_wait daemon error", - pipeline_id=pipeline_id, - error=str(exc), - ) - return - if not messages: - return - try: - wake_q.put_nowait(("message", messages)) - except _queue.Full: - logger.warning( - "status_wait message queue full; dropping message", - pipeline_id=pipeline_id, - ) - - _track_host_wait_start() - event_bus.subscribe(None, _on_event) - daemon: threading.Thread | None = None - try: - daemon = threading.Thread( - target=_on_message_store_wake, - name=f"status-wait-msg-{pipeline_id}", - daemon=True, - ) - daemon.start() - - try: - source, payload = wake_q.get(timeout=timeout) - except _queue.Empty: - source = None - payload = None - - # Re-load the pipeline once here so both paths share a - # consistent snapshot for the minimal envelope. - try: - _store2, fresh_pipeline = _resolve_pipeline(pipeline_id, repo_path) - except InvalidPipelineIdError, PipelineNotFoundError: - fresh_pipeline = pipeline - - if source == "event": - event = payload - # Issue #2464: never fall back to ``msg_since_id`` when the - # tip is empty. After a phase-boundary clear the caller's - # cursor is dead; re-emitting it here is what kept the - # ``since_id not found in store`` warning firing on every - # subsequent poll. ``since_id_stale: True`` in the envelope - # tells the consumer to drop its cached cursor. - tip_msg_id = _message_store_tip_id(pipeline_id) - cursor = _build_status_wait_cursor(tip_msg_id, event.sequence) - envelope = _build_minimal_status_envelope(fresh_pipeline, cursor) - envelope.update( - { - "changed": True, - "trigger": "event", - "event_type": event.event_type.value, - } - ) - if since_id_stale: - envelope["since_id_stale"] = True - return make_success_response("Event wake", data=envelope) - - if source == "message": - messages = payload - # Issue #2464: same as the event path — fall back to None - # when the message half is unavailable rather than re-emitting - # the stale ``msg_since_id``. - last_id = messages[-1].id if messages else None - # Delphi filter pass — currently a no-op for the host caller - # (role=None returns messages unchanged) but plumbed here so a - # future role parameter can enable reviewer-redaction (R13). - if _delphi is not None: - try: - messages = _delphi(pipeline_id, None, messages) - except Exception: # pragma: no cover - pass - tip_evt_seq = event_bus.current_sequence() - cursor = _build_status_wait_cursor(last_id, tip_evt_seq) - envelope = _build_minimal_status_envelope(fresh_pipeline, cursor) - envelope.update( - { - "changed": True, - "trigger": "message", - "messages": [m.to_dict() for m in messages], - } - ) - if since_id_stale: - envelope["since_id_stale"] = True - return make_success_response("Message wake", data=envelope) - - # Timeout path — minimal envelope only. - tip_msg_id = _message_store_tip_id(pipeline_id) - tip_evt_seq = event_bus.current_sequence() - cursor = _build_status_wait_cursor(tip_msg_id, tip_evt_seq) - envelope = _build_minimal_status_envelope(fresh_pipeline, cursor) - envelope.update({"changed": False, "no_change": True}) - if since_id_stale: - envelope["since_id_stale"] = True - return make_success_response("No change within wait window", data=envelope) - finally: - try: - event_bus.unsubscribe(None, _on_event) - except Exception: # pragma: no cover — unsubscribe is best-effort - pass - _track_host_wait_end() - # Daemon thread is deliberately left running — it exits on - # its own when ``message_store.get_messages`` returns or the - # timeout elapses (plan risk R14, accepted). - - -def _get_pr_info(pipeline: Pipeline) -> tuple[str | None, int | None]: - """Extract context-PR URL and number from the pipeline contract. - - Returns ``(pr_url, pr_number)`` or ``(None, None)`` when no PR has - been opened. Under #2777 the PR phase was removed and the context - PR opens up-front via ``_open_context_pr_at_implement_start`` which - persists ``context_pr_number`` to ``contract.pr.context_pr_number``; - we read that directly. ``pr_url`` is also persisted on the pipeline - record by ``_open_context_pr_at_implement_start`` for downstream - consumers (the JIRA reassess sweep at ``jira_reassess.py``). - """ - # ``Pipeline.pr_url`` / ``Pipeline.pr_number`` are populated by the - # up-front opener; they are the canonical surface for callers that - # used to read ``phases["pr"].artifacts["pr_url"]``. - pr_url = getattr(pipeline, "pr_url", None) - pr_number = getattr(pipeline, "pr_number", None) - if not pr_url: - return None, None - if pr_number is None: - match = re.search(r"/pull/(\d+)", pr_url) - pr_number = int(match.group(1)) if match else None - return pr_url, pr_number - - -def _consensus_block(consensus_state: dict) -> dict: - """Slim a tracker ``get_state()`` snapshot down to the status payload. - - Keeps the fields operators act on (per-role phases + confirmed - flags, the blocking set, and the unresolved-NACK details: who - NACKed whom, on which version, and why; #3481) and drops the - bulky ``approval_matrix`` / ``review_graph`` dumps. - - BRC trackers only emit dict-format agent entries (the legacy - AgentReadiness object came from the now-deleted ConsensusEvaluator, - cq-5 of #2777). - """ - return { - "agents": dict(consensus_state.get("agents", {})), - "is_complete": consensus_state.get("is_complete", False), - "blocking_agents": consensus_state.get("blocking_agents", []), - "has_unresolved_nacks": consensus_state.get("has_unresolved_nacks", False), - "unresolved_nacks": consensus_state.get("unresolved_nacks", []), - "protocol": consensus_state.get("protocol", "brc"), - } - - -def _get_concurrent_status(pipeline: Pipeline, slice_id: str | None = None) -> dict | None: - """Get concurrent execution monitoring data for a pipeline. - - Returns None if concurrent execution is not enabled for this pipeline. - Returns a dict with the following structure when concurrent mode is active:: - - { - "enabled": True, - "max_concurrent_agents": int, - "messages": {"total": int, "by_type": {"PROGRESS": int, ...}}, - "consensus": { - "agents": {"coder": {"state": "READY", ...}, ...}, - "is_complete": bool, - "blocking_agents": ["role", ...] # agents not yet READY - }, - "agents": [{"role": str, "status": str}, ...] # from phase execution - } - - Dependencies on other concurrent-mode modules (message_store, consensus) are - imported lazily and degrade gracefully to empty structures when unavailable. - - ``slice_id``: in a slice-DAG implement phase each slice runs its own - BRC consensus, keyed ``{pipeline_id}/{slice_id}``. The bare pipeline - id has no tracker, so a non-slice lookup reported a misleading - cross-slice reconstruction (#2761). Callers querying a per-slice - agent's consensus must pass that agent's ``slice_id``; the consensus - block then reflects exactly that slice's tracker. When omitted, only - pipeline-level (non-slice) consensus is reported in ``consensus``; - a slice-DAG pipeline queried without a slice yields no ``consensus`` - block rather than a fabricated one. Instead, live slice-scoped - trackers are surfaced under ``slice_consensus`` keyed by slice_id - (#3481), so operators still see each active round's real state. - """ - try: - from concurrent_executor import is_concurrent_execution - except ImportError: - from ..concurrent_executor import is_concurrent_execution # type: ignore[no-redef] - - current_phase = pipeline.current_phase.value if pipeline.current_phase else None - if not is_concurrent_execution(pipeline, phase=current_phase): - return None - - config = pipeline.config - result: dict = { - "enabled": True, - "max_concurrent_agents": getattr(config, "max_concurrent_agents", 6), - } - - # Message store provides aggregate counts of inter-agent messages by type. - # This module is implemented in phase-1 of the concurrent execution feature; - # ImportError is expected until that phase lands. - try: - from message_store import get_message_store - except ImportError: - try: - from ..message_store import get_message_store # type: ignore[no-redef] - except ImportError: - logger.debug("Message store not available for status") - get_message_store = None # type: ignore[assignment] - - if get_message_store is not None: - store = get_message_store() - msg_status = store.get_status(pipeline.id) - result["messages"] = { - "total": msg_status.get("total", 0), - "by_type": msg_status.get("by_type", {}), - } - else: - result["messages"] = {"total": 0, "by_type": {}} - - # Consensus evaluator tracks per-agent readiness states and determines - # whether all agents agree the phase is complete. Implemented in phase-3; - # blocking_agents lists roles that are not yet READY (WORKING or BLOCKED). - # BRC peer consensus (preferred) or legacy readiness-based - try: - try: - from peer_consensus import get_peer_consensus_tracker - except ImportError: - from ..peer_consensus import get_peer_consensus_tracker # type: ignore[no-redef] - - tracker = get_peer_consensus_tracker(pipeline.id, slice_id) - if not tracker: - # Attempt lazy reconstruction from message store for concurrent - # pipelines. ``slice_id`` scopes the replay to one slice's - # tracker; without it, only pipeline-level messages replay so a - # slice-DAG pipeline does not reconstruct cross-slice (#2761). - try: - from review_graph import get_review_graph_for_phase - - try: - from peer_consensus import reconstruct_tracker_from_messages - except ImportError: - from ..peer_consensus import ( - reconstruct_tracker_from_messages, # type: ignore[no-redef] - ) - - if is_concurrent_execution(pipeline, pipeline.current_phase): - graph = get_review_graph_for_phase( - pipeline.current_phase.value, repo=pipeline.repo - ) - tracker = reconstruct_tracker_from_messages( - pipeline.id, - graph, - slice_id=slice_id, - phase=pipeline.current_phase.value, - ) - except ImportError: - pass # Fall through to legacy evaluator - except Exception as e: - logger.warning( - "Tracker reconstruction failed", - error=str(e), - pipeline_id=pipeline.id, - slice_id=slice_id, - ) - if tracker: - consensus_state = tracker.get_state() - else: - # No BRC tracker available (slice-scoped query for a slice with - # no tracker yet, or a non-concurrent pipeline). The legacy - # ConsensusEvaluator was removed under cq-5 of #2777, so there - # is no fallback evaluator to consult. Report no consensus - # block; callers (e.g. the MCP get_consensus_status tool) fall - # back to message-based inference per the existing #1229 path. - consensus_state = None - except ImportError: - logger.debug("Peer consensus tracker not available for status") - consensus_state = None - - if consensus_state is not None: - result["consensus"] = _consensus_block(consensus_state) - else: - # Don't populate consensus with empty placeholder — callers (e.g. the - # MCP get_consensus_status tool) use truthiness to decide whether to - # fall back to message-based inference. An empty-but-truthy dict - # prevents that fallback from triggering (see issue #1229). - pass - - # Slice-id-less observability (#3481): in a slice-DAG implement phase - # the live trackers are keyed ``{pipeline_id}/{slice_id}``, so the - # pipeline-level lookup above finds nothing and an operator querying - # without a slice scope saw no structured consensus at all; the only - # way to see tracker state was tailing orchestrator pod logs. Surface - # each active slice's real snapshot, explicitly keyed by slice. This - # is NOT the #2761 cross-slice "soup" (that was mingling every - # slice's messages into ONE inferred tracker); the pipeline-level - # ``consensus`` block above still never reflects a slice tracker. - if slice_id is None: - try: - try: - from peer_consensus import get_slice_trackers - except ImportError: - from ..peer_consensus import get_slice_trackers # type: ignore[no-redef] - - slice_trackers = get_slice_trackers(pipeline.id) - except ImportError: - slice_trackers = {} - slice_consensus: dict[str, dict] = {} - for sid in sorted(slice_trackers): - try: - slice_consensus[sid] = _consensus_block(slice_trackers[sid].get_state()) - except Exception as e: # noqa: BLE001 - one bad slice must not hide the rest - logger.warning( - "Slice consensus snapshot failed", - pipeline_id=pipeline.id, - slice_id=sid, - error=str(e), - ) - if slice_consensus: - result["slice_consensus"] = slice_consensus - - # Agent lifecycle info from the phase execution record — shows which agents - # are spawned for the current phase and their container-level status. - # Includes ``container_id`` and server-computed ``elapsed_seconds`` so the - # sandboxed overseer can anchor stall-duration math on the live container's - # ``started_at`` rather than pre-restart message-bus events (issue #2084). - current_phase_name = pipeline.current_phase.value - phase_exec = pipeline.phases.get(current_phase_name) - agents_info: list[dict[str, Any]] = [] - if phase_exec and hasattr(phase_exec, "agents"): - now = datetime.now(UTC) - for agent in phase_exec.agents: - if hasattr(agent, "role"): - role = agent.role.value if hasattr(agent.role, "value") else str(agent.role) - else: - role = str(agent) - if hasattr(agent, "status"): - status = agent.status.value if hasattr(agent.status, "value") else "unknown" - else: - status = "unknown" - - entry: dict[str, Any] = {"role": role, "status": status} - - container_id = getattr(agent, "container_id", None) - if isinstance(container_id, str) and container_id: - entry["container_id"] = container_id - - started_at = getattr(agent, "started_at", None) - started_dt: datetime | None = None - if isinstance(started_at, datetime): - started_dt = started_at - elif isinstance(started_at, str) and started_at: - try: - started_dt = datetime.fromisoformat(started_at) - except ValueError: - started_dt = None - if started_dt is not None: - if started_dt.tzinfo is None: - started_dt = started_dt.replace(tzinfo=UTC) - entry["started_at"] = started_dt.isoformat() - entry["elapsed_seconds"] = max(0, int((now - started_dt).total_seconds())) - - agents_info.append(entry) - - # When the persisted phase-agent list is empty, backfill the - # running-pod view from live Job labels (#3230). Under the - # orchestrator-owned event loop (#3164) on-demand one-shot pods are - # never persisted into ``phase_exec.agents``, so without this the - # overseer's stall-duration math and the dashboard see "0 running - # agents" while role pods are demonstrably ``Running``. Empty stays - # empty when no pod is live, so legitimate between-spawn quiescence is - # not misreported as a cohort. - if not agents_info: - agents_info = _live_event_agents(pipeline.id, slice_id) - result["agents"] = agents_info - - return result - - -def _read_shared_criteria( - filename: str, - user_override: str | None = None, - repo_path: str | None = None, -) -> str | None: - """Read shared criteria from file, checking user override first. - - Search order: - 1. .egg/<user_override> in the repo (if user_override provided) - 2. shared/prompts/<filename> relative to source tree - 3. /app/prompts/<filename> (Docker container path) - - Returns the file content, or None if no file found (caller uses inline fallback). - """ - # Check user override first - if user_override and repo_path: - override_path = Path(repo_path) / ".egg" / user_override - if override_path.is_file() and override_path.stat().st_size > 0: - return override_path.read_text() - - # Try source tree path (development / tests) - source_path = Path(__file__).parent.parent.parent / "shared" / "prompts" / filename - if source_path.is_file(): - return source_path.read_text() - - # Try Docker container path (production) - docker_path = Path("/app/prompts") / filename - if docker_path.is_file(): - return docker_path.read_text() - - return None - - -def _get_agent_design_criteria() -> str: - """Return agent-mode design review criteria.""" - content = _read_shared_criteria("agent-design-criteria.md") - if content is not None: - return content - logger.warning("Shared agent-design-criteria.md not found, using inline fallback") - return ( - "Flag these **clear** anti-patterns:\n\n" - "1. **Excessive pre-fetching** — Baking large diffs (10KB+) or full file contents " - "into prompts instead of letting the agent fetch what it needs\n" - "2. **Structured output for humans** — Requiring JSON when output goes directly " - "to humans rather than machines\n" - "3. **Post-processing pipelines** — Scripts that parse agent output to take actions " - "the agent could take directly\n" - "4. **Rigid procedures** — Micromanaging step-by-step procedures when objectives " - "would suffice\n" - "5. **Prompt-level security** — Using instructions for constraints that should be " - "sandbox-enforced\n" - "6. **Direct LLM API calls outside sandbox** — Calling the Anthropic API from " - "orchestrator, gateway, or shared code instead of delegating to sandbox containers\n" - "7. **Direct API calls bypassing the Agent SDK** — Using raw HTTP calls to the " - "Anthropic API instead of run_agent() (in-sandbox) or build_agent_command() " - "(orchestrator-spawned containers). Unlike item 6 (scoped to infra code), " - "this applies everywhere including sandbox code.\n" - "8. **Hardcoded model identifiers** — Using full model IDs (date-pinned or " - "version-pinned) instead of short aliases (sonnet, opus, haiku)\n" - ) - - -def _get_code_review_criteria(repo_path: str | None = None) -> str: - """Return code review criteria.""" - content = _read_shared_criteria( - "code-review-criteria.md", - user_override="review-rules.md", - repo_path=repo_path, - ) - if content is not None: - return content - logger.warning("Shared code-review-criteria.md not found, using inline fallback") - return ( - "### Security (highest priority)\n" - "- Injection vulnerabilities (SQL, command, XSS, LDAP, path traversal)\n" - "- Authentication/authorization flaws\n" - "- Credential exposure, hardcoded secrets\n" - "- SSRF, open redirects, unsafe deserialization\n\n" - "### Correctness\n" - "- Logic errors, off-by-one, boundary conditions\n" - "- Race conditions, deadlocks, concurrency bugs\n" - "- Null/undefined handling, missing error paths\n" - "- Resource leaks (connections, file handles, memory)\n" - "- End-to-end feature functionality: verify new features work in their " - "real execution environment\n\n" - "### Robustness\n" - "- Missing input validation at trust boundaries\n" - "- Unhandled exceptions that could crash the system\n" - "- Missing retry logic for transient failures\n" - "- Inadequate timeouts for external calls\n\n" - "### Design\n" - "- Violations of existing codebase patterns\n" - "- Breaking changes to public interfaces\n" - "- Tight coupling that will hinder future changes\n\n" - "### Severity Classification\n\n" - "**Blocking** (request changes):\n" - "- Security vulnerabilities\n" - "- Non-functional features — the feature's core purpose does not work " - "end-to-end\n" - "- Logic errors that produce incorrect results\n" - "- Breaking changes to existing functionality\n" - "- Resource leaks or crashes\n" - "- Pre-existing broken or inconsistent behavior in code the PR " - "modifies\n\n" - "**Non-blocking** (suggestions):\n" - "- Code quality improvements (naming, structure, duplication)\n" - "- Defense-in-depth additions\n" - "- Missing edge case handling that doesn't affect the core feature\n" - "- Documentation gaps\n" - "- Style or convention deviations not caught by linters\n\n" - "**Do not dismiss issues as 'not a regression'**: If a PR modifies " - "code that has existing broken or inconsistent behavior, the issue is " - "blocking even if the PR didn't introduce it. A PR that adds a new " - "code path through already-inconsistent logic makes the inconsistency " - "worse.\n\n" - "**Beware of false analogies**: When comparing new code to existing " - "patterns, verify the analogy holds at the execution-model level. " - "Two features may look structurally similar in config but have " - "completely different execution paths. If the existing pattern works " - "via mechanism A but the new code relies on mechanism B that doesn't " - "exist, the comparison is invalid — classify based on actual " - "functionality, not superficial similarity.\n\n" - "### Skip\n\n" - "- Style issues handled by linters (formatting, import order)\n" - "- Type annotation completeness (type checkers handle this)\n" - "- Auto-generated files (migrations, lock files)\n" - "- `.egg-state/` pipeline artifacts (contracts, drafts, BRC history " - "— managed by the orchestrator)\n" - ) - - -def _get_contract_review_criteria(repo_path: str | None = None) -> str: - """Return contract verification criteria.""" - content = _read_shared_criteria( - "contract-review-criteria.md", - user_override="contract-rules.md", - repo_path=repo_path, - ) - if content is not None: - return content - logger.warning("Shared contract-review-criteria.md not found, using inline fallback") - return ( - "### Task Verification\n" - "For each task in the contract, verify:\n" - "1. The described functionality is present in the code\n" - "2. The acceptance criteria for the task is satisfied\n" - "3. If a commit is linked, verify it relates to the task\n" - "4. Where applicable, tests cover the new functionality\n\n" - "### Phase Consistency\n" - "- All tasks in completed phases are actually implemented\n" - "- Phase status matches task completion state\n" - "- No orphaned code exists that isn't covered by any task\n\n" - "### Acceptance Criteria Verification\n" - "For each acceptance criterion:\n" - "1. Examine the implementation to verify it meets the criterion\n" - "2. Note any gaps in your review\n\n" - "### Contract Integrity\n" - "- No implementation changes violate previously verified criteria\n" - "- New changes don't break existing contract compliance\n" - "- All required files listed in tasks are present\n" - ) - - -def _get_refine_review_criteria() -> str: - """Return review criteria for the dedicated refine reviewer.""" - return ( - "### 1. Problem Understanding\n" - "- Does the analysis correctly identify the core problem or feature request?\n" - "- Is the current behavior (if applicable) accurately described?\n" - "- Are the goals and desired outcomes clear?\n\n" - "### 2. Research Quality\n" - "- Has the agent explored the relevant parts of the codebase?\n" - "- Are existing patterns and conventions identified?\n" - "- Is the technical context accurate and thorough?\n\n" - "### 3. Options Analysis\n" - "- Are the proposed options meaningfully different?\n" - "- Are trade-offs clearly articulated for each option?\n" - "- Is the reasoning logical and well-founded?\n\n" - "### 4. Constraints and Dependencies\n" - "- Are technical constraints identified (performance, compatibility, etc.)?\n" - "- Are dependencies on other code or systems noted?\n" - "- Are potential risks or complications surfaced?\n\n" - "### 5. Open Questions\n" - "- Are open questions specific enough for a human to answer?\n" - "- Do questions address genuine ambiguities?\n" - "- Are questions actionable?\n" - "- **Does each question require a human, or could the planner decide it?** " - "NACK questions that ask about work decomposition / slice-DAG shape / " - "PR packaging — those belong to the plan phase's HITL gate, not the " - "refine gate. NACK questions about implementation strategy " - "(API shape, migration approach, fallback design, detector design) " - "unless the answer is a fact only the operator knows (product intent, " - "scope boundary, external commitment, user-visible behavior). Good " - "refine questions are about *what the problem is* and *what's in/out " - "of scope*; the planner handles *how to build it*.\n\n" - "### 6. Recommendation Quality\n" - "- Is there a clear recommended approach?\n" - "- Is the recommendation justified with specific reasons?\n" - "- Does the recommendation align with the analysis findings?\n\n" - "### 7. HITL Decision Registration & Un-surfaced Decisions (#3390)\n" - "- Run `egg-contract show` and verify a contract decision or feedback " - "item exists for every open question in the analysis, and that each " - "decision-bearing section cites its `cq-N` (the `--format markdown` " - "output of `egg-contract add-decision` embeds it). Open questions as " - "bare prose with no registered `cq-N` ⇒ **NACK** — the producer must " - "register each via `egg-contract add-decision` / " - "`egg-contract add-feedback` and re-propose. (Deterministic " - "propose-time checks already validate the producer's *attested* ids; " - "your job is the judgment half the validators cannot do.)\n" - "- **Un-surfaced decisions — NACK.** Read the draft for choices it " - "quietly *commits to* that should be the operator's call — e.g. " - '"we will drop the legacy filter", a scope narrowing/widening, a ' - "user-visible behavior change, abandoning a stated requirement — " - "with no registered `cq-N` backing the choice. Consensus must not " - "close on a draft that bakes in a human-grade decision outside the " - "HITL channel: the producer either registers the decision (the gate " - "then surfaces it) or rewrites the draft to remove the unilateral " - "commitment.\n" - "- **Calibration — do not over-NACK.** An implementation choice the " - "planner can make from the analysis (API shape, migration approach, " - "fallback design, detector shape) is NOT a human-grade decision; do " - "not force registration of those. The bar is the same as §5: answers " - "only the operator owns (product intent, scope boundaries, external " - "commitments, user-visible behavior).\n" - "- If the ledger is deliberately empty (the producer attested " - "`no_decisions_rationale`), verify the rationale holds: requirements " - "genuinely unambiguous, no assumptions made silently. NACK if you " - "find a hidden operator-grade choice.\n" - "- **Task-named decisions — NACK an explicit-none ledger (#3462).** " - "If the task description names decisions as the operator's to make " - "(or directs that decisions be surfaced as HITL questions), each " - "must have a registered `cq-N` — even when the draft argues prior " - 'context already resolves it. "Already resolved" is a recommended ' - "disposition to register (recommended option citing the resolving " - "context), not a reason to skip; a `no_decisions_rationale` " - "attestation on such a task is a **NACK** regardless of how " - "defensible the rationale reads.\n\n" - + _human_companion_review_criteria( - companion="`*-analysis-human.md`", - parent="the refine analysis", - producer="refiner", - ) - ) - - -def _human_companion_review_criteria(*, companion: str, parent: str, producer: str) -> str: - """Forcing checklist for verifying the simplifier's human companion. - - Shared by the refine and plan reviewer rubrics so the companion is - judged at VERDICT time (#3381), not merely mentioned in the reviewer's - "while waiting" preparation text. The simplifier is a producer-only role - whose companion is gated CRITICAL by this reviewer; the companion is the - only artifact with no automated content check, so this reviewer is the - sole gate on its format. Walk every item before ACKing the **simplifier** - (this section governs the simplifier's proposal, never the {producer}'s). - """ - return ( - f"### Human-Focused Companion ({companion} — the simplifier's " - "proposal)\n" - f"The simplifier produces {companion}, a plain-language companion to " - f"{parent} for a **broad audience — engineers, PMs, and managers**. " - "It is gated CRITICAL by you and has no automated content check, so " - "you are the only gate on its format. **You must walk this checklist " - "and answer every item before you ACK the simplifier** (this is a " - f"separate verdict from your review of the {producer}; NACK the " - "**simplifier**, never the " - f"{producer}, for companion defects):\n" - f"1. **Is it a summary, not a review?** Open {companion} and the full " - f"{parent} side-by-side. NACK if the companion reads as a " - "review/critique of the draft rather than a summary of it — any " - 'verdict/scoring framing ("verdict", "what I verified", "I ' - 'affirm", "sound", ACK/NACK language), any directives aimed at a ' - 'later phase ("the plan should commit to", "don\'t let the plan ' - 'inflate", "guardrails", "anti-pattern to reject"), or any ' - "constraint lists. The companion explains the change to a human; it " - "does not judge it.\n" - "2. **Is it free of implementation minutiae?** NACK if it contains " - "`file:line` references (e.g. `foo.go::Bar` / `L209`), function / " - "struct / field / type names or other code identifiers, or per-field " - "enumerations. It should describe behaviour and user-visible impact, " - "not the code.\n" - "3. **Is it materially lighter than the parent?** NACK if it is a " - "near-copy or as long/dense as the full draft. It must be " - "substantially shorter and more digestible — plain prose and short " - "lists.\n" - "4. **Is it readable by a non-engineer?** NACK if a PM or manager " - "could not follow *what is changing and why it matters*, or if it " - "leaks egg-internal jargon (BRC, consensus, slice-DAG, contract, " - "phase, agent-role terms).\n" - f"5. **Is it faithful?** NACK if it misrepresents {parent}, omits a " - "material point, or introduces new scope/claims.\n" - "A missing or empty companion is a NACK — it is mandatory.\n" - ) - - -def _get_first_principles_review_criteria() -> str: - """Return review criteria for the adversarial first-principles reviewer. - - The escalation instructions interpolate the accept-path's sentinel option - labels from ``routes.decisions`` so the labels the agent writes (here) and - the labels the resolve hook matches stay a single source of truth — they - cannot drift. Lazy import avoids a module-load cycle. - """ - from routes.decisions import ( - FIRST_PRINCIPLES_ADOPT_OPTION, - FIRST_PRINCIPLES_CANCEL_OPTION, - FIRST_PRINCIPLES_PROCEED_OPTION, - ) - - return ( - "You are the **first-principles reviewer**. Your subject is the " - "pipeline's **seed** — the operator's task statement (run " - "`egg-contract show` and read `task_description`, plus the linked " - "issue) — and the **direction** the refiner's analysis is taking. You " - "judge whether the *premise is sound and the direction is " - "appropriate*, NOT the quality of the analysis — that is " - "`reviewer_refine`'s job, so do not duplicate it.\n\n" - "### 1. Interrogate the premise\n" - "- Is the stated problem real, and is solving it worth the work?\n" - "- Is the premise contradicted by what's actually in the codebase — " - "the thing it proposes to build already exists, or the problem is " - "already handled?\n" - "- Will the stated direction actually achieve the stated goal, or does " - "it solve something adjacent?\n\n" - "### 2. Surface significant redirects (where warranted)\n" - "Raise a redirect only when you can name a concrete, evidence-backed " - "alternative — never a vague 'have you considered'. Valid redirects:\n" - "- A **materially simpler path** that achieves the same goal.\n" - "- A **fundamentally different approach** that is better on the " - "merits.\n" - "- A **scope change** — widen it if the seed under-reaches the real " - "goal, narrow it if it over-reaches.\n" - "- **Don't build it** — the work is unnecessary, already solved, or " - "solves a non-problem.\n" - "Back each redirect with evidence: a codebase fact (`file:line`), the " - "seed's own stated goal, or a specific contradiction. Raising more " - "than one is fine — it is acceptable to be relatively noisy — but " - "consolidate related concerns and hold every one to the " - "concrete-and-evidenced bar.\n\n" - "### 3. What NOT to raise (stay in your lane)\n" - "- Analysis-quality issues (research depth, option trade-offs, " - "completeness) — `reviewer_refine` owns those.\n" - "- Work decomposition, slice-DAG shape, PR packaging, or " - "implementation strategy (API shape, migration approach) — those " - "belong to the plan phase and the planner.\n" - "- Taste, stylistic preference, or 'did you consider X' with no " - "concrete better alternative.\n" - "If the premise and direction are sound, say so briefly and ACK — a " - "clean pass is a common and correct outcome. Do not manufacture an " - "objection to look diligent.\n\n" - "### 4. How to act — escalate, never NACK\n" - "- **Never NACK the refiner on first-principles grounds.** A NACK only " - "re-runs the refiner, which cannot change the operator-owned seed; " - "premise and direction are the operator's call, not the refiner's to " - "fix.\n" - "- When you have a redirect, **file one phase-scoped HITL decision** " - "via the `mcp__sdlc__register_open_question` tool so the operator can " - "act on it with one click (the **accept-path**). Pass these args:\n" - ' - `phase`: `"refine"`.\n' - " - `question`: state the concern, then the concrete redirect and " - "why (operator-facing prose).\n" - " - `options`: these EXACT labels, in this order — do NOT paraphrase, " - "the orchestrator matches them verbatim to drive the accept-path: " - f'`["{FIRST_PRINCIPLES_ADOPT_OPTION}", ' - f'"{FIRST_PRINCIPLES_PROCEED_OPTION}", ' - f'"{FIRST_PRINCIPLES_CANCEL_OPTION}"]`.\n' - " - `redirect_seed`: the FULL rewritten seed — the complete " - "`task_description` as it should read if the operator adopts your " - "redirect (not a diff, not just the objection). This rides the same " - "RPC that files the decision, so the orchestrator can read it back " - "directly; do NOT write it to a free-standing file (a reviewer " - "worktree has no path to carry one to the orchestrator).\n" - " On the operator's choice the orchestrator will: **adopt** → rewrite " - "the seed to your `redirect_seed` and re-run the refine phase against " - "it; **proceed** → leave the direction unchanged; **don't build** → " - "cancel the pipeline. If you have only an objection with no concrete " - "alternative direction, you do not have a redirect — do not file the " - "decision (omit `redirect_seed`).\n" - "- Then **ACK the refiner**: your first-principles pass is done and " - "any concerns are filed for the operator. Your ACK does not endorse " - "the direction — it records that you reviewed it; the open decision " - "independently holds the refine→plan gate until the operator resolves " - "it.\n" - ) - - -def _get_plan_review_criteria() -> str: - """Return review criteria for the dedicated plan reviewer.""" - return ( - "### 1. Alignment with Analysis\n" - "- Does the plan implement the recommended approach from the analysis?\n" - "- If the plan deviates from the analysis, is the reason explained?\n" - "- Are all requirements from the analysis addressed?\n\n" - "### 2. Task Breakdown\n" - "- Are tasks discrete, actionable, and properly scoped?\n" - "- Is each task small enough to implement in a single pass?\n" - "- Are task boundaries clear (no overlapping responsibilities)?\n\n" - "### 3. Acceptance Criteria\n" - "- Does each task have clear, testable acceptance criteria?\n" - "- Are criteria specific enough to verify completion?\n" - "- Do criteria cover both happy path and edge cases?\n\n" - "### 4. Dependency Ordering\n" - "- Are task dependencies correctly identified?\n" - "- Is the ordering logical (foundations before features)?\n" - "- Are there opportunities for parallelism that are missed?\n\n" - "### 5. Risk Assessment\n" - "- Are technical risks identified (security, performance, compatibility)?\n" - "- Are mitigation strategies concrete and actionable?\n" - "- Is the rollback plan realistic?\n\n" - "### 6. Test Strategy\n" - "- Is the test strategy appropriate for the scope of changes?\n" - "- Are both unit and integration tests considered?\n" - "- Are test scenarios aligned with acceptance criteria?\n\n" - "### 7. Completeness\n" - "- Does the plan cover all aspects of the original request?\n" - "- Are documentation updates included where needed?\n" - "- Are there any obvious gaps or missing tasks?\n\n" - "### 8. Task Role ↔ Files Alignment (deterministic, see #2527)\n" - "- Task role↔files alignment is enforced **orchestrator-side** at " - "`CONSENSUS_PROPOSE`: a planner proposal whose task `role:` " - "assignments cannot push their `files:` (per " - "`shared/egg_restrictions/patterns.py`, the same blocklist the " - "gateway uses) is rejected with HTTP 400 before the proposal " - "reaches you. By the time you act on a `CONSENSUS_PROPOSE`, " - "structural role↔files alignment is therefore already validated — " - "no manual check is required for this dimension.\n" - "- If you want belt-and-suspenders verification, you can run the " - "validator yourself against the proposed plan: " - '`python3 -c "from egg_contracts.plan_parser import parse_plan_file, ' - "validate_task_role_alignment as v; r = parse_plan_file('<plan-path>'); " - "print('\\n'.join(v(r.to_contract_slices())))\"`. " - "Errors here would predict a push-time `403 " - "restricted_path_modified` — NACK the planner and quote the " - "structured errors verbatim if any surface.\n\n" - "### 9. Primitive-Existence Audit (hard NACK, see #2594)\n" - "Plans are cheap to NACK at this phase and expensive to NACK " - "at implement-phase (8+ pod spawns per slice, ~60–90 min " - "wall clock per implement cycle). For #2474, a single " - "`grep -rn ScriptedProvider sandbox/ k8s/ orchestrator/` " - "returning zero hits would have prevented ~10.7 h of " - "compute. Do that grep **now**.\n\n" - "For every primitive the plan names — class, function, HTTP " - "route, env var, ConfigMap key, test fixture, CLI flag, " - "decorator — produce a small evidence table in your review " - "document. Example shape:\n\n" - "| primitive | kind | grep | result |\n" - "|-----------|------|------|--------|\n" - "| `ScriptedProvider` | class | `grep -rn 'class ScriptedProvider' sandbox/ k8s/ orchestrator/` | 0 hits → NACK |\n" - "| `orchestrator_url` fixture | fixture | `grep -rn 'def orchestrator_url' integration_tests/` | `integration_tests/local_pipeline/conftest.py:NN` — sibling, not parent (see §10) |\n\n" - "Prescribed greps by kind:\n" - "- **class / function**: `grep -rn '<NAME>' <relevant dirs>` " - "finds at least one definition site.\n" - "- **HTTP route**: blueprint registers the path + method the " - "plan uses (search `orchestrator/routes/` and `gateway/`).\n" - "- **env var / ConfigMap key**: a consumer the plan assumes " - "actually reads it.\n" - "- **test fixture**: defined in a conftest **reachable from " - "the test's directory** (parent vs sibling matters — see §10).\n" - "- **CLI flag**: parser registers it.\n\n" - "**NACK rule**: any named primitive whose grep returns zero " - "hits in the directories the plan implies is a hard NACK. " - "Quote the failed command verbatim in your verdict so the " - "planner can re-draft. If the primitive exists but in a " - "different form than the plan assumes (different module, " - "different signature, different scope — e.g. unit-test-only " - "vs deployed-pod), NACK and quote the actual `file:line`.\n\n" - "**Exception — `(NEW — task TASK-X-Y)` annotations.** Plans " - "introduce new primitives by design; the producer prompt " - "tells the planner to mark such primitives " - "`(NEW — task TASK-X-Y)` so the audit doesn't false-NACK " - "the very task that creates them. When you see this " - "annotation: **do not NACK on missing-grep evidence**. " - "Instead verify that the referenced task's acceptance " - "criteria genuinely create the primitive in the form the " - "plan uses (right kind, right module, right scope), and " - "that downstream tasks consuming the primitive depend on " - "the creating task. NACK only if the creating task does " - "not actually produce the primitive or the dependency " - "ordering is wrong.\n\n" - "### 10. Trust-Boundary Audit (hard NACK, see #2594)\n" - "Some primitives exist but are not available in the " - "execution context the plan assumes. The canonical example: " - "`ScriptedProvider` is a unit-test-only fake; deployed agent " - "pods (`sandbox/`) run the real provider, so a k3s " - "integration test cannot inject canned LLM trajectories " - "into a deployed pod without separate infra work. The " - "`integration_tests/` fixture layout encodes a parallel " - "distinction along the **pytest-fixture** axis: the " - "`gateway_url` and `orchestrator_url` fixtures are both " - "defined only in `integration_tests/local_pipeline/conftest.py` " - "and both transitively depend on `local_pipeline_stack`, " - "which `pytest.skip`s when kubectl is unavailable. The " - "parent `integration_tests/conftest.py` exposes `egg_stack` " - "(also kubectl-gated) — `egg_stack.gateway_url` is an " - "attribute on the `EggStack` dataclass, not a standalone " - "fixture. There is no `in-sandbox-agent`-runnable pytest " - "fixture in `integration_tests/` today; the in-sandbox-agent " - "tier reaches the gateway via the `GATEWAY_URL` env at " - "agent runtime, which is a separate surface from pytest " - "fixtures.\n\n" - "For each task that interacts with the orchestrator, " - "gateway, or k3s cluster, identify the **execution context** " - "and confirm the named primitives are available in that " - "context:\n\n" - "- **in-sandbox-agent** — driven by an egg agent pod. " - "Production code the agent writes reaches gateway-mediated " - "routes via the `GATEWAY_URL` env var. No `orchestrator_url`. " - "No lifecycle-secret-gated routes. Cannot inject " - "ScriptedProvider into a pod. **No pytest fixture in " - "`integration_tests/` resolves here today** — every fixture " - "is kubectl-gated and skips in the sandbox.\n" - "- **trusted-CI-runner** — driven by pytest from outside " - "the cluster (CI / dev machine running `make test` against " - "k3s). Sees every pytest fixture in `integration_tests/` " - "(parent and `local_pipeline/`), including `gateway_url`, " - "`orchestrator_url`, lifecycle-secret-gated routes, and " - "`kubectl` pod-log access. Test files live under " - "`integration_tests/` (gateway-only) or " - "`integration_tests/local_pipeline/` (orchestrator-scoped).\n" - "- **human-operator** — manual / `egg-orch` CLI. Not a " - "test-execution context; flag any task that implicitly " - "requires this.\n\n" - "See " - "`docs/architecture/integration-test-trust-boundary.md` " - "for the authoritative tier → fixture / route mapping.\n\n" - "**NACK rule**: if a task's named primitives are not " - "available in its declared (or implied) execution context, " - "NACK and name the specific mismatch. Common forms — NACK " - "each one:\n\n" - '- "task TASK-1-8 writes an in-sandbox-agent pytest test ' - "depending on the `gateway_url` fixture, but that fixture is " - '`trusted-CI-runner`-only and skips when kubectl is absent"\n' - '- "task TASK-2-3 places a test that imports ' - "`orchestrator_url` under `integration_tests/foo/` — pytest " - "resolves fixtures lexically from the nearest conftest " - "upward, so a sibling of `local_pipeline/` cannot see that " - 'fixture and the test fails at collection time"\n' - '- "task TASK-3-1 calls a `@require_lifecycle_secret` route ' - "from an `in-sandbox-agent`-context handler — " - "`EGG_LIFECYCLE_SECRET` is not present in sandbox pods, so " - 'the route returns 403"\n' - '- "task TASK-4-2 references `ScriptedProvider` from ' - "`sandbox/` (or any deployed-pod path) — it is a unit-test " - "double under `shared/tests/`, not a runtime-injectable " - 'provider"\n\n' - "### 11. Slice Sizing (hard NACK, judgment-based — see #2809)\n" - "Slice sizing is owned by the **architect**, not the " - "task_planner. ``reviewer_plan`` is empowered AND required to " - "hard-NACK the architect when a slice is oversized for one " - "BRC cycle. This is a separate rubric key from the slice-DAG " - "shape checks so the NACK is unambiguously routed to the " - "architect for slice re-shaping (re-spawn ``architect`` with " - "the subdivision feedback).\n\n" - "**No fixed tasks-per-slice budget.** Use judgment. NACK when " - "any of the following holds:\n\n" - "- A single slice touches **more than ~3 distinct " - "file-categories** (e.g. orchestrator + gateway + schema + " - "tests + docs all in one slice probably wants subdivision).\n" - "- A single slice combines **deletion-heavy work** with " - "**new-API-introduction work** — these usually want different " - "review attention and ship better as separate slices.\n" - "- A single slice would require the implementing producer to " - "**commit-propose-revise more than 3–4 times** to converge " - "(typical signal: many independent commit clusters with " - "different reviewer surfaces).\n" - "- A single slice contains **independent task groups with no " - "internal dependency** — natural seams for parallel " - "sub-slices.\n\n" - "**NACK format**: name the seam where subdivision is " - "appropriate so the architect's re-propose is actionable. " - "Examples:\n\n" - '- "slice-1 bundles gateway allowlist edits, orchestrator ' - "route handlers, and shared/egg_contracts schema changes — " - "three distinct file-categories with different reviewer " - "surfaces. Subdivide along the gateway / orchestrator " - '/ schema seam."\n' - '- "slice-2 bundles ~600 LOC of removals across "' - "orchestrator/* with ~200 LOC of new gateway-Jira routes — " - "deletion-heavy + new-API in one cycle. Ship the removals " - 'as one slice and the new routes as a downstream slice."\n' - '- "slice-3 contains 9 tasks across 4 independent feature ' - "areas (search, profile, settings, notifications) with no " - "cross-area dependency — subdivide into one slice per " - 'area."\n\n' - "The architect re-proposes with the subdivision applied (the " - "existing BRC re-review loop handles convergence). " - "task_planner re-consumes the revised " - "``architect-slices.yaml`` scaffold on the next BRC cycle. " - "**Refiner / operator can override sizing concerns** if there " - "is a deliberate reason to ship a large slice (e.g. atomic " - "schema migration that cannot be split safely) — in that " - "case the architect should cite the override in the analysis " - "and the reviewer can ACK once the rationale is on the " - "record.\n\n" - "### 12. Slice File-Overlap Ordering (deterministic hard NACK — see #3046)\n" - "Complements §11. When slices are subdivided, any two that touch " - "the **same file** must be **ordered** along one dependency chain — " - "one a transitive ``dependencies`` ancestor of the other — never " - "left as parallel roots or siblings. The implement phase cuts each " - "slice's integration branch off its dependency parent (roots off " - "``work``), so two overlapping slices with no edge between them fork " - "independently off the shared base and their edits to the shared " - "file collide at integration (a guaranteed modify/delete conflict — " - "the #3023 incident, where three slices all touched " - "``consensus_wrapper.py``, one deleting it).\n" - "This is enforced **orchestrator-side at plan ingestion**: an " - "overlapping-but-unordered DAG is rejected before the slices are " - "written to the contract, surfacing as a ``slice_overlap_violation`` " - "discriminator (or a 'Plan ingestion REJECTED: slices touch " - "overlapping files' block on ``plan_review_feedback``). When you see " - "it, NACK the **architect** and quote the structured errors " - "verbatim; instruct it to serialise the overlapping cluster into one " - "linear ``dependencies`` chain — a slice that deletes/retires a file " - "depends on every slice that modifies it — or to merge the slices. " - "Disjoint slices stay parallel so they still run concurrently.\n" - "Belt-and-suspenders self-check: " - '`python3 -c "from egg_contracts.plan_parser import parse_plan_file, ' - "validate_slice_file_overlap as v; r = parse_plan_file('<plan-path>'); " - "print('\\n'.join(v(r.to_contract_slices())))\"`.\n\n" - "### 13. Test Co-location (hard NACK — see #3411)\n" - "Complements §12 on the test dimension. When a slice removes, " - "renames, or rewrites code, the tests exercising that code must be " - "updated, removed, or skip-guarded **in the same slice** — never in " - "a later one. Every cumulative slice tip must be independently " - "green: the per-slice green gate (#3398) executes the repo's " - "checks at the slice tip before opening the PR and blocks while " - "any check is red, so a plan that parks test obsolescence in a " - "later slice guarantees gate blocks and repair-loop churn on " - "slices whose only sin is plan topology (the #3280 stack shipped " - "a 46-failure window across slices 3–4 exactly this way: slice-3 " - "removed ``spawn_overseer_*`` from the spawner, the tests " - "exercising them were only touched in slice-5).\n" - "For each slice whose tasks remove or rename symbols, check: do " - "the test files that statically reference those symbols appear in " - "that slice's task ``files:`` (or a ``dependencies`` ancestor's)? " - "If they appear only in a LATER slice — or nowhere — NACK the " - "**architect** (slice shape is architect-owned, #2809), naming " - "the code files, the referencing test files, and the slice each " - "currently sits in, so the re-propose moves the test updates into " - "the removing slice.\n" - "Belt-and-suspenders self-check (repos shipping the changeset-" - "aware selector; this repo does): `python3 " - "scripts/select_tests/__main__.py --impacted-tests <file>...` " - "prints every test file that transitively imports the named files " - "— the same import graph `make test` narrowing uses. Exit 2 means " - "the closure could not be computed: fall back to grepping the " - "removed symbols in the test trees, and never read empty output " - "on exit 2 as 'no impacted tests'.\n\n" - "### 14. HITL Decision Registration & Un-surfaced Decisions (#3390)\n" - "- Run `egg-contract show` and check the plan-phase decision ledger: " - "every plan-phase open question must be a registered contract " - "decision (`cq-N`), and the plan draft must cite the id where the " - "question is raised. A plan-grade question living only in prose ⇒ " - "**NACK** the producer that owns it (task_planner for the plan " - "draft, architect for slice-shape questions, risk_analyst for " - "risk-acceptance questions).\n" - "- **Un-surfaced decisions — NACK.** A plan that silently commits to " - "a choice only the operator owns — dropping a requirement, changing " - "user-visible behavior, accepting a risk the operator never saw, " - "de-scoping acceptance criteria — without a registered `cq-N` bakes " - "a human-grade decision into the pipeline outside the HITL channel. " - "NACK: the producer registers the decision or removes the " - "unilateral commitment.\n" - "- **Calibration — do not over-NACK.** Design calls the plan phase " - "legitimately owns (task decomposition, API shape, migration " - "approach, slice ordering within the architect's constraints) are " - "NOT operator decisions — do not force registration of those. The " - "bar is answers only the operator owns (product intent, scope " - "boundaries, external commitments, user-visible behavior).\n" - "- A deliberately empty ledger arrives as a producer's " - "`no_decisions_rationale` attestation — verify it holds; NACK if " - "the plan hides an operator-grade choice.\n" - "- **Task-named decisions — NACK an explicit-none ledger (#3462).** " - "If the task description or refine analysis names decisions as the " - "operator's to make (or directs that decisions be surfaced as HITL " - "questions), each must have a registered `cq-N` — even when the " - 'plan argues prior context already resolves it. "Already ' - 'resolved" is a recommended disposition to register (recommended ' - "option citing the resolving context), not a reason to skip; a " - "`no_decisions_rationale` attestation on such a task is a **NACK** " - "regardless of how defensible the rationale reads.\n\n" - + _human_companion_review_criteria( - companion="`*-plan-human.md`", - parent="the implementation plan", - producer="task_planner", - ) - ) - - -def _get_security_review_criteria(repo_path: str | None = None) -> str: - """Return security-lens review criteria (issue #1965). - - The shared file inherits from ``code-review-criteria.md`` and adds - lens-specific rules (cross-file allowlist mismatches, - handler-vs-validator path mismatches, info-disclosure / authz bypass, - uncommitted-artifact mismatches, secret leakage, OWASP cross-file - patterns). Falls back to a short inline placeholder when the shared - file isn't available. - """ - content = _read_shared_criteria( - "security-review-criteria.md", - user_override="security-review-rules.md", - repo_path=repo_path, - ) - if content is not None: - return content - logger.warning("Shared security-review-criteria.md not found, using inline fallback") - return ( - "Inherits from `code-review-criteria.md`; only lens-specific rules " - "below override or extend it.\n\n" - "### Security lens (focus areas)\n" - "- **Cross-file allowlist mismatch** — handler in one file references " - "a check defined / extended in a different file (the PR #1964 " - "`^project$` pattern).\n" - "- **Handler-vs-validator path mismatch** — verify the validator's " - "regex / allowlist actually covers every code path the handler " - "reaches.\n" - "- Information-disclosure and authorization-bypass patterns at " - "trust boundaries.\n" - "- Uncommitted-artifact / Dockerfile-symlink mismatches (the PR " - "#1964 `sandbox/scripts/jira` pattern).\n" - "- Secret leakage via logs, error text, environment dumps, or " - "version-controlled config.\n" - "- OWASP top-10 patterns spanning more than one changed file.\n" - ) - - -def _get_code_review_holistic_criteria(repo_path: str | None = None) -> str: - """Return holistic-lens review criteria (issue #2126). - - The shared file inherits from ``code-review-criteria.md`` and adds - holistic-lens rules (end-to-end use-case walk, doc↔code symmetry, - synthetic-key / sentinel cross-module audit, silent-fallback hunt). - """ - content = _read_shared_criteria( - "code-review-holistic-criteria.md", - user_override="code-review-holistic-rules.md", - repo_path=repo_path, - ) - if content is not None: - return content - logger.warning("Shared code-review-holistic-criteria.md not found, using inline fallback") - return ( - "Inherits from `code-review-criteria.md`; only holistic-lens rules " - "below override or extend it.\n\n" - "### Holistic lens (focus areas)\n" - "- Walk the primary advertised use case end-to-end across the " - "full diff. NACK silent dead-ends like the `__checkout__` bug " - "on PR #2105.\n" - "- Cross-check doc-claimed behaviour against what the code does. " - "NACK doc-claimed inference / migration paths that do not exist.\n" - "- Audit synthetic keys, sentinels, and magic values for " - "cross-module agreement.\n" - "- Hunt silent fallbacks that swallow operator-visible " - "misconfiguration.\n" - "- Defer line-by-line correctness to `reviewer_code`.\n" - ) - - -def _get_concurrency_review_criteria(repo_path: str | None = None) -> str: - """Return concurrency-lens review criteria (issue #1965). - - The shared file inherits from ``code-review-criteria.md`` and adds - lens-specific rules (race conditions, deadlocks, shared-state - mutation, async-context leakage, retry storms, resource-cleanup - ordering, BRC-protocol invariants). - """ - content = _read_shared_criteria( - "concurrency-review-criteria.md", - user_override="concurrency-review-rules.md", - repo_path=repo_path, - ) - if content is not None: - return content - logger.warning("Shared concurrency-review-criteria.md not found, using inline fallback") - return ( - "Inherits from `code-review-criteria.md`; only lens-specific rules " - "below override or extend it.\n\n" - "### Concurrency lens (focus areas)\n" - "- Race conditions and deadlocks.\n" - "- Shared-state mutation without proper synchronization.\n" - "- Async-context leakage and retry-storm patterns.\n" - "- Resource-cleanup ordering bugs.\n" - "- BRC-protocol invariants (send→wait ordering, cursor threading " - "per #1925, heartbeat-stall windows per #2012).\n" - ) - - -def _get_review_criteria_for_type( - reviewer_type: str, phase: str, repo_path: str | None = None -) -> str: - """Dispatch to the correct criteria function based on reviewer type.""" - if reviewer_type == "agent-design": - return _get_agent_design_criteria() - elif reviewer_type == "code": - return _get_code_review_criteria(repo_path=repo_path) - elif reviewer_type == "code-holistic": - return _get_code_review_holistic_criteria(repo_path=repo_path) - elif reviewer_type == "contract": - return _get_contract_review_criteria(repo_path=repo_path) - elif reviewer_type == "refine": - return _get_refine_review_criteria() - elif reviewer_type == "first-principles-reviewer": - return _get_first_principles_review_criteria() - elif reviewer_type == "plan": - return _get_plan_review_criteria() - elif reviewer_type == "security": - return _get_security_review_criteria(repo_path=repo_path) - elif reviewer_type == "concurrency": - return _get_concurrency_review_criteria(repo_path=repo_path) - else: - raise ValueError(f"Unknown reviewer type: {reviewer_type}") - - -def _get_reviewer_scope_preamble(reviewer_type: str, phase: str) -> str: - """Return a scope preamble that tells the reviewer what to focus on.""" - if reviewer_type == "agent-design": - return ( - "This is a specialized **agent-mode design review**. Focus ONLY on " - "agent-mode design principles. Do NOT review general code quality, " - "security, or correctness — other reviewers handle those.\n\n" - "**Only flag issues if you find clear agent-mode design anti-patterns.** " - "If the output has no agent-mode concerns, a brief approval is acceptable " - "— you do not need to produce a lengthy analysis when there are no concerns." - ) - elif reviewer_type == "code": - return ( - "This is a **comprehensive code review**. Focus on security, correctness, " - "and robustness. Agent-mode design alignment is handled by another reviewer.\n\n" - "**Be direct.** Do not soften feedback. State issues clearly and explain " - "why they matter.\n\n" - "**Be thorough.** Find ALL issues on the first pass. Do not stop after " - "identifying a few problems.\n\n" - "**Analysis format:** Provide file-by-file analysis covering each changed " - "file. For each file, note what changed, whether the change is correct, " - "and any issues or observations." - ) - elif reviewer_type == "code-holistic": - return ( - "This is a CRITICAL **holistic code review** (issue #2126). " - "You run alongside `reviewer_code` — your job is the " - "cross-module coherence question line-by-line review does not " - "own. **Don't verify every line; `reviewer_code` covers " - "that.**\n\n" - "**Lens scope:** read the diff once with the whole PR in mind, " - "then run all four passes from the criteria below: (1) walk " - "the primary advertised use case end-to-end (the `__checkout__` " - "dead-end on PR #2105 is the canonical miss); (2) check that " - "every doc-claimed behaviour is actually implemented and every " - "user-facing code path is documented; (3) confirm synthetic " - "keys / sentinels / magic values are recognised by every " - "consumer in another module; (4) hunt silent fallbacks " - "(`except Exception:`, swallowed `None`s, default no-op " - "branches) where the operator would expect a signal.\n\n" - "**Distinct CRITICAL role.** Your NACK gates consensus on its " - "own — it is not averaged against `reviewer_code`'s " - "verdict. If the architectural-coherence question fails, " - "NACK even when the line-by-line review is clean.\n\n" - "**Analysis format:** Name the pass that found the issue, the " - "producer / consumer modules the asymmetry spans, and the " - "user-visible failure shape. If all four passes come back " - "clean a concise ACK is acceptable, but the BRC bus enforces " - "a minimum content length on ACK / NACK bodies, so write at " - "least a sentence or two summarising what you checked." - ) - elif reviewer_type == "contract": - return ( - "This is a **contract verification review**. Verify that the implementation " - "matches the contract and all acceptance criteria are met. Do NOT review " - "general code quality or security — other reviewers handle those.\n\n" - "**Analysis format:** Provide a criterion-by-criterion verification — for each " - "acceptance criterion, state whether it is met and cite the specific evidence." - ) - elif reviewer_type == "refine": - return ( - "This is a **refine phase review**. Focus on the quality and completeness " - "of the analysis produced during the refine phase. Evaluate problem " - "understanding, codebase research, options analysis, and the recommended " - "approach. Agent-mode design alignment is handled by another reviewer.\n\n" - "**Analysis format:** Provide section-by-section evaluation of the refine " - "output — assess each major section for depth, accuracy, and completeness." - ) - elif reviewer_type == "first-principles-reviewer": - return ( - "This is an adversarial **first-principles review**. Focus ONLY on " - "whether the premise is sound and the direction appropriate — the " - "seed and where the refiner's analysis is heading. Do NOT review " - "analysis quality, code, or implementation detail; other agents " - "own those.\n\n" - "You escalate by surfacing HITL decisions for the operator, not by " - "NACKing the refiner. If the direction is sound, a brief approval " - "and ACK is the right outcome — do not manufacture an objection." - ) - elif reviewer_type == "plan": - return ( - "This is a **plan phase review**. Focus on the quality and completeness " - "of the implementation plan. Evaluate task breakdown, acceptance criteria, " - "dependency ordering, risk assessment, and test strategy. Agent-mode " - "design alignment is handled by another reviewer.\n\n" - "**Analysis format:** Provide section-by-section evaluation of the plan — " - "assess task decomposition, acceptance criteria quality, dependency ordering, " - "and risk coverage." - ) - elif reviewer_type == "security": - return ( - "This is a CRITICAL **security-lens review** (issue #2139). " - "A NACK from this lens blocks consensus until the producer " - "re-proposes. Focus ONLY on the security lens; defer code " - "quality, performance, and non-security findings to " - "`reviewer_code`.\n\n" - "**Lens scope:** cross-file allowlist mismatches, " - "handler-vs-validator path mismatches, information-disclosure / " - "authorization-bypass patterns at trust boundaries, " - "uncommitted-artifact / Dockerfile-symlink mismatches, secret " - "leakage, and OWASP top-10 patterns that span more than one " - "changed file. Be especially alert to allowlist-mismatch " - "patterns where a handler in one file accepts traffic that a " - "validator in another file was supposed to reject.\n\n" - "**Analysis format:** Provide a finding-by-finding lens report. " - "If the diff has no security concerns, a concise approval is " - "acceptable — verbose reports without findings are not required, " - "but the BRC bus enforces a minimum content length on ACK / " - "NACK bodies, so write at least a sentence or two summarizing " - 'what you checked (not a single-word "LGTM").' - ) - elif reviewer_type == "concurrency": - return ( - "This is a CRITICAL **concurrency-lens review** (issue #2139). " - "A NACK from this lens blocks consensus until the producer " - "re-proposes. Focus ONLY on the concurrency lens; defer code " - "quality, performance, and non-concurrency findings to " - "`reviewer_code`.\n\n" - "**Lens scope:** race conditions, deadlocks, shared-state " - "mutation without synchronization, async-context leakage, " - "retry-storm patterns, resource-cleanup ordering bugs, and " - "BRC-protocol invariants (send→wait ordering, cursor " - "threading per #1925, heartbeat-stall windows per #2012).\n\n" - "**Analysis format:** Provide a finding-by-finding lens report. " - "If the diff has no concurrency concerns, a concise approval is " - "acceptable — verbose reports without findings are not required, " - "but the BRC bus enforces a minimum content length on ACK / " - "NACK bodies, so write at least a sentence or two summarizing " - 'what you checked (not a single-word "LGTM").' - ) - else: - raise ValueError(f"Unknown reviewer type: {reviewer_type}") - - -def _verdict_path_for_type( - phase: str, - reviewer_type: str, - issue_number: int | None = None, - pipeline_id: str | None = None, -) -> str: - """Return the relative verdict file path for a given reviewer type. - - Uses issue_number as prefix when available, otherwise pipeline_id. - """ - prefix = _pipeline_identifier(issue_number, pipeline_id or "unknown") - return f".egg-state/reviews/{prefix}-{phase}-{reviewer_type}-review.json" - - -def _draft_filename(phase: str) -> str | None: - """Return the draft filename for a phase, without any prefix. - - Centralises the phase-to-filename mapping so that - ``_get_draft_path`` and ``_get_generic_draft_path`` stay in sync. - """ - if phase == "refine": - return "analysis.md" - elif phase == "implement": - return None - else: - return f"{phase}.md" - - -def _get_draft_path( - phase: str, - issue_number: int | None = None, - pipeline_id: str | None = None, -) -> str | None: - """Return relative path to the draft file for a phase. - - Spec-driven (#3077 slice-3): the registered ``refine`` and ``plan`` - phases route through :func:`egg_contracts.artifact_spec.resolve_artifact_path` - so the registry is the single source of truth that propose-time - validation (:func:`orchestrator.routes.signals._validate_producer_artifacts`) - and every draft reader in this module share. Slice-2 of #3077 pins - the equality with a mandatory consistency test - (``TestConsistencyB_GetDraftPathEquality`` in - ``shared/egg_contracts/tests/test_artifact_spec.py``); the slice-3 - rewrite below makes that equality structural rather than incidental - — refine-risk-1's "no second copy of path knowledge" ratchet. - - Phases not yet registered in the spec (currently ``pr``) keep their - legacy path via the centralised ``_draft_filename`` mapping, so - pre-existing PR-phase callers stay byte-identical. ``implement`` - has no draft and falls out as ``None`` here. - - Uses ``issue_number`` as prefix when available, otherwise - ``pipeline_id``; falls back to ``"unknown"`` when neither is supplied. - """ - _SPEC_BY_PHASE = {"refine": "analysis-draft", "plan": "plan-draft"} - spec_name = _SPEC_BY_PHASE.get(phase) - if spec_name is not None: - # Lazy import: the spec module is pure Python and has no - # orchestrator/gateway deps, but importing it at module load - # time would still pull egg_contracts into pipelines.py's - # import graph regardless of whether _get_draft_path is called - # — keep the deferral so the import cost only lands on actual - # invocations. - from egg_contracts.artifact_spec import resolve_artifact_path - - identifier = _pipeline_identifier(issue_number, pipeline_id or "unknown") - return resolve_artifact_path(spec_name, identifier) - - filename = _draft_filename(phase) - if not filename: - return None - prefix = _pipeline_identifier(issue_number, pipeline_id or "unknown") - return f".egg-state/drafts/{prefix}-{filename}" - - -# Human-focused companion drafts (mandatory, produced by the simplifier). -# Resolved through the same artifact-spec registry as the agent drafts so -# the path knowledge lives in exactly one place. Kept separate from -# ``_get_draft_path`` (which is pinned byte-for-byte by a consistency test -# and switches on the real phase value) rather than overloading its phase -# argument with a synthetic ``refine-human`` key. -_HUMAN_SPEC_BY_PHASE = {"refine": "analysis-draft-human", "plan": "plan-draft-human"} - - -def _get_human_draft_path( - phase: str, - issue_number: int | None = None, - pipeline_id: str | None = None, -) -> str | None: - """Return the relative path to the human-focused companion draft. - - Returns ``None`` for phases without a registered human companion - (currently only ``refine`` and ``plan`` have one). - """ - spec_name = _HUMAN_SPEC_BY_PHASE.get(phase) - if spec_name is None: - return None - from egg_contracts.artifact_spec import resolve_artifact_path - - identifier = _pipeline_identifier(issue_number, pipeline_id or "unknown") - return resolve_artifact_path(spec_name, identifier) - - -def _cleanup_stale_generic_drafts(worktree_path: Path) -> bool: - """Remove unprefixed generic draft files from a worktree. - - Legacy pipelines left behind ``analysis.md`` and ``plan.md`` (without - an issue-number or pipeline-id prefix) in ``.egg-state/drafts/``. - These stale files can confuse downstream draft-reading logic. This - helper deletes only the exact unprefixed filenames; prefixed files - (e.g. ``1553-analysis.md``) are left untouched. - - Uses ``git rm`` so the deletions are staged and can be committed - immediately. Falls back to ``os.unlink`` if the file is untracked. - - Safe to call when the drafts directory does not exist (no-op). - - Returns ``True`` if a commit was made (i.e. tracked files were removed - and committed), ``False`` otherwise. - """ - drafts_dir = worktree_path / ".egg-state" / "drafts" - if not drafts_dir.is_dir(): - return False - - git_base = [ - "git", - "-c", - "core.hooksPath=/dev/null", - "-c", - f"safe.directory={worktree_path}", - "-C", - str(worktree_path), - ] - removed = False - - stale_names = ("analysis.md", "plan.md") - for name in stale_names: - stale = drafts_dir / name - if stale.exists(): - logger.info( - "Removing stale generic draft", - path=str(stale), - ) - try: - subprocess.run( - [*git_base, "rm", "-f", str(stale.relative_to(worktree_path))], - capture_output=True, - text=True, - check=True, - timeout=10, - ) - removed = True - except subprocess.CalledProcessError as exc: - # File may be untracked — just delete it from disk. - # Warn so that unexpected git rm failures (e.g. index - # lock) are diagnosable. - logger.warning( - "git rm failed for stale draft, falling back to unlink", - path=str(stale), - error=str(exc), - ) - stale.unlink(missing_ok=True) - - if removed: - try: - subprocess.run( - [ - *git_base, - "commit", - "--no-verify", - "-m", - "Remove stale generic draft files", - ], - capture_output=True, - text=True, - check=True, - timeout=30, - ) - return True - except subprocess.CalledProcessError as commit_err: - logger.debug( - "No changes to commit after stale draft cleanup", - error=str(commit_err), - ) - - return False - - -def _get_generic_draft_path(phase: str) -> str | None: - """Return the generic (unprefixed) draft path for a phase. - - Used as a fallback when the issue-specific draft file is missing. - """ - filename = _draft_filename(phase) - if not filename: - return None - return f".egg-state/drafts/{filename}" - - -def _git_show_draft( - repo_path: Path, - branch: str, - rel_path: str, - timeout: int = 15, -) -> str | None: - """Read a file from ``origin/{branch}`` via ``git show``. - - Returns the file content as a string, or ``None`` if the file does - not exist on the remote ref or the git command fails. This is a - read-only operation that does not modify the worktree. - - Note: this function does **not** ``git fetch`` itself. The caller is - responsible for ensuring ``origin/{branch}`` is fresh (e.g., by - running ``git fetch origin {branch}`` before calling this helper). - """ - git_base = [ - "git", - "-c", - "core.hooksPath=/dev/null", - "-c", - f"safe.directory={repo_path}", - "-C", - str(repo_path), - ] - try: - result = subprocess.run( - [*git_base, "show", f"origin/{branch}:{rel_path}"], - capture_output=True, - text=True, - timeout=timeout, - check=False, - ) - if result.returncode == 0 and result.stdout: - return result.stdout - if result.returncode != 0: - logger.debug( - "git show returned non-zero", - branch=branch, - rel_path=rel_path, - returncode=result.returncode, - stderr=result.stderr.strip()[:200], - ) - except Exception as exc: - logger.debug( - "git show failed for draft", - branch=branch, - rel_path=rel_path, - error=str(exc), - ) - return None - - -def _read_source_branch_artifacts( - repo_path: Path, - source_branch: str, - issue_number: int | None, - pipeline_id: str, - store: Any, - pipeline: Any, - source_artifact_prefix: str | None = None, - spawner: Any | None = None, - gateway_mode: str = "public", -) -> bool: - """Read plan and analysis artifacts from a source branch. - - Reads draft files from ``origin/<source_branch>`` via ``git show``. - Only populates ``pipeline.plan`` and ``pipeline.analysis`` when they - are not already set (inline values take precedence). - - Prefix resolution order for the exact-path lookup: - - 1. ``source_artifact_prefix`` (explicit override, e.g. ``"issue-1570-v3"``) - 2. ``pipeline_id`` (includes qualifier, e.g. ``"issue-1570-v7"``) - 3. ``issue_number`` (bare issue number, e.g. ``1570``) - - Falls back to listing available files via ``git ls-tree`` when none - of the prefixes match. - - Args: - repo_path: Path to the repository (worktree or main). - source_branch: Branch name to read artifacts from. - issue_number: Pipeline issue number (for deriving prefix). - pipeline_id: Pipeline ID (includes qualifier when present). - store: StateStore instance for saving updated pipeline. - pipeline: Pipeline model instance to populate. - source_artifact_prefix: Explicit prefix override for draft - filenames on the source branch (e.g. ``"issue-1570-v3"``). - When set, only this prefix is tried before the ls-tree - fallback. - spawner: ContainerSpawner instance for gateway-authenticated git - operations. When provided, the fetch uses the gateway API - (which injects GitHub credentials) instead of a raw - ``git fetch`` that lacks auth in the sandboxed environment. - gateway_mode: Network mode for the gateway session (``"public"`` - or ``"private"``). - - Returns: - True if any artifacts were read, False otherwise. - """ - git_base = [ - "git", - "-c", - "core.hooksPath=/dev/null", - "-c", - f"safe.directory={repo_path}", - "-C", - str(repo_path), - ] - # Bare prefix is the issue number when available — used as a fallback - # after the full pipeline_id prefix. Do NOT use _pipeline_identifier() - # here because it returns pipeline_id for qualifier-tagged pipelines, - # which defeats the fallback chain (pipeline_id → bare issue number). - bare_prefix: int | str = issue_number if issue_number is not None else pipeline_id - updated = False - - # Fetch the source branch so origin/{source_branch} is up-to-date. - # Without this, git show fails because the remote ref isn't cached - # locally. Use the gateway-authenticated fetch when available — - # raw git commands in the sandboxed environment lack GitHub - # credentials (the gateway sidecar injects them). - if spawner is not None: - try: - spawner.gateway.fetch_branch( - pipeline_id=pipeline_id, - repo_path=str(repo_path), - args=[source_branch], - mode=gateway_mode, - ) - except Exception: - logger.warning( - "Gateway fetch of source branch failed (will try git show anyway)", - source_branch=source_branch, - pipeline_id=pipeline_id, - exc_info=True, - ) - else: - # Fallback for tests or environments without a gateway. - try: - subprocess.run( - [*git_base, "fetch", "origin", source_branch], - capture_output=True, - text=True, - timeout=30, - check=False, - ) - except Exception: - logger.debug( - "Failed to fetch source branch (will try git show anyway)", - source_branch=source_branch, - exc_info=True, - ) - - # Build ordered list of prefixes to try. Duplicates are removed so - # we don't hit git show twice for the same path. - if source_artifact_prefix is not None: - # Explicit override — try only this prefix before ls-tree fallback. - prefixes: list[str | int] = [source_artifact_prefix] - else: - # Default: try pipeline_id first (includes qualifier), then bare - # issue number. When pipeline_id == bare_prefix (e.g. no qualifier - # and no issue number), the dedup below collapses them. - prefixes = [] - if pipeline_id and str(pipeline_id) != str(bare_prefix): - prefixes.append(pipeline_id) - prefixes.append(bare_prefix) - - for field_name, suffix in [("analysis", "-analysis.md"), ("plan", "-plan.md")]: - # Skip if already populated (inline values take precedence). - # Use ``is not None`` so empty strings are not silently overwritten. - if getattr(pipeline, field_name) is not None: - continue - - drafts_prefix = ".egg-state/drafts/" - content = None - - # Try each prefix in order (exact path lookup). - for pfx in prefixes: - expected_path = f"{drafts_prefix}{pfx}{suffix}" - content = _git_show_draft(repo_path, source_branch, expected_path) - if content: - logger.info( - "Read artifact from source branch (exact prefix)", - field=field_name, - source_branch=source_branch, - path=expected_path, - ) - break - - if content is None: - # Fallback: list available files and find a match - try: - result = subprocess.run( - [ - *git_base, - "ls-tree", - "--name-only", - f"origin/{source_branch}:{drafts_prefix.rstrip('/')}", - ], - capture_output=True, - text=True, - timeout=15, - check=False, - ) - if result.returncode == 0 and result.stdout.strip(): - matches = [f for f in result.stdout.strip().splitlines() if f.endswith(suffix)] - # Filter by issue number to avoid picking up artifacts - # from other issues on the same branch (#1654). - if issue_number is not None: - issue_matches = [f for f in matches if f.startswith(f"{issue_number}-")] - if issue_matches: - matches = issue_matches - else: - logger.warning( - "No fallback match for issue number — skipping", - field=field_name, - issue_number=issue_number, - source_branch=source_branch, - available=matches, - ) - continue - if len(matches) > 1: - logger.warning( - "Multiple fallback matches for artifact — using first", - field=field_name, - source_branch=source_branch, - matches=matches, - ) - for filename in matches: - fallback_path = f"{drafts_prefix}{filename}" - content = _git_show_draft(repo_path, source_branch, fallback_path) - if content: - logger.info( - "Read artifact from source branch via fallback", - field=field_name, - source_branch=source_branch, - path=fallback_path, - ) - break - except Exception as exc: - logger.debug( - "git ls-tree failed for source branch drafts", - source_branch=source_branch, - error=str(exc), - ) - - if content: - setattr(pipeline, field_name, content) - updated = True - logger.info( - "Read artifact from source branch", - field=field_name, - source_branch=source_branch, - pipeline_id=pipeline_id, - length=len(content), - ) - - if updated: - # Clear source_branch after successful read to avoid re-reading on - # pipeline restart (same pattern as plan/analysis clearing after - # draft files are pushed). - pipeline.source_branch = None - pipeline.source_artifact_prefix = None - store.save_pipeline( - pipeline, message=f"Populate artifacts from source branch {source_branch}" - ) - else: - logger.warning( - "No artifacts found on source branch", - source_branch=source_branch, - pipeline_id=pipeline_id, - source_artifact_prefix=source_artifact_prefix, - ) - - return updated - - -def _pull_contract_from_source_branch( - repo_path: Path, - source_branch: str, - issue_number: int | None, - pipeline_id: str, - spawner: Any | None = None, - gateway_mode: str = "public", - task_description: str | None = None, -) -> bool: - """Load a persisted contract from ``origin/<source_branch>`` into the worktree. - - When ``submit_task`` is called with ``source_branch``, the source branch - carries ``.egg-state/contracts/<pipeline>.json`` (with any resolved HITL - decisions). Without this helper, ``_run_pipeline`` calls - ``create_contract()`` unconditionally and overwrites those decisions with - a zero-state contract (#2035). This helper fetches the source branch, - reads the contract via ``git show``, rebinds its pipeline_id to the new - pipeline, and writes it into the worktree so the caller can skip - ``create_contract()`` and proceed to commit+push the pulled contract. - - ``task_description`` is the NEW submit's composed task statement - (``compose_task_description`` at the call site — identity anchor + - resubmit prompt, #3163). The pulled contract carries the SOURCE - pipeline's ``task_description``, but the new submit's statement is - authoritative for THIS pipeline and is where operators put binding - resume directives (e.g. "adopt prior branch X, do not reimplement" - — #3123). When non-empty it replaces the pulled value; the source - value stays recoverable from the source branch's git history. This - replacement is also what keeps a fork from leaking the source - pipeline's task into the new pipeline's per-event prompts: issue - and JIRA pipelines always compose a non-empty anchor, so the pulled - cross-pipeline text never survives. Only a free-text resume with a - blank prompt preserves the pulled value (a plain resume of the same - task). - - Returns True when a contract was successfully pulled, False otherwise. - Best-effort: missing, invalid, or unreachable source contracts all yield - False so the caller falls back to ``create_contract()``. - """ - from egg_contracts.loader import ( - ContractNotFoundError, - ContractValidationError, - load_contract_from_branch, - save_contract, - ) - - # Fetch the source branch so origin/<source_branch> is current. Mirrors - # the pattern in _read_source_branch_artifacts — use the gateway when - # available, fall back to raw git for tests / non-sandboxed callers. - if spawner is not None: - try: - spawner.gateway.fetch_branch( - pipeline_id=pipeline_id, - repo_path=str(repo_path), - args=[source_branch], - mode=gateway_mode, - ) - except Exception: - logger.warning( - "Gateway fetch of source branch failed (will try git show anyway)", - source_branch=source_branch, - pipeline_id=pipeline_id, - exc_info=True, - ) - else: - try: - subprocess.run( - [ - "git", - "-c", - "core.hooksPath=/dev/null", - "-c", - f"safe.directory={repo_path}", - "-C", - str(repo_path), - "fetch", - "origin", - source_branch, - ], - capture_output=True, - text=True, - timeout=30, - check=False, - ) - except Exception: - logger.debug( - "Failed to fetch source branch for contract pull", - source_branch=source_branch, - exc_info=True, - ) - - identifier: int | str = issue_number if issue_number is not None else pipeline_id - - try: - contract = load_contract_from_branch( - identifier, - repo_path, - branch=f"origin/{source_branch}", - ) - except ContractNotFoundError: - logger.debug( - "No contract on source branch", - pipeline_id=pipeline_id, - source_branch=source_branch, - ) - return False - except ContractValidationError as e: - logger.warning( - "Contract on source branch failed validation, falling back to fresh contract", - pipeline_id=pipeline_id, - source_branch=source_branch, - error=str(e), - ) - return False - except Exception: - logger.warning( - "Failed to load contract from source branch", - pipeline_id=pipeline_id, - source_branch=source_branch, - exc_info=True, - ) - return False - - # Rebind to the new pipeline_id so save_contract writes under the new - # canonical key when the pipeline was forked with a qualifier - # (e.g. source=issue-1965, new=issue-1965-v2). - contract.pipeline_id = pipeline_id - # Refresh the task statement from the new submit (#3123/#3163): - # without this, the resubmit's composed statement — identity anchor - # plus any operator resume directives — never reaches any - # agent-visible surface, because the caller skips create_contract() - # (the only other writer of ``task_description``) whenever the pull - # succeeds. Issue/JIRA pipelines always compose non-blank (the - # anchor at minimum), so the replace also prevents a fork from - # carrying the SOURCE pipeline's task text into this pipeline's - # per-event prompts. A blank/None value (free-text resume with no - # new prompt) preserves the pulled value so the source pipeline's - # task statement still drives the resumed run. - if task_description is not None and task_description.strip(): - contract.task_description = task_description - save_contract(contract, repo_path) - - logger.info( - "Loaded contract from source branch", - pipeline_id=pipeline_id, - source_branch=source_branch, - decision_count=len(contract.decisions), - phase_count=len(contract.slices), - ) - return True - - -def _read_phase_draft( - repo_path: Path, - phase: str, - issue_number: int | None = None, - pipeline_id: str | None = None, - max_chars: int = 32000, - branch: str | None = None, -) -> str | None: - """Read draft file contents. Truncates at max_chars. - - Returns None when the draft cannot be found (no path configured or - file missing on disk). - - Attempts in order: - - 1. Primary (issue-specific) path on disk - 2. Generic (unprefixed) path on disk - 3. Primary path via ``git show origin/{branch}:`` - 4. Generic path via ``git show origin/{branch}:`` - - The ``git show`` fallback (steps 3–4) handles cases where - ``_sync_worktree_with_remote`` failed silently and the draft exists - on the remote branch but not in the local checkout. - """ - draft_rel = _get_draft_path(phase, issue_number=issue_number, pipeline_id=pipeline_id) - if not draft_rel: - return None - - def _truncate(content: str) -> str: - if len(content) > max_chars: - return content[:max_chars] + f"\n\n... (truncated, {len(content)} chars total)" - return content - - draft_path = repo_path / draft_rel - generic_rel = _get_generic_draft_path(phase) - - # Try primary (issue-specific) path first. - if draft_path.exists(): - return _truncate(draft_path.read_text(encoding="utf-8")) - - logger.debug( - "Draft file not found", - path=str(draft_path), - phase=phase, - issue_number=issue_number, - pipeline_id=pipeline_id, - ) - - # Fallback: try the generic (unprefixed) path on disk. - if generic_rel: - generic_path = repo_path / generic_rel - if generic_path.exists(): - logger.debug( - "Using generic fallback draft path", - primary_path=str(draft_path), - fallback_path=str(generic_path), - phase=phase, - ) - return _truncate(generic_path.read_text(encoding="utf-8")) - - # Fallback: try reading from remote tracking ref via git show. - # This handles cases where _sync_worktree_with_remote() failed - # silently (fetch failure, detached HEAD, divergence, etc.) and - # the draft exists on origin but not in the local checkout. - if branch: - content = _git_show_draft(repo_path, branch, draft_rel) - if content is None and generic_rel: - content = _git_show_draft(repo_path, branch, generic_rel) - if content is not None: - logger.info( - "Read draft from remote tracking ref (local copy missing)", - phase=phase, - branch=branch, - ) - return _truncate(content) - - return None - - -def _read_human_phase_draft( - repo_path: Path, - phase: str, - issue_number: int | None = None, - pipeline_id: str | None = None, - max_chars: int = 32000, - branch: str | None = None, -) -> str | None: - """Read the human-focused companion draft for a phase. - - Mirrors :func:`_read_phase_draft` (disk first, then the - ``git show origin/{branch}`` fallback for a copy that only landed on - the remote branch), but resolves the path via - :func:`_get_human_draft_path` and has no generic-path variant — the - companion is always pipeline-identified. Returns ``None`` when the - companion is absent (so the gate falls back to the agent draft). - """ - human_rel = _get_human_draft_path(phase, issue_number=issue_number, pipeline_id=pipeline_id) - if not human_rel: - return None - - def _truncate(content: str) -> str: - if len(content) > max_chars: - return content[:max_chars] + f"\n\n... (truncated, {len(content)} chars total)" - return content - - human_path = repo_path / human_rel - if human_path.exists(): - return _truncate(human_path.read_text(encoding="utf-8")) - - if branch: - content = _git_show_draft(repo_path, branch, human_rel) - if content is not None: - logger.info( - "Read human companion draft from remote tracking ref (local copy missing)", - phase=phase, - branch=branch, - ) - return _truncate(content) - - return None - - -def _summarize_issue(prompt: str | None, issue_number: int | None = None) -> str: - """Extract a 1-2 sentence summary from the issue title and first paragraph. - - Used to give execution agents (tester, documenter) a brief - orientation without embedding the full issue body. Analysis agents - (architect, task_planner, risk_analyst) still receive the full issue. - - Extracts the first markdown heading (or first non-empty line) as the title, - then the first paragraph as supporting context. - """ - if not prompt or not prompt.strip(): - return f"Working on issue #{issue_number}." if issue_number else "" - - lines = prompt.strip().splitlines() - - # Extract title: first markdown heading, or first non-empty line - title = "" - body_start = 0 - for i, line in enumerate(lines): - s = line.strip() - if not s: - continue - if s.startswith("#"): - title = s.lstrip("# ").strip() - else: - title = s - body_start = i + 1 - break - - # Extract first paragraph after title (up to ~300 chars) - first_para_lines: list[str] = [] - for line in lines[body_start:]: - s = line.strip() - if not s: - if first_para_lines: - break - continue - first_para_lines.append(s) - - first_para = " ".join(first_para_lines) - if len(first_para) > 300: - first_para = first_para[:297] + "..." - - # Build summary - issue_ref = f" (issue #{issue_number})" if issue_number else "" - summary = f"**Background**: {title}{issue_ref}" - if first_para: - summary += f"\n\n{first_para}" - - return summary - - -def _extract_plan_overview(plan_text: str) -> str: - """Extract the plan overview section (before individual phase details). - - Returns the summary/overview portion of the plan, stopping before - individual phase task listings (### Phase N: ...) and the yaml-tasks - appendix. This gives the coder high-level context without the full plan. - """ - lines = plan_text.splitlines() - overview_lines: list[str] = [] - - for line in lines: - stripped = line.strip() - # Stop at individual phase headings - if stripped.startswith("### Phase ") or stripped.startswith("### phase-"): - break - # Stop at the yaml-tasks appendix - if "yaml-tasks" in stripped: - break - # Stop at structured task appendix - if stripped.startswith("## Structured Task Appendix"): - break - # Stop at issue-to-task mapping (detailed reference section) - if stripped.startswith("## Issue-to-Task Mapping"): - break - overview_lines.append(line) - - # Trim trailing blank lines - while overview_lines and not overview_lines[-1].strip(): - overview_lines.pop() - - return "\n".join(overview_lines) - - -def _build_role_context( - role_value: str, - prompt: str | None, - issue_number: int | None = None, - phase_obj=None, - all_phases=None, - base_branch: str | None = None, -) -> str: - """Build role-appropriate context to replace raw issue body embedding. - - Analysis roles (architect, task_planner, risk_analyst) receive the full - issue body since they need it for problem understanding and planning. - - Execution roles (tester, documenter) receive a brief summary - with structured task information and pointers to full context. - - Args: - role_value: Agent role string - prompt: Original task prompt (full issue body) - issue_number: GitHub issue number - phase_obj: Current plan phase object (phase context) - all_phases: All contract phases (phase context) - - Returns: - Role-appropriate context string to embed in the agent prompt - """ - from egg_contracts.agent_roles import EXECUTION_ROLE_VALUES - - # Analysis roles need the full issue body for problem understanding - if role_value in ("architect", "task_planner", "risk_analyst"): - if prompt: - return f"## Task Description\n\n{prompt}\n" - return "" - - lines: list[str] = [] - - # Brief summary for execution roles - summary = _summarize_issue(prompt, issue_number) - if summary: - lines.append(f"## Background\n\n{summary}\n") - - # Phase-specific context - if phase_obj is not None: - lines.append(f"## Phase Scope: {phase_obj.name} ({phase_obj.id})\n") - - if role_value == "tester": - lines.append( - f"Focus your testing on code changed in plan phase `{phase_obj.id}`. " - "The following tasks were implemented in this phase:\n" - ) - elif role_value == "documenter": - lines.append( - "Document the current state of the code in the areas these tasks " - "touch — a snapshot of how the system works now, not a log of what " - "changed. The following tasks were implemented in this phase:\n" - ) - else: - lines.append("The following tasks were implemented in this phase:\n") - - # Filter tasks by role for execution agents. - # Only apply role-based filtering when at least one task has a role - # assigned — legacy plans (all role=None) show all tasks to all agents, - # preserving backward compatibility. - _has_any_role = any(t.role is not None for t in phase_obj.tasks) - if role_value in EXECUTION_ROLE_VALUES and _has_any_role: - # Unassigned tasks (role=None) default to coder. - filtered_tasks = [ - task - for task in phase_obj.tasks - if task.role == role_value or (task.role is None and role_value == "coder") - ] - else: - filtered_tasks = list(phase_obj.tasks) - - for task in filtered_tasks: - lines.append(f"- **{task.id}**: {task.description}") - if getattr(task, "acceptance_criteria", None): - lines.append(f" - Acceptance: {task.acceptance_criteria}") - if getattr(task, "files_affected", None): - lines.append(f" - Files: {', '.join(task.files_affected)}") - lines.append("") - - if all_phases and phase_obj is not None and role_value in ("tester", "documenter"): - # Brief orientation about other phases for context - other_phases = [p for p in all_phases if p.id != phase_obj.id] - if other_phases: - lines.append("### Other Phases (for orientation)\n") - for phase in other_phases: - status = getattr(phase, "status", "unknown") - lines.append(f"- {phase.id}: {phase.name} [{status}]") - lines.append("") - - # Context pointers — agents can get more detail on demand - lines.append("## For More Context\n") - if issue_number: - lines.append(f"- Full issue: `gh issue view {issue_number}`") - _rc_base_ref = _resolve_origin_ref(base_branch) - lines.append(f"- Changed files: `git diff {_rc_base_ref}...HEAD` or check handoff data") - lines.append("- Coder output: check `EGG_HANDOFF_DATA` environment variable") - lines.append("") - - return "\n".join(lines) - - -def _build_role_restrictions_section(repo: str | None = None) -> str: - """Build a prompt section describing file access restrictions per execution role. - - This section is injected into the task_planner prompt so that it can - assign each task to the correct execution role (coder, tester, documenter) - based on which files the task will modify. - - Args: - repo: Optional ``owner/repo`` for per-repo pattern overrides - (#2528). When set, the rendered patterns reflect - ``role_patterns:`` from ``repositories.yaml`` for the repo - so the planner sees the same boundaries the gateway will - enforce. When ``None``, falls back to global defaults. - - Returns: - Formatted markdown string describing role file boundaries. - """ - from egg_restrictions.patterns import get_agent_patterns_for_repo - - lines: list[str] = [ - "## Execution Role File Restrictions", - "", - "Each task should include a `role` field (coder, tester, or documenter) " - "indicating which agent should execute it. Assign roles based on the file " - "access restrictions below. Tasks without a `role` field default to coder.", - "", - ] - - patterns_by_role = get_agent_patterns_for_repo(repo) - for role_name in ("coder", "tester", "documenter"): - pattern = patterns_by_role.get(role_name) - if pattern is None: - continue - lines.append(f"### {role_name}") - if pattern.allowed_patterns: - lines.append(f"- **Allowed**: {', '.join(f'`{p}`' for p in pattern.allowed_patterns)}") - if pattern.blocked_patterns: - lines.append(f"- **Blocked**: {', '.join(f'`{p}`' for p in pattern.blocked_patterns)}") - # Hard blocks are rejected even when they'd match the allow list or a - # fixture/docs exemption (#3396) — the planner must see them so it - # doesn't route a hard-blocked path (e.g. a fixture under `.github/` - # or any `.egg-state/` subdir) to this role. - if pattern.hard_blocked_patterns: - hard = f"- **Hard-blocked (never pushable)**: {', '.join(f'`{p}`' for p in pattern.hard_blocked_patterns)}" - if pattern.hard_block_exempt_patterns: - hard += ( - f" (except {', '.join(f'`{p}`' for p in pattern.hard_block_exempt_patterns)})" - ) - lines.append(hard) - lines.append("") - - lines.append( - "Assign `role: tester` to tasks that only touch test files, " - "`role: documenter` to tasks that only touch docs/README files, " - "and `role: coder` (or omit the field) for everything else. " - "If a task spans multiple roles, split it into separate tasks per role." - ) - lines.append("") - - # Staging-dir convention for `.github/` (issue #2508). - lines.append("### `.github/` changes — use the `.github-staging/` convention") - lines.append("") - lines.append( - "Every producer role is blocked from writing under `.github/` " - "(CI workflows, CODEOWNERS, dependabot config) — this is a " - "branch-protection invariant, not a planner mistake. Tasks that " - "need to modify those files must instead write the proposed " - "end-state to top-level `.github-staging/`, mirroring the " - "`.github/` structure (e.g. a proposed change to " - "`.github/workflows/ci.yml` is staged at " - "`.github-staging/workflows/ci.yml`). The producing agent must " - "call the staged files out in the PR body so the human " - "reviewer moves them into `.github/` before merge. Assign such " - "tasks to `role: coder` and make the staging path explicit in " - "the task's `files_affected`. `.github-staging/` must remain " - "tracked by git (do not add it to `.gitignore`); otherwise the " - "staged files won't be in the PR commit and the reviewer's " - "`git mv` will fail." - ) - lines.append("") - - # Runtime escape hatch — the actionable producer-side guidance (the - # "call these two tools, do not invent a workaround, exit cleanly" - # text) lives in ``_build_impasse_escape_hatch_section`` and is - # injected into producer prompts (coder/tester/documenter); see - # issue #2529. Here we tell the planner only that the post-failure - # delegation path exists, so it knows the orchestrator can rewire a - # mis-assigned task without re-planning. The planner does not emit - # impasses itself. - lines.append("### Runtime delegation (post-failure)") - lines.append("") - lines.append( - "If a producer discovers mid-execution that its assigned task " - "is structurally impossible, it emits a typed Impasse via " - "``mcp__sdlc__report_impasse`` and the orchestrator may " - "auto-delegate the task to a different producer role (see " - "issue #2529). You don't need to plan for this — it's a " - "runtime safety net for plan bugs, role-restriction " - "mismatches, and external blockers." - ) - lines.append("") - - return "\n".join(lines) - - -def _build_impasse_escape_hatch_section() -> str: - """Build the producer-facing runtime escape hatch section (#2529). - - Injected into the coder/tester/documenter prompts so producers know - to call ``mcp__sdlc__check_file_restriction`` / - ``mcp__sdlc__report_impasse`` instead of inventing workarounds when - they hit a structurally impossible task. The planner never emits - impasses, so this section is omitted from its prompt — see - ``_build_role_restrictions_section`` for the planner-facing - summary. - """ - return "\n".join( - [ - "## Impossible task? Use the runtime escape hatch — DO NOT invent workarounds", - "", - ( - "If you discover mid-execution that the task you've been " - "assigned is structurally impossible (file restrictions " - "block your role, the plan is buggy, an external " - "dependency is missing), STOP. Do not invent a " - "workaround like staging the files in another directory " - "or asking another agent to do it via a freeform handoff " - "document — past pipelines (#2474, #2529) wasted ~10+ " - "min and triggered downstream NACKs that way." - ), - "", - "Instead, use the two MCP tools:", - "", - ( - '1. `mcp__sdlc__check_file_restriction({path: "..."})` — ' - "cheap pure-local read against `shared/egg_restrictions/" - "patterns.py`. Confirms whether your role can write the " - "path and returns `alternative_role` (the producer role " - "that *can* write it, when exactly one covers it). Call " - "this BEFORE exploring a file you suspect is outside " - "your boundary." - ), - "", - ( - "2. `mcp__sdlc__report_impasse({category, reason, " - "task_id, suggested_role, blocked_files})` — emits a " - "typed Impasse signal and exits cleanly. **`task_id` is " - "required for ``wrong_role`` impasses** (look it up in " - "your spawn prompt or via `egg-contract show`); without " - "it the orchestrator cannot route precisely and " - "escalates to HITL. The orchestrator detects the " - "impasse post-phase and either delegates to " - "``suggested_role`` (first attempt) or escalates to " - "HITL (second attempt or no eligible role). Categories: " - "``wrong_role`` (file restrictions; auto-delegateable), " - "``plan_bug`` / ``external_blocker`` / ``unknown`` " - "(always HITL). Once you've called this tool, do NOT " - "commit code or call any other producer tool — just " - "exit." - ), - "", - ] - ) - - -def _render_contract_tasks( - repo_path: str, - pipeline_id: str, - pipeline_mode: str, - issue_number: int | None = None, -) -> str | None: - """Load contract and render tasks as a markdown checklist. - - Returns None if the contract cannot be loaded. - """ - try: - from egg_contracts.loader import load_contract - from egg_contracts.models import TaskStatus - except ImportError: - return None - - # Contracts are keyed by pipeline_id (loader's compat shim handles - # legacy paths for in-flight pipelines that predate key unification). - try: - contract = load_contract(pipeline_id, Path(repo_path)) - except Exception: - return None - - if not contract.slices: - return None - - lines = ["## Contract Tasks\n"] - for slice_ in contract.slices: - if not slice_.tasks: - continue - lines.append(f"### {slice_.name}\n") - for task in slice_.tasks: - check = "x" if task.status == TaskStatus.COMPLETE else " " - lines.append(f"- [{check}] **{task.id}**: {task.description}") - if task.acceptance_criteria: - lines.append(f" - Acceptance: {task.acceptance_criteria}") - if task.files_affected: - lines.append(f" - Files: {', '.join(task.files_affected)}") - lines.append("") - - return "\n".join(lines) if len(lines) > 1 else None - - -def _build_review_prompt( - phase: str, - pipeline_id: str, - pipeline_mode: str, - reviewer_type: str = "code", - issue_number: int | None = None, - review_cycle: int = 1, - prior_feedback: str | None = None, - repo_path: str | None = None, - last_reviewed_commit: str | None = None, - base_branch: str | None = None, - concurrent: bool = False, - operator_directives: list[OperatorDirective] | None = None, - iteration_history: list[IterationSummary] | None = None, -) -> str: - """Build a review prompt for the reviewer agent. - - In sequential mode, tells the reviewer to write a typed verdict JSON - file to .egg-state/reviews/. In concurrent (BRC) mode, the reviewer's - ACK/NACK reason IS the review output — no verdict file is written. - """ - draft_path = _get_draft_path(phase, issue_number=issue_number, pipeline_id=pipeline_id) - - verdict_path: str | None = None - if not concurrent: - verdict_path = _verdict_path_for_type( - phase, - reviewer_type, - issue_number=issue_number, - pipeline_id=pipeline_id, - ) - - lines = [ - f"You are reviewing the **{phase}** phase output of the SDLC pipeline " - f"({reviewer_type} reviewer).\n", - "## Scope\n", - _get_reviewer_scope_preamble(reviewer_type, phase), - "", - "## Context\n", - f"Pipeline ID: {pipeline_id}", - f"Phase: {phase}", - f"Reviewer: {reviewer_type}", - f"Review cycle: {review_cycle}", - "", - "## Your Task\n", - ] - - # Delta review: for re-reviews with a known last-reviewed commit, - # instruct the reviewer to focus on the delta. - # - # Two-dot `git diff A..HEAD` would wrongly include any base-branch merges - # landed between A and HEAD. `git log A..HEAD --not origin/<base> -p` - # explicitly excludes commits reachable from the base branch, so the - # reviewer sees only PR-authored work (issue #1758). - is_delta_review = review_cycle > 1 and last_reviewed_commit and not draft_path - _base_ref = _resolve_origin_ref(base_branch) - _delta_base_branch = _base_ref.removeprefix("origin/") - diff_command = ( - f"git log {last_reviewed_commit}..HEAD --not {_base_ref} -p" - if is_delta_review - else f"git diff {_base_ref}...HEAD" - ) - - if draft_path: - lines.append(f"1. Read the draft at `{draft_path}`") - elif is_delta_review: - lines.append( - f"1. First run `git fetch origin {_delta_base_branch}`, then review " - f"the delta using `{diff_command}` (see **Delta Review** below)" - ) - else: - lines.append( - f"1. Review the implementation using `git log --oneline -10` and `{diff_command}`" - ) - - # Add procedural steps for code reviewers (matching GHA reviewer thoroughness). - # Both ``code`` and ``code-holistic`` get the same numbered procedural-step - # scaffold, but steps 2 and 8 differ by lens: ``code`` reviews every file - # systematically and evaluates against the code-review criteria, while - # ``code-holistic`` skims the diff once and runs the four cross-module - # passes from the holistic criteria file. See issue #2126 — the prior - # unified wording told the holistic reviewer to "review every changed - # file systematically", which contradicted the holistic criteria's - # "don't verify every line". - # - # The operator-copy-paste framing (step 5) and pre-existing-broken-behavior - # clause (added after step 8) are deliberately scoped to code/code-holistic - # only. The shapes generalize — a security reviewer reading a `curl | bash` - # snippet, a concurrency reviewer reading a `gunicorn` launch line, or a - # contract reviewer reading an acceptance-criterion snippet would all - # benefit from "would this command execute as written?" — but the four - # #2724 misses that motivated these additions were code-lens issues - # (`pip install -r requirements.txt`, `${ANSWER}` shell-interpolated, - # `datetime.utcnow()` deprecation, non-atomic write). Keeping these on the - # code-lens branch avoids prompt bloat for narrower-lens reviewers whose - # rubrics already cover the same ground in lens-specific shape. Expand - # scope only if observed misses in other-lens reviews motivate it. - if reviewer_type in ("code", "code-holistic") and not draft_path: - if reviewer_type == "code-holistic": - lines.append( - "2. **Skim the full diff once** to build a mental map of " - "what the PR adds, who the user is, and what the user's " - "primary path through the change looks like — do not " - "re-verify every line; that is the code reviewer's job" - ) - else: - lines.append("2. Get the full diff and **review every changed file systematically**") - lines.append( - "3. Read surrounding context — check how changed code integrates with the rest of the codebase" - ) - lines.append( - "4. Trace data flow from input to output, especially for security-sensitive paths" - ) - lines.append( - "5. Verify end-to-end functionality — for new features, trace the complete " - "execution path in the real deployment environment. Check that config files, " - "environment variables, and dependencies are actually available where the code runs. " - "**Read every documented snippet, install command, and code example " - "as an operator about to copy-paste it.** Apply this verification " - "ladder to each snippet:\n" - " - Would the command execute as written?\n" - " - Does the documented file exist (`ls` or `find` it)?\n" - " - Does the library/API the snippet calls match the actual " - "signature (use WebSearch for deprecations and version-dependent " - "behavior)?\n" - "\n" - " The four blocking findings on PR #2724 (escaped to the GitHub " - "bot) were all of this shape — `pip install -r requirements.txt` " - "against a non-existent file, `${ANSWER}` shell-interpolated as a " - "bare Python identifier, `datetime.utcnow()` deprecated since " - "Python 3.12, non-atomic file write — and would all have been " - "caught by reading the snippet as a copy-paster instead of as a " - "documentation reader." - ) - lines.append( - "6. Research when uncertain — use WebSearch and WebFetch (when available) " - "to look up library behavior, check official documentation, verify " - "API usage patterns, and confirm the code follows current best practices" - ) - lines.append("7. Consider edge cases the author may not have tested") - if reviewer_type == "code-holistic": - lines.append( - "8. Run the four mandatory passes from the criteria below " - "(end-to-end primary use case, doc ↔ code symmetry, " - "synthetic-key / sentinel coordination, silent-fallback hunt)" - ) - else: - lines.append("8. Evaluate against the criteria below") - # Procedural surfacing of the pre-existing-broken-behavior clause - # from code-review-criteria.md:71. Buried in the rubric body it's - # easy to skim past — the #2724 misses on `pip install -r - # requirements.txt` and the Python-version mismatch both lived - # in lines the PR reflowed but did not author, and the reviewer - # treated them as out-of-scope context. Promoting it to a - # numbered step (read before reviewing, not consulted mid-review) - # makes it fire on lines the PR touches by reflowing, not only - # on lines it authors fresh. - lines.append( - "**(Pre-existing broken behavior in modified code is blocking.)** " - "Any unchanged line the PR reflows, surrounds, or otherwise " - "modifies its area of belongs to this review's scope. If the " - "PR's hunks reflow an install section, a documented snippet, or " - "a config example, verify the *whole section* works as advertised " - "— not just the lines marked `+`. The code is already being " - "changed in that area; this is the natural place to fix it. " - "Pre-existing bugs in modified code are NACK-blocking — do not " - 'dismiss as "not a regression."' - ) - if concurrent: - lines.append( - "9. Deliver your full review via ACK/NACK (see BRC protocol below). " - "Your `--reason` IS your review — include all findings there." - ) - else: - lines.append(f"9. Write your verdict to `{verdict_path}` as JSON") - lines.append("10. Commit the verdict file") - lines.append("") - lines.append( - "**Find ALL issues on the first pass** — do not stop after identifying " - "a few problems. You are the last line of defense before code reaches " - "production." - ) - elif draft_path: - # Expanded procedural steps for draft-based (non-code) reviewers - lines.append("2. Read the draft thoroughly — do not skim") - lines.append( - "3. Cross-reference each section of the draft against the review criteria below" - ) - lines.append("4. Cite specific sections, quotes, or omissions as evidence in your analysis") - lines.append("5. Evaluate completeness — identify any criteria not adequately addressed") - lines.append("6. Assess overall quality and coherence of the draft") - if concurrent: - lines.append( - "7. Deliver your full review via ACK/NACK (see BRC protocol below). " - "Your `--reason` IS your review — include all findings there." - ) - else: - lines.append(f"7. Write your verdict to `{verdict_path}` as JSON") - lines.append("8. Commit the verdict file") - else: - lines.append("2. Evaluate it against the criteria below") - if concurrent: - lines.append( - "3. Deliver your full review via ACK/NACK (see BRC protocol below). " - "Your `--reason` IS your review — include all findings there." - ) - else: - lines.append(f"3. Write your verdict to `{verdict_path}` as JSON") - lines.append("4. Commit the verdict file") - lines.append("") - - # Review criteria - lines.append("## Review Criteria\n") - lines.append(_get_review_criteria_for_type(reviewer_type, phase, repo_path=repo_path)) - lines.append("") - - # Review conventions — quality standards aligned with PR reviewer thoroughness - lines.append("## Review Conventions\n") - if reviewer_type in ("code", "code-holistic"): - lines.append( - "You are a critical part of the engineering infrastructure — the last line " - "of defense before code reaches production. Your review must meet these " - "quality standards:\n" - ) - else: - lines.append("Your review must meet these quality standards:\n") - lines.append( - "1. **Be comprehensive.** Review the entire scope, not just the obvious parts. " - "Do not stop after finding the first few issues." - ) - lines.append( - "2. **Be specific.** Reference exact file paths, line numbers, function names, " - "and code snippets. Vague feedback is not actionable." - ) - lines.append( - "3. **Be direct.** State issues plainly without hedging or softening language. " - '"This will fail when X" not "you might want to consider X".' - ) - lines.append( - "4. **Suggest fixes.** When identifying a problem, include a concrete suggestion " - "for how to resolve it." - ) - lines.append( - "5. **Provide context.** Explain *why* something is an issue — the impact, " - "the risk, or the principle being violated." - ) - lines.append("") - - # Verdict classification — only for code reviewers (aligned with review-conventions.md) - # Non-code reviewers get appropriate guidance from their type-specific criteria - # (e.g., _get_plan_review_criteria() already says "flag as needs_revision") - if reviewer_type in ("code", "code-holistic"): - _nack_label = "NACK" if concurrent else "`needs_revision`" - _ack_label = "ACK" if concurrent else "`approved`" - lines.append(f"### When to {_nack_label} vs {_ack_label}\n") - lines.append( - f"**{_nack_label} for**: Security vulnerabilities, logic errors, correctness " - "issues, non-functional features (core purpose doesn't work end-to-end), missing " - "error handling, resource leaks, breaking changes, violations of codebase patterns. " - f"When in doubt, {_nack_label}." - ) - lines.append( - f"**{_ack_label} for**: No blocking issues found after thorough review. " - "Non-blocking suggestions should still be included." - ) - lines.append("") - lines.append( - "**Key distinction**: A feature that doesn't work is a correctness issue, not a " - "style issue. If the feature's core functionality is broken — not just degraded or " - f"missing edge cases — always {_nack_label}, even if the code structure looks " - "reasonable or matches an existing pattern." - ) - lines.append("") - - # Delta review directive for re-reviews - if is_delta_review: - lines.append("## Delta Review\n") - lines.append( - f"This is review cycle {review_cycle}. Focus on new changes since your " - f"last review. First run `git fetch origin {_delta_base_branch}` to " - f"ensure the base branch is available, then use " - f"`git log {last_reviewed_commit}..HEAD --not {_base_ref} -p` to see " - "the delta — this excludes any base-branch commits that were merged " - "in since your last review, so you only see PR-authored changes. " - "Verify prior feedback was addressed AND review new code thoroughly." - ) - lines.append("") - - # Phase iteration context: operator directives + prior iteration - # history. Surfaced to reviewers so they cannot faithfully NACK a - # directive-driven change against a stale default rubric (#2795). - iteration_context = _build_phase_iteration_context(operator_directives, iteration_history) - if iteration_context: - lines.append(iteration_context) - - # Prior feedback for re-reviews - if review_cycle > 1 and prior_feedback: - lines.append("## Prior Review Feedback\n") - lines.append( - "This is a re-review. The previous review found issues. " - "Verify that the following feedback was addressed:\n" - ) - lines.append(prior_feedback) - lines.append("") - - # Verdict format — only for sequential (non-concurrent) reviewers. - # In concurrent/BRC mode, the ACK/NACK reason IS the review output. - if not concurrent: - lines.append("## Verdict Format\n") - lines.append(f"Write the following JSON to `{verdict_path}`:\n") - lines.append("```json") - lines.append("{") - lines.append(f' "reviewer": "{reviewer_type}",') - lines.append(' "verdict": "approved" or "needs_revision",') - lines.append(' "summary": "Brief summary of findings (1-2 sentences)",') - lines.append(' "analysis": "Detailed analysis of the reviewed work (see below)",') - lines.append(' "suggestions": "Non-blocking suggestions for improvement",') - lines.append(' "feedback": "Blocking issues requiring revision before approval",') - lines.append(' "timestamp": "ISO 8601 timestamp"') - lines.append("}") - lines.append("```\n") - lines.append("**Field guidelines:**\n") - lines.append( - "- **analysis**: Always provide detailed analysis regardless of verdict. " - "Describe what you reviewed, what you found, and your reasoning." - ) - lines.append( - "- **suggestions**: Non-blocking observations and improvement ideas. " - "Include these even when approving — they help the team improve over time." - ) - lines.append( - "- **feedback**: Reserved for **blocking issues only** — problems that must " - "be fixed before the work can be approved. Leave empty when approving." - ) - lines.append( - "\nIf the work meets all criteria, set verdict to `approved`. " - "If significant issues remain, set verdict to `needs_revision` " - "and provide actionable feedback in the `feedback` field." - ) - - # Phase restrictions for reviewers - lines.append("") - lines.append("## Phase Restrictions\n") - lines.append("- You CAN read all source files and review artifacts") - if not concurrent: - lines.append("- You CAN write verdict files to `.egg-state/reviews/`") - if reviewer_type == "contract": - lines.append( - "- You CAN update the contract in `.egg-state/contracts/` (e.g. marking items as done)" - ) - lines.append("- You CANNOT push code (git push)") - lines.append("- You CANNOT create or update PRs") - lines.append("- You CANNOT modify source files (src/, lib/, docs/, tests/)") - lines.append("") - - return "\n".join(lines) - - -def _read_review_verdict( - repo_path: Path, - phase: str, - reviewer_type: str = "code", - pipeline_mode: str = "issue", - issue_number: int | None = None, - pipeline_id: str | None = None, -) -> ReviewVerdict | None: - """Read a typed review verdict JSON from the repo. - - Returns None if the file is missing or malformed (treated as approved - for graceful degradation). - """ - verdict_rel = _verdict_path_for_type( - phase, - reviewer_type, - issue_number=issue_number, - pipeline_id=pipeline_id, - ) - verdict_file = repo_path / verdict_rel - - if not verdict_file.exists(): - logger.warning( - "Verdict file not found, treating as approved", - path=str(verdict_file), - reviewer_type=reviewer_type, - ) - return None - - try: - raw = verdict_file.read_text() - data = json.loads(raw) - return ReviewVerdict(**data) - except (json.JSONDecodeError, Exception) as e: - logger.warning( - "Failed to parse verdict file, treating as approved", - path=str(verdict_file), - reviewer_type=reviewer_type, - error=str(e), - ) - return None - - -def _read_tester_gaps( - repo_path: Path, - identifier: int | str | None = None, -) -> str | None: - """Read tester output and extract gap findings for feedback to the coder. - - Reads `.egg-state/agent-outputs/{identifier}-tester-output.json` (with - fallback to `tester-output.json`) and formats any test failures and gaps - found into a summary string. - - Falls back to scanning the `summary` field for failure keywords when - `gaps_found` is not present (backwards compat with old tester outputs). - - Args: - repo_path: Path to the repository. - identifier: Pipeline/issue identifier for namespaced filenames. - - Returns: - Formatted gap summary string, or None if no gaps found. - """ - outputs_dir = repo_path / ".egg-state" / "agent-outputs" - - # Try prefixed filename first, fall back to old global filename - tester_output_file = None - if identifier is not None: - prefixed = outputs_dir / f"{identifier}-tester-output.json" - if prefixed.exists(): - tester_output_file = prefixed - if tester_output_file is None: - tester_output_file = outputs_dir / "tester-output.json" - - if not tester_output_file.exists(): - return None - - try: - raw = tester_output_file.read_text() - data = json.loads(raw) - except (json.JSONDecodeError, OSError) as e: - logger.warning( - "Failed to parse tester output file", - path=str(tester_output_file), - error=str(e), - ) - return None - - if not isinstance(data, dict): - return None - - sections: list[str] = [] - - tests_failed = data.get("tests_failed", 0) - if tests_failed: - sections.append(f"- **{tests_failed}** test(s) failed") - - gaps_found = data.get("gaps_found") - if gaps_found and isinstance(gaps_found, list): - # Cap at 10 gaps to avoid prompt bloat - capped = gaps_found[:10] - for gap in capped: - gap_str = str(gap)[:200] - sections.append(f"- {gap_str}") - if len(gaps_found) > 10: - sections.append(f"- ... and {len(gaps_found) - 10} more gaps") - elif not tests_failed: - # Backwards compat: scan summary for failure keywords - summary = data.get("summary", "") - if isinstance(summary, str) and any( - kw in summary.lower() for kw in ("fail", "gap", "missing", "error", "deficien") - ): - sections.append(f"- Tester summary: {summary}") - - if not sections: - return None - - return f"{TESTER_FINDINGS_HEADER}\n" + "\n".join(sections) - - -def _aggregate_review_verdicts( - verdicts: dict[str, ReviewVerdict | None], -) -> AggregatedReviewResult: - """Aggregate multiple typed review verdicts into an overall result. - - Returns: - AggregatedReviewResult with: - - verdict: "approved" or "needs_revision" (any needs_revision → overall needs_revision) - - blocking_feedback: combined feedback from needs_revision verdicts only - - advisory_content: analysis and suggestions from ALL verdicts (including approved) - - Missing/None verdicts are skipped. - """ - overall = "approved" - feedback_sections: list[str] = [] - advisory_sections: list[str] = [] - - for reviewer_type, verdict in verdicts.items(): - if verdict is None: - continue - - # Collect blocking feedback from needs_revision verdicts - if verdict.verdict == "needs_revision": - overall = "needs_revision" - section = f"### {reviewer_type} reviewer\n" - if verdict.feedback: - section += verdict.feedback - elif verdict.summary: - section += verdict.summary - feedback_sections.append(section) - - # Collect analysis and suggestions from ALL verdicts (including approved) - advisory_parts: list[str] = [] - if verdict.analysis: - advisory_parts.append(verdict.analysis) - if verdict.suggestions: - advisory_parts.append(f"**Suggestions:** {verdict.suggestions}") - if advisory_parts: - advisory_sections.append( - f"### {reviewer_type} reviewer\n" + "\n\n".join(advisory_parts) - ) - - blocking_feedback = "\n\n".join(feedback_sections) if feedback_sections else "" - advisory_content = "\n\n".join(advisory_sections) if advisory_sections else "" - return AggregatedReviewResult( - verdict=overall, - blocking_feedback=blocking_feedback, - advisory_content=advisory_content, - ) - - -class WorktreeSyncOutcome(NamedTuple): - """Structured outcome from :func:`_sync_worktree_with_remote` (#2792, #2979). - - Phase-boundary callers inspect ``diverged_unreconciled`` to decide - whether to pause the pipeline for a manual reconcile. Best-effort - callers can ignore the return value entirely — every field has a - safe default and the sync still does the same in-band work whether - or not the outcome is consumed. - - ``case`` is the same discriminator the function emits to its - ``worktree_sync_outcome`` log line, so the field can be cross- - referenced against operator-grep patterns. - - ``diverged_unreconciled`` is True when local and remote had truly - diverged (ahead AND behind) and the rebase autoresolve could not - reconcile them. Since #2979 the helper does **not** hard-reset in - that case — the rebase autoresolve already aborted (restoring the - worktree to the clean local HEAD with the orchestrator's committed - work intact), so the helper leaves the worktree there and reports - the unreconciled divergence so the caller can pause for a manual - reconcile rather than discarding committed work. - - ``backup_ref`` is the full ref name (``refs/egg-backup/sync-recovery/ - <pipeline_id>/<unix_ts>``) pinning the local HEAD when divergence is - unreconciled — a stable handle the operator can inspect/reset to. - ``None`` means the (best-effort) backup write failed; the commits are - still on the live HEAD, and the local-only SHAs go into the WARN log - inline so they're at least in the audit trail (see the helper body). - - ``local_only_commit_shas`` is the list of local-only short SHAs (with - summaries) that are on HEAD but not yet on origin. Empty when the - rev-list itself failed; the divergence is still reported, but the - operator can't be given the exact commit list inline. - - ``rebase_category`` / ``rebase_detail`` carry the failing rebase's - ``PushResult.category`` / ``detail`` (conflicting paths, the rebase - argv, and a git-output excerpt) when ``diverged_unreconciled`` is - True. They exist so the reconcile HITL can show the operator *what* - failed instead of an unfalsifiable generic claim (#3416) — the log - lines carry the same data but roll; the decision persists. - """ - - case: str - diverged_unreconciled: bool = False - backup_ref: str | None = None - local_only_commit_shas: tuple[str, ...] = () - rebase_category: str | None = None - rebase_detail: str | None = None - - -def _build_sync_recovery_backup_ref(pipeline_id: str, unix_ts: int) -> str: - """Return the canonical ``refs/egg-backup/sync-recovery/<pid>/<ts>`` name (#2792). - - Pulled out so the test, the writer, and any future opportunistic - pruner share a single ref-name convention. The slash-segment - layout lets ``git for-each-ref refs/egg-backup/sync-recovery/<pid>`` - enumerate just this pipeline's backups. - """ - return f"refs/egg-backup/sync-recovery/{pipeline_id}/{unix_ts}" - - -def _collect_local_only_commits( - git_base: list[str], - *, - pipeline_id: str, - branch: str, - remote_branch: str, -) -> tuple[str, ...]: - """Enumerate local-only commits between HEAD and ``origin/<remote_branch>``. - - Returns a tuple of ``"<short-sha> <summary>"`` strings, oldest first. - A failure (subprocess error, nonzero rc, parse error) returns an - empty tuple and emits a WARN — the hard-reset fallback proceeds - with an unknown discard list rather than blocking on best-effort - forensic enumeration (#2792 section 5). - """ - try: - result = subprocess.run( - [ - *git_base, - "rev-list", - "--reverse", - "--pretty=format:%h %s", - "--no-commit-header", - f"origin/{remote_branch}..HEAD", - ], - capture_output=True, - text=True, - timeout=10, - check=False, - ) - if result.returncode != 0: - logger.warning( - "Failed to enumerate local-only commits before hard reset", - pipeline_id=pipeline_id, - branch=branch, - remote_branch=remote_branch, - rc=result.returncode, - stderr=result.stderr.strip()[:200], - ) - return () - lines = [ln.strip() for ln in result.stdout.splitlines() if ln.strip()] - return tuple(lines) - except Exception as exc: - logger.warning( - "Local-only commit enumeration raised before hard reset", - pipeline_id=pipeline_id, - branch=branch, - remote_branch=remote_branch, - error=str(exc), - ) - return () - - -def _create_sync_recovery_backup_ref( - git_base: list[str], - *, - pipeline_id: str, - ref_name: str, -) -> bool: - """Pin current HEAD under ``ref_name`` via ``git update-ref`` (#2792). - - Returns True on success. On failure logs WARN and returns False; - the caller proceeds with the destructive reset regardless — the - backup is best-effort, the reset is the reconcile primitive. - """ - try: - result = subprocess.run( - [*git_base, "update-ref", ref_name, "HEAD"], - capture_output=True, - text=True, - timeout=10, - check=False, - ) - if result.returncode != 0: - logger.warning( - "Failed to create sync-recovery backup ref", - pipeline_id=pipeline_id, - ref_name=ref_name, - rc=result.returncode, - stderr=result.stderr.strip()[:200], - ) - return False - return True - except Exception as exc: - logger.warning( - "Sync-recovery backup-ref write raised", - pipeline_id=pipeline_id, - ref_name=ref_name, - error=str(exc), - ) - return False - - -def _sync_worktree_with_remote( - spawner: "ContainerSpawner", # noqa: UP037 - pipeline_id: str, - worktree_repo_path: Path, - prior_phase_succeeded: bool = True, - gateway_mode: Literal["public", "private"] = "public", - base_branch: str | None = None, - *, - pipeline_branch: str | None = None, -) -> WorktreeSyncOutcome: - """Sync a worktree with its remote branch (best-effort). - - After an orchestrator restart or a phase boundary, the local worktree - branch may be behind the remote: commits pushed during previous phases - (contracts, drafts, statefiles) exist on origin but not in the local - checkout. This function fetches those commits and reconciles the - worktree so that all downstream code (contract loading, draft reading, - populator, etc.) sees the full pipeline state. - - ``pipeline_branch`` is the **remote** branch name to reconcile against. - Since #2399 the pipeline tip lives at ``egg/<pid>/work`` on origin so - slice integration branches at ``egg/<pid>/slice-N`` can coexist as - siblings; ``pipeline.branch`` already carries that ``/work`` suffix - (set by :func:`_ensure_pipeline_work_ref` at submission time), so - callers should pass ``pipeline_branch=pipeline.branch`` directly — - the local worktree branch and the remote ref now match. Without an - explicit ``pipeline_branch``, the function reads - ``git branch --show-current`` and looks up ``origin/<that-name>``, - which always misses on real pipelines and exits at - ``case=no_remote_tracking`` (#2367). Callers with a pipeline in - scope MUST pass ``pipeline_branch=pipeline.branch``. When omitted, - the function falls back to the local branch name for backward - compatibility with non-pipeline scripts. - - When local is ahead of remote: - - If the prior phase succeeded, push local commits to remote first. - On a successful push, reset to origin (a no-op fast-forward that - keeps the worktree clean). If the push FAILS, the local commits - are preserved as-is and the function returns without resetting — - ``remote_ahead == 0`` means origin holds nothing to incorporate, so - a ``reset --hard origin`` would only discard completed, committed - work (e.g. agent-registered HITL contract decisions) before the - phase_gate decision bridge could surface them (#2972). - - If the prior phase failed or was killed, discard local commits and - reset to remote (discards incomplete work). - - When local has diverged (ahead AND behind), rebase local commits onto - ``origin/{pipeline_branch}`` via the same helper used by the - gateway-side push-reject reconcile path. ``--ff-only`` cannot - reconcile real divergence by definition, so the pre-#2337 - implementation silently left the worktree stale and downstream - populator/decision-sync paths consumed the stale state. - - When the rebase itself fails (#2792, made non-destructive in #2979), - the autoresolve has already run ``git rebase --abort`` — which - restores the worktree to the clean local HEAD and reapplies the - autostash, so the orchestrator's committed work is intact on HEAD. - The helper does **not** hard-reset (the pre-#2979 behaviour, which - discarded that committed work to a backup ref and FAILed the - pipeline). It pins HEAD under ``refs/egg-backup/sync-recovery/ - <pipeline_id>/<unix_ts>`` as a stable operator handle and returns - ``diverged_unreconciled=True`` so phase-boundary callers pause the - pipeline for a manual reconcile (AWAITING_HUMAN) rather than - consuming the un-reconciled state or discarding work. - - Every return path emits at least one ``worktree_sync_outcome`` log - line with a ``case`` discriminator so production logs name which - path fired. The ``rev_list_failed`` and ``divergence_unreconciled`` - cases bail non-destructively (no ``reset --hard``); only the - local-behind and prior-phase-failed-discard cases reach the step-4 - reset, neither of which can lose committed work that isn't already - on origin. - - Safe to call on every pipeline start because it is idempotent when the - local branch is already up to date. - - Returns a :class:`WorktreeSyncOutcome` describing what the helper - did. Most callers can ignore the return value; phase-boundary - callers inspect ``diverged_unreconciled`` to decide whether to pause - the pipeline for a manual reconcile (#2979). - """ - base_branch_for_reconcile = base_branch - git_base = [ - "git", - "-c", - "core.hooksPath=/dev/null", - "-c", - f"safe.directory={worktree_repo_path}", - "-C", - str(worktree_repo_path), - ] - - # Step 1: Authenticated fetch via gateway (gateway holds GitHub credentials) - fetch_ok = spawner.gateway.fetch_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(worktree_repo_path), - mode=gateway_mode, - ) - if not fetch_ok: - logger.info( - "worktree_sync_outcome", - pipeline_id=pipeline_id, - case="fetch_failed", - ) - return WorktreeSyncOutcome(case="fetch_failed") - - # Step 2: Determine current branch - try: - result = subprocess.run( - [*git_base, "branch", "--show-current"], - capture_output=True, - text=True, - timeout=10, - check=False, - ) - branch = result.stdout.strip() - if not branch: - logger.info( - "worktree_sync_outcome", - pipeline_id=pipeline_id, - case="detached_head", - ) - return WorktreeSyncOutcome(case="detached_head") - except Exception as branch_err: - logger.info( - "worktree_sync_outcome", - pipeline_id=pipeline_id, - case="branch_detect_failed", - error=str(branch_err), - ) - return WorktreeSyncOutcome(case="branch_detect_failed") - - # ``branch`` is the **local** branch name (e.g. ``egg/<pid>/work`` on - # orchestrator worktrees). ``remote_branch`` is the remote-side name - # we look up on origin and push/reset against. When the caller - # passes ``pipeline_branch`` (the canonical, agent-facing branch), - # use it for every remote-side ref so the ``/work`` suffix mismatch - # in #2367 cannot strand a pipeline in ``no_remote_tracking``. - remote_branch = pipeline_branch or branch - - # Step 3: Verify remote tracking branch exists - try: - result = subprocess.run( - [*git_base, "rev-parse", "--verify", f"origin/{remote_branch}"], - capture_output=True, - text=True, - timeout=10, - check=False, - ) - if result.returncode != 0: - logger.info( - "worktree_sync_outcome", - pipeline_id=pipeline_id, - branch=branch, - remote_branch=remote_branch, - case="no_remote_tracking", - ) - return WorktreeSyncOutcome(case="no_remote_tracking") - except Exception as rev_parse_err: - logger.info( - "worktree_sync_outcome", - pipeline_id=pipeline_id, - branch=branch, - remote_branch=remote_branch, - case="rev_parse_failed", - error=str(rev_parse_err), - ) - return WorktreeSyncOutcome(case="rev_parse_failed") - - # Step 3b: Check divergence between local and remote. - local_ahead = 0 - remote_ahead = 0 - rev_list_ok = False - try: - result = subprocess.run( - [ - *git_base, - "rev-list", - "--left-right", - "--count", - f"HEAD...origin/{remote_branch}", - ], - capture_output=True, - text=True, - timeout=10, - check=False, - ) - parts = result.stdout.strip().split() - if result.returncode == 0 and len(parts) == 2: - local_ahead = int(parts[0]) - remote_ahead = int(parts[1]) - rev_list_ok = True - else: - logger.warning( - "worktree_sync_outcome", - pipeline_id=pipeline_id, - branch=branch, - remote_branch=remote_branch, - case="rev_list_failed", - rc=result.returncode, - stdout=result.stdout.strip()[:200], - ) - # #2979: the ahead/behind counts are unknown, so a Step-4 - # ``reset --hard origin`` here could discard local-only - # commits that are NOT on origin — a destructive reset over - # un-provably-pushed work with no backup ref. Bail - # non-destructively instead, leaving the worktree untouched. - return WorktreeSyncOutcome(case="rev_list_failed") - except Exception as rev_list_err: - logger.warning( - "worktree_sync_outcome", - pipeline_id=pipeline_id, - branch=branch, - remote_branch=remote_branch, - case="rev_list_failed", - error=str(rev_list_err), - ) - # #2979: unknown ahead/behind counts — bail non-destructively - # rather than fall through to the Step-4 ``reset --hard`` (which - # would risk discarding un-pushed local work without a backup). - return WorktreeSyncOutcome(case="rev_list_failed") - - # Step 3c: Handle local-ahead commits. - if local_ahead == 0 and remote_ahead == 0 and rev_list_ok: - # Local and remote are already in sync — skip the no-op reset entirely - # so the outcome is distinguishable from a true behind-remote sync. - logger.info( - "worktree_sync_outcome", - pipeline_id=pipeline_id, - branch=branch, - remote_branch=remote_branch, - case="already_in_sync", - local_ahead=0, - remote_ahead=0, - ) - return WorktreeSyncOutcome(case="already_in_sync") - - if local_ahead > 0 and remote_ahead == 0: - # Local is strictly ahead of remote (no divergence). - if prior_phase_succeeded: - # Prior phase completed successfully — push local work to remote - # before resetting, so it's not lost. Pushing to ``remote_branch`` - # (not the local ``/work`` name) so the agent-facing branch - # receives the commits — the gateway builds - # ``HEAD:refs/heads/{branch}`` from this argument. - push_result = spawner.gateway.push_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(worktree_repo_path), - branch=remote_branch, - mode=gateway_mode, - base_branch=base_branch_for_reconcile, - ) - if push_result: - # Push succeeded — local and remote are now in sync. - # Re-fetch to update the remote tracking ref so that - # origin/{remote_branch} reflects the pushed commits. - spawner.gateway.fetch_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(worktree_repo_path), - mode=gateway_mode, - ) - logger.info( - "worktree_sync_outcome", - pipeline_id=pipeline_id, - branch=branch, - remote_branch=remote_branch, - case="local_ahead_pushed", - local_ahead=local_ahead, - remote_ahead=remote_ahead, - ) - return WorktreeSyncOutcome(case="local_ahead_pushed") - else: - # Push failed. ``remote_ahead == 0`` in this branch, so - # origin holds nothing the worktree lacks — resetting to - # origin here would discard the completed, committed local - # work (e.g. the agent-registered HITL contract decisions - # the pre-sync ``_commit_statefiles_to_worktree`` just - # committed) for ZERO reconcile benefit, then advance - # silently. That is exactly how #2972 dropped a refiner's - # ``register_open_question`` / ``request_feedback`` items - # before the phase_gate decision bridge could surface them: - # the prior code fell through to the Step-4 ``reset --hard`` - # and returned ``reset_succeeded`` (``hard_reset_performed`` - # False), so no operator signal fired. Preserve the local - # commits instead — they remain in the worktree for - # downstream reads (the decision bridge, populator) and for - # the next push attempt. The WARNING below is the loud - # breadcrumb that the tip is unpushed; unlike the divergence - # path (``remote_ahead > 0``) there is no remote work to - # rebase onto, so non-destructive preservation is correct. - logger.warning( - "worktree_sync_outcome", - pipeline_id=pipeline_id, - branch=branch, - remote_branch=remote_branch, - case="local_ahead_push_failed", - local_ahead=local_ahead, - remote_ahead=remote_ahead, - category=push_result.category, - error=push_result.detail, - ) - return WorktreeSyncOutcome(case="local_ahead_push_failed") - else: - # Prior phase failed — incomplete local work will be discarded by - # the step-4 reset. Emit a distinct case so operators can grep - # this branch of the taxonomy without inferring it from - # reset_succeeded with local_ahead > 0. - logger.info( - "worktree_sync_outcome", - pipeline_id=pipeline_id, - branch=branch, - remote_branch=remote_branch, - case="local_ahead_discarded", - local_ahead=local_ahead, - remote_ahead=remote_ahead, - ) - # Fall through to reset (Step 4) — discards incomplete local work - # from a failed/killed prior phase. (The successful-phase - # push-failure case returns above without resetting so completed - # work is never silently dropped — #2972.) - - elif local_ahead > 0 and remote_ahead > 0: - # True divergence. Reconcile by rebasing local commits onto - # origin/{branch} via the same helper used by the gateway-side - # push-reject reconcile path (#2337). --ff-only cannot reconcile - # real divergence by definition, so the pre-#2337 implementation - # silently left the worktree stale. - # - # ⚠️ When ``base_branch_for_reconcile`` is None, - # ``_build_rebase_cmd`` falls back to the plain - # ``git rebase origin/{branch}`` form — the same form that - # triggered #2222 main-contamination on the gateway-side - # push-reject path. That fallback is the contamination vector: - # with HEAD at current main and origin/{branch} on a stale - # snapshot, the plain form replays merge-base..HEAD on the - # stale tip, producing a PR full of duplicate-by-content - # commits. Callers should always thread ``pipeline.base_branch`` - # so the helper emits the safer - # ``--onto origin/{branch} origin/{base_branch}`` form. Logging - # the None case so the next person debugging contamination has a - # breadcrumb. - if base_branch_for_reconcile is None: - logger.warning( - "worktree_sync divergence_rebase with base_branch=None — " - "falling back to bare-rebase form (#2222 contamination risk)", - pipeline_id=pipeline_id, - branch=branch, - remote_branch=remote_branch, - ) - logger.info( - "Local and remote have diverged — rebasing local onto origin", - pipeline_id=pipeline_id, - branch=branch, - remote_branch=remote_branch, - local_ahead=local_ahead, - remote_ahead=remote_ahead, - ) - rebase_outcome = _rebase_with_agent_output_autoresolve( - git_base=git_base, - pipeline_id=pipeline_id, - branch=remote_branch, - base_branch=base_branch_for_reconcile, - ) - if rebase_outcome.ok: - logger.info( - "worktree_sync_outcome", - pipeline_id=pipeline_id, - branch=branch, - remote_branch=remote_branch, - case="divergence_rebased", - local_ahead=local_ahead, - remote_ahead=remote_ahead, - ) - return WorktreeSyncOutcome(case="divergence_rebased") - logger.error( - "worktree_sync_outcome", - pipeline_id=pipeline_id, - branch=branch, - remote_branch=remote_branch, - case="divergence_rebase_failed", - local_ahead=local_ahead, - remote_ahead=remote_ahead, - category=rebase_outcome.category, - detail=rebase_outcome.detail, - ) - - # #2979: non-destructive divergence reconcile. The rebase - # autoresolve could not reconcile the divergence — a conflict on - # a path outside ``.egg-state/agent-outputs/``. In normal - # operation this is now unreachable: #2979 stopped agents from - # git-pushing ``.egg-state/contracts/`` (they mutate contracts - # through the contract API), so the orchestrator is the sole - # writer of the only non-agent-outputs path both sides touched on - # the work branch, and the rebase only ever replays disjoint - # paths. When it *does* fire (an unexpected residual conflict, a - # restart mid-flight), the autoresolve has already run - # ``git rebase --abort``, which restored the worktree to the - # clean local HEAD and reapplied the autostash — the - # orchestrator's committed work is intact on HEAD. - # - # #2792/#2797 used to ``git reset --hard origin`` here, discarding - # that committed work (operator-bound contract decisions included) - # to a backup ref the operator had to spelunk, then FAIL the - # pipeline. Instead, leave the worktree at local HEAD and report - # the unreconciled divergence; the caller pauses the pipeline for - # a manual reconcile (AWAITING_HUMAN, not FAILED). Downstream - # consumers — populator, decision-sync, plan-complete — never run - # against the un-reconciled state because the pause halts the - # phase before them, which is the silent-stale-read failure #2337 - # raised an error for, addressed without discarding work. - # - # Pin HEAD under a backup ref anyway: a stable, enumerable handle - # the operator can ``git log`` / ``git reset`` against, and a - # guard against any later worktree mutation. Best-effort — a - # failed write inlines the SHAs into the WARN log for the audit - # trail (the commits remain on the live HEAD regardless). - # Nanosecond precision so two reconcile attempts within the same - # second on the same pipeline cannot collide on the ref name. - local_only = _collect_local_only_commits( - git_base, - pipeline_id=pipeline_id, - branch=branch, - remote_branch=remote_branch, - ) - unix_ts = time.time_ns() - backup_ref = _build_sync_recovery_backup_ref(pipeline_id, unix_ts) - backup_ok = _create_sync_recovery_backup_ref( - git_base, - pipeline_id=pipeline_id, - ref_name=backup_ref, - ) - if not backup_ok and local_only: - logger.warning( - "Divergence-reconcile backup ref write failed; local-only " - "SHAs inlined for audit (commits remain on the live HEAD)", - pipeline_id=pipeline_id, - branch=branch, - remote_branch=remote_branch, - local_only_commit_shas=list(local_only), - ) - logger.warning( - "worktree_sync_outcome", - pipeline_id=pipeline_id, - branch=branch, - remote_branch=remote_branch, - case="divergence_unreconciled", - local_ahead=local_ahead, - remote_ahead=remote_ahead, - backup_ref=backup_ref if backup_ok else None, - local_only_commit_count=len(local_only), - rebase_category=rebase_outcome.category, - ) - return WorktreeSyncOutcome( - case="divergence_unreconciled", - diverged_unreconciled=True, - backup_ref=backup_ref if backup_ok else None, - local_only_commit_shas=local_only, - rebase_category=rebase_outcome.category, - rebase_detail=rebase_outcome.detail, - ) - - # Step 4: Reset local branch to remote. - # This handles: local behind remote (origin strictly ahead — nothing - # local to lose) and the prior-phase-failed local-ahead discard (the - # incomplete work is intentionally dropped). The already-in-sync case - # returns early above; the rev-list-failed and unreconciled-divergence - # cases now bail non-destructively before reaching here (#2979), so - # this reset never runs over un-provably-pushed committed work. - try: - result = subprocess.run( - [*git_base, "reset", "--hard", f"origin/{remote_branch}"], - capture_output=True, - text=True, - timeout=30, - check=False, - ) - if result.returncode != 0: - logger.warning( - "worktree_sync_outcome", - pipeline_id=pipeline_id, - branch=branch, - remote_branch=remote_branch, - case="reset_failed", - local_ahead=local_ahead, - remote_ahead=remote_ahead, - error=result.stderr.strip(), - ) - return WorktreeSyncOutcome(case="reset_failed") - logger.info( - "worktree_sync_outcome", - pipeline_id=pipeline_id, - branch=branch, - remote_branch=remote_branch, - case="reset_succeeded", - local_ahead=local_ahead, - remote_ahead=remote_ahead, - ) - return WorktreeSyncOutcome(case="reset_succeeded") - except Exception as sync_err: - logger.warning( - "worktree_sync_outcome", - pipeline_id=pipeline_id, - branch=branch, - remote_branch=remote_branch, - case="reset_failed", - local_ahead=local_ahead, - remote_ahead=remote_ahead, - error=str(sync_err), - ) - return WorktreeSyncOutcome(case="reset_failed") - - -class StalePipelineBranchError(RuntimeError): - """Raised when ``origin/<pipeline.branch>`` is behind base and the - rebase to bring it up to date hit a conflict. - - Phase-startup callers convert this into a FAILED pipeline with a - clear ``error`` so the operator knows to manually rebase or start - fresh — vastly preferable to silently producing a PR with 70+ - cherry-picked-variant commits buried in it (#2098). - """ - - -def _rebase_pipeline_branch_onto_base( - spawner: "ContainerSpawner", # noqa: UP037 - pipeline_id: str, - worktree_repo_path: Path, - pipeline_branch: str, - base_branch: str, - gateway_mode: Literal["public", "private"] = "public", -) -> None: - """Rebase a stale ``origin/<pipeline_branch>`` onto ``origin/<base_branch>``. - - When ``submit_task`` resumes a pipeline whose branch has been sitting - on the remote for days/weeks while ``main`` advanced, the existing - pipeline branch tip carries old-SHA copies of commits that have since - been rebased onto main. Without this helper, the first orchestrator - push hits non-fast-forward, the reconcile path rebases ``HEAD`` onto - the stale tip, and every downstream commit inherits 70+ stale-from- - main commits as ancestors — producing a final PR diff that buries - the actual feature work under contamination (#2098). - - This helper runs on the orchestrator-side worktree and treats it as - scratch space for the rebase: - - 1. Skip when ``pipeline_branch`` doesn't exist on the remote (fresh - run — there's nothing to rebase). - 2. Skip when ``origin/<pipeline_branch>`` is not behind - ``origin/<base_branch>`` (already up to date). - 3. Skip when ``HEAD`` is an ancestor of *neither* - ``origin/<pipeline_branch>`` *nor* ``origin/<base_branch>``. Two - real resume paths satisfy the ancestry check: - - (a) **Preserved worktree** (canonical #2098 case): the - orchestrator-side worktree was kept across a cancel/resubmit, - so ``HEAD`` carries state-file commits that were already - pushed to ``origin/<branch>``. ``HEAD`` is a strict ancestor - of ``origin/<branch>``. - (b) **Fresh worktree**: the worktree volume was wiped between - cancel and resubmit (e.g. orchestrator redeploy onto a fresh - PVC), so the gateway recreated it from ``origin/<base>``. - ``HEAD == origin/<base>`` is a (trivial) ancestor of - ``origin/<base>``; resetting to ``origin/<branch>`` discards - no unique commits because every base commit is preserved as - the rebase target. - - If neither ancestry holds, ``HEAD`` carries truly unpublished - work and we defer rather than overwrite it. - 4. Reset the worktree to ``origin/<pipeline_branch>``, ``git rebase - origin/<base_branch>``, and force-push the rebased tip. Git's - built-in cherry-pick-skip drops commits already content-equivalent - to ones on the new base. - 5. On conflict: abort the rebase, restore the worktree to - ``origin/<base_branch>``, and raise ``StalePipelineBranchError`` - so phase startup fails fast with an actionable error. - - Best-effort fetch+rev-list errors are logged and swallowed so a - transient gateway hiccup doesn't block pipeline startup; only a - rebase that *started* but couldn't finish raises. - """ - if not pipeline_branch or not base_branch or pipeline_branch == base_branch: - return - - git_base = [ - "git", - "-c", - "core.hooksPath=/dev/null", - "-c", - f"safe.directory={worktree_repo_path}", - "-C", - str(worktree_repo_path), - ] - - def _run_git( - args: list[str], - timeout: int, - ) -> subprocess.CompletedProcess[str] | None: - """Run a git command and convert ``TimeoutExpired`` / ``OSError`` - into a ``None`` return so callers can decide what to do. - - Mirrors the defensive pattern in ``_sync_worktree_with_remote``. - """ - try: - return subprocess.run( - [*git_base, *args], - capture_output=True, - text=True, - timeout=timeout, - check=False, - ) - except (subprocess.TimeoutExpired, OSError) as exc: - logger.warning( - "rebase-on-resume: git command failed to run", - pipeline_id=pipeline_id, - branch=pipeline_branch, - git_args=args, - error=str(exc), - ) - return None - - # Step 1: Fetch both refs through the gateway so we have current - # origin/<branch> and origin/<base> tips locally. fetch_worktree_branch - # already runs `git fetch origin` (no refspec) which updates all - # remote-tracking refs in one call. - fetch_ok = spawner.gateway.fetch_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(worktree_repo_path), - mode=gateway_mode, - ) - if not fetch_ok: - logger.warning( - "rebase-on-resume: fetch failed, skipping rebase check", - pipeline_id=pipeline_id, - branch=pipeline_branch, - ) - return - - # Step 2: Verify origin/<pipeline_branch> exists. Fresh pipelines - # haven't pushed yet, so there's nothing to rebase. - verify_branch = _run_git(["rev-parse", "--verify", f"origin/{pipeline_branch}"], timeout=10) - if verify_branch is None or verify_branch.returncode != 0: - return - - verify_base = _run_git(["rev-parse", "--verify", f"origin/{base_branch}"], timeout=10) - if verify_base is None or verify_base.returncode != 0: - logger.warning( - "rebase-on-resume: origin/<base_branch> not resolvable, skipping", - pipeline_id=pipeline_id, - branch=pipeline_branch, - base_branch=base_branch, - ) - return - - # Step 3: Is the pipeline branch actually behind base? If not, no-op. - behind = _run_git( - [ - "rev-list", - "--count", - f"origin/{pipeline_branch}..origin/{base_branch}", - ], - timeout=10, - ) - if behind is None or behind.returncode != 0: - logger.warning( - "rebase-on-resume: rev-list failed, skipping", - pipeline_id=pipeline_id, - branch=pipeline_branch, - stderr=(behind.stderr.strip() if behind is not None else None), - ) - return - try: - behind_count = int(behind.stdout.strip() or "0") - except ValueError: - behind_count = 0 - if behind_count == 0: - return - - # Step 4: Confirm reset-to-origin/<branch> is lossless before we - # overwrite HEAD. Three worktree states are handled: - # - # (a) Preserved-worktree resume (#2098 canonical): the orchestrator- - # side worktree was kept across a cancel/resubmit, so HEAD - # carries state-file commits that were already pushed to - # origin/<branch>. HEAD is a strict ancestor of - # origin/<branch> — resetting drops nothing. - # (b) Fresh-worktree resume: the worktree volume was wiped between - # cancel and resubmit (e.g. orchestrator redeploy onto a fresh - # PVC, manual cleanup), so the gateway recreated the worktree - # from origin/<base>. HEAD == origin/<base>; resetting to - # origin/<branch> discards no unique commits because every - # commit on origin/<base> is preserved as the rebase target. - # (c) Confused-HEAD resume (#2222): the worktree carries a local- - # only commit (e.g. a half-pushed statefiles commit) on top of - # a stale origin/<branch> tip — HEAD is on neither ref. The - # previous behaviour was to "defer to push-reconcile", but the - # reconcile path's _build_rebase_cmd fallback is the - # contamination producer in #2222. Recover by hard-resetting - # to origin/<base>: any local-only work is dropped (it would - # be re-created by agents on the next phase, vastly preferable - # to a contaminated PR). - def _head_on(ref: str) -> bool: - result = _run_git(["merge-base", "--is-ancestor", "HEAD", ref], timeout=10) - return result is not None and result.returncode == 0 - - if not (_head_on(f"origin/{pipeline_branch}") or _head_on(f"origin/{base_branch}")): - logger.warning( - "rebase-on-resume: HEAD on neither origin/<branch> nor origin/<base> — " - "resetting to origin/<base> to avoid push-reconcile contamination (#2222)", - pipeline_id=pipeline_id, - branch=pipeline_branch, - base_branch=base_branch, - behind_base=behind_count, - ) - # Step 4a: hard-reset to ``origin/<base>`` first. Note that step 5 - # immediately overwrites HEAD again with ``reset --hard - # origin/<branch>`` in the success path, so this reset's effect on - # HEAD is short-lived — its purpose is to act as a safe-state floor: - # if step 5 itself fails (network blip, ref vanishes), we leave the - # worktree on a known-good ref (``origin/<base>``) instead of the - # ambiguous pre-recovery state that prompted the rescue. Don't - # "simplify" by dropping this — the back-to-back hard resets are - # intentional. - recovery_reset = _run_git(["reset", "--hard", f"origin/{base_branch}"], timeout=30) - if recovery_reset is None or recovery_reset.returncode != 0: - logger.warning( - "rebase-on-resume: recovery reset to origin/<base> failed, skipping", - pipeline_id=pipeline_id, - branch=pipeline_branch, - base_branch=base_branch, - stderr=(recovery_reset.stderr.strip() if recovery_reset is not None else None), - ) - return - - logger.info( - "rebase-on-resume: pipeline branch is behind base, attempting rebase", - pipeline_id=pipeline_id, - branch=pipeline_branch, - base_branch=base_branch, - behind_base=behind_count, - ) - - # Step 5: Reset the worktree to the stale pipeline branch tip so we - # can rebase it onto current base. - reset_to_branch = _run_git(["reset", "--hard", f"origin/{pipeline_branch}"], timeout=30) - if reset_to_branch is None or reset_to_branch.returncode != 0: - logger.warning( - "rebase-on-resume: reset to pipeline branch failed, skipping", - pipeline_id=pipeline_id, - branch=pipeline_branch, - stderr=(reset_to_branch.stderr.strip() if reset_to_branch is not None else None), - ) - return - - # Step 6: Rebase onto current base. Plain ``git rebase - # origin/<base>`` — git's cherry-pick-skip drops content-equivalent - # commits already on base (the 70+ stale-variant commits in #2098). - rebase = _run_git(["rebase", f"origin/{base_branch}"], timeout=120) - if rebase is None or rebase.returncode != 0: - # Conflict, timeout, or other rebase failure. Abort the rebase, - # restore the worktree to origin/<base> so it isn't left mid- - # rebase for downstream callers, and raise so the operator gets - # an actionable error rather than a contaminated PR. ``rebase - # is None`` covers the timeout case where ``_run_git`` already - # logged the underlying exception. - _run_git(["rebase", "--abort"], timeout=30) - _run_git(["reset", "--hard", f"origin/{base_branch}"], timeout=30) - stderr_text = rebase.stderr.strip() if rebase is not None else "rebase command timed out" - logger.error( - "rebase-on-resume: rebase failed — aborting pipeline start", - pipeline_id=pipeline_id, - branch=pipeline_branch, - base_branch=base_branch, - stderr=stderr_text, - timed_out=rebase is None, - ) - raise StalePipelineBranchError( - f"origin/{pipeline_branch} is {behind_count} commits behind " - f"origin/{base_branch} and rebasing it failed. " - f"Manually rebase the branch (or delete it to start fresh) " - f"and resubmit. Stderr: {stderr_text}" - ) - - # Git emits ``warning: skipped previously applied commit <sha>`` on - # stderr for every cherry-pick-equivalent it dropped. Counting them - # gives operators a quick sanity check that the helper actually - # discarded the stale-from-main commits (vs. e.g. silently no-op'd). - skipped_via_rebase = sum( - 1 for line in rebase.stderr.splitlines() if "skipped previously applied commit" in line - ) - - # Step 7: Force-push the rebased branch. ``force=True`` is required - # because the rebased tip has different SHAs from origin/<branch>; - # this is exactly the contamination we just removed, so overwriting - # is the desired behavior. - push_result = spawner.gateway.push_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(worktree_repo_path), - branch=pipeline_branch, - mode=gateway_mode, - base_branch=base_branch, - force=True, - ) - if not push_result.ok: - # Restore HEAD to origin/<base> so the worktree is in a known - # state for downstream callers (the rebased commits stay in the - # local reflog if needed for recovery). - _run_git(["reset", "--hard", f"origin/{base_branch}"], timeout=30) - logger.error( - "rebase-on-resume: force-push of rebased branch failed", - pipeline_id=pipeline_id, - branch=pipeline_branch, - category=push_result.category, - detail=push_result.detail, - ) - raise StalePipelineBranchError( - f"Rebased {pipeline_branch} onto origin/{base_branch} but " - f"force-push to remote failed ({push_result.category}): " - f"{push_result.detail}" - ) - - # Re-fetch so origin/<pipeline_branch> reflects the rebased tip for - # any subsequent rev-parse in the same pipeline-start path. - spawner.gateway.fetch_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(worktree_repo_path), - mode=gateway_mode, - ) - logger.info( - "rebase-on-resume: rebased and force-pushed pipeline branch", - pipeline_id=pipeline_id, - branch=pipeline_branch, - base_branch=base_branch, - dropped_stale_commits=behind_count, - skipped_via_rebase=skipped_via_rebase, - ) - - -def _refresh_pipeline_branch_against_current_base( - spawner: "ContainerSpawner", # noqa: UP037 - pipeline_id: str, - worktree_repo_path: Path, - pipeline_branch: str, - base_branch: str, - gateway_mode: Literal["public", "private"] = "public", -) -> bool: - """Rebase ``origin/<pipeline_branch>`` onto current ``origin/<base_branch>`` - immediately before opening the PR (#2224 PR 2). - - ``_rebase_pipeline_branch_onto_base`` runs at the start of each - phase iteration to clean up stale branch state on resume. Nothing - between branch-cut and PR-open refreshes against - ``origin/<base_branch>``; if ``base_branch`` advances *during* the - PR phase's own work, the resulting PR is behind. This helper - closes that gap. - - The pipeline branch is the only ref this helper writes to: the - rebase replays pipeline-branch commits onto current - ``origin/<base_branch>``, and the force-push targets - ``pipeline_branch``. ``base_branch`` is read-only here — no - commits are ever pushed to it, even when it happens to be - ``main``. - - On success, force-pushes the rebased branch so the open PR's head - SHA reflects the rebase. - - On *any* failure (rebase conflict, push rejection, transient gateway - error), restores the worktree to ``origin/<pipeline_branch>``, - logs at WARNING, and returns ``False`` — the caller still opens the - PR against the un-rebased tip. This is intentional: a merge conflict - at PR-open time is better surfaced to the human reviewer than - swallowed by failing the whole pipeline. - - Returns ``True`` when a rebase was performed and pushed; ``False`` - when no rebase was needed or any step failed (in which case the - caller proceeds with the un-rebased tip). - """ - if not pipeline_branch or not base_branch or pipeline_branch == base_branch: - return False - - git_base = [ - "git", - "-c", - "core.hooksPath=/dev/null", - "-c", - f"safe.directory={worktree_repo_path}", - "-C", - str(worktree_repo_path), - ] - - def _run_git( - args: list[str], - timeout: int, - ) -> subprocess.CompletedProcess[str] | None: - try: - return subprocess.run( - [*git_base, *args], - capture_output=True, - text=True, - timeout=timeout, - check=False, - ) - except (subprocess.TimeoutExpired, OSError) as exc: - logger.warning( - "pr-open rebase: git command failed", - pipeline_id=pipeline_id, - branch=pipeline_branch, - git_args=args, - error=str(exc), - ) - return None - - # Step 1: Fetch fresh refs. Without this we'd rebase against the - # base tip we saw at branch-cut, defeating the whole point. - fetch_ok = spawner.gateway.fetch_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(worktree_repo_path), - mode=gateway_mode, - ) - if not fetch_ok: - logger.warning( - "pr-open rebase: fetch failed, skipping", - pipeline_id=pipeline_id, - branch=pipeline_branch, - ) - return False - - # Step 2: Verify both refs resolve. - verify_branch = _run_git(["rev-parse", "--verify", f"origin/{pipeline_branch}"], timeout=10) - if verify_branch is None or verify_branch.returncode != 0: - return False - verify_base = _run_git(["rev-parse", "--verify", f"origin/{base_branch}"], timeout=10) - if verify_base is None or verify_base.returncode != 0: - return False - - # Step 3: No-op when the branch is already up-to-date with current base - # (no commits behind). Saves a force-push when none is needed. - behind = _run_git( - [ - "rev-list", - "--count", - f"origin/{pipeline_branch}..origin/{base_branch}", - ], - timeout=10, - ) - if behind is None or behind.returncode != 0: - return False - try: - behind_count = int((behind.stdout or "0").strip() or "0") - except ValueError: - behind_count = 0 - if behind_count == 0: - return False - - # Step 4: Compute the merge-base so we can use the safe - # ``--onto <new_base> <upstream>`` form (HEAD is the implicit branch - # being rebased after the step-5 reset). The merge-base is the - # commit where the branch diverged from base_branch; using it as - # ``<upstream>`` tells git "replay only the commits unique to HEAD - # onto <new_base>" — no base-branch commits get absorbed into the - # branch's linear history, which is the contamination shape #2222 - # hardened against in the push-reconcile path. - merge_base_proc = _run_git( - ["merge-base", f"origin/{pipeline_branch}", f"origin/{base_branch}"], - timeout=15, - ) - if merge_base_proc is None or merge_base_proc.returncode != 0: - logger.warning( - "pr-open rebase: merge-base resolution failed, skipping", - pipeline_id=pipeline_id, - branch=pipeline_branch, - base_branch=base_branch, - stderr=(merge_base_proc.stderr.strip() if merge_base_proc is not None else None), - ) - return False - merge_base = (merge_base_proc.stdout or "").strip() - if not merge_base: - return False - - # Step 5: Reset the worktree to the current branch tip so the - # rebase operates on the right starting state. The reset target - # is ``origin/<pipeline_branch>`` — fresh from fetch in step 1 — - # so we are not rebasing on top of stale local state. - # - # Unlike ``_rebase_pipeline_branch_onto_base`` (resume-time helper), - # there is no ``_head_on(...)`` ancestry guard before this reset. - # That is intentional at this PR-open call site: any local-ahead - # commits at this point are orchestrator housekeeping commits that - # are orphan-by-design (the agents' work is already on - # ``origin/<branch>`` via the per-cycle push) so nothing needs to - # be preserved here. - reset = _run_git(["reset", "--hard", f"origin/{pipeline_branch}"], timeout=30) - if reset is None or reset.returncode != 0: - logger.warning( - "pr-open rebase: reset to origin/<branch> failed, skipping", - pipeline_id=pipeline_id, - branch=pipeline_branch, - stderr=(reset.stderr.strip() if reset is not None else None), - ) - return False - - # Step 6: Rebase using the safe ``--onto <new_base> <upstream>`` - # form. HEAD is the implicit branch being rebased (set by the - # step-5 reset above). The closest argv-shape prior art is - # ``gateway_client._build_rebase_cmd`` — that one rebases in the - # opposite direction (replay HEAD onto a stale branch tip) but uses - # the same explicit-upstream pattern that pins the replay range to - # ``<upstream>..HEAD`` and so sidesteps the bare-form contamination - # shape behind #2222. - rebase = _run_git( - [ - "rebase", - "--onto", - f"origin/{base_branch}", - merge_base, - ], - timeout=120, - ) - if rebase is None or rebase.returncode != 0: - # Conflict, timeout, or any failure: abort cleanly and restore - # to origin/<branch> so the caller can still open the PR - # against the un-rebased tip. Unlike the resume-time helper, - # we *don't* raise here — pipeline failure for a merge conflict - # at PR-open time is worse than a slightly-behind PR. - _run_git(["rebase", "--abort"], timeout=30) - _run_git(["reset", "--hard", f"origin/{pipeline_branch}"], timeout=30) - stderr_text = rebase.stderr.strip() if rebase is not None else "rebase command timed out" - logger.warning( - "pr-open rebase: rebase failed, opening PR against un-rebased tip", - pipeline_id=pipeline_id, - branch=pipeline_branch, - base_branch=base_branch, - behind_base=behind_count, - stderr=stderr_text, - timed_out=rebase is None, - ) - return False - - # Step 7: Force-push the rebased tip so origin/<branch> matches the - # SHAs the PR will be opened against. - push_result = spawner.gateway.push_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(worktree_repo_path), - branch=pipeline_branch, - mode=gateway_mode, - base_branch=base_branch, - force=True, - ) - if not push_result.ok: - # Best-effort restore so the worktree state is predictable for - # downstream callers; the PR still opens against the pre-rebase - # remote tip (which is what origin/<branch> still reflects). - _run_git(["reset", "--hard", f"origin/{pipeline_branch}"], timeout=30) - logger.warning( - "pr-open rebase: force-push of rebased branch failed, " - "opening PR against un-rebased remote tip", - pipeline_id=pipeline_id, - branch=pipeline_branch, - category=push_result.category, - detail=push_result.detail, - ) - return False - - # Re-fetch so origin/<branch> reflects the pushed tip locally. - spawner.gateway.fetch_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(worktree_repo_path), - mode=gateway_mode, - ) - logger.info( - "pr-open rebase: rebased and force-pushed pipeline branch", - pipeline_id=pipeline_id, - branch=pipeline_branch, - base_branch=base_branch, - behind_base_at_start=behind_count, - ) - return True - - -def _read_tree_head(git_base: list[str]) -> None: - """Refresh the index from HEAD without touching the working tree. - - Defends ``_commit_statefiles_to_worktree`` against a cross-worktree - branch-ref advance. When an agent runs the gateway-allowed recovery - primitive ``git update-ref refs/heads/<pipeline-branch> <sha>`` from - a sibling worktree (see ``sandbox/agent-config/rules/branch-recovery.md`` - and the detached-HEAD hint in ``gateway/gateway.py``), the shared local - branch ref advances out from under this worktree. ``update-ref`` does - not honour per-worktree locks, so this worktree's HEAD symref - silently jumps to the agent's commit while the index and working tree - stay at the prior state. Without this refresh, the stale index - reports every agent-pushed file as a *staged deletion* against HEAD, - and the subsequent ``git commit`` lands them as a real delete commit - (the symptom in #2626). ``read-tree HEAD`` repoints the index to - the new HEAD without touching the working tree; the immediately - following ``git add --force`` then stages only the orchestrator's - on-disk writes. - """ - subprocess.run( - [*git_base, "read-tree", "HEAD"], - capture_output=True, - text=True, - check=True, - timeout=30, - ) - - -def _restore_missing_state_files_from_head( - git_base: list[str], - worktree_path: Path, - pipeline_id: str | None = None, -) -> None: - """Materialize tracked ``.egg-state/`` files that HEAD has but disk doesn't. - - Companion to :func:`_read_tree_head`: the same cross-worktree - ``update-ref`` advance behind #2626 leaves the working tree stale - relative to the just-advanced HEAD. The #2626 fix protected the - *commit* (no delete-commit lands), but downstream readers go through - the working tree — :func:`_populate_contract_from_plan` reads the - plan draft via ``Path(...).read_text()``, which fails with the - natural ``PlanDraftMissingOnLocalError`` even though HEAD itself - carries the agent-pushed draft (the #2721 symptom; recovery in the - field was ``git checkout HEAD -- .egg-state/drafts/ - .egg-state/agent-outputs/``). - - ``git ls-files -z --deleted -- .egg-state/`` lists tracked files - that are missing on disk. ``-z`` switches the output to - NUL-separated raw bytes so paths with non-ASCII chars, newlines, or - quote chars survive parsing intact (with the default - ``core.quotePath=true`` the non-``-z`` form C-quote-encodes those - paths and ``splitlines()`` would misparse them). Must be called - AFTER :func:`_read_tree_head` so the index reflects HEAD; otherwise - a stale index can leave the delete-list incomplete. ``git checkout - HEAD --pathspec-from-file=- --pathspec-file-nul`` then restores each - missing path in both the index and the working tree (the index - reset is a no-op because read-tree HEAD already aligned it); piping - the NUL-separated list via stdin sidesteps any ARG_MAX limit on the - argv path even for pathological ``.egg-state/`` populations. - Confined to ``.egg-state/`` so the restoration cannot resurrect a - sibling-pipeline file the orchestrator deliberately removed - elsewhere in the tree. - - Fail-open: any subprocess error logs and returns silently — the - downstream populator still has its own missing-draft guard, so a - failure here cannot silently hide a true draft-missing case. - """ - try: - deleted = subprocess.run( - [*git_base, "ls-files", "-z", "--deleted", "--", ".egg-state/"], - capture_output=True, - check=False, - timeout=30, - ) - except (subprocess.TimeoutExpired, OSError) as ls_err: - logger.warning( - "_restore_missing_state_files_from_head: ls-files probe failed", - worktree_path=str(worktree_path), - pipeline_id=pipeline_id, - error=str(ls_err), - ) - return - if deleted.returncode != 0: - logger.warning( - "_restore_missing_state_files_from_head: ls-files probe failed", - worktree_path=str(worktree_path), - pipeline_id=pipeline_id, - returncode=deleted.returncode, - stderr=deleted.stderr.decode("utf-8", errors="replace").strip()[:200], - ) - return - missing_paths = [p for p in deleted.stdout.split(b"\0") if p] - if not missing_paths: - return - pathspec_stdin = b"\0".join(missing_paths) + b"\0" - try: - restore = subprocess.run( - [ - *git_base, - "checkout", - "HEAD", - "--pathspec-from-file=-", - "--pathspec-file-nul", - ], - input=pathspec_stdin, - capture_output=True, - check=False, - timeout=30, - ) - except (subprocess.TimeoutExpired, OSError) as checkout_err: - logger.warning( - "_restore_missing_state_files_from_head: checkout failed", - worktree_path=str(worktree_path), - pipeline_id=pipeline_id, - missing_count=len(missing_paths), - error=str(checkout_err), - ) - return - if restore.returncode != 0: - logger.warning( - "_restore_missing_state_files_from_head: checkout failed", - worktree_path=str(worktree_path), - pipeline_id=pipeline_id, - missing_count=len(missing_paths), - returncode=restore.returncode, - stderr=restore.stderr.decode("utf-8", errors="replace").strip()[:200], - ) - return - logger.info( - "_restore_missing_state_files_from_head: restored tracked-but-missing files", - worktree_path=str(worktree_path), - pipeline_id=pipeline_id, - restored_count=len(missing_paths), - restored_sample=[p.decode("utf-8", errors="replace") for p in missing_paths[:5]], - ) - - -def _commit_statefiles_to_worktree( - worktree_path: Path, - message: str, - pipeline_identifier: int | str | None = None, - *, - pipeline_id: str | None = None, -) -> bool: - """Stage and commit ``.egg-state/`` files in *worktree_path*. - - When *pipeline_identifier* is provided, only files whose names start - with the identifier (followed by ``.`` or ``-``) are staged. This - prevents concurrent pipelines from leaking each other's state files - into unrelated PRs (see #1390). - - Most ``.egg-state/`` files are prefixed with the issue number (drafts, - reviews, BRC history, agent-outputs), but contract files are keyed by - ``pipeline_id`` (e.g. ``issue-1759-v3.json``) and don't share the - issue-number prefix. When *pipeline_id* is provided alongside - *pipeline_identifier*, files matching either prefix are staged — this - closes the gap where plan-phase contract updates were written to disk - but never committed because the glob only saw the issue-number prefix - (see #1829). - - Falls back to staging the entire ``.egg-state/`` directory when both - *pipeline_identifier* and *pipeline_id* are ``None`` (backwards-compat). - - Any pre-existing staged changes in the worktree's index are discarded - on entry — the helper runs ``git read-tree HEAD`` before staging (see - :func:`_read_tree_head` for the cross-worktree-ref-advance defence - from #2626). Only files matching the pipeline scope and present on - disk are committed; callers must not pre-stage state they expect this - helper to preserve. - - The commit is idempotent (skips when nothing is staged). - Raises ``subprocess.CalledProcessError`` on git failure. - Call sites decide whether to abort or continue. - - Returns ``True`` when a commit was actually made, ``False`` when the - helper short-circuited (no .egg-state dir, no prefix match, or - nothing staged after add). Lets call sites skip a follow-up push - that would be a no-op fast-forward (#2548 review suggestion D). - """ - state_dir = worktree_path / ".egg-state" - logger.info( - "_commit_statefiles_to_worktree: entering", - worktree_path=str(worktree_path), - pipeline_identifier=str(pipeline_identifier), - pipeline_id=str(pipeline_id), - commit_message=message, - ) - if not state_dir.exists(): - logger.info( - "_commit_statefiles_to_worktree: no .egg-state directory — exiting", - worktree_path=str(worktree_path), - pipeline_identifier=str(pipeline_identifier), - pipeline_id=str(pipeline_id), - ) - return False # Nothing to commit yet - - git_base = [ - "git", - "-c", - "core.hooksPath=/dev/null", - "-c", - f"safe.directory={worktree_path}", - "-C", - str(worktree_path), - ] - - if pipeline_identifier is not None or pipeline_id is not None: - # Scope to files belonging to this pipeline only (#1390). - # Use prefix-anchored patterns with delimiter boundaries to avoid - # substring false positives (e.g. pipeline 4 matching pipeline 42). - # Union both prefixes so issue-number-prefixed files (drafts, - # reviews, BRC history) and pipeline-id-keyed files (contracts) - # are all staged (#1829). - prefixes: list[str] = [] - if pipeline_identifier is not None: - prefixes.append(str(pipeline_identifier)) - if pipeline_id is not None and pipeline_id not in prefixes: - prefixes.append(pipeline_id) - - matched_set: set[str] = set() - for pid in prefixes: - escaped = glob.escape(pid) - pattern_dot = str(state_dir / "**" / f"{escaped}.*") - pattern_dash = str(state_dir / "**" / f"{escaped}-*") - for f in glob.glob(pattern_dot, recursive=True) + glob.glob( - pattern_dash, recursive=True - ): - if Path(f).is_file(): - matched_set.add(f) - matched = sorted(matched_set) - logger.info( - "_commit_statefiles_to_worktree: glob match results", - pipeline_identifier=str(pipeline_identifier), - pipeline_id=str(pipeline_id), - prefixes=prefixes, - match_count=len(matched), - matched_paths=[str(Path(f).relative_to(worktree_path)) for f in matched[:20]], - truncated=len(matched) > 20, - ) - if not matched: - return False # No state files for this pipeline yet - - rel_paths = [str(Path(f).relative_to(worktree_path)) for f in matched] - _read_tree_head(git_base) - # Restore scope is intentionally broader than the staging glob: - # the helper operates over all of ``.egg-state/`` to maintain - # HEAD↔disk parity (so other readers — e.g. peer-artifact loads — - # see what HEAD says). Each pipeline has its own worktree, so - # broader scope cannot resurrect a sibling-pipeline file. - _restore_missing_state_files_from_head(git_base, worktree_path, pipeline_id) - subprocess.run( - [*git_base, "add", "--force", "--"] + rel_paths, - capture_output=True, - text=True, - check=True, - timeout=30, - ) - else: - _read_tree_head(git_base) - # Restore scope matches the staging scope here — both span all of - # ``.egg-state/`` — so the broader restore is trivially safe. - _restore_missing_state_files_from_head(git_base, worktree_path, pipeline_id) - subprocess.run( - [*git_base, "add", "--force", ".egg-state/"], - capture_output=True, - text=True, - check=True, - timeout=30, - ) - - # Only commit if there are staged changes (idempotent on re-runs). - # No pathspec: match the diff scope to the commit scope below so the - # early-out fires iff the commit would have nothing to write. A - # scoped diff (``-- .egg-state/``) paired with the unscoped commit - # below would short-circuit when only non-``.egg-state/`` content - # is staged, dropping that content on the floor instead of - # committing it. Nothing in this code path stages outside - # ``.egg-state/`` today, so this is belt-and-suspenders, but the - # two scopes must stay symmetric to keep the invariant local. - result = subprocess.run( - [*git_base, "diff", "--cached", "--quiet"], - capture_output=True, - text=True, - check=False, - timeout=30, - ) - if result.returncode == 0: - logger.info( - "_commit_statefiles_to_worktree: nothing staged — skipping commit", - pipeline_identifier=str(pipeline_identifier), - commit_message=message, - ) - return False # Nothing to commit - - logger.info( - "_commit_statefiles_to_worktree: staged changes detected — committing", - pipeline_identifier=str(pipeline_identifier), - commit_message=message, - ) - # Commit WITHOUT a trailing ``-- .egg-state/`` pathspec. ``git commit`` - # with a pathspec defaults to ``--only`` semantics, which auto-stages - # working-tree changes (including unstaged *deletions*) for the - # matching paths — i.e. ``git commit -- .egg-state/`` silently picks - # up files that disappeared from disk even though the explicit - # ``git add`` above only staged the on-disk hits from the glob. This - # surfaces as two distinct failure shapes that share the same - # mechanism (HEAD references a draft that is not on disk locally): - # #2625, where agents push drafts to ``origin/<branch>`` from their - # own worktrees so the orchestrator's local checkout sits at a HEAD - # containing files it never materialised; and #2626, where the - # agent-side ``git update-ref`` recovery (plumbing, no per-worktree - # branch lock) advances the shared pipeline-branch ref under the - # orchestrator's worktree, leaving every agent-pushed file looking - # like a staged deletion. In both cases the pathspec form turned a - # benign working-tree gap into a delete-commit against agent-pushed - # work. Without the pathspec, only the explicit ``git add`` staging - # above is committed. - subprocess.run( - [*git_base, "commit", "--no-verify", "-m", message], - capture_output=True, - text=True, - check=True, - timeout=30, - ) - logger.info( - "_commit_statefiles_to_worktree: commit succeeded", - pipeline_identifier=str(pipeline_identifier), - commit_message=message, - ) - return True - - -def persist_contract_statefiles( - pipeline_id: str, - worktree_path: Path, - message: str, - *, - pipeline: Pipeline | None = None, -) -> bool: - """Durably persist a contract decision write: commit + push to the work branch. - - Contract HITL decisions (``cq-N`` registrations and resolutions) are - written to the shared pipeline worktree's contract file with no git - commit; the file was only serialized to the work branch at slice/phase - checkpoints. Both phase-(re)start syncs — the gateway's worktree-reuse - reset and ``_sync_worktree_with_remote`` step 4 — run - ``git reset --hard origin/<work>``, so any decision write that had not - been committed AND pushed by then was silently reverted, letting the - bootstrap reconciler re-mint the same ``cq-N`` ids and clobber - just-resolved operator decisions (#3427). Committing and pushing at - write time makes the reset target already contain the decision. - - Best-effort by design: failures are logged and swallowed — the write is - still live on the worktree file and the next checkpoint commit retries. - Returns ``True`` only when the state was committed and pushed (or there - was nothing new to commit). - """ - try: - if pipeline is None: - _, pipeline = _resolve_pipeline(pipeline_id, get_repo_path()) - identifier = _pipeline_identifier(getattr(pipeline, "issue_number", None), pipeline_id) - committed = _commit_statefiles_to_worktree( - worktree_path, - message, - identifier, - pipeline_id=pipeline_id, - ) - if not committed: - return True # Nothing new on disk — already durable. - branch = getattr(pipeline, "branch", None) - if not branch: - logger.warning( - "Contract decision write committed but pipeline has no work " - "branch to push to; the commit is local-only and a worktree " - "reset may still discard it (#3427)", - pipeline_id=pipeline_id, - ) - return False - gateway_mode, _ = _compute_gateway_mode(pipeline) - _get_spawner().gateway.push_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(worktree_path), - branch=branch, - mode=gateway_mode, - base_branch=getattr(pipeline, "base_branch", None), - ) - logger.info( - "Contract decision write persisted to work branch (#3427)", - pipeline_id=pipeline_id, - branch=branch, - commit_message=message, - ) - return True - except Exception as persist_err: # noqa: BLE001 — best-effort durability - logger.warning( - "Failed to durably persist contract decision write; the decision " - "is live on the worktree file but will not survive a worktree " - "reset until the next checkpoint commit (#3427)", - pipeline_id=pipeline_id, - error=str(persist_err), - ) - return False - - -def _ensure_statefiles_on_branch( - worktree_repo_path: Path, - pipeline: Pipeline, -) -> bool: - """Verify the contract file exists in the worktree and re-create if missing. - - This is a safety net for short-flow pipelines where the initial contract - push may have failed or where subsequent pushes diverged. - - Returns True if the contract exists (or was successfully restored), - False if restoration failed. - """ - from egg_contracts.loader import contract_exists, create_contract, get_contract_path - - # Contract lookup uses pipeline.id directly (canonical key). - if contract_exists(pipeline.id, worktree_repo_path): - return True - - canonical_path = get_contract_path(pipeline.id, worktree_repo_path) - - logger.warning( - "Contract file missing from worktree — attempting restoration", - pipeline_id=pipeline.id, - expected_path=str(canonical_path), - ) - - try: - # Mirror the primary creation site: the composed task statement - # (identity anchor + submit description, #3163) lands on the - # restored contract too, for every entry path. - from egg_contracts.loader import compose_task_description - - issue_url = ( - f"https://github.com/{pipeline.repo}/issues/{pipeline.issue_number}" - if pipeline.issue_number is not None - else None - ) - task_description = compose_task_description( - description=pipeline.prompt, - issue_number=pipeline.issue_number, - issue_url=issue_url, - jira_ticket=pipeline.jira_ticket, - ) - if pipeline.issue_number is not None: - create_contract( - issue_number=pipeline.issue_number, - title=f"Issue #{pipeline.issue_number}", - url=issue_url or "", - pipeline_id=pipeline.id, - repo_root=worktree_repo_path, - task_description=task_description, - ) - else: - create_contract( - pipeline_id=pipeline.id, - title=(pipeline.prompt or "")[:100], - task_description=task_description, - repo_root=worktree_repo_path, - ) - - # Restore plan/analysis drafts from remote if missing locally. - # These were pushed during init but may be lost from the worktree - # after agent activity during the implement phase (#1454). - if pipeline.branch: - git_base = [ - "git", - "-c", - "core.hooksPath=/dev/null", - "-c", - f"safe.directory={worktree_repo_path}", - "-C", - str(worktree_repo_path), - ] - # Ensure remote-tracking ref is fresh before reading from it. - try: - subprocess.run( - [*git_base, "fetch", "origin", pipeline.branch], - capture_output=True, - text=True, - timeout=30, - check=False, - ) - except Exception: - pass # Best-effort; git show may still work with cached ref - for draft_phase in ("plan", "refine"): - draft_rel = _get_draft_path( - draft_phase, - issue_number=pipeline.issue_number, - pipeline_id=pipeline.id, - ) - if not draft_rel: - continue - draft_path = worktree_repo_path / draft_rel - if draft_path.exists(): - continue - try: - result = subprocess.run( - [*git_base, "show", f"origin/{pipeline.branch}:{draft_rel}"], - capture_output=True, - text=True, - timeout=15, - check=False, - ) - if result.returncode == 0 and result.stdout: - draft_path.parent.mkdir(parents=True, exist_ok=True) - draft_path.write_text(result.stdout, encoding="utf-8") - logger.info( - "Restored draft from remote branch", - pipeline_id=pipeline.id, - draft_path=draft_rel, - ) - except Exception as e: - logger.warning( - "Could not restore draft from remote", - pipeline_id=pipeline.id, - draft_path=draft_rel, - error=str(e), - ) - - # Final fallback: write plan/analysis from pipeline model if still - # missing after remote restoration attempt. This handles the case - # where the draft was never pushed to the remote (#1460). - for draft_phase, field_value in [("plan", pipeline.plan), ("refine", pipeline.analysis)]: - if not field_value: - continue - draft_rel = _get_draft_path( - draft_phase, - issue_number=pipeline.issue_number, - pipeline_id=pipeline.id, - ) - if not draft_rel: - continue - draft_path = worktree_repo_path / draft_rel - if draft_path.exists(): - continue - draft_path.parent.mkdir(parents=True, exist_ok=True) - draft_path.write_text(field_value, encoding="utf-8") - logger.info( - "Restored draft from pipeline model (remote unavailable)", - pipeline_id=pipeline.id, - draft_path=draft_rel, - ) - - # Re-populate tasks and PR metadata from plan draft if available. - # Without this, recreated contracts lose the planner-generated PR - # title/description and fall back to the generic pipeline ID title. - # See: https://github.com/jwbron/egg/issues/1432 - _restore_populate_result = _populate_contract_from_plan( - worktree_repo_path, - pipeline.id, - pipeline.mode.value if pipeline.mode else "issue", - pipeline.issue_number, - ) - # #2627 follow-up: this is a best-effort restoration path on a - # recreated contract — failure here is recoverable on later - # pipeline steps, so we just log the structured outcome. - if _restore_populate_result.outcome != PopulateOutcome.POPULATED: - logger.info( - "Restored-contract populate produced non-POPULATED outcome", - pipeline_id=pipeline.id, - outcome=_restore_populate_result.outcome.value, - ) - - # File-staging identifier still uses _pipeline_identifier convention. - identifier = _pipeline_identifier(pipeline.issue_number, pipeline.id) - _commit_statefiles_to_worktree( - worktree_repo_path, - f"Restore missing contract for {identifier}", - pipeline_identifier=identifier, - pipeline_id=pipeline.id, - ) - logger.info( - "Contract file restored successfully", - pipeline_id=pipeline.id, - ) - return True - except Exception as restore_err: - logger.error( - "Failed to restore contract file", - pipeline_id=pipeline.id, - error=str(restore_err), - ) - return False - - -def _detect_default_branch(worktree_repo_path: Path) -> str: - """Detect the remote's default branch from a worktree. - - Tries in order: - 1. origin/HEAD symbolic ref (most reliable) - 2. origin/main - 3. origin/master - 4. Fallback to "main" - - Returns: - The branch name (e.g., "main" or "master"), without the "origin/" prefix. - """ - # Try origin/HEAD symbolic ref - try: - result = subprocess.run( - ["git", "symbolic-ref", "refs/remotes/origin/HEAD", "--short"], - capture_output=True, - text=True, - cwd=str(worktree_repo_path), - timeout=10, - check=False, - ) - if result.returncode == 0 and result.stdout.strip(): - ref = result.stdout.strip() # e.g. "origin/main" - return ref.removeprefix("origin/") - except Exception: - pass - - # Try origin/main - try: - result = subprocess.run( - ["git", "rev-parse", "--verify", "origin/main"], - capture_output=True, - text=True, - cwd=str(worktree_repo_path), - timeout=10, - check=False, - ) - if result.returncode == 0: - return "main" - except Exception: - pass - - # Try origin/master - try: - result = subprocess.run( - ["git", "rev-parse", "--verify", "origin/master"], - capture_output=True, - text=True, - cwd=str(worktree_repo_path), - timeout=10, - check=False, - ) - if result.returncode == 0: - return "master" - except Exception: - pass - - logger.warning( - "Could not detect default branch, falling back to 'main'", - worktree_path=str(worktree_repo_path), - ) - return "main" - - -def _resolve_origin_ref(base_branch: str | None) -> str: - """Return ``origin/<branch>``, falling back to ``origin/main``. - - Centralises the ``f"origin/{base_branch}" if base_branch else "origin/main"`` - pattern so every orient-prompt / diff-command call site honours the - resolved base branch consistently. - """ - ref = (base_branch or "main").strip() or "main" - # Tolerate callers that already passed ``origin/<x>`` by mistake. - if ref.startswith("origin/"): - return ref - return f"origin/{ref}" - - -def _fetch_pr_state(pr_number: int, repo: str | None = None) -> dict[str, Any]: - """Fetch PR state, base/head refs, and fork-hint via ``gh pr view``. - - Returns a dict with keys ``state`` (str, e.g. "OPEN"/"MERGED"/"CLOSED"), - ``base_ref`` (str or None), ``head_ref`` (str or None), ``head_sha`` - (str or None), ``is_fork`` (bool), ``changed_files`` (int), and - ``head_repository_name_with_owner`` (str or None). Returns an empty - dict when ``gh`` is unavailable or the PR cannot be looked up. - """ - if pr_number is None: - return {} - fields = ( - "state,baseRefName,headRefName,headRefOid,isCrossRepository," - "changedFiles,headRepositoryOwner,headRepository" - ) - cmd = ["gh", "pr", "view", str(pr_number), "--json", fields] - if repo: - cmd.extend(["--repo", repo]) - try: - result = subprocess.run( - cmd, - capture_output=True, - text=True, - timeout=20, - check=False, - ) - except Exception as exc: # pragma: no cover - defensive - logger.warning( - "_fetch_pr_state: gh pr view raised", - pr_number=pr_number, - repo=repo, - error=str(exc), - ) - return {} - if result.returncode != 0: - logger.warning( - "_fetch_pr_state: gh pr view failed", - pr_number=pr_number, - repo=repo, - returncode=result.returncode, - stderr=result.stderr.strip()[:200], - ) - return {} - try: - data = json.loads(result.stdout) - except json.JSONDecodeError, ValueError: - return {} - - head_repo = data.get("headRepository") or {} - head_owner = data.get("headRepositoryOwner") or {} - head_repo_name = head_repo.get("name") if isinstance(head_repo, dict) else None - head_owner_login = head_owner.get("login") if isinstance(head_owner, dict) else None - head_repo_full = ( - f"{head_owner_login}/{head_repo_name}" if head_owner_login and head_repo_name else None - ) - return { - "state": data.get("state"), - "base_ref": data.get("baseRefName"), - "head_ref": data.get("headRefName"), - "head_sha": data.get("headRefOid"), - "is_fork": bool(data.get("isCrossRepository")), - "changed_files": data.get("changedFiles") or 0, - "head_repository_name_with_owner": head_repo_full, - } - - -BRC_HISTORY_TYPES = frozenset( - { - "CONSENSUS_PROPOSE", - "CONSENSUS_ACK", - "CONSENSUS_NACK", - "CONSENSUS_WITHDRAW", - "CONSENSUS_CONFIRMED", - "CONSENSUS_RE_REVIEW", - # In-cycle conditional-ACK obligation resolution (#2338). Captured - # in the BRC history file so the audit trail survives orchestrator - # teardown — closes the gap that resolution was previously only - # an in-memory event. - "CONSENSUS_OBLIGATION_RESOLVED", - "STATUS", - "HANDOFF", - "AGENT_FAILED", - "NUDGE", - "OVERSEER_ALERT", - # HEARTBEAT (issue #1897) — structured per-agent state messages. - "HEARTBEAT", - # QUESTION removed per issue #1897 Phase 7. The enum member - # remains for backward-compat until the tester updates - # test_brc_history / test_checkpoint fixtures; see - # MessageType.QUESTION. - } -) - -# Subset of BRC_HISTORY_TYPES that the orchestrator's CONSENSUS_* signal -# handlers tag with ``metadata['slice_id']`` for slice-aware implement -# pipelines (#2548). The implement-phase BRC writer treats a missing -# ``slice_id`` on these as a contract violation (drop with WARNING), -# while the remaining BRC_HISTORY_TYPES (HEARTBEAT, STATUS, HANDOFF, -# AGENT_FAILED, NUDGE, OVERSEER_ALERT) come from emitters that do not -# uniformly carry slice scope — those are routed to the unattributed -# sibling file rather than dropped, so the audit trail stays complete. -CONSENSUS_BRC_TYPES = frozenset( - { - "CONSENSUS_PROPOSE", - "CONSENSUS_ACK", - "CONSENSUS_NACK", - "CONSENSUS_WITHDRAW", - "CONSENSUS_CONFIRMED", - "CONSENSUS_RE_REVIEW", - "CONSENSUS_OBLIGATION_RESOLVED", - } -) - - -def _get_message_store(): - """Import and return the message store factory function, or None if unavailable.""" - try: - from message_store import get_message_store - except ImportError: - try: - from ..message_store import get_message_store # type: ignore[import-not-found] - except ImportError: - return None - return get_message_store - - -def _render_brc_history_markdown( - brc_messages: list[Any], - pipeline_id: str, - phase: str, - *, - slice_id: str | None = None, -) -> str: - """Render *brc_messages* as a chronological markdown log. - - The output shape mirrors the legacy aggregate file: a heading line, - a generated-timestamp footer, and one ``### [ts] role (TYPE): subject`` - section per message with a fenced YAML metadata block. - - ``Generated:`` is derived from the *latest* message timestamp (not - wall-clock time) so regenerating the file from the same message set - produces byte-identical output. This keeps the PR-phase safety-net - rewrite (:func:`_rewrite_brc_history_for_pr`) idempotent: when no new - BRC messages arrived between phase completion and PR creation, the - rewritten file matches the previous commit and the follow-up commit is - skipped by :func:`_commit_statefiles_to_worktree`. See #1714. - """ - message_timestamps = [m.timestamp for m in brc_messages if m.timestamp is not None] - if message_timestamps: - generated_str = max(message_timestamps).strftime("%Y-%m-%dT%H:%M:%SZ") - else: - generated_str = "unknown" - # The "unattributed" bucket is not a slice — it holds cross-cutting - # non-CONSENSUS messages that lack canonical slice scope (HEARTBEAT, - # OVERSEER_ALERT, AGENT_FAILED, …) routed to a sibling file so the - # audit trail stays complete. Rendering it as "Slice: unattributed" - # would mislead a reviewer who lands on the file via a link line — - # special-case the heading and metadata block instead. - is_unattributed = slice_id == "unattributed" - lines: list[str] = [] - if is_unattributed: - lines.append(f"# BRC Consensus History — {phase} phase, cross-cutting (unattributed)") - elif slice_id: - lines.append(f"# BRC Consensus History — {phase} phase, {slice_id}") - else: - lines.append(f"# BRC Consensus History — {phase} phase") - lines.append("") - lines.append(f"Generated: {generated_str}") - lines.append(f"Pipeline: {pipeline_id}") - if is_unattributed: - lines.append("Section: cross-cutting (unattributed)") - elif slice_id: - lines.append(f"Slice: {slice_id}") - lines.append("") - - for msg in brc_messages: - ts = msg.timestamp.strftime("%Y-%m-%dT%H:%M:%SZ") if msg.timestamp else "unknown" - # Include to_role for directed messages (not broadcast "all") - if msg.to_role and msg.to_role != "all": - header = ( - f"### [{ts}] {msg.from_role} → {msg.to_role} ({msg.message_type}): {msg.subject}" - ) - else: - header = f"### [{ts}] {msg.from_role} ({msg.message_type}): {msg.subject}" - lines.append(header) - if msg.body: - lines.append("") - lines.append(msg.body) - - # Emit a YAML metadata block with id, phase, and non-empty metadata - meta_block: dict[str, Any] = {} - if msg.id: - meta_block["id"] = msg.id - if msg.phase: - meta_block["phase"] = msg.phase - if msg.metadata: - meta_block["metadata"] = msg.metadata - if meta_block: - lines.append("") - lines.append("````yaml") - lines.append( - yaml.safe_dump(meta_block, sort_keys=False, default_flow_style=False).rstrip() - ) - lines.append("````") - lines.append("") - return "\n".join(lines) - - -def _write_brc_history_file( - worktree_path: Path, - pipeline_id: str, - phase: str, - identifier: int | str, - brc_messages: list[Any], - *, - slice_id: str | None = None, -) -> None: - """Render and persist the markdown + JSON companion files for one bucket. - - ``slice_id``, when provided, switches the on-disk filename from the - aggregate ``{identifier}-{phase}.{md,json}`` shape used by - refine/plan/pr to the per-slice ``{identifier}-{phase}-{slice_id}.{md,json}`` - shape used by implement (#2548 — hard switchover, no aggregate - implement file is produced). - """ - if slice_id: - stem = f"{identifier}-{phase}-{slice_id}" - else: - stem = f"{identifier}-{phase}" - - history_dir = worktree_path / ".egg-state" / "brc-history" - history_dir.mkdir(parents=True, exist_ok=True) - history_file = history_dir / f"{stem}.md" - - # Write the markdown history file - try: - history_file.write_text( - _render_brc_history_markdown( - brc_messages, - pipeline_id, - phase, - slice_id=slice_id, - ) - ) - except Exception as md_err: - logger.warning( - "Failed to write BRC history markdown file", - pipeline_id=pipeline_id, - phase=phase, - slice_id=slice_id, - error=str(md_err), - ) - - # Write a JSON companion artifact containing the full message dicts - json_file = history_dir / f"{stem}.json" - try: - json_data = [msg.to_dict() for msg in brc_messages] - json_file.write_text(json.dumps(json_data, indent=2, default=str)) - except Exception as json_err: - logger.warning( - "Failed to write BRC history JSON companion file", - pipeline_id=pipeline_id, - phase=phase, - slice_id=slice_id, - error=str(json_err), - ) - - logger.info( - "Wrote BRC history file", - pipeline_id=pipeline_id, - phase=phase, - slice_id=slice_id, - path=str(history_file), - message_count=len(brc_messages), - ) - - -def _write_brc_history( - worktree_path: Path, - pipeline_id: str, - phase: str, - identifier: int | str, - *, - write_per_slice: bool = True, -) -> None: - """Write BRC consensus message history for a phase to .egg-state. - - Retrieves BRC-related messages for the given phase from the message store - and writes them as a chronological markdown log to - ``.egg-state/brc-history/{identifier}-{phase}.md``. - - For the ``implement`` phase the writer auto-detects slice-aware vs - aggregate mode (#2548): - - * If at least one BRC message carries a canonical - ``metadata['slice_id']`` (validated against - ``SLICE_ID_PATTERN``), the writer partitions messages per-slice - and writes one file per slice as - ``{identifier}-implement-{slice_id}.{md,json}``. - Per-message attribution rules: - - - ``CONSENSUS_*`` messages without a canonical slice_id are - dropped with a single aggregate WARNING — the orchestrator's - CONSENSUS_* signal handlers tag every implement-phase write - under D4, so a missing slice_id is a contract violation. - - Other ``BRC_HISTORY_TYPES`` (HEARTBEAT, STATUS, HANDOFF, - AGENT_FAILED, NUDGE, OVERSEER_ALERT) come from emitters that - do not uniformly carry slice scope. When they lack a - canonical slice_id they are routed to a sibling - ``{identifier}-implement-unattributed.{md,json}`` file rather - than dropped, so the audit trail stays complete and reviewers - of any per-slice transcript can cross-reference. - - * If **no** BRC message carries a slice_id (non-slice pipelines), - the writer falls back to the aggregate - ``{identifier}-implement.{md,json}`` filename. - - No-ops gracefully when the message store is unavailable or contains no - BRC messages for the pipeline and phase. - - Args: - worktree_path: Path to the worktree repo directory - pipeline_id: The pipeline ID to retrieve messages for - phase: The pipeline phase name (e.g. "implement", "plan") - identifier: The pipeline identifier for file naming - write_per_slice: When False and ``phase == "implement"`` in a - slice-aware pipeline, skip writing the per-slice - ``{identifier}-implement-{slice_id}.{md,json}`` files. The - ``unattributed`` sibling and any non-slice aggregate file - are still written. Per-slice files are owned by their - slice's integration branch (committed by - :func:`_commit_slice_brc_history_to_integration_branch`); - duplicating them onto ``work`` causes add/add merge - conflicts when slice PRs target ``work`` (#2755). Default - ``True`` preserves the historical behavior for the slice - hook itself and for any out-of-tree callers. - """ - logger.info( - "_write_brc_history: entering", - pipeline_id=pipeline_id, - phase=phase, - identifier=str(identifier), - ) - - store_fn = _get_message_store() - if store_fn is None: - logger.info( - "_write_brc_history: early return — message store unavailable", - pipeline_id=pipeline_id, - phase=phase, - ) - return - - try: - store = store_fn() - messages = store.get_messages(pipeline_id, limit=10000) - except Exception as e: - logger.warning( - "_write_brc_history: early return — failed to retrieve messages", - pipeline_id=pipeline_id, - phase=phase, - error=str(e), - ) - return - - if not messages: - logger.info( - "_write_brc_history: early return — no messages in store", - pipeline_id=pipeline_id, - phase=phase, - ) - return - - brc_messages = [m for m in messages if m.message_type in BRC_HISTORY_TYPES and m.phase == phase] - if not brc_messages: - logger.info( - "_write_brc_history: early return — no BRC messages for phase", - pipeline_id=pipeline_id, - phase=phase, - total_messages=len(messages), - ) - return - - if phase == "implement": - # Implement-phase BRC messages are partitioned per-slice (#2548) - # for slice-aware pipelines (issue mode with `contract.slices`): - # the orchestrator's CONSENSUS_* signal handlers tag every - # implement-phase consensus message with `metadata['slice_id']`, - # and this writer buckets them into one transcript file per - # slice. Non-slice pipelines have no slice scope on any message, - # so they fall back to the aggregate - # `{identifier}-implement.{md,json}` filename. - # - # ``metadata['slice_id']`` is interpolated into the on-disk - # filename below, so this is a gateway-facing seam in the same - # sense as ``signals.py`` / the restart route / - # ``concurrent_executor`` branch builders: every value MUST be - # validated against the canonical ``SLICE_ID_PATTERN`` before - # use, otherwise an attacker-controlled metadata blob (any role - # can post arbitrary metadata via ``messages.py``) could smuggle - # path separators into the filename and write outside - # ``.egg-state/brc-history/``. See ``slice_id_validation.py`` - # for the invariant. ``SLICE_ID_PATTERN`` is already imported at - # module top (the same try/except sandbox-vs-orchestrator dual - # import that imports ``extract_slice_id``); no local re-import - # is needed. - - buckets: dict[str, list[Any]] = {} - # ``unattributed_consensus`` holds CONSENSUS_* messages that lack - # a canonical slice_id — those are a D4 contract violation and - # are dropped with a single aggregate WARNING. ``unattributed_other`` - # holds non-CONSENSUS BRC types (HEARTBEAT, STATUS, HANDOFF, - # AGENT_FAILED, NUDGE, OVERSEER_ALERT) whose emitters do not - # uniformly carry slice scope; those are written to the - # ``unattributed`` sibling file so the audit trail stays complete. - unattributed_consensus: list[Any] = [] - unattributed_other: list[Any] = [] - for msg in brc_messages: - # ``Message.metadata`` is a Pydantic dict[str, Any] field with a - # default_factory=dict (message_store.Message), so it is always a - # dict at this point — no need to guard with getattr/isinstance. - raw_slice_id = msg.metadata.get("slice_id") - if isinstance(raw_slice_id, str) and SLICE_ID_PATTERN.fullmatch(raw_slice_id): - buckets.setdefault(raw_slice_id, []).append(msg) - continue - if str(getattr(msg, "message_type", "")) in CONSENSUS_BRC_TYPES: - unattributed_consensus.append(msg) - else: - unattributed_other.append(msg) - - if not buckets: - # No slice-attributed messages anywhere — this is a non-slice - # pipeline (an implement-phase run that never spawned slice - # scopes). Fall back to the aggregate - # `{identifier}-implement.{md,json}` filename so we never - # silently drop the entire BRC stream when no per-slice - # bucketing is possible. See #2548 reviewer_code_holistic - # finding #3. - logger.info( - "_write_brc_history: no slice-attributed implement-phase " - "messages — writing aggregate file (non-slice pipeline)", - pipeline_id=pipeline_id, - phase=phase, - total_brc_messages=len(brc_messages), - ) - _write_brc_history_file( - worktree_path, - pipeline_id, - phase, - identifier, - brc_messages, - ) - return - - # Slice-aware pipeline: at least one canonical slice_id was - # observed. CONSENSUS_* messages that lack a canonical slice_id - # are a D4 hard-switchover contract violation — drop them with - # a loud aggregate WARNING (count + sample types) so an operator - # notices the asymmetry rather than silently shipping a thinned- - # out transcript. - if unattributed_consensus: - sample_types = sorted( - {str(getattr(m, "message_type", "")) for m in unattributed_consensus[:8]} - ) - logger.warning( - "_write_brc_history: dropped implement-phase CONSENSUS_* messages " - "without canonical metadata.slice_id (hard switchover, #2548)", - pipeline_id=pipeline_id, - phase=phase, - dropped_count=len(unattributed_consensus), - sample_message_types=sample_types, - attributed_count=sum(len(v) for v in buckets.values()), - ) - - # Non-CONSENSUS BRC types without a canonical slice_id come from - # emitters that do not uniformly attach slice scope (HealthMonitor - # nudges, overseer respawn alerts, AGENT_FAILED broadcasts, - # CLI-routed HANDOFF/NUDGE messages, etc.). Route them to a - # sibling ``{identifier}-implement-unattributed.{md,json}`` file - # so the audit trail stays complete — reviewers reading any - # per-slice transcript can cross-reference. See #2548 - # reviewer_code blocking finding. - if unattributed_other: - _write_brc_history_file( - worktree_path, - pipeline_id, - phase, - identifier, - unattributed_other, - slice_id="unattributed", - ) - - if not write_per_slice: - # Caller opted out of per-slice writes (#2755). The - # ``unattributed`` sibling has already been written above - # (when ``unattributed_other`` was non-empty); skip the - # per-slice bucket loop so we don't add files that the - # slice branches already own. See the docstring's - # ``write_per_slice`` arg for the merge-conflict rationale. - logger.info( - "_write_brc_history: skipping per-slice writes (write_per_slice=False)", - pipeline_id=pipeline_id, - phase=phase, - slice_bucket_count=len(buckets), - ) - return - - # Natural sort by the integer suffix so a 12-slice pipeline iterates - # `slice-1, slice-2, ..., slice-12` rather than the lexicographic - # `slice-1, slice-10, slice-11, slice-12, slice-2`. Every key is - # already SLICE_ID_PATTERN-validated (`^slice-[0-9]+$`) above, so the - # int() parse is total. - for slice_id, slice_msgs in sorted( - buckets.items(), key=lambda kv: int(kv[0].rsplit("-", 1)[1]) - ): - _write_brc_history_file( - worktree_path, - pipeline_id, - phase, - identifier, - slice_msgs, - slice_id=slice_id, - ) - return - - # Refine, plan, and pr phases continue to write the aggregate - # `{identifier}-{phase}.{md,json}` file — only implement is per-slice. - _write_brc_history_file( - worktree_path, - pipeline_id, - phase, - identifier, - brc_messages, - ) - - -def _rewrite_brc_history_for_pr( - worktree_path: Path, - pipeline_id: str, - pipeline_phases: dict, - identifier: int | str, -) -> None: - """Re-write BRC history for all completed phases before PR creation. - - Iterates ``pipeline_phases`` (a mapping of phase name → phase execution - objects with a ``.status`` attribute) and calls :func:`_write_brc_history` - for each phase whose status is ``PipelineStatus.COMPLETE``. - - Errors from individual phase writes are logged at warning level and - do not prevent other phases from being processed. - - After re-writing history files, commits the results via - :func:`_commit_statefiles_to_worktree`. Commit failures are also - logged and swallowed so the PR creation can proceed. - """ - completed_phases = [ - name for name, ex in pipeline_phases.items() if ex.status == PipelineStatus.COMPLETE - ] - logger.info( - "_rewrite_brc_history_for_pr: entering", - pipeline_id=pipeline_id, - total_phases=len(pipeline_phases), - completed_phase_count=len(completed_phases), - completed_phases=completed_phases, - ) - for phase_name, phase_exec in pipeline_phases.items(): - if phase_exec.status == PipelineStatus.COMPLETE: - try: - _write_brc_history( - worktree_path, - pipeline_id, - phase_name, - identifier, - # Per-slice implement-phase files are owned by each - # slice's integration branch (#2548 D2/D5); committing - # them onto ``work`` would re-introduce the add/add - # merge conflict from #2755. Only the aggregate / - # unattributed sibling lands on ``work``. - write_per_slice=False, - ) - except Exception as brc_err: - logger.warning( - "Failed to re-write BRC history for PR (continuing)", - pipeline_id=pipeline_id, - phase=phase_name, - error=str(brc_err), - ) - try: - _commit_statefiles_to_worktree( - worktree_path, - "Persist BRC history files for PR", - pipeline_identifier=identifier, - pipeline_id=pipeline_id, - ) - logger.info( - "_rewrite_brc_history_for_pr: commit step completed successfully", - pipeline_id=pipeline_id, - ) - except subprocess.CalledProcessError as git_err: - logger.warning( - "Failed to commit BRC history for PR (continuing)", - pipeline_id=pipeline_id, - error=str(git_err), - ) - logger.info( - "_rewrite_brc_history_for_pr: exiting", - pipeline_id=pipeline_id, - ) - - -def _resolve_pipeline_worktree_path(pipeline: Pipeline, fallback: Path) -> Path: - """Resolve the on-disk worktree path for *pipeline*. - - Prefers ``WORKTREE_BASE_DIR / pipeline.id / <repo_short>`` when it - exists (the same layout _run_pipeline materialises at spawn time; - see pipelines.py spawn block). Falls back to *fallback* — typically - the state store's ``repo_path`` — when no worktree is materialised. - """ - repo_short = pipeline.repo.split("/")[-1] if pipeline.repo else None - if repo_short: - candidate = WORKTREE_BASE_DIR / pipeline.id / repo_short - if candidate.exists(): - return candidate - pipeline_wt_dir = WORKTREE_BASE_DIR / pipeline.id - if pipeline_wt_dir.exists(): - # sorted() for deterministic selection when multiple subdirs exist - for sub in sorted(pipeline_wt_dir.iterdir()): - if sub.is_dir() and (sub / ".git").exists(): - return sub - return fallback - - -def _resolve_slice_gate_repo(slice_obj, pipeline: Pipeline) -> str | None: - """The repo every implement-phase gate for *slice_obj* is scoped to (#3393). - - Single source of truth for slice → gate-repo resolution (task-6-1): the - test gate, the reviewer diff base, the per-repo check/lint commands, and - the slice agent's cwd all key off this one accessor. It is exactly - :func:`models.resolve_slice_repo` — the slice's own ``repo`` when set, - else the pipeline's primary repo (so a repoless slice, or any slice in an - N=1 pipeline, scopes to the single/primary repo). Returns ``None`` only - for a genuinely repoless pipeline (test scaffolds with no repo at all). - """ - try: - from models import resolve_slice_repo # type: ignore[no-redef] - except ImportError: - from ..models import resolve_slice_repo # type: ignore[no-redef] - return resolve_slice_repo(slice_obj, pipeline) - - -def _resolve_slice_worktree_path( - pipeline: Pipeline, slice_repo: str | None, fallback: Path -) -> Path: - """Resolve the on-disk worktree path for a slice's repo (#3393 task-6-1). - - A multi-repo pipeline materialises one worktree per participating repo - under ``WORKTREE_BASE_DIR / pipeline.id / <repo_short>`` — the same - owner/repo-keyed layout as :func:`_resolve_pipeline_worktree_path`, one - directory per repo. Given a slice's resolved repo (``owner/name``), this - returns that repo's worktree when it exists on disk, else *fallback* - (the pipeline-primary worktree). For an N=1 pipeline the slice's repo IS - the primary, so ``slice_repo`` matches ``pipeline.repo`` and the answer - is byte-identical to the pipeline-primary worktree — callers therefore - only reach here for a genuine secondary-repo slice. - """ - repo_short = slice_repo.split("/")[-1] if slice_repo else None - if repo_short: - candidate = WORKTREE_BASE_DIR / pipeline.id / repo_short - if candidate.exists(): - return candidate - return fallback - - -def _persist_phase_brc_history( - pipeline: Pipeline, - store: StateStore, - phase: str, -) -> None: - """Persist BRC history for *phase* and commit it, best-effort. - - Mirrors the per-phase write+commit sequence that ``_run_pipeline`` - runs inline at phase completion, so external phase-transition paths - (the ``complete_phase`` / ``advance_phase`` REST+MCP handlers) do - not silently drop BRC transcripts when ``_clear_concurrent_state`` - wipes the message store. See #1827. - - Note: this commits but does **not** push. Callers must ensure a - push happens downstream — in ``advance_phase`` the spawned - ``_run_pipeline`` thread pushes the branch, carrying this commit - along; in a standalone ``complete_phase`` the caller is expected to - trigger a subsequent advance or push. - """ - worktree_path = _resolve_pipeline_worktree_path(pipeline, store.repo_path) - try: - _write_brc_history( - worktree_path, - pipeline.id, - phase, - _brc_history_identifier(pipeline), - # Per-slice implement-phase files are owned by the slice's - # integration branch (committed by - # :func:`_commit_slice_brc_history_to_integration_branch`); - # the work-branch worktree must not duplicate them, otherwise - # slice PRs targeting ``work`` hit add/add merge conflicts - # (#2755). The parameter is a no-op for non-implement phases. - write_per_slice=False, - ) - except Exception as brc_err: - logger.warning( - "Failed to persist BRC history before phase transition (continuing)", - pipeline_id=pipeline.id, - phase=phase, - error=str(brc_err), - ) - return - - try: - _commit_statefiles_to_worktree( - worktree_path, - f"Persist statefiles after {phase} phase", - pipeline_identifier=_pipeline_identifier(pipeline.issue_number, pipeline.id), - # Contract files are keyed by pipeline_id, not the issue-number - # prefix; without this the restart-time persist skipped the - # contract entirely (#1829 gap, observed in #3427). - pipeline_id=pipeline.id, - ) - except subprocess.CalledProcessError as git_err: - logger.warning( - "Failed to commit BRC history before phase transition (continuing)", - pipeline_id=pipeline.id, - phase=phase, - error=str(git_err), - ) - - -def _build_pre_merge_obligations_section( - pipeline_id: str, - contract_deferred_actions: list[Any] | None = None, -) -> str: - """Render the "Pre-merge Obligations" section from active conditional ACKs. - - Two sources, in order of preference: - - 1. ``contract_deferred_actions`` — ``DeferredAction`` objects (or legacy - strings) previously persisted to ``contract.pr.deferred_actions`` when - a human approved the conditional-ACK HITL gate (#2004). This is the - durable path: the tracker may have been torn down by the time PR - creation runs, and the contract survives. - 2. The live consensus tracker (#1998). Used when the contract has - no deferred_actions — either because the gate landed before - tracker teardown, or the gate was never required. - - The markdown composition (open vs. resolved sections, banner copy) - is delegated to :mod:`orchestrator.pr_obligations`. Pre-#2777 cq-6 - the slice-DAG terminal slice rendered the same section from this - shared shape; under cq-4 the obligations live solely on the - up-front context PR (``egg/<id>/work → main``) opened by - :func:`_open_context_pr_at_implement_start`, so only this - ``_auto_create_pr`` callsite renders them now. The shared shape - stays so a future caller (re-introducing per-slice obligation - rendering, etc.) has parity. - - Returns an empty string if neither source yields obligations, so - callers can unconditionally append the result to the PR body. - """ - try: - from pr_obligations import render_obligations_section_from_normalized - except ImportError: - from ..pr_obligations import ( # type: ignore[import-not-found,no-redef] - render_obligations_section_from_normalized, - ) - obligations = _collect_pre_merge_obligations(pipeline_id, contract_deferred_actions) - return render_obligations_section_from_normalized(obligations) - - -def _collect_pre_merge_obligations( - pipeline_id: str, - contract_deferred_actions: list[Any] | None, -) -> list[dict[str, str]]: - """Normalize obligations from contract or live tracker into a uniform shape. - - Returns a list of ``{reviewer, condition, resolved_in_diff}`` dicts. The - contract source takes precedence over the live tracker when present. - - .. note:: - - Under #2777 cq-4 obligations live on the up-front context PR - (``egg/<id>/work → main``) opened by - :func:`_open_context_pr_at_implement_start`, not on individual - slice PRs — so the slice-loop no longer calls this helper. The - pipeline-level tracker fallback survives because the - ``_auto_create_pr`` path that still calls this helper uses the - pipeline-level tracker; future re-introducers of per-slice - obligation rendering would need to thread a slice-keyed tracker - (see ``peer_consensus._tracker_key`` ⇒ - ``{pipeline_id}/{slice_id}``) through here. - """ - try: - from pr_obligations import normalize_deferred_actions - except ImportError: - from ..pr_obligations import ( # type: ignore[import-not-found,no-redef] - normalize_deferred_actions, - ) - normalized = normalize_deferred_actions(contract_deferred_actions) - if normalized: - return normalized - - # Tier 2 — live tracker (pre-#2004 path; kept so conditions still - # render if the HITL gate hasn't resolved yet, e.g. under force=true). - try: - from peer_consensus import get_peer_consensus_tracker - except ImportError: - from ..peer_consensus import get_peer_consensus_tracker # type: ignore[import-not-found] - tracker = get_peer_consensus_tracker(pipeline_id) - if tracker is None: - return [] - try: - conditions = tracker.get_pre_merge_conditions() - except Exception as e: # defensive — never block PR creation on this - logger.warning( - "Failed to read pre-merge conditions from tracker", - pipeline_id=pipeline_id, - error=str(e), - ) - return [] - - tracker_normalized: list[dict[str, str]] = [] - for c in conditions: - condition = str(c.get("condition", "")).strip() - if not condition: - continue - tracker_normalized.append( - { - "reviewer": str(c.get("reviewer", "") or "").strip(), - "condition": condition, - "resolved_in_diff": str(c.get("resolved_in_diff", "") or "").strip(), - } - ) - return tracker_normalized - - -def _build_brc_history_link_line( - worktree_repo_path: Path, - identifier: int | str | None, - link_base: str | None = None, -) -> str: - """Build a one-line pointer to the committed BRC history transcripts. - - Scans ``.egg-state/brc-history/`` for ``{identifier}-<phase>.md`` files - written by :func:`_write_brc_history` and returns a sentence linking - each phase's transcript, ordered by canonical execution order - (``refine`` → ``plan`` → ``implement`` → ``pr``; unknown names sorted - alphabetically after). - - ``link_base`` (#3115): when set (e.g. - ``https://github.com/<repo>/blob/<branch>``), links are rendered as - branch-qualified absolute URLs instead of the default ``./``-relative - form. GitHub resolves relative links in PR bodies against the repo's - default branch, where ``.egg-state/`` does not exist — so any caller - embedding this line in a PR body must pass ``link_base``. - - Returns an empty string when ``identifier`` is ``None`` or no - transcripts exist on disk. - """ - if identifier is None: - return "" - history_dir = worktree_repo_path / ".egg-state" / "brc-history" - if not history_dir.is_dir(): - return "" - prefix = f"{identifier}-" - phases: list[str] = [] - for path in history_dir.glob(f"{prefix}*.md"): - stem = path.stem - if stem.startswith(prefix): - phases.append(stem[len(prefix) :]) - if not phases: - return "" - - canonical = [p.value for p in PipelinePhase] - rank = {name: i for i, name in enumerate(canonical)} - - # Per-slice implement files (#2548) carry the stem - # ``implement-slice-{N}``; cluster them at the canonical ``implement`` - # rank so the rendered link order is - # ``refine → plan → implement[-slice-N] → implement-unattributed → - # pr`` instead of pushing the per-slice files past pr to the end of - # the list. Within the implement cluster, sort by the integer slice - # index so a 12-slice pipeline renders ``slice-1, slice-2, …, - # slice-12`` rather than the lexicographic ``slice-1, slice-10, - # slice-11, slice-12, slice-2``. The ``implement-unattributed`` - # sibling (cross-cutting non-CONSENSUS BRC types without slice scope, - # see ``_write_brc_history``) sorts after every per-slice file so a - # reviewer reads each slice transcript first, then the cross-cutting - # context. - def _sort_key(name: str) -> tuple[int, int, str]: - if name == "implement": - return (rank["implement"], -1, "") - if name == "implement-unattributed": - return (rank["implement"], 1 << 30, name) - if name.startswith("implement-slice-"): - try: - idx = int(name.rsplit("-", 1)[1]) - except ValueError: - idx = 1 << 30 # malformed → sort last within cluster - return (rank["implement"], idx, name) - return (rank.get(name, len(canonical)), 0, name) - - phases.sort(key=_sort_key) - - prefix_url = f"{link_base.rstrip('/')}/" if link_base else "./" - links = ", ".join( - f"[`{phase}`]({prefix_url}.egg-state/brc-history/{identifier}-{phase}.md)" - for phase in phases - ) - return f"_Per-phase BRC transcripts: {links}._" - - -def _compose_context_pr_body( - *, - contract, - pipeline, - worktree_repo_path: Path, - identifier: int | str, - context_repo: str | None = None, - sibling_context_prs: list[dict[str, Any]] | None = None, -) -> str: - """Compose the context-PR body from contract + pipeline state (#3115). - - Before #3115 the context PR's body was ``contract.pr.description`` - verbatim, which dropped ``test_plan`` / ``manual_steps`` on the - floor (the composer that rendered them died with the PR phase in - #2777 even though the plan preflight still requires both fields) - and linked to none of the pipeline artifacts the orchestrator - deterministically knows about. This helper restores the full shape: - - 1. The planner's ``description`` (narrative, verbatim). - 2. ``## Test Plan`` / ``## Manual Steps`` from the contract fields - (Title Case matches the global PR template and the slice PR's - inline-narrative branch in ``gateway_client.py``). - 3. A generated ``## Pipeline context`` footer: pipeline id, - originating issue, the slice table, and links to the refine - analysis draft, the plan draft, and the per-phase BRC - transcripts committed on the work branch. - - Artifact links are branch-qualified absolute URLs - (``https://github.com/<repo>/blob/<work-branch>/...``) — GitHub - resolves relative links in PR bodies against the default branch, - where ``.egg-state/`` does not exist. Draft links are only emitted - for files that exist in the worktree, so a pipeline that skipped - refine does not link a 404. - - Pure string composition over already-loaded state — no git or - gateway calls — so the opener's failure surface is unchanged. - """ - pr = contract.pr - sections: list[str] = [] - - # Soft-break unwrapping (#3122): the ``pr:`` block fields arrive as - # YAML block scalars hard-wrapped at ~75 chars, and GitHub renders - # every newline in a PR body as a line break — join the wraps back - # into paragraphs, leaving real markdown structure alone. - description = unwrap_soft_breaks(pr.description if pr else None).strip() - if description: - sections.append(description) - - test_plan = unwrap_soft_breaks(pr.test_plan if pr else None).strip() - if test_plan: - sections.append(f"## Test Plan\n\n{test_plan}") - - manual_steps = unwrap_soft_breaks(pr.manual_steps if pr else None).strip() - if manual_steps: - sections.append(f"## Manual Steps\n\n{manual_steps}") - - # Build the footer body first; only emit the ``## Pipeline context`` - # header when *more than* the bare pipeline-id line gets added (a - # single ``- Pipeline: <id>`` line under its own ``##`` header is - # noise — every reviewer can read that off the URL). - body_lines: list[str] = [f"- Pipeline: `{pipeline.id}`"] - has_meaningful_content = False - if pipeline.issue_number: - # Bare ``#N`` autolinks within the same repo, which is where - # the pipeline's originating issue lives. - body_lines.append(f"- Issue: #{pipeline.issue_number}") - has_meaningful_content = True - - # #3393 slice-4 / task-4-2: the repo this context PR lives in. A - # slice PR in this same repo cross-links as a bare ``#N`` autolink; - # a slice PR in a DIFFERENT repo of the pipeline must be qualified - # as ``owner/repo#N`` (a bare ``#N`` would resolve against the wrong - # repo). Defaults to the pipeline primary — the repo the up-front - # opener composes the primary context PR for. For an N=1 pipeline - # every slice resolves to the primary, so every link stays bare and - # the body is byte-identical to the single-repo shape. - this_context_repo = context_repo or getattr(pipeline, "primary_repo", None) or pipeline.repo - try: - from models import resolve_slice_repo # type: ignore[no-redef] - except ImportError: - from ..models import resolve_slice_repo # type: ignore[no-redef] - - slices = list(contract.slices or []) - if slices: - body_lines.append(f"- Slices ({len(slices)}):") - for s in slices: - name = " ".join((s.name or s.id).split()) - # Strip both ``slice-`` and the legacy ``phase-`` prefix — - # ``Slice.id`` still permits the latter (models.py) and - # ``_migrate_phases_to_slices`` only rewrites it on JSON - # load, so a directly-constructed Slice can still carry it. - number = s.id.removeprefix("slice-").removeprefix("phase-") - line = f" {number}. {name} (`{s.id}`)" - # Cross-link the stack (#3122): once the slice's PR is open - # its number is persisted on the contract and the run loop - # re-composes this body, so the entry gains a link. - if getattr(s, "pr_number", None): - s_repo = resolve_slice_repo(s, pipeline) - if s_repo and this_context_repo and s_repo != this_context_repo: - # Cross-repo sibling — repo-qualify so GitHub resolves - # the autolink to the right repo (#3393 slice-4). - line += f" — {s_repo}#{s.pr_number}" - else: - # Same-repo (or repo unknown): bare ``#N`` autolinks - # within the repo this context PR lives in. - line += f" — #{s.pr_number}" - body_lines.append(line) - has_meaningful_content = True - - link_base: str | None = None - if pipeline.repo and pipeline.branch: - link_base = f"https://github.com/{pipeline.repo}/blob/{pipeline.branch}" - - if link_base: - doc_links: list[str] = [] - for phase, label in (("refine", "Refine analysis"), ("plan", "Implementation plan")): - rel_path = _get_draft_path( - phase, issue_number=pipeline.issue_number, pipeline_id=pipeline.id - ) - if rel_path and (worktree_repo_path / rel_path).is_file(): - doc_links.append(f"[{label}]({link_base}/{rel_path})") - # Human-focused companion (the simplifier's ``*-human.md``), when present. - human_rel = _get_human_draft_path( - phase, issue_number=pipeline.issue_number, pipeline_id=pipeline.id - ) - if human_rel and (worktree_repo_path / human_rel).is_file(): - doc_links.append(f"[{label} (human summary)]({link_base}/{human_rel})") - if doc_links: - body_lines.append(f"- Docs: {', '.join(doc_links)}") - has_meaningful_content = True - brc_line = _build_brc_history_link_line(worktree_repo_path, identifier, link_base=link_base) - if brc_line: - body_lines.append("") - body_lines.append(brc_line) - has_meaningful_content = True - - if has_meaningful_content: - sections.append("\n".join(["## Pipeline context", "", *body_lines])) - - # #3393 slice-4 / task-4-2: cross-reference the pipeline's context - # PRs in OTHER repos. Rendered only for a multi-repo pipeline (the - # opener passes ``sibling_context_prs`` when it coordinates >1 - # repo); an N=1 pipeline passes ``None`` and this section is - # omitted, keeping the body byte-identical to the single-repo shape. - coord_lines: list[str] = [] - for ref in sibling_context_prs or []: - ref_repo = (ref.get("repo") or "").strip() - ref_number = ref.get("number") - if not ref_repo or not isinstance(ref_number, int) or isinstance(ref_number, bool): - continue - if ref_number < 1: - continue - # ``owner/repo#N`` autolinks cross-repo (a bare ``#N`` would - # resolve against the repo this body lives in). - coord_lines.append(f"- {ref_repo}#{ref_number}") - if coord_lines: - sections.append( - "\n".join( - [ - "## Coordinated repos", - "", - "This pipeline coordinates PRs across multiple repos (#3393):", - "", - *coord_lines, - ] - ) - ) - return "\n\n".join(sections) - - -def _persist_context_pr_number( - pipeline_id: str, - pr_number: int, - *, - worktree_repo_path: Path, - identifier: int | str, - pr_url: str | None = None, -) -> None: - """Persist context-PR linkage on both the contract and the pipeline (#2777). - - Single-purpose helper extracted so the new - :func:`_open_context_pr_at_implement_start` opener is not a - non-transactional state mutator. Wraps the contract write under - the existing per-pipeline state lock so concurrent advance_phase / - backstop callers serialise on the same lock instance the rest of - the orchestrator uses, then calls ``save_contract`` to atomically - rewrite ``.egg-state/contracts/...`` on disk. - - The helper is the SOLE writer of ``context_pr_number`` after - slice-2 (#2777, TASK-2-1) deleted the legacy - ``_persist_context_pr_linkage_on_contract``. It is called exactly - once per ``_open_context_pr_at_implement_start`` invocation, - immediately after either the ``gh pr list`` idempotency hit or the - successful ``gh pr create``. The same persistence write fires on - the idempotent path so a resume-from-orphaned-pipeline where the - contract lost ``context_pr_number`` mid-run still recovers (the - unit test in TASK-3-8 asserts this). - - In slice-2 (#2777 TASK-2-2 cross-reviewer NACK fix) the helper was - extended to ALSO write ``pipeline.pr_url`` and ``pipeline.pr_number`` - on the pipeline record. Three downstream consumers depend on these - pipeline-level fields: - - * :func:`_get_pr_info` at the pipeline-status endpoint - (``/api/v1/pipelines/<id>/status``) reports them. - * :meth:`PipelineToolHandler._make_pipeline_summary` (the MCP - ``get_pipeline_status`` tool) reports them. - * ``orchestrator.jira_reassess.pipelines_for_ticket_pr_url`` powers - the #1557 reverse-index in-flight detection that prevents the - Jira reassess sweep from re-mutating issues whose parent egg run - still has an open PR. - - Before this rewire the dedicated writer for the pipeline fields was - the deleted ``_finalize_pr_phase_failed`` (TASK-2-2 of #2777 - deleted it lock-step with the PR phase). Without the explicit - rewrite each of the three consumers above would silently report - ``None``. - - ``pr_url`` is synthesised from ``pipeline.repo`` + ``pr_number`` - when not supplied (the idempotent ``gh pr list`` hit only carries - the number; the create_pr path knows the URL directly from gh's - stdout). The synthesis mirrors GitHub's canonical PR URL shape and - keeps ``_get_pr_info``'s regex parse working unchanged. - - Persistence surface (egg-reviewer non-blocking #3): - - ``save_contract`` is a file-level atomic write — it rewrites - the contract on disk but does NOT commit-and-push it to the - worktree branch. The legacy - ``_persist_context_pr_linkage_on_contract`` (slice-2 deletes - it) wrapped the save in - ``_commit_statefiles_to_worktree`` + ``push_worktree_branch``; - the new opener intentionally does NOT, because the opener - runs at the canonical advance_phase REST site BEFORE - ``_spawn_pipeline_run_thread`` spawns the runner. That makes - the on-disk write durable for the runner's first read, but - the runner's ``_sync_worktree_with_remote`` has hard-reset - paths that can later wipe an uncommitted contract change. - Convergence is by the four runner-side backstops (slice-loop - entry, implement-entry backstop, ``_run_pipeline`` auto- - advance, HITL resume), which call the opener again — its - ``gh pr list`` idempotency hit re-persists ``context_pr_number`` - on disk after a reset. Across the full lifecycle the persisted - value converges; within a single advance_phase call the helper - is best-effort-on-disk-pending-runner-commit, not transactional. - - Raises: - ContextPrCreationError: when the contract cannot be loaded or - saved. Unlike the soft-fail legacy helper this propagates - so the caller surfaces a typed failure rather than leaving - the contract out-of-sync with GitHub. - """ - try: - from egg_contracts.loader import load_contract, save_contract - except ImportError as imp_err: - raise ContextPrCreationError( - "egg_contracts.loader unavailable while persisting context_pr_number", - reason="loader_unavailable", - cause=imp_err, - ) from imp_err - - try: - with get_pipeline_state_lock(pipeline_id): - contract_local = load_contract(identifier, worktree_repo_path) - if contract_local.pr is None: - # The contract MUST have a PR record by the time we - # reach the plan→implement boundary — populate writes - # it from the plan's ``pr:`` block. Missing PRMetadata - # here is a structural failure, not a persistence - # nuance; surface it loudly. - raise ContextPrCreationError( - "contract has no PRMetadata; cannot persist " - "context_pr_number (populate-from-plan must run first)", - reason="missing_pr_metadata", - ) - contract_local.pr.context_pr_number = pr_number - save_contract(contract_local, worktree_repo_path) - - # Pipeline-level mirror (#2777 cross-reviewer NACK fix). - # Load → mutate → save under the same lock so the contract - # write and pipeline write are atomic for downstream - # observers (status endpoint, MCP tool, jira_reassess). - # Pull the state store via the same lazy-import pattern the - # rest of pipelines.py uses; the soft-fail import shape is - # intentional so a stripped-down test harness that mocks - # only the contract loader does not crash here. - # ``get_state_store`` requires the repo path explicitly - # (state_store.py:1356); pass ``worktree_repo_path`` so the - # store resolves under the same root we just wrote the - # contract to. - try: - from state_store import get_state_store # type: ignore[no-redef] - except ImportError: - from ..state_store import get_state_store # type: ignore[no-redef] - store = get_state_store(worktree_repo_path) - try: - reloaded = store.load_pipeline(pipeline_id) - except Exception as pipe_load_err: # noqa: BLE001 - # Don't fail the whole opener because the pipeline - # mirror couldn't be loaded — the contract write - # already succeeded above. Log + continue so the - # context PR opens; the mirror will be re-applied - # on the next idempotent opener tick. - logger.warning( - "Context PR opener: could not mirror pipeline.pr_url / " - "pipeline.pr_number (continuing — contract write succeeded)", - pipeline_id=pipeline_id, - pr_number=pr_number, - error=str(pipe_load_err), - ) - return - mirror_url = pr_url - if mirror_url is None: - # Idempotent path (``gh pr list`` hit) only carries the - # number; synthesise the canonical PR URL from - # pipeline.repo + pr_number so all three consumers - # still see a populated ``pr_url`` string. Skip the - # synthesis when ``repo`` is unset (local-mode - # pipelines have no remote PR). - if reloaded.repo: - mirror_url = f"https://github.com/{reloaded.repo}/pull/{pr_number}" - reloaded.pr_number = pr_number - if mirror_url: - reloaded.pr_url = mirror_url - store.save_pipeline(reloaded) - except ContextPrCreationError: - raise - except Exception as save_err: # noqa: BLE001 - raise ContextPrCreationError( - f"failed to persist context_pr_number={pr_number}: {save_err}", - reason="save_failed", - cause=save_err, - ) from save_err - - -def _refresh_context_pr_body( - pipeline_id: str, - *, - pipeline: Any, - spawner: Any, - worktree_repo_path: Path, - identifier: int | str, - gateway_mode: str = "public", -) -> bool: - """Re-compose and push the context PR's body to GitHub (#3122). - - Called by the run loop after a slice PR opens and its number is - persisted on the contract, so the context PR's slice table gains a - link to each slice PR as the stack materialises - (:func:`_compose_context_pr_body` renders ``— #N`` for every slice - with a recorded ``pr_number``). - - The context PR body is machine-owned: the refresh fully regenerates - it from contract + pipeline state through the same composer the - opener used, clobbering any manual edits. Best-effort by design — - a body refresh is cosmetic, so every failure (contract load, - composition, gateway) logs a warning and returns ``False`` without - raising; no slice outcome may depend on it. - - **Concurrency contract**: the caller must hold - ``get_pipeline_state_lock(pipeline_id)`` for the entire load + - compose + push sequence — without it, two slices completing in the - same wave can interleave so the slice whose refresh lands later - clobbers a body that already included both links. Because no - later slice fires a refresh after the last one, the final slice's - ``— #N`` link would stay missing forever if the race fired on it. - Serializing inside the per-pipeline lock eliminates the race; the - sole production caller (``_run_implement_phase_slices``) already - holds it. - """ - if not pipeline.repo: - return False - - try: - from egg_contracts.loader import load_contract - - contract = load_contract(identifier, worktree_repo_path) - except Exception as load_err: # noqa: BLE001 - # Lazy import + contract load: ImportError, loader validation - # errors, OSError on the contract file read. - logger.warning( - "Context PR body refresh: contract load failed (skipping)", - pipeline_id=pipeline_id, - error=str(load_err), - ) - return False - - context_pr_number = ( - contract.pr.context_pr_number if contract.pr else None - ) or pipeline.pr_number - if not context_pr_number: - # No context PR to refresh — reachable on #3100-degraded - # contracts where the opener never persisted linkage. - return False - - try: - body = _compose_context_pr_body( - contract=contract, - pipeline=pipeline, - worktree_repo_path=worktree_repo_path, - identifier=identifier, - ) - except Exception as compose_err: # noqa: BLE001 - # Pure string composition over loaded state; a raise here is a - # programming error, but the cosmetic-refresh contract still - # holds — log and skip rather than fail the slice. - logger.warning( - "Context PR body refresh: composition failed (skipping)", - pipeline_id=pipeline_id, - pr_number=context_pr_number, - error=str(compose_err), - ) - return False - - return spawner.gateway.update_pr_body( - pipeline_id, - pipeline.repo, - pr_number=context_pr_number, - body=body, - issue_number=pipeline.issue_number, - # Attribute the action in the gateway audit log; matches - # sibling orchestrator-driven PR mutations (create_slice_pr, - # rebase_onto). - agent_role="orchestrator", - mode=gateway_mode, - ) - - -def _open_context_pr_at_implement_start( - pipeline_id: str, repo_path: Path | None = None -) -> int | None: - """Hard-required, idempotent up-front context PR opener (#2777, cq-4). - - Single up-front context-PR opener for the plan→implement boundary. - Replaces the soft-fail ``_maybe_open_base_pr_for_plan_to_implement`` - wrapper (deleted by slice-2 TASK-2-1 in #2777) that swallowed every - gateway failure with ``return None`` and the four retry-point call - sites it required. Under the new topology the context PR is - ``egg/<id>/work → main`` (rather than a dedicated - ``egg/<id>/context`` branch) and is opened ONCE at the plan→implement - transition; the slice stack cascades onto it. - - Behaviour: - - 1. Look up the pipeline + worktree from ``pipeline_id``. - 2. If the pipeline has neither ``repo`` nor ``base_branch`` set - (local mode), return ``None`` without raising — there is no - remote PR to open. This matches the legacy wrapper's silent-skip - behaviour for local pipelines so the new hard-required contract - does not regress in-house test pipelines. A ``repo`` with no - ``base_branch`` is the normal "auto-detect the default branch" - state (#3031), NOT a misconfiguration: the base is resolved via - :func:`_detect_default_branch` and used for the lookup + create. - A ``base_branch`` with no ``repo`` is a genuine misconfiguration - and raises ``ContextPrCreationError(reason="missing_repo")``. - 3. Otherwise call ``GatewayClient.lookup_open_pr(head, base)`` — the - same control-plane idempotency primitive ``create_slice_pr`` uses - — to find the open PR whose head is the pipeline's work branch and - whose base is the pipeline's base branch. The gateway runs the - narrow ``gh pr list --head --base --state open`` filter server-side - (launcher auth, ``/api/v1/gh/find_open_pr``), so both PR-idempotency - sites share one seam instead of this opener enumerating every open - PR and filtering client-side (#2934). On hit, persist the PR number - via :func:`_persist_context_pr_number` and return it (no - ``gh pr create`` invocation). - 4. On miss, read ``contract.pr.title`` and compose the body via - :func:`_compose_context_pr_body` (#3115) — the planner's - ``description`` plus rendered ``test_plan`` / ``manual_steps`` - and a generated pipeline-context footer (issue, slice table, - analysis/plan draft + BRC transcript links on the work branch). - Call ``GatewayClient.create_pr`` to open the PR, persist the PR - number, and return it. - - Raises: - ContextPrCreationError: on any of (a) pipeline lookup failure, - (b) contract load failure / missing PR metadata, - (c) an unexpected ``lookup_open_pr`` failure (the primitive - itself soft-fails a transient gateway/parse error to ``None``, - so this only fires on a programming error), (d) ``create_pr`` - failure, (e) persistence failure. NO soft-fail - ``return None`` for any of these — - the failure must reach the BRC NACK / 422 surface so the - operator sees the failure rather than silently stranding - the slice stack on ``/work``. The test in TASK-3-8 asserts - no swallow path exists. - - Returns: - Existing or newly-created PR number on the happy path, OR - ``None`` ONLY when the pipeline legitimately has no remote - (local mode). The two outcomes are disambiguated by inspecting - the pipeline's ``repo`` / ``base_branch`` ahead of the call; - the run-loop never needs to branch on ``None`` because - local-mode pipelines never reach the slice loop with remote - operations queued. - - Idempotency contract: - Calling the function twice for the same pipeline is safe — the - second call sees the already-open PR via ``lookup_open_pr`` and - re-persists the number through :func:`_persist_context_pr_number`. - No second ``create_pr`` invocation occurs. Tests in TASK-3-8 - verify this by asserting ``create_pr`` is called zero times on - the idempotent path AND ``_persist_context_pr_number`` IS - called with the existing PR number. - """ - # Step 1: resolve the pipeline + worktree path. ``get_state_store_for_pipeline`` - # handles the multi-repo case so the opener works the same way the - # legacy wrapper did from every call site. - try: - from routes import get_state_store_for_pipeline, resolve_worktree_path - except ImportError as imp_err: - raise ContextPrCreationError( - "routes helpers unavailable while resolving pipeline", - reason="routes_unavailable", - cause=imp_err, - ) from imp_err - - try: - store, pipeline = get_state_store_for_pipeline(pipeline_id, repo_path=repo_path) - except Exception as load_err: - raise ContextPrCreationError( - f"pipeline {pipeline_id!r} could not be loaded: {load_err}", - reason="pipeline_load_failed", - cause=load_err, - ) from load_err - - # Step 2: local-mode short-circuit + base-branch resolution. - # - # ``repo`` AND ``base_branch`` both empty ⇒ local mode (no remote PR - # to open); return ``None`` without raising. - # - # ``repo`` set but ``base_branch`` empty is the NORMAL state, not a - # misconfiguration: ``Pipeline.base_branch`` defaults to ``None`` - # ("auto-detected from repo's default branch") and the standard - # ``submit_task`` path never populates it, so essentially every - # remote pipeline reaches here with ``base_branch=None``. #2777 cq-4 - # collapsed the final ``work → main`` PR into this up-front opener - # but dropped the default-branch resolution the deleted PR phase did, - # making the opener the only ``base_branch`` consumer that hard- - # raised on ``None`` instead of resolving it — stranding every - # standard pipeline's slice stack on ``/work`` (#3031). Resolve it - # here the way every other consumer does - # (``base_branch or _detect_default_branch``) and thread the resolved - # value through both the idempotency lookup and ``create_pr``. - # - # A ``base_branch`` set with no ``repo`` IS a genuine - # misconfiguration (nothing to open a PR against); surface it as a - # typed error so the operator notices. - repo_set = bool(pipeline.repo) - base_set = bool(pipeline.base_branch) - if not repo_set and not base_set: - logger.info( - "Context PR opener: skipping local-mode pipeline (no repo, no base_branch)", - pipeline_id=pipeline_id, - ) - return None - if base_set and not repo_set: - raise ContextPrCreationError( - f"pipeline {pipeline_id!r} has a base_branch " - f"({pipeline.base_branch!r}) but no repo; cannot open a context " - "PR with no remote", - reason="missing_repo", - ) - - if not pipeline.branch: - # A remote pipeline without a configured work branch is a - # structural failure; raise so the operator notices instead of - # silently skipping (which would re-introduce the soft-fail - # behaviour cq-4 explicitly removes). - raise ContextPrCreationError( - f"pipeline {pipeline_id!r} has no branch set; cannot open context PR", - reason="missing_branch", - ) - - worktree_repo_path = resolve_worktree_path(pipeline_id, store.repo_path) - # Resolve ``base_branch=None`` to the repo's default branch (#3031). - # ``_detect_default_branch`` reads ``origin/HEAD`` from the worktree - # and falls back to ``main``/``master`` then the literal ``"main"``, - # so it never raises and always yields a concrete base ref for the - # lookup + create_pr calls below. - effective_base = pipeline.base_branch or _detect_default_branch(worktree_repo_path) - identifier = _pipeline_identifier(pipeline.issue_number, pipeline_id) - gateway_mode, _vis = _compute_gateway_mode(pipeline) - - # Step 3: idempotency pre-flight. Reuse the same control-plane - # ``lookup_open_pr(head, base)`` primitive the per-slice path - # (``create_slice_pr``) uses, so both PR-idempotency sites share the - # narrow server-side ``gh pr list --head --base`` filter on the - # launcher-auth route rather than this opener enumerating every open - # PR and filtering client-side (#2934). ``lookup_open_pr`` returns a - # clean ``int | None`` (the head/base discrimination and number - # coercion happen server-side + in the primitive), so the client-side - # match loop and the malformed-``number`` guard the old - # ``list_open_prs`` path needed are gone. The primitive soft-fails a - # transient gateway/parse error to ``None`` — matching the slice path, - # and safe because ``gh pr create`` would reject a duplicate - # ``head → base`` PR server-side anyway. The ``try`` is the opener's - # typed-error backstop for an unexpected raise (e.g. a misconfigured - # gateway client), preserving the cq-4 no-raw-exception contract. - spawner = _get_spawner() - try: - existing_pr_number = spawner.gateway.lookup_open_pr( - pipeline_id=pipeline_id, - repo=pipeline.repo, - head=pipeline.branch, - base=effective_base, - ) - except Exception as lookup_err: - raise ContextPrCreationError( - f"gateway lookup_open_pr failed for context-PR idempotency check: {lookup_err}", - reason="lookup_failed", - cause=lookup_err, - ) from lookup_err - - if existing_pr_number is not None: - # Idempotent path. Persist the number even though it MAY - # already be on the contract: the resume-from-orphaned-pipeline - # case (contract lost ``context_pr_number`` mid-run) recovers - # here. The TASK-3-8 unit test asserts the persistence call. - _persist_context_pr_number( - pipeline_id, - existing_pr_number, - worktree_repo_path=worktree_repo_path, - identifier=identifier, - ) - logger.info( - "Context PR opener: idempotent hit on existing PR (no create_pr call)", - pipeline_id=pipeline_id, - pr_number=existing_pr_number, - head=pipeline.branch, - base=effective_base, - ) - _maybe_open_secondary_context_prs( - pipeline_id, - pipeline=pipeline, - primary_pr_number=existing_pr_number, - work_branch=pipeline.branch, - worktree_repo_path=worktree_repo_path, - identifier=identifier, - gateway_mode=gateway_mode, - spawner=spawner, - ) - return existing_pr_number - - # Step 4: open a new context PR. Read title/description from the - # canonical ``contract.pr`` fields (populated from the plan's - # ``pr:`` block by ``_populate_contract_from_plan``). - try: - from egg_contracts.loader import load_contract - except ImportError as imp_err: - raise ContextPrCreationError( - "egg_contracts.loader unavailable while reading PR metadata", - reason="loader_unavailable", - cause=imp_err, - ) from imp_err - - try: - contract = load_contract(identifier, worktree_repo_path) - except Exception as load_err: - raise ContextPrCreationError( - f"failed to load contract for {identifier!r}: {load_err}", - reason="contract_load_failed", - cause=load_err, - ) from load_err - - if contract.pr is None or not (contract.pr.title or "").strip(): - raise ContextPrCreationError( - "contract.pr.title is missing or empty; cannot open context PR", - reason="missing_pr_metadata", - ) - pr_title = contract.pr.title.strip() - # #3115: render the full context-PR body (description + test plan + - # manual steps + generated pipeline-context footer) instead of the - # bare ``contract.pr.description``. - pr_body = _compose_context_pr_body( - contract=contract, - pipeline=pipeline, - worktree_repo_path=worktree_repo_path, - identifier=identifier, - ) - - try: - pr_url = spawner.gateway.create_pr( - pipeline_id=pipeline_id, - repo=pipeline.repo, - title=pr_title, - body=pr_body, - head=pipeline.branch, - base=effective_base, - issue_number=pipeline.issue_number, - mode=gateway_mode, - ) - except Exception as create_err: - raise ContextPrCreationError( - f"gateway create_pr failed for context PR: {create_err}", - reason="gateway_error", - cause=create_err, - ) from create_err - - if not pr_url: - raise ContextPrCreationError( - "gateway create_pr returned no URL; cannot derive context PR number", - reason="gateway_no_url", - ) - - # Extract the PR number from the URL — gh prints - # ``https://github.com/<owner>/<repo>/pull/<N>`` on stdout. - # Use a trailing-boundary pattern (end-of-string OR a non-digit - # path/query separator) so that a hypothetical - # ``/pull/12345/files`` or ``/pull/12345?diff=split`` URL still - # parses correctly but a digit-suffixed slug like - # ``/pulled-files/12345`` cannot smuggle a wrong number through - # (reviewer_concurrency non-blocking #2 hardening). - match = re.search(r"/pull/(\d+)(?:[/?#]|$)", pr_url) - if not match: - raise ContextPrCreationError( - f"could not parse PR number from create_pr URL: {pr_url!r}", - reason="gateway_bad_url", - ) - try: - new_pr_number = int(match.group(1)) - except (TypeError, ValueError) as parse_err: - raise ContextPrCreationError( - f"could not coerce PR number from create_pr URL: {pr_url!r}", - reason="gateway_bad_url", - cause=parse_err, - ) from parse_err - - _persist_context_pr_number( - pipeline_id, - new_pr_number, - worktree_repo_path=worktree_repo_path, - identifier=identifier, - pr_url=pr_url, - ) - - logger.info( - "Context PR opener: opened new PR at plan→implement boundary (#2777)", - pipeline_id=pipeline_id, - pr_number=new_pr_number, - head=pipeline.branch, - base=effective_base, - url=pr_url, - ) - _maybe_open_secondary_context_prs( - pipeline_id, - pipeline=pipeline, - primary_pr_number=new_pr_number, - work_branch=pipeline.branch, - worktree_repo_path=worktree_repo_path, - identifier=identifier, - gateway_mode=gateway_mode, - spawner=spawner, - ) - return new_pr_number - - -def _repos_with_slices(contract, pipeline) -> list[str]: - """Repos that own ≥1 slice — the lazy-per-repo participation set (#3393, slice-4). - - A repo *participates* (gets its own ``egg/<id>/work`` branch + context - PR) iff at least one slice resolves to it via - :func:`models.resolve_slice_repo`. The result is ordered by - ``pipeline.repos`` and de-duplicated; a submitted repo that ends up - owning no slices is excluded (operator ruling #1). For an N=1 pipeline - this returns the single repo. This is the invariant the context-PR - opener's per-repo iteration honours (task-4-2). - """ - try: - from models import resolve_slice_repo # type: ignore[no-redef] - except ImportError: - from ..models import resolve_slice_repo # type: ignore[no-redef] - - slices = getattr(contract, "slices", None) or [] - owning = {resolve_slice_repo(s, pipeline) for s in slices} - return [spec.repo for spec in (pipeline.repos or []) if spec.repo in owning] - - -def _maybe_open_secondary_context_prs( - pipeline_id: str, - *, - pipeline: Any, - primary_pr_number: int, - work_branch: str | None, - worktree_repo_path: Path, - identifier: int | str, - gateway_mode: str, - spawner: Any, -) -> None: - """Guarded, never-raising entry to the lazy per-repo context opener (#3393). - - No-op unless the pipeline coordinates more than one repo, so the N=1 - single-repo path in :func:`_open_context_pr_at_implement_start` - performs zero extra work (no contract load, no gateway calls) and is - byte-for-byte unchanged. Requires a resolvable primary repo + work - branch; both are guaranteed set on the multi-repo remote path that - reaches here (the opener already returned for local-mode pipelines). - """ - if len(getattr(pipeline, "repos", None) or []) <= 1: - return - primary_repo = pipeline.primary_repo - if not primary_repo or not work_branch: - return - try: - _open_secondary_context_prs( - pipeline_id, - pipeline=pipeline, - primary_repo=primary_repo, - primary_pr_number=primary_pr_number, - work_branch=work_branch, - worktree_repo_path=worktree_repo_path, - identifier=identifier, - gateway_mode=gateway_mode, - spawner=spawner, - ) - except Exception as sec_err: # noqa: BLE001 - logger.warning( - "Lazy per-repo context PRs raised (continuing — primary context PR unaffected) (#3393)", - pipeline_id=pipeline_id, - error=str(sec_err), - ) - - -def _open_secondary_context_prs( - pipeline_id: str, - *, - pipeline: Any, - primary_repo: str, - primary_pr_number: int, - work_branch: str, - worktree_repo_path: Path, - identifier: int | str, - gateway_mode: str, - spawner: Any, -) -> dict[str, int]: - """Open the lazy per-repo context PRs for a multi-repo pipeline (#3393, slice-4 / task-4-2). - - :func:`_open_context_pr_at_implement_start` opens the PRIMARY repo's - context PR (``egg/<id>/work → base``) exactly as it always has. This - helper adds the *other* repos: it iterates the set of repos that own - ≥1 slice (via ``resolve_slice_repo`` over the contract's slices), - drops the primary, and for each remaining repo opens that repo's own - ``egg/<id>/work`` context PR (same branch naming, per repo). A - submitted repo with NO slices is skipped — lazy-per-repo, operator - ruling #1. Every opened context PR (primary + secondaries) then has - its body refreshed to cross-reference the sibling context PRs in the - other repos (``## Coordinated repos``). - - It is only invoked when ``len(pipeline.repos) > 1``; for an N=1 - pipeline the caller never reaches here, so the single-repo path is - byte-for-byte unchanged. - - Prerequisite / current limit (honest scope note): opening a context - PR in a secondary repo requires that repo's ``egg/<id>/work`` branch - to exist on its remote, which in turn needs a secondary-repo worktree - to push it. Threading the full repo set into worktree CREATION was - explicitly deferred by slice-3 (the worktree map is owner/repo-keyed - and list-shaped, but only the primary repo is materialised today), so - until that later wiring lands the secondary ``create_pr`` will - typically fail on a missing head branch. This helper therefore: - - * uses the launcher-auth ``lookup_open_pr`` idempotency primitive - (which works per-repo with no worktree) to ADOPT an already-open - secondary context PR, and - * ATTEMPTS ``create_pr`` otherwise, soft-failing (log, continue) so a - missing secondary branch never strands the pipeline. - - The iteration + cross-referencing structure is therefore complete and - forward-compatible: once secondary-repo worktree/branch creation is - wired, secondary context PRs open with no further change here. - - Every failure is caught and logged; the helper never raises. Returns - the ``{repo: pr_number}`` map of context PRs known after the pass - (always including the primary), for logging / tests. - """ - opened: dict[str, int] = {primary_repo: primary_pr_number} - - try: - from egg_contracts.loader import load_contract - except ImportError: - logger.warning( - "Secondary context PRs: egg_contracts.loader unavailable (skipping) (#3393)", - pipeline_id=pipeline_id, - ) - return opened - - try: - contract = load_contract(identifier, worktree_repo_path) - except Exception as load_err: # noqa: BLE001 - logger.warning( - "Secondary context PRs: contract load failed (skipping) (#3393)", - pipeline_id=pipeline_id, - error=str(load_err), - ) - return opened - - # Repos owning ≥1 slice (ordered by ``pipeline.repos``), minus the - # primary — the lazy-per-repo participation set (task-4-2). - secondary_repos = [r for r in _repos_with_slices(contract, pipeline) if r != primary_repo] - - if not secondary_repos: - # Multi-repo pipeline whose slices all resolve to the primary - # (e.g. no slice pinned a secondary repo). Nothing lazy to open. - return opened - - base_by_repo = {spec.repo: spec.base_branch for spec in (pipeline.repos or [])} - context_pr_title = ( - contract.pr.title.strip() - if contract.pr and (contract.pr.title or "").strip() - else f"{identifier} context" - ) - - for repo in secondary_repos: - # ``base_branch=None`` ⇒ the repo's default branch. Without a - # secondary worktree we cannot run ``_detect_default_branch`` - # here, so fall back to ``main`` (the create call resolves the - # real default server-side when base is omitted anyway). - base = base_by_repo.get(repo) or "main" - try: - existing = spawner.gateway.lookup_open_pr( - pipeline_id=pipeline_id, - repo=repo, - head=work_branch, - base=base, - ) - if existing is not None: - opened[repo] = existing - logger.info( - "Secondary context PR: adopted existing PR (#3393)", - pipeline_id=pipeline_id, - repo=repo, - pr_number=existing, - ) - continue - - body = _compose_context_pr_body( - contract=contract, - pipeline=pipeline, - worktree_repo_path=worktree_repo_path, - identifier=identifier, - context_repo=repo, - sibling_context_prs=[ - {"repo": r, "number": n} for r, n in opened.items() if r != repo - ], - ) - pr_url = spawner.gateway.create_pr( - pipeline_id=pipeline_id, - repo=repo, - title=context_pr_title, - body=body, - head=work_branch, - base=base, - issue_number=pipeline.issue_number, - mode=gateway_mode, # type: ignore[arg-type] - ) - match = re.search(r"/pull/(\d+)(?:[/?#]|$)", pr_url or "") - if match: - opened[repo] = int(match.group(1)) - logger.info( - "Secondary context PR: opened new PR (#3393)", - pipeline_id=pipeline_id, - repo=repo, - pr_number=opened[repo], - head=work_branch, - base=base, - ) - else: - logger.warning( - "Secondary context PR: create returned no parseable URL (#3393)", - pipeline_id=pipeline_id, - repo=repo, - url=pr_url, - ) - except Exception as sec_err: # noqa: BLE001 - # Best-effort: a missing secondary ``egg/<id>/work`` branch - # (the deferred-worktree limit above) surfaces here as a - # gateway create failure. Log + continue so the primary - # context PR + slice stack are unaffected. - logger.warning( - "Secondary context PR deferred (continuing) — secondary-repo " - "work branch likely absent until secondary worktree creation " - "is wired (#3393)", - pipeline_id=pipeline_id, - repo=repo, - error=str(sec_err), - ) - - # Cross-reference pass: refresh every opened context PR body so each - # links the sibling context PRs in the other repos. Best-effort and - # cosmetic — a failed refresh never affects the slice stack. - if len(opened) > 1: - for repo, number in opened.items(): - try: - body = _compose_context_pr_body( - contract=contract, - pipeline=pipeline, - worktree_repo_path=worktree_repo_path, - identifier=identifier, - context_repo=repo, - sibling_context_prs=[ - {"repo": r, "number": n} for r, n in opened.items() if r != repo - ], - ) - spawner.gateway.update_pr_body( - pipeline_id=pipeline_id, - repo=repo, - pr_number=number, - body=body, - issue_number=pipeline.issue_number, - mode=gateway_mode, # type: ignore[arg-type] - ) - except Exception as refresh_err: # noqa: BLE001 - logger.warning( - "Coordinated-repos cross-reference refresh failed (continuing) (#3393)", - pipeline_id=pipeline_id, - repo=repo, - error=str(refresh_err), - ) - - return opened - - -def _is_slice_dag_mode(contract) -> bool: - """Return True when the contract represents a multi-slice DAG (#2777, cq-10). - - Dedupes the bare ``len(contract.slices) > 1`` recompute that - appears at the ``_run_implement_phase_slices`` entry and inside the - run loop's per-slice handling. A single helper means future changes - to "what counts as DAG mode" — e.g. treating a single slice with - explicit dependencies as DAG — only need to land in one place. - The third site under the deleted ``_should_skip_pr_phase_auto_pr`` - is gone since slice-2 of #2777 removed the PR phase. - - Returns False for ``None`` or a contract without a populated - ``slices`` list (monolithic / pre-populate phase pipelines). - """ - if contract is None: - return False - slices = getattr(contract, "slices", None) or [] - return len(slices) > 1 - - -def _resolve_slice_base_branch( - contract, - slice_id: str, - *, - pipeline_id: str, - pipeline_branch: str, - extant_branches: set[str] | None = None, - parent_branch_exists: Callable[[str], bool] | None = None, -) -> str: - """Return the parent branch for a slice's integration branch (#2777, cq-9). - - Replaces the deleted slice-1 resolver helper (removed by slice-2 - TASK-2-1) with a single resolver that handles both root and - non-root slices. - - Three-tier resolution (default — ``extant_branches is None``): - - 1. **Eager-persisted parent** (post-slice-4 TASK-4-2). If - ``parent_branch_at_creation`` is set on the slice record, - return it. This is the primary path post-slice-4 — slices - created after the eager persist landed always go through - this arm. - 2. **Dependency-derived parent, gated on parent existence - (#2928)**. For a non-root slice whose - ``parent_branch_at_creation`` is empty (the normal first-run - case), the stack target is its dependency parent's - integration branch ``{issue_branch}/{dependencies[0]}``. When - a ``parent_branch_exists`` callback is provided, the resolver - probes whether that parent branch is still present on origin: - - * parent branch **exists** → return the dependency-derived - parent. This is the correct target for both fresh slices - (whose own integration branch does not exist yet) and - legacy slices. - * parent branch **absent** → the parent slice's PR was merged - into ``work`` and its branch deleted by the cascade, so - ``work`` already contains the parent's commits. Fall back - to ``pipeline_branch``. - * probe **raises** → conservative default: assume the parent - exists and return the derived parent. Never silently swap a - real slice onto ``work`` because of a flaky gateway. - - This replaces the pre-#2928 merge-base check, which probed the - *slice's own* integration branch for a fork point and routed a - ``None`` result (no fork point) to ``pipeline_branch``. That - conflated a FRESH slice (integration branch not yet created — - the common first-run case) with a genuinely orphaned slice, - silently mis-basing fresh slices onto ``work`` whenever - ``work`` had advanced ahead of the parent (the wedge in - #2928). - 3. **Final fallback** to ``pipeline_branch`` (``egg/<id>/work``) - when (a) no eager-persisted parent, (b) the slice is a root - (no dependencies), OR (c) the slice's dependency parent branch - is absent from origin. Root-targeted branches are never - deleted by the cascade so this is always a safe terminal - candidate. - - **Orphan-reconciler mode (``extant_branches`` non-None)**: the - stacked-PR reconciler at ``orchestrator/stacked_pr_reconciler.py`` - needs the resolver to SKIP ancestors whose branches are no longer - on origin (the primary trigger for orphan reconciliation is "parent - branch was deleted by the cascade merge"). When ``extant_branches`` - is supplied, each candidate (including ``parent_branch_at_creation`` - and any walked ancestor) is filtered against the set; if no extant - candidate is found the resolver falls back to ``pipeline_branch`` - (which is always extant — root-targeted branches are never deleted - by the stacked-PR flow). - - Args: - contract: The pipeline contract (must carry ``slices``). - slice_id: The slice whose base branch to resolve. - pipeline_id: Used only for log diagnostics; the resolver does - NOT consult the state store. - pipeline_branch: The pipeline's work branch (``egg/<id>/work``). - Returned for root slices when no - ``parent_branch_at_creation`` is recorded, and as the - final fallback in orphan-reconciler mode and the - merge-base "no fork point" arm. - extant_branches: Optional set of branch names known to exist - on origin. When supplied, the resolver filters every - candidate (recorded parent + walked ancestors) against - this set and skips any that are absent. The reconciler - uses this to escape from the deleted parent branch up the - DAG until an extant ancestor is reached. - parent_branch_exists: Optional callback (#2928) used to - decide whether a non-root slice's dependency parent - branch is still on origin. When provided, the resolver - invokes ``parent_branch_exists(parent_branch)`` with the - dependency-derived parent branch name. ``True`` returns - the derived parent; ``False`` routes to - ``pipeline_branch`` (parent merged + cascade-deleted); a - raised exception is treated conservatively as ``True``. - The default ``_run_one_slice_inner`` caller wires this - against ``spawner.gateway.ls_remote_branch_strict`` — the - strict variant is required so a gateway / network / - policy failure RAISES into this resolver's ``try/except`` - instead of being collapsed to ``False`` (which would - silently route a real slice onto ``pipeline_branch`` on - any gateway flake — re-creating the #2928 wedge). The - stacked-PR reconciler leaves it ``None`` (it has already - verified extant branches via the ``extant_branches`` - set). - - Mutually exclusive with ``extant_branches`` in practice: - the production caller (``_run_one_slice_inner``) passes - only this gate, and the stacked-PR reconciler passes only - ``extant_branches``. If a future caller passed both, this - gate would short-circuit to ``pipeline_branch`` on a - ``False`` return BEFORE the ``extant_branches`` walk - could find an extant ancestor; callers that have already - built the extant set should leave this ``None``. - - Returns: - The branch name to use as the slice integration branch's - parent. Never an empty string. - - Raises: - ValueError: When the requested slice id is absent from the - contract — a structural bug that the slice loop's earlier - forest-validation step should have caught. - """ - slices = getattr(contract, "slices", None) or [] - slice_record = next((s for s in slices if s.id == slice_id), None) - if slice_record is None: - raise ValueError( - f"slice {slice_id!r} not present in contract for pipeline " - f"{pipeline_id!r}; available slices: " - f"{[s.id for s in slices]}" - ) - - def _extant(candidate: str) -> bool: - """True when ``candidate`` passes the orphan-reconciler filter. - - When ``extant_branches`` is None, every non-empty candidate - passes (the default resolver doesn't validate liveness). - """ - if not candidate: - return False - if extant_branches is None: - return True - return candidate in extant_branches - - # (1) Eager-persisted parent (post-slice-4 TASK-4-2). Treated as - # authoritative regardless of root-status: if the persist landed, - # it's the resolved parent — UNLESS the orphan-reconciler caller - # told us this branch was deleted on origin (extant_branches - # filter). - parent_recorded = getattr(slice_record, "parent_branch_at_creation", None) or "" - if parent_recorded and _extant(parent_recorded): - return parent_recorded - - # Build the slice-id → slice-record lookup once for the DAG walk - # below (used in both the default and orphan-reconciler modes). - slices_by_id = {s.id: s for s in slices} - - deps = getattr(slice_record, "dependencies", None) or [] - parent_slice_id = deps[0] if deps else None - - # (2) Root slice — under the new topology (cq-4), the context PR - # is ``egg/<id>/work → main`` so root slices stack directly on the - # work branch rather than a separate ``egg/<id>/context`` branch. - if parent_slice_id is None: - return pipeline_branch - - # (3) Non-root slice — derive from the first dependency. Mirrors - # the existing ``f"{issue_branch}/{parent_slice_id}"`` convention - # at the legacy slice-loop call site. - issue_branch = _slice_namespace_root(pipeline_branch) - derived_parent = f"{issue_branch}/{parent_slice_id}" - - # #2928: parent-existence gate. When eager-persist did not land - # (``parent_recorded`` empty above) AND a ``parent_branch_exists`` - # callback is provided, decide between the dependency-derived - # parent and ``pipeline_branch`` by probing whether the parent - # slice's integration branch is still on origin — NOT by probing - # the slice's own branch for a fork point. - # - # The pre-#2928 implementation computed - # ``merge_base(integration_branch, derived_parent)`` and routed a - # ``None`` result to ``pipeline_branch``. That conflated a FRESH - # slice (its integration branch is created *after* this resolver - # runs, so it has no fork point on the first run — the common - # case) with a genuinely orphaned slice, silently mis-basing - # fresh slices onto ``work`` whenever ``work`` had advanced ahead - # of the parent (e.g. a stray contract-state commit on ``work``). - # The correct discriminator is parent-branch existence: - # - # * parent exists → stack on it (fresh OR legacy slice). - # * parent absent → the parent PR merged into ``work`` and its - # branch was cascade-deleted, so ``work`` already contains the - # parent's commits → ``pipeline_branch`` is the right base. - # * probe raises → conservative: assume the parent exists and - # return the derived parent; never silently swap a real slice - # onto ``work`` because the gateway was flaky. - if parent_branch_exists is not None: - try: - exists = parent_branch_exists(derived_parent) - except Exception as probe_err: # noqa: BLE001 - logger.warning( - "parent_branch_exists probe raised; assuming parent " - "exists and returning dependency-derived parent (#2928)", - pipeline_id=pipeline_id, - slice_id=slice_id, - derived_parent=derived_parent, - error=str(probe_err), - ) - exists = True - if not exists: - logger.warning( - "Dependency-parent branch absent on origin; parent " - "appears merged into work — basing slice on pipeline " - "branch (#2928)", - pipeline_id=pipeline_id, - slice_id=slice_id, - derived_parent=derived_parent, - pipeline_branch=pipeline_branch, - ) - return pipeline_branch - - # Default mode (no extant filter): return the immediate parent - # branch synthesised from the slice DAG. This is the unchanged - # pre-extant-kwarg behaviour. - if extant_branches is None: - return f"{issue_branch}/{parent_slice_id}" - - # Orphan-reconciler mode: walk up the DAG via ``dependencies[0]`` - # until an extant ancestor branch is found. The forest constraint - # at ``shared/egg_contracts/models.py:341`` guarantees ≤1 parent - # per slice, so a single traversal pointer suffices. - cursor: str | None = parent_slice_id - while cursor: - candidate = f"{issue_branch}/{cursor}" - if _extant(candidate): - return candidate - cursor_slice = slices_by_id.get(cursor) - if cursor_slice is None: - break - next_deps = getattr(cursor_slice, "dependencies", None) or [] - cursor = next_deps[0] if next_deps else None - - # Every ancestor's branch has been deleted (cascading merge). Fall - # back to the pipeline branch — stable across the stacked-PR flow - # because root-targeted branches are never deleted by the cascade. - return pipeline_branch - - -def _lookup_peer_consensus_tracker_or_none(pipeline_id: str, slice_id: str | None) -> Any | None: - """Look up a per-slice PeerConsensusTracker; return None on import failure. - - Slice-4 TASK-4-4 helper. The bootstrap classifier needs to inspect - consensus state (``tracker.evaluate()['is_complete']``) for - IN_PROGRESS slices with commits on origin to differentiate case (2) - (consensus not reached → mark spawned) from case (3) (consensus - reached → mark COMPLETE so the slice-PR opener fires). This thin - wrapper centralises the lazy import + None-on-import-failure dance - so the classifier itself stays declarative and easily unit-tested. - """ - try: - from orchestrator.peer_consensus import ( - get_peer_consensus_tracker as _gpct, - ) - except ImportError: - try: - from peer_consensus import ( # type: ignore[no-redef] - get_peer_consensus_tracker as _gpct, - ) - except ImportError: - return None - try: - return _gpct(pipeline_id, slice_id=slice_id) - except Exception: # noqa: BLE001 - return None - - -def _slice_has_pending_decision(slice_id: str, decisions: list[Any]) -> bool: - """Return True iff the contract has any unresolved HITL decision. - - Slice-4 TASK-4-4 case (4) helper. The classifier treats a BLOCKED - slice with no pending decision as a state-machine anomaly (the - slice was waiting on a HITL that has since been resolved without - flipping the slice status forward). Surface that to the operator - via OVERSEER_ALERT. - - The contract's :class:`egg_contracts.models.Decision` does NOT - carry a structured ``slice_id`` tag — decisions are scoped by - phase (``decision.phase``) rather than by slice. The conservative - interpretation: any unresolved decision is *potentially* the - reason this slice is BLOCKED, so we return ``True`` (suppress the - "missing-HITL" alert) whenever the contract carries ANY unresolved - decision. The function only returns ``False`` when ZERO unresolved - decisions exist on the contract — at which point a BLOCKED slice - is provably unexplained and the overseer alert is warranted. - - Practically: this errs on the side of NOT alerting (suppressing - a real cross-slice mismatch in favour of skipping a spurious - alert), because alert noise during normal multi-slice HITL flows - is worse than a missed anomaly that the next bootstrap pass will - re-check anyway. - - ``slice_id`` is currently unused; kept in the signature so a - future contract schema bump that adds a structured slice tag can - use the existing call sites verbatim. - """ - del slice_id # contract decisions are not tagged by slice yet - for d in decisions: - if not getattr(d, "resolved", False): - return True - return False - - -def _classify_non_complete_slice( - *, - pipeline_id: str, - slice_obj: Any, - issue_branch: str, - pipeline_repo: Any, - worktree_repo_path: Path, - gateway: Any, - gateway_mode: Literal["public", "private"], - consensus_tracker_lookup: Callable[[str, str | None], Any | None], -) -> str: - """Classify a non-COMPLETE slice for Layer-C bootstrap reconciliation. - - Slice-4 TASK-4-4. Returns one of the five classification labels: - - * ``"fresh"`` — case (1) IN_PROGRESS/PENDING with no commits on - origin. No Layer-C action; the scheduler re-yields READY and - the run loop spawns fresh agents. - * ``"resume"`` — case (2) IN_PROGRESS with commits on origin and - consensus NOT reached. Caller calls - ``scheduler.mark_spawned(slice_id)`` so the run loop does NOT - respawn. - * ``"consensus_complete"`` — case (3) IN_PROGRESS with commits - and ``tracker.evaluate()['is_complete']`` True. Caller marks - the slice COMPLETE so the next loop iteration runs the slice-PR - opener via its idempotent pre-flight. - * ``"blocked"`` — case (4) BLOCKED slice (HITL pending). Caller - preserves status. If no pending HITL is found on the contract, - caller escalates via ``_escalate_blocked_slice_to_hitl`` - which writes a new ``Decision`` to the contract. - * ``"corrupt"`` — case (5) impossible status enum or - contradictory state combination (PENDING with commits, etc.). - Caller escalates via ``_escalate_corrupt_slice_to_hitl`` - which writes a new ``Decision`` to the contract. - - The classifier is intentionally a pure function modulo the - injected ``gateway`` probe + ``consensus_tracker_lookup`` — - unit tests in TASK-4-6 fake both. - """ - try: - from egg_contracts.models import SliceStatus - except ImportError: - return "corrupt" - - status = getattr(slice_obj, "status", None) - if status == SliceStatus.BLOCKED: - # Case 4 — caller (Layer-C loop) validates the HITL via - # ``_slice_has_pending_decision`` and escalates if absent. - # The classifier itself just reports the BLOCKED state. - return "blocked" - - if status not in (SliceStatus.PENDING, SliceStatus.IN_PROGRESS): - # Case 5 — unknown / corrupt status enum value. The - # SliceStatus StrEnum has exactly four members; any other - # value (None, a string that didn't deserialise to the enum, - # a future enum addition we don't recognise yet) is treated - # as corrupt rather than silently re-yielded as READY. - return "corrupt" - - # Probe the slice's integration branch for commits on origin. - integration_branch = f"{issue_branch}/{slice_obj.id}" - has_commits: bool - if pipeline_repo is None: - # Repoless pipelines (test scaffolds) — no origin to consult. - # Treat as no-commits → fresh, which mirrors the default - # scheduler behaviour. - has_commits = False - else: - try: - sha = gateway.get_remote_branch_sha( - pipeline_id, - str(worktree_repo_path), - f"refs/heads/{integration_branch}", - mode=gateway_mode, - ) - has_commits = sha is not None - except Exception as probe_err: # noqa: BLE001 - # Probe failure (gateway down, transient HTTP). Conservative - # default: treat as has_commits=False so the slice is - # re-yielded READY rather than silently mark-spawned with - # no agents alive. - # - # NOTE on asymmetry vs. ``_resolve_slice_base_branch`` - # (slice-4 TASK-4-3, ~line 10510): the resolver defaults - # the *opposite* direction — probe failure → "has fork - # point → derived parent" — because mis-routing onto - # ``pipeline_branch`` on a transient probe error would - # silently change a slice's stack target. Here in Layer C, - # a "fresh" mis-classification just causes the scheduler - # to re-yield the slice as READY (fresh-agent spawn, which - # then sync-then-fetches and continues correctly). The - # asymmetry is deliberate: each direction picks the safer - # default for its own caller. - logger.warning( - "Layer-C bootstrap probe raised; treating slice as fresh (slice-4 TASK-4-4)", - pipeline_id=pipeline_id, - slice_id=slice_obj.id, - error=str(probe_err), - ) - has_commits = False - - if not has_commits: - # PENDING-without-commits is the normal fresh slice case. - # IN_PROGRESS-without-commits means a crash between the - # eager-persist (TASK-4-2) and ``create_slice_integration_branch`` - # — also fresh from the scheduler's perspective. - return "fresh" - - if status == SliceStatus.PENDING and has_commits: - # Case 5 — PENDING with commits on origin is a state-machine - # impossibility (the eager-persist (TASK-4-2) flips PENDING → - # IN_PROGRESS in the same contract write that records the - # parent branch BEFORE any commits could land). Treat as - # corrupt. - return "corrupt" - - # IN_PROGRESS with commits — distinguish (2) vs (3) via the - # consensus tracker reconstructed by startup_reconciliation.py - # (slice-4 TASK-4-5). - tracker = consensus_tracker_lookup(pipeline_id, slice_obj.id) - consensus_complete = False - if tracker is not None: - try: - evaluation = tracker.evaluate() - consensus_complete = bool(evaluation.get("is_complete")) - except Exception as eval_err: # noqa: BLE001 - logger.warning( - "Layer-C bootstrap tracker.evaluate() raised; treating slice " - "as consensus-incomplete (slice-4 TASK-4-4)", - pipeline_id=pipeline_id, - slice_id=slice_obj.id, - error=str(eval_err), - ) - - return "consensus_complete" if consensus_complete else "resume" - - -def _escalate_layer_c_hitl( - *, - pipeline_id: str, - slice_id: str, - worktree_repo_path: Path, - current_phase: PipelinePhase | None, - question: str, -) -> None: - """Create an HITL Decision on the contract for a Layer-C anomaly (slice-4 TASK-4-4). - - Shared transport for case (4) blocked-without-HITL and case (5) - corrupt-status escalations. Per the plan task body — "escalate - via ``mcp__sdlc__register_open_question`` (do NOT silently - re-yield as READY — silent classification error is worse than - an operator pause)" — Layer C must create an unresolved - ``Decision`` on the contract that pauses the slice until the - operator picks an option, not just a message-bus broadcast. - - The caller supplies ``worktree_repo_path`` (the per-pipeline - worktree where the live contract lives — Layer C runs inside - ``_run_implement_phase_slices`` which already has it in scope) - and ``current_phase`` (the live pipeline phase, so a Decision - surfaces under the phase the operator is debugging rather than - a hard-coded literal). - - **Lock-nesting invariant (reviewer_code v2 blocker 3)**: the - caller MUST NOT already hold ``get_pipeline_state_lock`` for - this pipeline. Today the Layer-C dispatch loop in - ``_run_implement_phase_slices`` calls this helper at the - top-level slice-loop scope BEFORE any per-slice lock - acquisition (the eager-persist site at - ``_run_one_slice_inner`` is the only nested-lock contract - write today). The current lock IS an ``threading.RLock`` so - re-entry would not deadlock, but if a future refactor narrows - the lock to a plain ``Lock`` (e.g. for monitor visibility), - a Layer-C call from inside another lock-holding scope would - deadlock the entire bootstrap. - - Pattern mirrors ``_persist_hitl_decision`` (above) but loads the - contract from the per-pipeline worktree directly (the caller - has it in scope) so the decision lands on the live contract - that ``/sdlc`` reads. Best-effort: contract-load / save - failures are logged and swallowed (consistent with the rest of - Layer C). The decision is tagged with a ``context`` prefix so a - dispatch handler in - ``routes/decisions.py`` can route on a stable discriminator if - one is added in a follow-up. - """ - try: - from egg_contracts.decisions import ( - find_duplicate_open_question, - find_resolved_question, - next_cq_id, - ) - from egg_contracts.loader import load_contract, save_contract - from egg_contracts.models import Decision, DecisionOption, DecisionType - except ImportError: - try: - from orchestrator.egg_contracts.decisions import ( # type: ignore[no-redef] - find_duplicate_open_question, - find_resolved_question, - next_cq_id, - ) - from orchestrator.egg_contracts.loader import ( # type: ignore[no-redef] - load_contract, - save_contract, - ) - from orchestrator.egg_contracts.models import ( # type: ignore[no-redef] - Decision, - DecisionOption, - DecisionType, - ) - except ImportError: - logger.warning( - "Layer-C HITL escalation skipped: egg_contracts not importable (slice-4 TASK-4-4)", - pipeline_id=pipeline_id, - slice_id=slice_id, - ) - return - decision_id: str = "" - try: - with get_pipeline_state_lock(pipeline_id): - contract_local = load_contract(pipeline_id, worktree_repo_path) - existing_decisions = contract_local.decisions or [] - decision_phase = current_phase or PipelinePhase.IMPLEMENT - # Dedupe/carry-forward — parity with ``register_open_question`` - # (#3374/#3392). The Layer-C question text is deterministic per - # (case, slice, pipeline), so every bootstrap re-run after a - # ``restart_phase`` re-derives the identical question. Without - # this guard each re-run minted a fresh ``cq-N`` (or, against a - # reset-stale contract, re-minted an existing one), making the - # operator re-answer questions they had already answered (#3427). - duplicate = find_duplicate_open_question(existing_decisions, question, decision_phase) - if duplicate is not None: - logger.info( - "Layer-C HITL escalation adopted existing open decision (slice-4 TASK-4-4)", - pipeline_id=pipeline_id, - slice_id=slice_id, - decision_id=getattr(duplicate, "id", None), - ) - return - carried = find_resolved_question(existing_decisions, question, decision_phase) - if carried is not None: - logger.info( - "Layer-C HITL escalation skipped: identical question " - "already resolved by the operator (slice-4 TASK-4-4)", - pipeline_id=pipeline_id, - slice_id=slice_id, - decision_id=getattr(carried, "id", None), - resolution=str(getattr(carried, "resolution", None))[:200], - ) - return - # Use the canonical ``cq-N`` allocator from - # ``shared/egg_contracts/decisions.py``. Orchestrator-side - # HITL escalations write to the ``cq-N`` namespace; the - # pipeline-side bridge owns ``decision-N``. The split was - # introduced by #2616 to prevent the - # ``len(decisions)+1`` collision between the two - # allocators (see the docstring at - # ``shared/egg_contracts/decisions.py``). - decision_id = next_cq_id(contract_local.decisions) - options = [ - DecisionOption(id="opt-1", label="Mark slice complete and continue"), - DecisionOption(id="opt-2", label="Restart slice from scratch"), - DecisionOption(id="opt-3", label="Cancel pipeline for manual investigation"), - ] - # Use the live pipeline phase rather than a hard-coded - # ``PipelinePhase.IMPLEMENT`` — Layer C fires during - # bootstrap which can run before any phase walk, and - # future slice-DAG topologies may span phases. - # - # The ``or PipelinePhase.IMPLEMENT`` arm (folded into - # ``decision_phase`` above) is defensive: the - # ``Pipeline.current_phase`` field is non-Optional with a - # default at the schema layer (``models.py:1032``), so - # in-tree callers should always populate it. The fallback - # exists for future non-``Pipeline``-shaped callers (e.g. - # contract-only loads during cold-start reconciliation - # that may construct a lighter object) — *not* a known-bug - # papering exercise for current shapes. - contract_local.decisions.append( - Decision( - id=decision_id, - question=question, - type=DecisionType.HITL, - phase=decision_phase, - options=options, - ) - ) - save_contract(contract_local, worktree_repo_path) - logger.info( - "Layer-C HITL escalation persisted on contract (slice-4 TASK-4-4)", - pipeline_id=pipeline_id, - slice_id=slice_id, - decision_id=decision_id, - ) - # Durably land the new decision on the work branch so the next - # phase-(re)start worktree reset cannot revert it (#3427). - persist_contract_statefiles( - pipeline_id, - worktree_repo_path, - f"Persist Layer-C HITL escalation {decision_id} (#3427)", - ) - except Exception as escalate_err: # noqa: BLE001 - logger.warning( - "Layer-C HITL escalation failed (slice-4 TASK-4-4); slice will " - "remain in its current contract status", - pipeline_id=pipeline_id, - slice_id=slice_id, - error=str(escalate_err), - ) - - -def _escalate_corrupt_slice_to_hitl( - *, - pipeline_id: str, - slice_id: str, - worktree_repo_path: Path, - current_phase: PipelinePhase | None, -) -> None: - """Escalate a Layer-C case-5 corrupt-state slice to HITL (slice-4 TASK-4-4). - - Question text is prefixed with ``[#2777 slice-4 TASK-4-4 case 5]`` - so a future dispatch handler in ``routes/decisions.py`` can route - on the literal substring without a separate context field on the - contract-level ``Decision`` model. - """ - _escalate_layer_c_hitl( - pipeline_id=pipeline_id, - slice_id=slice_id, - worktree_repo_path=worktree_repo_path, - current_phase=current_phase, - question=( - f"[#2777 slice-4 TASK-4-4 case 5] Slice {slice_id} of pipeline " - f"{pipeline_id} has an impossible status enum value or state " - f"combination (e.g. status not in PENDING/IN_PROGRESS/COMPLETE/" - f"BLOCKED, or PENDING with commits on the integration branch). " - f"Bootstrap reconciliation cannot classify the slice safely. " - f"How should the orchestrator proceed?" - ), - ) - - -def _escalate_blocked_slice_to_hitl( - *, - pipeline_id: str, - slice_id: str, - reason: str, - worktree_repo_path: Path, - current_phase: PipelinePhase | None, -) -> None: - """Escalate a Layer-C case-4 blocked-without-HITL slice to HITL (slice-4 TASK-4-4). - - Question text is prefixed with ``[#2777 slice-4 TASK-4-4 case 4]`` - so a future dispatch handler in ``routes/decisions.py`` can route - on the literal substring without a separate context field on the - contract-level ``Decision`` model. - """ - _escalate_layer_c_hitl( - pipeline_id=pipeline_id, - slice_id=slice_id, - worktree_repo_path=worktree_repo_path, - current_phase=current_phase, - question=( - f"[#2777 slice-4 TASK-4-4 case 4] Slice {slice_id} of pipeline " - f"{pipeline_id} is in BLOCKED status, but no PENDING HITL " - f"decision was found on the contract that matches the slice. " - f"{reason}. How should the orchestrator proceed?" - ), - ) - - -# --- #3393 slice-5: cross-repo merge-sequencing HITL holds ------------------- -# Stable discriminator prefix on the cross-repo-hold Decision question so -# (a) the poll can idempotently detect an already-registered hold for a -# gate across reconciler ticks / orchestrator restarts, and (b) a future -# dispatch handler in ``routes/decisions.py`` can route on the literal -# substring without a separate context field on the contract Decision. -_CROSS_REPO_HOLD_MARKER_PREFIX = "[#3393 cross-repo-hold" - - -def _cross_repo_hold_marker(slice_id: str) -> str: - """Return the stable per-gate discriminator embedded in the hold question.""" - return f"{_CROSS_REPO_HOLD_MARKER_PREFIX} slice={slice_id}]" - - -_CROSS_REPO_HOLD_REASON_TEXT = { - "closed_unmerged": ( - "the upstream cross-repo PR was CLOSED without merging, so the " - "automated merge-state hold cannot auto-ready this slice's PR" - ), - "timeout": ( - "the upstream cross-repo PR did not merge within the poll bound, so " - "the automated merge-state hold timed out rather than leaving this " - "slice's PR draft indefinitely" - ), - "beyond_merge_state": ( - "the plan declared this cross-repo dependency a beyond-merge-state " - "condition (release/publish, version-pin, or cannot-continue block), " - "which is released by human decision, never automated detection" - ), -} - - -# The two operator-selectable options on a cross-repo hold Decision. The -# RELEASE option readies the PR; the KEEP option leaves it draft for manual -# handling. Kept as constants so the registration (options list) and the -# resolution reader agree on one shape. -_CROSS_REPO_HOLD_RELEASE_OPTION_ID = "opt-release" -_CROSS_REPO_HOLD_RELEASE_OPTION_LABEL = "Release the hold and mark the PR ready" -_CROSS_REPO_HOLD_KEEP_OPTION_ID = "opt-keep" -_CROSS_REPO_HOLD_KEEP_OPTION_LABEL = "Keep the PR held for manual handling" - - -def _cross_repo_hold_resolution(contract: Any, slice_id: str) -> str | None: - """Return the human's verdict on the cross-repo hold Decision for a slice. - - Scans the (freshly-loaded) contract for the Decision carrying this gate's - :func:`_cross_repo_hold_marker` and, when it is resolved, maps the - operator's SELECTED option to a gate verdict: - - * :data:`cross_repo_merge_gate.RELEASE` — the release option was chosen - (mark the PR ready), else - * :data:`cross_repo_merge_gate.KEEP` — the keep-held option was chosen, OR - the resolution is present but unrecognized (fail-safe: an ambiguous - resolution must NOT auto-ready — cq-1 "human owns the release"). - - Returns ``None`` when the Decision is absent or not yet resolved (keep - waiting). The stored ``Decision.resolution`` may be the option label, the - option id, or a ``{"action":"select","selected":<label>}`` envelope (the - SDLC HITL CLI shape), so we unwrap the envelope and match on both id and a - distinctive keyword. This is the release path that honours the operator's - choice rather than readying on the bare resolved-boolean - (reviewer_code_holistic v1 NACK). - """ - try: - from cross_repo_merge_gate import KEEP, RELEASE - except ImportError: - from ..cross_repo_merge_gate import KEEP, RELEASE # type: ignore[no-redef] - - marker = _cross_repo_hold_marker(slice_id) - decision = None - for d in getattr(contract, "decisions", None) or []: - if marker in (getattr(d, "question", "") or ""): - decision = d - break - if decision is None or not getattr(decision, "resolved", False): - return None - - raw = getattr(decision, "resolution", None) or "" - # Unwrap the ``{"action":"select","selected":<label>}`` envelope the SDLC - # HITL CLI sends (mirrors routes.decisions._normalize_choice_resolution), - # tolerating a bare string / non-JSON resolution unchanged. - selected = raw - try: - import json as _json - - payload = _json.loads(raw) - if isinstance(payload, dict) and payload.get("action") == "select": - sel = payload.get("selected") - if isinstance(sel, str): - selected = sel - except ValueError, TypeError: - pass - - text = selected.strip().lower() - # #3393 task-5-1/gap-2 (defends the operator's cq-1 fail-safe ruling): - # release ONLY on an EXACT match against the release option's id or - # label. The prior ``"release" in text`` substring check failed OPEN - # — a freeform "Other" resolution that merely CONTAINS the word - # "release" in a negating sense (e.g. "do NOT release yet") would have - # auto-readied a PR the human meant to keep held, a narrower - # reintroduction of the "keep-held is a lie" class reviewer_code_holistic - # NACK'd. Exact equality (after envelope-unwrap + strip + lower) keeps - # the designed path (selecting opt-release / its label) working while - # every ambiguous or negated value falls through to the KEEP fail-safe. - if text in ( - _CROSS_REPO_HOLD_RELEASE_OPTION_ID.lower(), - _CROSS_REPO_HOLD_RELEASE_OPTION_LABEL.lower(), - ): - return RELEASE - # Any other resolved value (the keep option, or an unrecognized/freeform - # string) keeps the PR held — never ready on an ambiguous selection. - return KEEP - - -def _register_cross_repo_hold( - *, - pipeline_id: str, - slice_id: str, - repo: str, - pr_number: int, - reason: str, - worktree_repo_path: Path, - current_phase: PipelinePhase | None, -) -> bool: - """Ensure a cross-repo merge-sequencing HITL hold exists on the contract. - - Idempotent: if a Decision carrying this gate's marker already exists - (pending OR resolved), no new Decision is created. Returns ``True`` - when a hold now exists for the gate (freshly registered or already - present), ``False`` only when registration could not be persisted — - the poll uses the return to decide whether the gate has been handed - off to the HITL release path. Modelled on :func:`_escalate_layer_c_hitl` - (loads the live contract from the per-pipeline worktree, allocates a - ``cq-N`` id, appends an unresolved HITL Decision, saves). The hold - surfaces on ``/status`` via the existing pending-decision collector. - """ - try: - from egg_contracts.decisions import next_cq_id - from egg_contracts.loader import load_contract, save_contract - from egg_contracts.models import Decision, DecisionOption, DecisionType - except ImportError: - try: - from orchestrator.egg_contracts.decisions import ( # type: ignore[no-redef] - next_cq_id, - ) - from orchestrator.egg_contracts.loader import ( # type: ignore[no-redef] - load_contract, - save_contract, - ) - from orchestrator.egg_contracts.models import ( # type: ignore[no-redef] - Decision, - DecisionOption, - DecisionType, - ) - except ImportError: - logger.warning( - "Cross-repo hold skipped: egg_contracts not importable (#3393)", - pipeline_id=pipeline_id, - slice_id=slice_id, - ) - return False - - marker = _cross_repo_hold_marker(slice_id) - reason_text = _CROSS_REPO_HOLD_REASON_TEXT.get(reason, reason) - try: - with get_pipeline_state_lock(pipeline_id): - contract_local = load_contract(pipeline_id, worktree_repo_path) - # Idempotent: a hold Decision for this gate already exists. - for d in contract_local.decisions or []: - if marker in (getattr(d, "question", "") or ""): - return True - decision_id = next_cq_id(contract_local.decisions) - question = ( - f"{marker} Slice {slice_id} of pipeline {pipeline_id} opened PR " - f"{repo}#{pr_number} as a draft behind a cross-repo dependency, " - f"but {reason_text}. Choose how the orchestrator should proceed: " - f"selecting '{_CROSS_REPO_HOLD_RELEASE_OPTION_LABEL}' marks the PR " - f"ready; selecting '{_CROSS_REPO_HOLD_KEEP_OPTION_LABEL}' leaves it " - f"draft for you to handle manually." - ) - options = [ - DecisionOption( - id=_CROSS_REPO_HOLD_RELEASE_OPTION_ID, - label=_CROSS_REPO_HOLD_RELEASE_OPTION_LABEL, - ), - DecisionOption( - id=_CROSS_REPO_HOLD_KEEP_OPTION_ID, - label=_CROSS_REPO_HOLD_KEEP_OPTION_LABEL, - ), - ] - contract_local.decisions.append( - Decision( - id=decision_id, - question=question, - type=DecisionType.HITL, - phase=current_phase or PipelinePhase.IMPLEMENT, - options=options, - ) - ) - save_contract(contract_local, worktree_repo_path) - logger.info( - "Registered cross-repo merge-sequencing HITL hold (#3393)", - pipeline_id=pipeline_id, - slice_id=slice_id, - repo=repo, - pr_number=pr_number, - reason=reason, - decision_id=decision_id, - ) - return True - except Exception as hold_err: # noqa: BLE001 - logger.warning( - "Cross-repo hold registration failed (#3393); PR stays draft, " - "poll will retry next tick", - pipeline_id=pipeline_id, - slice_id=slice_id, - reason=reason, - error=str(hold_err), - ) - return False - - -def _check_slice_evidence_reachability( - pipeline_id: str, - spawner: "ContainerSpawner", # noqa: UP037 - worktree_repo_path: Path, - slice_id: str, - integration_branch: str, - *, - gateway_mode: Literal["public", "private"] = "public", - contract: Any | None = None, -) -> str | None: - """Verify the slice's cited evidence commits reached the integration branch (#3125). - - The integration branch only advances when a producer pushes - (``consensus_push`` at propose time). A commit recorded by - ``egg-contract complete-task --commit <sha>`` *after* that producer - confirmed — the prescribed HITL unblock flow for a post-confirmation - task reassignment (#3124) — lives only on the agent's local worktree - branch, so the slice would otherwise close and open its PR without - the deliverable while the contract task record points at a commit - nothing retains. - - Runs after slice consensus and before any close side effects (BRC - transcript commit, slice PR). Returns ``None`` when the slice may - close, or a human-readable failure string listing every task row - whose cited commit is not an ancestor of the integration branch tip - — the caller records the slice failure with it, which routes - through the existing cascade + HITL escalation machinery instead of - closing silently. - - Only role-bound task rows are gated (#3339): the check exists for - the producer-scoped #3124 flow, so a ``role=unassigned`` row's - orphan commit is bookkeeping, not a gated deliverable, and must not - fail a consensus-reached slice. See ``cc.evidence_commits``. - - Failure posture mirrors the other completeness checks (#3081 / - #3114): the gate degrades to ``None`` (close proceeds, warning - logged) when the contract cannot be read, the slice id does not - resolve, or the gateway reachability probe cannot be evaluated. - Only a definitive "this cited commit is not on the branch" verdict - fails the close. ``EGG_EVIDENCE_REACHABILITY_GATE`` is the operator - kill switch. - - ``contract`` is an optional pre-loaded contract: the close path - already needs the contract one stretch later for the slice PR data - snapshot, so threading the same load through saves one file read - and one ``get_pipeline_state_lock`` acquisition. When ``None`` - (the default — keeps the gate self-contained for tests), the gate - loads the contract itself under the lock. - """ - try: - import contract_completeness as cc - except ImportError: - from .. import contract_completeness as cc # type: ignore[no-redef] - - if not cc.evidence_gate_enabled(): - logger.info( - "Evidence-reachability gate disabled by kill switch (#3125)", - pipeline_id=pipeline_id, - slice_id=slice_id, - ) - return None - - if contract is None: - from egg_contracts.loader import load_contract as _load_contract - - try: - with get_pipeline_state_lock(pipeline_id): - contract = _load_contract(pipeline_id, worktree_repo_path) - except Exception as load_err: # noqa: BLE001 - logger.warning( - "Evidence-reachability gate skipped: contract load failed (#3125)", - pipeline_id=pipeline_id, - slice_id=slice_id, - error=str(load_err), - ) - return None - - rows = cc.evidence_commits(contract, slice_id) - if rows is None: - logger.warning( - "Evidence-reachability gate skipped: slice not found in contract (#3125)", - pipeline_id=pipeline_id, - slice_id=slice_id, - ) - return None - if not rows: - return None - - # De-duplicate while preserving first-seen order: multiple task rows - # can cite the same commit (the prescribed unblock flow #3124 often - # links one commit across two adjacent rows). Each duplicate would - # otherwise burn one merge-base round-trip per dupe. The membership - # join below re-attaches the verdict to every row that cites it. - probe_shas = list(dict.fromkeys(r["commit"] for r in rows)) - unreachable_shas = spawner.gateway.find_unreachable_evidence_commits( - pipeline_id, - str(worktree_repo_path), - commit_shas=probe_shas, - integration_branch=integration_branch, - mode=gateway_mode, - ) - if unreachable_shas is None: - # The probe itself could not be evaluated (gateway/network). - # find_unreachable_evidence_commits already logged the cause. - return None - if not unreachable_shas: - return None - - lost = [r for r in rows if r["commit"] in set(unreachable_shas)] - summary = cc.format_evidence_rows(lost) - logger.error( - "Slice close blocked: task records cite commits unreachable from " - "the integration branch (#3125)", - pipeline_id=pipeline_id, - slice_id=slice_id, - integration_branch=integration_branch, - unreachable=summary, - ) - return ( - f"slice {slice_id}: evidence-reachability gate failed — contract task " - f"records cite commits that are not on integration branch " - f"{integration_branch}: {summary}. Cherry-pick (or push) the cited " - f"commits onto {integration_branch}, then re-run the slice close; " - f"set {cc.EVIDENCE_GATE_ENV_VAR}=off to bypass." - ) - - -def _commit_slice_brc_history_to_integration_branch( - pipeline, - spawner: "ContainerSpawner", # noqa: UP037 - worktree_repo_path: Path, - slice_id: str, - integration_branch: str, - *, - gateway_mode: Literal["public", "private"] = "public", -) -> bool: - """Commit a slice's per-slice BRC history onto its integration branch (#2548). - - Runs after the slice's implement-phase consensus is reached and - before the slice PR is opened, so reviewers approaching the slice - PR see the full BRC consensus transcript that approved the slice's - code as part of the diff. - - Steps: - - 1. Materialise a per-tick temp directory under - ``WORKTREE_BASE_DIR`` (gateway-allowlisted; see #2684) and - render the per-slice BRC history files into a ``staging/`` - subdirectory via :func:`_write_brc_history`. The writer pulls - messages from the message store; the staging directory is - scoped to this hook tick so concurrent slice hooks do not - cross-write each other (#2755). - 2. Materialise a temporary **detached** git worktree on - ``origin/<integration_branch>`` (the slice's integration branch). - A detached worktree claims no branch ref, so it never collides - with the slice's own agent worktrees — which hold the - integration branch checked out for the duration of the slice - run — nor with a prior tick that crashed mid-flight (#2778). - 3. Copy ONLY this slice's per-slice BRC files - (``<identifier>-implement-<slice_id>.{json,md}``) from the - staging directory to the integration worktree. Other slices' - files (or the unattributed sibling) are deliberately not - copied — each slice PR carries only its own BRC transcript per - D2 / D5 of #2548. - 4. Commit via :func:`_commit_statefiles_to_worktree` - (orchestrator-authored, ``--no-verify``, idempotent: skips when - staged is empty). - 5. Push via :meth:`GatewayClient.push_worktree_branch` (launcher- - auth so we bypass agent-facing push restrictions on - ``.egg-state/brc-history/``). - - Returns ``True`` on success or no-op (files already committed and - push is a fast-forward no-op). Returns ``False`` on any failure; - the caller treats this as best-effort and proceeds with PR - creation. The per-slice BRC files do not exist on the work - worktree under this design (#2755) — the integration branch is - the only on-disk surface that carries them, so a failure here - means the slice PR opens without its consensus transcript. - - Idempotency: every step is convergent — re-running mid-flight - against an already-committed integration branch produces no new - commit (``_commit_statefiles_to_worktree`` skips when nothing is - staged) and a no-op fast-forward push. - - Concurrency: this hook runs from ``_run_one_slice_inner``, which - is itself invoked concurrently across slices in a thread pool. - Each invocation creates its own ``mkdtemp``-rooted staging - directory, so two slices reaching consensus near-simultaneously - do not share any filesystem state (#2755 fix). Each slice copies - only its own per-slice files to its integration worktree - (Step 3), so concurrent writes do not cross-pollinate slice PRs. - """ - pipeline_id = pipeline.id - - if not pipeline.repo: - logger.info( - "Per-slice BRC commit: pipeline has no remote repo, skipping (#2548)", - pipeline_id=pipeline_id, - slice_id=slice_id, - ) - return False - - identifier = _brc_history_identifier(pipeline) - - import shutil - import tempfile - - git_base = [ - "git", - "-c", - "core.hooksPath=/dev/null", - "-c", - f"safe.directory={worktree_repo_path}", - "-C", - str(worktree_repo_path), - ] - - # Root under WORKTREE_BASE_DIR so the temp path falls inside the - # gateway's repo-path allowlist (gateway/git_client.py - # ALLOWED_REPO_PATHS). A ``/tmp`` location is rejected by - # ``validate_repo_path``, which silently failed the BRC-history - # push and left slice PRs without their consensus transcript - # (#2684). Falls back to system temp when the base dir is absent - # (e.g. unit tests) — emit a warning on that branch so a broken - # docker volume mount in production is noisy rather than silently - # recreating the #2684 push-rejection. - if WORKTREE_BASE_DIR.exists(): - tmp_dir_base = str(WORKTREE_BASE_DIR) - else: - logger.warning( - "Per-slice BRC commit: WORKTREE_BASE_DIR missing — falling " - "back to system temp (likely a broken volume mount in " - "production; the push to the integration branch will be " - "rejected by the gateway allowlist) (#2684)", - pipeline_id=pipeline_id, - slice_id=slice_id, - integration_branch=integration_branch, - worktree_base_dir=str(WORKTREE_BASE_DIR), - ) - tmp_dir_base = None - tmp_worktree = Path( - tempfile.mkdtemp( - prefix=f"egg-slice-brc-{pipeline_id}-{slice_id}-", - dir=tmp_dir_base, - ) - ) - # Per-tick staging directory so concurrent slice hooks do not - # share the writer's output (#2755). ``_write_brc_history`` - # renders into ``<staging>/.egg-state/brc-history/`` — same - # relative layout it uses against a worktree — so the - # ``Path.relative_to(staging)`` step below preserves the - # canonical on-disk path when copying onto the integration - # worktree. - staging = tmp_worktree / "staging" - wt_path = tmp_worktree / "wt" - - try: - # --- Step 1: render the per-slice BRC files into the staging - # directory. The writer pulls messages from the message store - # and writes all per-slice files for the implement phase; we - # filter to this slice's files below. - try: - _write_brc_history( - staging, - pipeline_id, - "implement", - identifier, - ) - except Exception as brc_err: # noqa: BLE001 - logger.warning( - "Per-slice BRC commit: failed to render BRC history into " - "staging dir, skipping integration-branch commit (#2548)", - pipeline_id=pipeline_id, - slice_id=slice_id, - error=str(brc_err), - ) - return False - - # The per-slice files we will copy onto the integration worktree. - # Both files are produced by ``_write_brc_history`` (markdown and - # JSON companion). Missing files are tolerated — the writer logs - # at warning level but still succeeds on the other format, so we - # copy whichever exists. - history_dir = staging / ".egg-state" / "brc-history" - per_slice_files: list[Path] = [] - for ext in ("md", "json"): - candidate = history_dir / f"{identifier}-implement-{slice_id}.{ext}" - if candidate.is_symlink(): - # Defense-in-depth: a planted symlink could point outside - # ``.egg-state/`` and leak unrelated content onto the slice - # PR. The staging directory is freshly minted under - # ``tempfile.mkdtemp`` per hook tick, so a symlink at this - # path would have to come from the writer itself — the - # check is cheap and protects against any future writer - # change that might honour an attacker-controlled - # metadata blob when synthesising the filename. - logger.warning( - "Per-slice BRC commit: skipping symlink in brc-history (#2548)", - pipeline_id=pipeline_id, - slice_id=slice_id, - path=str(candidate), - ) - continue - if candidate.is_file(): - per_slice_files.append(candidate) - - if not per_slice_files: - logger.warning( - "Per-slice BRC commit: no per-slice BRC files produced " - "for slice — skipping integration-branch commit (#2548)", - pipeline_id=pipeline_id, - slice_id=slice_id, - identifier=str(identifier), - ) - return False - - # --- Step 2: refresh the local remote-tracking ref for the - # integration branch. The slice's agents pushed directly to - # ``origin/<integration_branch>`` during the run, so the work - # worktree's local tracking ref may lag. Best-effort: a failure - # here usually means the agent-side push has not yet propagated; - # the worktree-add below would then fail and we'd return False. - try: - spawner.gateway.fetch_branch( - pipeline_id, - str(worktree_repo_path), - args=[f"+refs/heads/{integration_branch}:refs/remotes/origin/{integration_branch}"], - mode=gateway_mode, # type: ignore[arg-type] - ) - except Exception as fetch_err: # noqa: BLE001 - logger.warning( - "Per-slice BRC commit: fetch of integration branch failed (continuing) (#2548)", - pipeline_id=pipeline_id, - slice_id=slice_id, - integration_branch=integration_branch, - error=str(fetch_err), - ) - - try: - subprocess.run( - [ - *git_base, - "worktree", - "add", - # Detached, not ``-B <integration_branch>``: a branch - # can live in only one linked worktree, and the - # slice's agent worktrees already hold it — ``-B`` - # lost that race with ``fatal: ... already used by - # worktree`` (#2778). See Step 2 in the docstring. - "--detach", - str(wt_path), - f"origin/{integration_branch}", - ], - capture_output=True, - text=True, - check=True, - timeout=60, - ) - except subprocess.CalledProcessError as wt_err: - logger.warning( - "Per-slice BRC commit: worktree add failed, skipping (#2548)", - pipeline_id=pipeline_id, - slice_id=slice_id, - integration_branch=integration_branch, - stderr=(wt_err.stderr or "")[:500], - ) - return False - - # --- Step 3: copy ONLY this slice's BRC files onto the integration - # worktree. Each file lands at the same relative path it occupies - # in the staging dir (``.egg-state/brc-history/...``). - for src in per_slice_files: - try: - rel = src.relative_to(staging) - except ValueError: - logger.warning( - "Per-slice BRC commit: file outside staging dir, skipping it (#2548)", - pipeline_id=pipeline_id, - slice_id=slice_id, - src=str(src), - ) - continue - dst = wt_path / rel - dst.parent.mkdir(parents=True, exist_ok=True) - shutil.copy2(src, dst) - - # --- Step 4: commit (idempotent — skips when staged is empty) --- - try: - _commit_statefiles_to_worktree( - wt_path, - f"Persist BRC history for {slice_id} (#2548)", - pipeline_identifier=identifier, - pipeline_id=pipeline_id, - ) - except Exception as commit_err: # noqa: BLE001 - logger.warning( - "Per-slice BRC commit: commit failed, skipping (#2548)", - pipeline_id=pipeline_id, - slice_id=slice_id, - error=str(commit_err), - ) - return False - - # --- Step 5: push to origin/<integration_branch>. Fast-forward - # no-op when the local tip matches origin (e.g. when the - # commit step was a no-op because everything was already - # committed on a prior tick). - try: - push_result = spawner.gateway.push_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(wt_path), - branch=integration_branch, - mode=gateway_mode, # type: ignore[arg-type] - base_branch=pipeline.base_branch, - ) - except Exception as push_err: # noqa: BLE001 - logger.warning( - "Per-slice BRC commit: push raised, skipping (#2548)", - pipeline_id=pipeline_id, - slice_id=slice_id, - error=str(push_err), - ) - return False - if not push_result.ok: - logger.warning( - "Per-slice BRC commit: push failed, skipping (#2548)", - pipeline_id=pipeline_id, - slice_id=slice_id, - category=getattr(push_result, "category", None), - detail=getattr(push_result, "detail", None), - ) - return False - - logger.info( - "Per-slice BRC commit: pushed BRC history to integration branch (#2548)", - pipeline_id=pipeline_id, - slice_id=slice_id, - integration_branch=integration_branch, - files=[str(p.relative_to(staging)) for p in per_slice_files], - ) - return True - finally: - # Best-effort cleanup of the temp worktree. A failure here is a - # housekeeping problem, not a pipeline-blocker. - try: - subprocess.run( - [*git_base, "worktree", "remove", "--force", str(wt_path)], - capture_output=True, - text=True, - check=False, - timeout=30, - ) - except Exception as cleanup_err: # noqa: BLE001 - logger.debug( - "Per-slice BRC commit: worktree remove failed (continuing) (#2548)", - pipeline_id=pipeline_id, - slice_id=slice_id, - error=str(cleanup_err), - ) - try: - shutil.rmtree(tmp_worktree, ignore_errors=True) - except Exception: # noqa: BLE001 - pass - - -def _build_slice_diff_summary( - pipeline, - spawner: "ContainerSpawner", # noqa: UP037 - worktree_repo_path: Path, - integration_branch: str, - parent_branch: str, - gateway_mode: Literal["public", "private"] = "public", -) -> tuple[list[str] | None, str | None]: - """Compute commit subjects + diffstat for a slice PR body (#3115). - - The slice PR body's task list is plan-derived — it describes intent, - not what the pushed branch actually contains. This helper reads the - real git state so ``create_slice_pr`` can render a ``## What's in - this PR`` section: the slice's commit subjects - (``git log origin/<parent>..origin/<head>``) and a diffstat against - the merge base (``git diff --stat origin/<parent>...origin/<head>``, - three-dot to match GitHub's PR diff semantics). - - Both remote-tracking refs are refreshed first via - ``GatewayClient.fetch_branch`` — the slice's agents push directly to - origin, so the orchestrator worktree's tracking refs may lag (same - pattern as :func:`_commit_slice_brc_history_to_integration_branch`, - which runs immediately before this in the slice loop). ``gateway_mode`` - must be threaded from the pipeline-computed mode at the call site; - defaulting to ``public`` against a private/internal repo causes the - gateway to refuse the session and the whole diff section silently - no-ops. - - Strictly best-effort: returns ``(None, None)`` on any failure - (fetch, git error, timeout) and never raises — a missing diff - summary must not block slice PR creation. - """ - pipeline_id = pipeline.id - try: - for branch in (parent_branch, integration_branch): - # ``fetch_branch`` swallows exceptions and returns False; - # a stale parent ref degrades the diffstat, it doesn't - # break it, so we just continue. - spawner.gateway.fetch_branch( - pipeline_id, - str(worktree_repo_path), - args=[f"+refs/heads/{branch}:refs/remotes/origin/{branch}"], - mode=gateway_mode, - ) - - git_base = [ - "git", - "-c", - "core.hooksPath=/dev/null", - "-c", - f"safe.directory={worktree_repo_path}", - "-C", - str(worktree_repo_path), - ] - span = f"origin/{parent_branch}..origin/{integration_branch}" - log_proc = subprocess.run( - [*git_base, "log", "--no-merges", "--format=%s", span], - capture_output=True, - text=True, - check=False, - timeout=30, - ) - commit_subjects = ( - [line.strip() for line in log_proc.stdout.splitlines() if line.strip()] - if log_proc.returncode == 0 - else None - ) - # ``--stat=100,80,40``: 100-col output, then git truncates past - # 40 entries with an ellipsis line — a slice touching hundreds - # of files must not produce a body longer than the task dump - # this section exists to displace. - diff_proc = subprocess.run( - [ - *git_base, - "diff", - "--stat=100,80,40", - f"origin/{parent_branch}...origin/{integration_branch}", - ], - capture_output=True, - text=True, - check=False, - timeout=60, - ) - diffstat = diff_proc.stdout.strip() if diff_proc.returncode == 0 else None - if not commit_subjects and not diffstat: - return None, None - return commit_subjects or None, diffstat or None - except Exception as err: # noqa: BLE001 - logger.warning( - "Slice diff summary failed (slice PR opens without it) (#3115)", - pipeline_id=pipeline_id, - integration_branch=integration_branch, - parent_branch=parent_branch, - error=str(err), - ) - return None, None - - -# Shared PR description guidance injected into planner prompts. -# Kept as a constant so both _build_phase_prompt and _build_agent_prompt -# stay in sync when the guidance evolves. -_PR_DESCRIPTION_GUIDANCE = [ - "**PR description quality**: The `pr.description` field becomes the PR body " - "that reviewers read first. Write 2-3 paragraphs following this structure:", - "1. **Context** — what problem exists and why it matters", - "2. **Changes** — what this PR does, with specifics (e.g. numbered list of " - "key changes with bold headers)", - "3. **Impact** — what behavior changes for users or other components", - "", - "Do NOT write a one-liner — reviewers need enough detail to understand " - "the problem, the approach, and why it was chosen without reading every file.", -] - -_PR_DESCRIPTION_YAML_EXAMPLE = [ - " Explain the problem or need this PR addresses and why it matters.", - "", - " Describe the key changes, ideally as a numbered or bulleted list", - " with bold headers so reviewers can scan quickly. For each change,", - " explain what it does and why.", - "", - " Summarize the impact — what changes for users, callers, or other", - " components as a result.", -] - -# YAML safety guidance for planner prompts. Plain (unquoted) scalars break -# when they contain ``: `` sequences — e.g. "Add `sequence: int = 0` field" -# parses as a nested mapping and raises ScannerError. Block scalars (``|-``) -# take the whole indented block literally, so backticks, colons, quotes, and -# other punctuation are safe. See issue #1974. -_YAML_TASKS_SAFETY_GUIDANCE = [ - "**YAML safety**: Use block scalars (`|-`) for every prose field — " - "`name`, `goal`, `description`, `acceptance`. Plain unquoted scalars " - "break when the text contains `` `code: type` ``, colons in URLs, or " - "other `: ` sequences, because PyYAML reads them as nested mappings " - "and the parser drops back to markdown fallback (silently losing the " - "`pr:` block). Follow the example above literally — do not inline these " - "values on the same line as the key.", -] - -# Permissive subagent-exploration guidance for producer prompts (#2814). -# Producers may delegate deep grep/Read exploration to the Claude Code -# `general-purpose` subagent so large tool-result payloads don't accumulate -# in the producer's main context window. Mitigates the failure surface of -# #2804 (Agent SDK 1MB JSON buffer overflow). Reused across all seven -# producer prompts so the wording stays uniform. -# -# `general-purpose` is the only subagent the Agent SDK ships out of the -# box; we deliberately do not name `Explore` here because the sandbox -# runtime does not register an `Explore` AgentDefinition (no `agents=` -# on ClaudeAgentOptions and no filesystem `.claude/agents/Explore.md`), -# so the example would burn a turn on an unknown-subagent retry. -_EXPLORATION_SUBAGENT_HEADER = "## Subagent use for exploration" -_EXPLORATION_SUBAGENT_GUIDANCE = [ - f"{_EXPLORATION_SUBAGENT_HEADER}\n", - "You **may** use the Agent tool (`subagent_type: general-purpose`) " - "when exploration would otherwise dominate your context window. Use " - "your judgment — a one-off grep or short read doesn't need a " - "subagent; deep investigation of a large file or many call sites " - "usually does. The producer's main context stays lean for synthesis; " - "the subagent returns a focused summary.\n", - "Example signals where subagent use often pays off:", - "- More than ~3 grep/read calls on the same target file or directory.", - "- Walking a primitive's call sites — delegate; ask for `file:line` " - "citations + a few lines of context.", - "- Reading large files (> ~500 lines) — get a subagent summary first; " - "only `Read` the main file yourself if the summary identifies specific " - "line ranges you need to author at.\n", - "Subagent summaries are part of your authoritative work. Verify " - "critical claims (e.g. `file:line` citations) before committing them " - "to your output.", - "", -] - - -def _build_phase_iteration_context( - operator_directives: list[OperatorDirective] | None, - iteration_history: list[IterationSummary] | None, -) -> str: - """Render operator directives + prior iteration history as a prompt section. - - Issued in iteration N+1 prompts (for **both** producers and reviewers) - after one or more HITL phase-gate kickbacks. Replaces the unstructured - ``## Review Feedback`` rendering that previously squatted on the - agentic-cycle feedback channel — operator directives now have their own - section with explicit precedence prose so reviewers cannot faithfully - NACK a directive-driven change against a stale default rubric (#2795). - - Returns an empty string when there are no directives and no history - so the caller can unconditionally append the result. - """ - directives = operator_directives or [] - history = iteration_history or [] - if not directives and not history: - return "" - - lines: list[str] = ["## Phase Iteration Context\n"] - if directives: - lines.append( - "The operator has kicked this phase back through HITL one or " - "more times. The directives below **override prompt-template " - "defaults**. If a rubric item in your role's instructions " - "conflicts with a directive, the directive wins. Later " - "directives override earlier ones.\n" - ) - lines.append("### Operator Directives (chronological)\n") - for idx, directive in enumerate(directives, start=1): - ts = directive.created_at.isoformat() - lines.append(f"**Directive {idx}** (iteration {directive.iteration_n}, {ts}):") - lines.append("") - lines.append(directive.feedback_text.rstrip()) - lines.append("") - - if history: - lines.append("### Prior Iteration History\n") - lines.append( - "Each entry below is a frozen snapshot of a previously kicked-" - "back iteration's BRC outcome — what the reviewers concluded " - "and why. Use it to see which rubric items tripped last round " - "so you do not repeat the same NACKs.\n" - ) - for summary in history: - ts = summary.completed_at.isoformat() - lines.append(f"**Iteration {summary.iteration_n}** (completed {ts}):") - if summary.final_proposal_commit: - # SHAs are pre-filtered by _build_iteration_summary_from_tracker - # (empty + RECONSTRUCTED_NO_SHA dropped before the dict is - # populated), so every value here is a real commit. - commit_parts = [ - f"{producer}={sha[:12]}" - for producer, sha in sorted(summary.final_proposal_commit.items()) - ] - lines.append(f"- Final proposal commits: {', '.join(commit_parts)}") - if summary.verdict_matrix: - verdicts = "; ".join( - f"{edge}: {state}" for edge, state in sorted(summary.verdict_matrix.items()) - ) - lines.append(f"- Verdict matrix: {verdicts}") - if summary.nack_reasons: - lines.append(f"- NACK reasons ({len(summary.nack_reasons)}):") - for reason in summary.nack_reasons: - lines.append(f" - {reason}") - if summary.artifacts_snapshot: - arts = ", ".join(sorted(summary.artifacts_snapshot.keys())) - lines.append(f"- Artifacts at iteration close: {arts}") - lines.append("") - - return "\n".join(lines) - - -def _build_iteration_summary_from_tracker( - tracker: Any, - iteration_n: int, - artifacts: dict[str, str] | None = None, - completed_at: datetime | None = None, -) -> IterationSummary: - """Capture an :class:`IterationSummary` from a live BRC tracker. - - Called by the HITL kickback handler **before** ``_clear_concurrent_state`` - wipes the tracker so the iteration N+1 prompt can render what tripped - iteration N. Tolerates a ``None`` tracker — returns a summary with only - the iteration index + completion timestamp populated, which still lets - downstream prompts mention that a kickback occurred without claiming - false verdict detail. - """ - completion = completed_at or datetime.now(UTC) - summary = IterationSummary( - iteration_n=iteration_n, - completed_at=completion, - artifacts_snapshot=dict(artifacts or {}), - ) - if tracker is None: - return summary - - try: - matrix = getattr(tracker, "matrix", None) - if matrix is None: - return summary - # Snapshot the matrix entries + commit SHAs under the tracker's - # lock so concurrent mutations from a still-live tracker can't - # tear the read. RLock means re-entry is safe if callers already - # hold it. Iteration below runs on the local copies. - lock = getattr(tracker, "_lock", None) - commits_snapshot: dict[str, str] = {} - if lock is not None: - with lock: - entries_snapshot = list(getattr(matrix, "_entries", {}).items()) - commits_snapshot = dict(getattr(tracker, "_proposal_commit_shas", {})) - else: - entries_snapshot = list(getattr(matrix, "_entries", {}).items()) - commits_snapshot = dict(getattr(tracker, "_proposal_commit_shas", {})) - - verdict_matrix: dict[str, str] = {} - nack_reasons: list[str] = [] - for (reviewer, producer), entry in entries_snapshot: - state = getattr(entry, "state", None) - state_val = state.value if state is not None else "unknown" - verdict_matrix[f"{reviewer}->{producer}"] = state_val - if state_val == "nacked" and getattr(entry, "reason", ""): - nack_reasons.append(f"{reviewer}→{producer}: {entry.reason}") - summary.verdict_matrix = verdict_matrix - summary.nack_reasons = nack_reasons - - producers = {producer for _, producer in (k for k, _ in entries_snapshot)} - commits: dict[str, str] = {} - for producer in producers: - sha = commits_snapshot.get(producer, "") - if sha and sha != "RECONSTRUCTED_NO_SHA": - commits[producer] = sha - summary.final_proposal_commit = commits - except Exception as e: # noqa: BLE001 - logger.debug( - "Failed to snapshot iteration summary from tracker", - iteration_n=iteration_n, - error=str(e), - ) - return summary - - -def _apply_inline_hitl_kickback_to_phase( - phase_execution: PhaseExecution, - revision_feedback: str, - tracker: Any = None, -) -> list[ContainerInfo]: - """Apply the inline HITL kickback's phase-state mutations. - - Extracted from the inline ``request_changes`` handler so tests can - drive the assertion through production code rather than constructing - a fixture by hand (#2795 review). The caller is still responsible for - the wrapping concerns: clearing the message store + consensus tracker - via ``_clear_concurrent_state``, persisting the pipeline via - ``store.save_pipeline``, and stopping the stale containers returned - here (the K8s delete is asynchronous so an explicit stop is required - to avoid iteration N+1 racing iteration N's still-terminating pods). - - Returns the snapshot of containers that were running at kickback - time, for the caller to issue the defensive stop on. - """ - # Monotone across the legacy-hitl_feedback migration boundary: a - # pre-#2795 phase migrates with iteration_history empty but a - # synthetic OperatorDirective carrying iteration_n derived from - # hitl_review_cycles. ``len(iteration_history)`` alone would - # restart at 0 and label two distinct iterations identically; use - # one past the maximum existing directive index as the floor so - # the displayed "iteration X" labels stay monotone. - iteration_n = max( - len(phase_execution.iteration_history), - max( - (d.iteration_n for d in phase_execution.operator_directives), - default=-1, - ) - + 1, - ) - phase_execution.operator_directives.append( - OperatorDirective( - iteration_n=iteration_n, - feedback_text=revision_feedback, - ) - ) - phase_execution.iteration_history.append( - _build_iteration_summary_from_tracker( - tracker, - iteration_n=iteration_n, - artifacts=phase_execution.artifacts, - ) - ) - stale_containers = list(phase_execution.containers) - phase_execution.containers = [] - phase_execution.agents = [] - phase_execution.artifacts = {} - phase_execution.review_cycles = 0 - return stale_containers - - -def _broadcast_hitl_nonconvergence_alert( - pipeline_id: str, - pipeline: Pipeline, - current_phase: PipelinePhase, - cycles: int, - threshold: int, -) -> None: - """Non-fatal overseer alert when the HITL converge loop runs long (#3392). - - The converge-before-advance loop is human-gated every round (the - operator resolves decisions before each re-run), so a long-running loop - cannot burn compute silently and is never force-advanced. After - ``threshold`` rounds we surface an ``OVERSEER_ALERT`` so a pathological - non-convergence — a real carry-forward bug, or a genuinely churning - design — is visible. Best-effort: a broadcast failure never blocks the - re-run. - """ - try: - from message_store import Message, MessageType - - store_fn = _get_message_store() - if store_fn is None: - return - msg_store = store_fn() - phase = current_phase.value if current_phase else None - msg_store.add_message( - Message( - pipeline_id=pipeline_id, - from_role="orchestrator", - to_role="all", - message_type=MessageType.OVERSEER_ALERT, - subject="hitl_nonconvergence: orchestrator [medium]", - body=( - f"The {phase} phase HITL converge-before-advance loop has run " - f"{cycles} rounds (>= {threshold}) without reaching a fixpoint. " - f"Each round is human-gated, so this is surfaced for visibility, " - f"not force-advanced. Investigate whether a decision keeps " - f"re-surfacing (carry-forward bug) or the design is genuinely " - f"churning. See #3392." - ), - metadata={"reason": "hitl_nonconvergence", "cycles": cycles}, - phase=phase, - ) - ) - except Exception as alert_err: # noqa: BLE001 - logger.warning( - "Failed to broadcast HITL non-convergence alert (non-fatal)", - pipeline_id=pipeline_id, - error=str(alert_err), - ) - - -def _perform_hitl_phase_rerun( - *, - store: Any, - spawner: Any, - pipeline: Pipeline, - phase_execution: PhaseExecution, - pipeline_id: str, - current_phase: PipelinePhase, - feedback_text: str, - event_message: str, -) -> None: - """Tear down the current phase iteration and arm a re-run (#3392). - - Shared by the two re-run triggers in the converge-before-advance HITL - loop: the operator-feedback kickback (``request_changes`` / - ``change_approach``) and the decision-driven re-run that folds resolved - HITL answers back into the phase documents. Snapshots the BRC tracker - for the next iteration's prompt, appends the operator directive + - iteration summary (#2795), clears concurrent state so the re-run does - not short-circuit on stale ``CONSENSUS_CONFIRMED`` messages (#1296), - persists, and stops the stale containers (the K8s delete is async, so an - explicit idempotent stop prevents iteration N+1 racing iteration N's - still-terminating pods). - - The caller must already hold the pipeline state lock, have set the - pipeline/phase status back to RUNNING, and incremented - ``phase_execution.hitl_review_cycles``. The caller issues the - ``continue`` that re-enters the outer loop. - """ - # Capture the BRC tracker state BEFORE _clear_concurrent_state drops - # it — that's our only chance to snapshot this iteration's verdicts for - # the next iteration's prompt. - rerun_tracker = None - try: - from peer_consensus import get_peer_consensus_tracker as _gpct - - rerun_tracker = _gpct(pipeline_id) - except Exception as tracker_err: # noqa: BLE001 - logger.debug( - "Tracker lookup failed during HITL re-run snapshot", - pipeline_id=pipeline_id, - error=str(tracker_err), - ) - - stale_containers = _apply_inline_hitl_kickback_to_phase( - phase_execution, - feedback_text, - tracker=rerun_tracker, - ) - - from routes.phases import _clear_concurrent_state - - _clear_concurrent_state(pipeline_id) - - store.save_pipeline(pipeline) - - for _ctr in stale_containers: - if _ctr.container_id and _ctr.status == ContainerStatus.RUNNING: - try: - spawner.backend.stop_container(_ctr.container_id, timeout=10) - except Exception as stop_err: # noqa: BLE001 - logger.debug( - "Best-effort HITL re-run teardown failed", - pipeline_id=pipeline_id, - container_id=_ctr.container_id, - error=str(stop_err), - ) - - report_pipeline_status( - pipeline, - event_type="phase.revision_requested", - message=event_message, - ) - _emit_pipeline_event(pipeline, "phase.revision_requested") - - -def _build_phase_prompt( - phase: str, - pipeline_id: str, - pipeline_mode: str, - prompt: str | None = None, - issue_number: int | None = None, - repo: str | None = None, - branch: str | None = None, - review_feedback: str | None = None, - review_cycle: int = 0, - repo_path: str | None = None, - operator_directives: list[OperatorDirective] | None = None, - iteration_history: list[IterationSummary] | None = None, -) -> str: - """Build a phase-specific prompt for the sandbox Claude invocation. - - Follows a structured prompt format: - Context → Task → Restrictions → Completion. - """ - # --- Context header --- - lines = [f"You are in the **{phase}** phase of the SDLC pipeline.\n"] - lines.append("## Context\n") - lines.append(f"Pipeline ID: {pipeline_id}") - lines.append(f"Phase: {phase}") - if repo: - lines.append(f"Repository: {repo}") - if branch: - lines.append(f"Branch: {branch}") - if issue_number is not None: - lines.append(f"Issue: #{issue_number}") - lines.append("") - - # --- Phase iteration context (HITL kickbacks) --- - # Operator directives have their own section with explicit precedence - # prose so reviewers cannot faithfully NACK a directive-driven change - # against a stale default rubric. See issue #2795. - iteration_context = _build_phase_iteration_context(operator_directives, iteration_history) - if iteration_context: - lines.append(iteration_context) - - # --- Prior review feedback (agentic revision cycles only) --- - # Scoped to agentic-cycle review feedback since #2795 — HITL kickback - # feedback now flows through ``operator_directives`` / the iteration - # context section above. - if review_feedback: - if review_cycle > 0: - lines.append(f"## Prior Review Feedback (Cycle {review_cycle})\n") - else: - lines.append("## Prior Review Feedback\n") - has_tester_findings = TESTER_FINDINGS_HEADER in review_feedback - if phase == "implement": - revision_action = "Address the feedback below and revise your implementation." - else: - revision_action = ( - "Address the feedback below and revise your draft **in-place** " - "(overwrite the same file)." - ) - if review_cycle == 0: - consensus_override = ( - " Even if an existing draft appears " - "to have reached consensus previously, that consensus is " - "superseded — you must revise to address this feedback before " - "proposing a new consensus." - ) - else: - consensus_override = "" - if has_tester_findings: - lines.append( - "The reviewer and tester found issues with your previous work. " - f"{revision_action}{consensus_override}\n" - ) - else: - preamble_noun = "implementation" if phase == "implement" else "draft" - lines.append( - f"The reviewer found issues with your previous {preamble_noun}. " - f"{revision_action}{consensus_override}\n" - ) - lines.append(review_feedback) - lines.append("") - - # --- Task description --- - # Skip re-embedding the full task description on revision cycles for - # implement phase — the coder already knows the task from cycle 0. - if prompt and not (phase == "implement" and review_cycle > 0): - lines.append("## Task Description\n") - lines.append(prompt) - lines.append("") - - # --- Phase-specific instructions --- - lines.append("## Your Task\n") - - # Get the correct draft path based on mode - analysis_path = _get_draft_path("refine", issue_number=issue_number, pipeline_id=pipeline_id) - plan_path = _get_draft_path("plan", issue_number=issue_number, pipeline_id=pipeline_id) - - if phase == "refine": - lines.extend( - [ - "Analyze this issue and produce a structured analysis document. Your goal is to:\n", - "1. Understand the problem or feature request", - "2. Research the current codebase to understand existing patterns", - "3. Research externally when the task involves third-party libraries, APIs, " - "or integrations — use WebSearch and WebFetch (when available) to look up " - "current documentation, best practices, and known issues. Skip external " - "research for purely internal changes where codebase context is sufficient.", - "4. Identify constraints and dependencies", - "5. Consider multiple implementation approaches", - "6. Recommend an approach with justification", - "7. Surface the questions and uncertainties that genuinely need a " - "human to answer — see `## How to Populate Open Questions` below for " - "the filter (slice/PR packaging, implementation strategy, and " - "API/schema details belong to the planner, not the refiner)", - "", - "**IMPORTANT**: Do NOT create an implementation plan, task breakdown, " - "or phased rollout. That is the **plan** phase's job. Stay focused on " - "**analysis**: understanding the problem, researching the codebase, " - "evaluating options, and surfacing decisions for the human.", - "", - "## Output Format\n", - "Create an analysis document following the template below. The " - "fenced block is the **template literal** — copy it as-is and fill " - "in the bracketed placeholders. The unfenced sections that follow " - "(`## How to Populate Open Questions`, `## Complexity Assessment`) " - "are **meta-guidance** — do **not** transcribe them into your " - "analysis document.\n", - "````markdown", - "# Analysis: [Issue Title]\n", - "> Issue: #[number] | Phase: refine\n", - "## Problem Statement\n", - "[Describe the problem or feature request. " - "What is the current state? What is the desired outcome?]\n", - "## Current Behavior\n", - "[Describe how the system currently works in the relevant area. " - "Include code references where helpful.]\n", - "## Constraints\n", - "- [Technical constraints (compatibility, performance, security)]", - "- [Business constraints (timeline, scope)]", - "- [Dependencies on other systems or features]\n", - "## Options Considered\n", - "### Option A: [Name]\n", - "**Approach**: [Brief description]\n", - "**Pros**:", - "- [Advantage 1]\n", - "**Cons**:", - "- [Disadvantage 1]\n", - "### Option B: [Name]\n", - "**Approach**: [Brief description]\n", - "**Pros**:", - "- [Advantage 1]\n", - "**Cons**:", - "- [Disadvantage 1]\n", - "## Recommended Approach\n", - "[Which option is recommended and why. Reference the option above.]\n", - "## Open Questions\n", - "[Register every open question by following the protocol in " - "`## How to Populate Open Questions` below the template, then paste " - "the markdown output of each registration command into this section. " - "Do **not** copy the protocol instructions themselves into this " - "document.]\n", - "---\n", - "*Authored-by: egg*", - "````\n", - "", - "## How to Populate Open Questions\n", - "These instructions tell you how to handle the `## Open Questions` " - "section of the template above. They are **meta-guidance**, not " - "template content — do **not** transcribe this section into the " - "analysis document you write.\n", - "**Every open question MUST be registered as a contract decision or " - "feedback item using `egg-contract`.** Do not just write questions " - "as prose — they will not be seen by the human unless registered.\n", - "**Skip already-resolved questions.** If the Task Description above " - "includes an `## Additional Context` section, treat anything addressed " - "there as already decided by the operator (those came from a pre-refine " - "HITL round). Do NOT call `egg-contract add-decision` or " - "`egg-contract add-feedback` for questions whose answers are already " - "captured in `## Additional Context` — re-registering them wastes turns " - "and produces no-op decisions. Read that section first; if it settles " - "anything, list those items in a `### Resolved in Pre-Refine` " - "subsection at the top of `## Open Questions` (one bullet per resolved " - "item, citing the answer). Only register questions that go beyond what " - "`## Additional Context` covers. This skip rule is NARROW: it covers " - "only answers THIS pipeline's operator recorded in " - "`## Additional Context`. It never covers decisions the task " - "description names as operator-owned, and never answers inherited " - "from a prior or cancelled run's seeded context — register those " - "(see the next rule).\n", - "**Task-named decisions are non-optional (#3462).** If the task " - "description or contract names specific decisions as the operator's " - "to make — or contains any directive to surface decisions as HITL " - "questions — you MUST register each one via " - "`egg-contract add-decision`, even when you believe prior context " - "already resolves it, or that it is non-blocking or deferred. " - "Belief about resolution is a *recommended disposition*, not a " - "reason to skip registration: make your recommended answer the " - "first option (suffix its label with `(recommended)`) and cite the " - "resolving context in that option's description, so the operator " - "can confirm in one click while retaining the authority to choose " - "differently. Documenting a decision in draft prose is a " - "supplement to registration, never a substitute — unregistered " - "decisions never reach the operator's decision surface.\n", - "Surface uncertainties, ambiguities, and assumptions **that genuinely " - "need a human to answer**. Filter ruthlessly: a good open question is " - "one the operator must answer because the answer changes what we're " - "building. A bad open question is one the planner phase will decide on " - "its own once it sees the analysis — those waste the operator's " - "attention and pre-anchor the planner. Err toward registering questions " - "about *what the problem actually is* and *what's in or out of scope* " - "rather than *how to build it*.\n", - "**Out of scope for refine open questions** — do NOT register decisions " - "about:\n" - "- **Work decomposition / slice-DAG shape / PR packaging** — " - "**Slice / PR packaging is NOT a refine-phase decision.** The " - "plan phase owns slice-DAG construction (see " - "`docs/architecture/slice-dag.md`) and the operator approves the " - "proposed slice shape at the plan HITL gate. Do not register " - "`add-decision` items asking how the work should be sliced, how " - "many PRs to ship, or which parts should run in parallel. If " - "the task obviously spans multiple parts, name them in Problem " - "Statement or Constraints — the planner will propose a shape " - "from the analysis it reads.\n" - "- **Implementation strategy choices** that the planner can decide " - 'from Problem Statement + Constraints (e.g. "which migration ' - 'approach", "which fallback design", "which detector shape"). ' - "Surface these as Options Considered / Recommended Approach in the " - "analysis prose, not as `add-decision` items.\n" - "- **API / schema details** the planner phase will work out once it " - "starts designing. If the operator must constrain the API shape, " - "frame it as a *constraint* in `## Constraints`, not an open question.\n" - "Register questions when the answer is a fact only the human knows " - "(product intent, scope boundaries, external commitments, " - "user-visible behavior) — not when the answer is a design call the " - "planner will make.\n", - "**Multiple-choice questions** — RUN this command for each question " - "where the human must pick from discrete options:", - "```bash", - 'egg-contract add-decision --question "Which approach should we use?" \\', - ' --options "Option A" "Option B" "Option C" --format markdown', - "```", - "Copy the markdown output into your analysis. The human can check " - 'a checkbox to select an option. An "Other (explain in reply)" ' - "option is auto-appended.\n", - "**Open-ended questions** — EXECUTE this command for free-form " - "questions where you need the human to provide text answers:", - "```bash", - "egg-contract add-feedback \\", - ' --question "What is the expected request volume?" \\', - ' --question "Are there any constraints on third-party dependencies?" \\', - " --format markdown", - "```", - "This creates a dedicated comment for the human to fill in answers. " - 'They edit the comment to add their responses and check "Submit ' - 'feedback" when done. The pipeline will resume with the feedback ' - "available in the contract.\n", - "**Advisory seam-listing is fine** — if the task obviously spans " - "independently-implementable parts, you MAY name them in Problem " - 'Statement or Constraints (e.g. "the change touches the gateway, ' - 'the orchestrator, and the sandbox") so the planner has the seam ' - "information. Make it **explicitly advisory**: the planner is free " - "to slice differently if it sees a better seam. Do not pre-number " - "parts as `slice-1 / slice-2`, do not draw a DAG, and do not pick " - "a 1-PR-vs-3-PR shape — those choices belong to the planner.\n", - "**DO NOT:**", - "- Write questions as plain markdown text without running " - "`egg-contract add-decision` or `egg-contract add-feedback`", - "- Use custom HTML comment markers like " - "`<!-- DECISION: ... -->` instead of the contract CLI", - "- Skip registration because you think the questions are minor — " - "register every question", - "- Skip registration because you believe a decision is already " - "resolved, non-blocking, or deferred — register it with your " - "recommended disposition instead (#3462)", - "- Attest `no_decisions_rationale` when the task names decisions " - "to surface — the attestation is presented to the operator as its " - "own confirmable decision, and a rejected 'none' sends the phase " - "back for a re-run (#3462)", - "- Transcribe this `## How to Populate Open Questions` section " - "into your analysis document — it is meta-guidance, not template " - "content\n", - "**Attest your decision ledger when proposing (#3390).** Your " - "consensus propose is REJECTED unless its attestation carries " - "the ledger: `--decisions-registered cq-1 cq-2 ...` (every id " - "you registered this phase) or " - '`--no-decisions-rationale "<why>"` when you deliberately ' - "registered none. Attested ids must exist on the contract for " - "this phase, and the draft must cite each one — the " - "`--format markdown` output you copied above embeds the id, so " - "the registration flow satisfies the citation automatically. " - "If your only open questions went into an `add-feedback` " - "request (no `cq-N` decisions), attest the rationale form and " - "name the feedback request in it. This is what lets the " - "operator trust that an empty gate means *deliberately no " - "decisions*, not *forgot to register*. The explicit-none form " - "is not a shortcut (#3462): the orchestrator surfaces it to " - "the operator as its own confirmable decision before the " - "phase gate, and it is only valid when the phase genuinely " - "raises no meaningful decision — never when the task names " - "decisions to surface, and never as a substitute for " - "registering a decision you believe is already resolved.\n", - "", - ] - ) - lines.extend( - [ - "## Complexity Assessment\n", - "After completing your analysis, assess the task complexity:", - "- **low**: Single-file change, straightforward bug fix, small config update, typo fix", - "- **medium**: Multi-file change with clear scope, feature addition with known patterns", - "- **high**: Architectural change, new subsystem, cross-cutting concern, " - "many independent phases that could be parallelized", - "", - ] - ) - lines.extend(_EXPLORATION_SUBAGENT_GUIDANCE) - lines.extend( - [ - f"Write your analysis to `{analysis_path}`.", - "Commit and push the draft when done.\n", - "**IMPORTANT**: Do NOT post your analysis directly to the issue. " - "The pipeline will have an internal reviewer check your analysis. " - "If revisions are needed, you'll be re-invoked with feedback. " - "Only after internal review passes will the analysis be posted " - "for human approval.", - "", - ] - ) - - elif phase == "plan": - lines.extend( - [ - "Create a detailed implementation plan, decomposing the work into " - "slices per the slice-DAG guidance at the end of this section. The " - "implement-phase pipeline ships each slice as its own stacked PR. " - "**Slice shape is your call.** A single-slice plan is fine when the " - "work is cohesive; pick a multi-slice shape when the work has clean " - "seams that ship independently. If the refine analysis sketched a " - "decomposition (e.g. naming the components touched), treat it as " - "**advisory context** — you are free to slice differently if a " - "better seam exists. The only thing that binds your slice shape is " - "an explicit slice-DAG HITL decision recorded by the operator on " - "the contract; if you believe such a decision is wrong, raise it as " - "an open question in your plan rather than silently overriding.", - "", - "Steps:", - "1. Review any prior analysis", - "2. Break down the work into phases with discrete tasks", - "3. Define clear acceptance criteria for each task", - "4. Identify test strategy — what automated tests cover the changes, " - "and what manual verification is needed", - "5. Identify any manual pre-merge or post-merge steps " - "(migrations, config changes, deployments)", - "6. Consider rollback and risks", - "", - "## Output Format", - "", - "Write a markdown plan with a **yaml-tasks** structured appendix at the end.", - "The prose section explains the approach; the appendix is machine-parsed.", - "", - *_PR_DESCRIPTION_GUIDANCE, - "", - "End your document with a fenced YAML block like this:", - "", - "````", - "```yaml", - "# yaml-tasks", - "pr:", - ' title: "Short imperative summary (≤70 chars)"', - " description: |", - *_PR_DESCRIPTION_YAML_EXAMPLE, - " test_plan: |", - " - Automated: describe which tests cover the changes", - " - Manual: specific steps a reviewer should take to verify", - " manual_steps: |", - " Pre-merge: any required steps before merging", - " Post-merge: any required steps after merging", - "slices:", - " - id: 1", - " name: |-", - " Slice Name", - " goal: |-", - " What this slice achieves, written for a reviewer of the", - " target repo. This text is rendered verbatim as the lead", - " paragraph of the slice's PR body (#3115), so keep it 1-3", - " plain-language sentences with no plan-internal", - " cross-references (reviewer codes, section numbers, draft", - " version markers).", - " tasks:", - " - id: TASK-1-1", - " description: |-", - " What to do — safe to include `code: type` snippets,", - " URLs, and other punctuation inside a block scalar.", - " acceptance: |-", - " How to verify it is done", - " files:", - " - path/to/file.py", - "```", - "````", - "", - *_YAML_TASKS_SAFETY_GUIDANCE, - "", - "Do NOT use a `pr_plan` key — slice packaging is owned by the " - "slice-DAG section below, not by an ad-hoc PR list.", - "", - "The `test_plan` field is **required** — describe both automated test " - "coverage and any manual verification steps. The `manual_steps` field " - "should list any pre-merge or post-merge actions required by the reviewer " - "or deployer; use an empty string if none.", - "", - # ---------------------------------------------------- - # #2137 — slice-DAG planner guidance (mirrors the - # concurrent task_planner block; keep the two paths - # aligned so the slice-shape rules behave the same way - # regardless of which planner runs). - # ---------------------------------------------------- - "## Slice-DAG guidance (#2137)", - "", - "The implement-phase pipeline ships each plan **slice** (formerly " - "**phase**) as its own stacked PR. The plan you emit drives that " - "DAG; the rules below are mandatory.", - "", - "**Yaml key swap**: prefer the canonical ``slices:`` key in your " - "``# yaml-tasks`` block (the parser also accepts ``phases:`` for " - "backward compatibility). New plans should use ``slices:``.", - "", - "**Slice-sizing NACK (hard, judgment-based — #2809)**: the plan " - "reviewer will hard-NACK an oversized slice. Use judgment when " - "shaping — no fixed LOC budget, but avoid bundling more than ~3 " - "distinct file-categories in one slice, avoid combining " - "deletion-heavy work with new-API-introduction work, avoid " - "slices that would require >3–4 commit-propose-revise cycles, " - "and avoid bundling independent task groups with no internal " - "dependency. Subdivide along those seams up front rather than " - "earning a NACK.", - "", - "**Forest constraint (HARD)**: every slice must have at most ONE " - "DAG parent — the implement-phase pipeline ships every slice as a " - "stacked PR with exactly one base branch. Multi-parent slices " - "break the stacking invariant and are rejected at plan ingestion.", - "", - "**Auto-serialization rule for would-be multi-parent slices**: " - "when a slice would naturally have >1 parents, serialise the " - "upstream slices into a linear chain and record the chosen " - "ordering on the downstream slice's ``serialized_chain_order`` " - "field. The list names the upstream slice IDs in their chosen " - "serialization order.", - "", - "**File-overlap rule (HARD, enforced at plan ingestion — " - "#3046)**: two slices that touch the SAME file must be ordered " - "on one dependency chain — one a transitive ``dependencies`` " - "ancestor of the other — never left as parallel roots or " - "siblings. The implement phase cuts each slice's branch off " - "its dependency parent, so an unordered overlapping pair forks " - "independently off the shared base and its edits to the shared " - "file collide at integration (a guaranteed modify/delete " - "conflict). Deletion/retirement slices are the classic trap: a " - "slice that removes a file must depend on every slice that " - "modifies it. Slices with disjoint file sets stay parallel.", - "", - "**Test co-location rule (HARD — #3411)**: a slice that " - "removes, renames, or rewrites code carries the matching " - "test updates (skip-guards, deletions, rewrites) in the SAME " - "slice — never a later one — with the test files listed in " - "that slice's task ``files:``. Each cumulative slice tip " - "must be independently green: the per-slice green gate " - "(#3398) runs the repo's checks at the slice tip before the " - "PR opens and blocks while any check is red, so deferring " - "test obsolescence to a later slice guarantees a blocked " - "slice. Discover the tests that statically reach the " - "changed files with the changeset-aware selector where the " - "repo ships it (this repo: ``python3 " - "scripts/select_tests/__main__.py --impacted-tests " - "<file>...``; exit 2 = closure unavailable — fall back to " - "grepping the removed symbols in the test trees).", - "", - "Worked example: if ``slice-3`` would naturally have " - "parents ``[slice-1, slice-2]``, instead emit:", - "", - "```yaml", - " - id: 1", - " name: |-", - " Foundations", - " # ... (root)", - " - id: 2", - " name: |-", - " Middle", - " dependencies:", - " - slice-1", - " - id: 3", - " name: |-", - " Downstream", - " dependencies:", - " - slice-2 # serialised — slice-2 is the only DAG parent", - " serialized_chain_order:", - " - slice-1", - " - slice-2 # records that you deliberately picked", - " # slice-1 → slice-2 → slice-3", - "```", - "", - "Your judgement is the source of truth. The fallback heuristic " - "when you have no preference is: cluster would-be parents by " - "``files_affected`` Jaccard overlap (>0.3), then order by " - "descending downstream fan-out.", - "", - f"Write your plan to `{plan_path}`.", - "Commit and push the draft when done.", - "", - ] - ) - - elif phase == "implement": - # Embed plan or analysis text directly on first cycle - # (avoids file-I/O turns inside the sandbox). - draft_embedded = False - if repo_path and review_cycle == 0: - draft_text = _read_phase_draft( - Path(repo_path), - "plan", - issue_number=issue_number, - pipeline_id=pipeline_id, - branch=branch, - ) - if draft_text: - lines.append("## Plan\n") - lines.append(f"```markdown\n{draft_text}\n```\n") - draft_embedded = True - - # Embed contract task checklist on first cycle - contract_tasks = _render_contract_tasks( - repo_path, pipeline_id, pipeline_mode, issue_number - ) - if contract_tasks: - lines.append(contract_tasks) - lines.append("") - - if review_cycle == 0: - # Build numbered step list; only include the "review" step - # when the draft wasn't already embedded above. - lines.append("Implement the changes described in the task and plan:") - lines.append("") - - steps: list[str] = [] - if not draft_embedded: - steps.append("Review the plan (check `.egg-state/drafts/`)") - steps.extend( - [ - "Implement the required changes — when working with third-party " - "libraries or APIs, use WebSearch and WebFetch (when available) to " - "look up current documentation, usage examples, and best practices", - "After completing each plan phase or task group, commit and push " - "immediately — do not batch all work into a final commit. Mark " - "tasks done: `egg-contract complete-task --task <id> --commit <sha>`", - "Run tests to verify correctness, then commit any fixes", - ] - ) - for i, step in enumerate(steps, 1): - lines.append(f"{i}. {step}") - lines.append("") - - lines.append("## Parallel Execution with Subagents\n") - lines.append( - "You have access to Claude Code's **Agent tool** for spawning subagents. " - "Use it to parallelize independent work:\n" - ) - lines.append( - "- If the plan has multiple independent phases or task groups that don't touch " - "overlapping files, implement them in parallel by launching one subagent per " - "phase/group." - ) - lines.append( - "- Each subagent gets a clear, self-contained prompt describing its scope " - "(files to modify, tasks to complete, acceptance criteria)." - ) - lines.append( - "- Subagents share your working directory and git state. Ensure parallel " - "subagents work on **non-overlapping files** to avoid conflicts." - ) - lines.append( - "- Subagents should only edit files — do NOT stage or commit from subagents. " - "After each group of parallel subagents completes, **immediately** commit and " - "push their combined changes before launching the next group." - ) - lines.append( - "- After subagents complete, verify the combined changes compile, pass tests, " - "and integrate correctly. Do NOT defer all commits to the end." - ) - lines.append( - "- For small or sequential tasks, just implement directly — don't over-parallelize." - ) - lines.append("") - lines.extend(_EXPLORATION_SUBAGENT_GUIDANCE) - else: - # Revision cycle: slim delta-focused prompt. - # Guard: if review_feedback is unexpectedly missing, fall - # back to including the task description so the coder isn't - # left with a nearly empty prompt. - if not review_feedback: - if prompt: - lines.append("## Task Description\n") - lines.append(prompt) - lines.append("") - - lines.append("## Revision Instructions\n") - if review_feedback: - has_tester_findings = TESTER_FINDINGS_HEADER in review_feedback - if has_tester_findings: - lines.extend( - [ - "The reviewer and tester found issues with your implementation. " - "Focus on addressing the specific feedback above.\n", - "1. Review the feedback in the **Prior Review Feedback** section above", - "2. Check `git diff` to understand the current state of changes", - f"3. Check `.egg-state/agent-outputs/" - f"{_pipeline_identifier(issue_number, pipeline_id)}" - f"-tester-output.json` for test failures and gaps", - "4. Fix the specific issues raised", - "5. Run tests to verify your fixes", - "6. Commit with descriptive messages", - "", - ] - ) - else: - lines.extend( - [ - "The reviewer found issues with your implementation. " - "Focus on addressing the specific feedback above.\n", - "1. Review the feedback in the **Prior Review Feedback** section above", - "2. Check `git diff` to understand the current state of changes", - "3. Fix the specific issues raised by the reviewer", - "4. Run tests to verify your fixes", - "5. Commit with descriptive messages", - "", - ] - ) - else: - lines.extend( - [ - "A revision was requested but no specific feedback was provided. " - "Review the task description above and check `git diff` for the current state.\n", - "1. Review the task description above and check `git diff`", - "2. Verify the implementation meets the requirements", - "3. Run tests to verify correctness", - "4. Commit with descriptive messages", - "", - ] - ) - - # Contract CLI instructions for both local and issue mode - lines.extend( - [ - "Use the contract CLI to track progress incrementally — update after " - "each commit, not in a batch at the end:", - "- `egg-contract show` — View current contract state", - "- `egg-contract complete-task --task <id> --commit <sha>` — Mark task done and link commit", - "- `egg-contract complete-phase --phase <id> --commit <sha>` — Mark phase done and link commit", - "- `egg-contract add-commit --task <id> --commit <sha>` — Link commit to task without marking done", - "", - ] - ) - - else: - lines.append(f"Execute the {phase} phase.\n") - - # --- Phase restrictions --- - lines.append("## Phase Restrictions\n") - if issue_number is None and phase in ("refine", "plan"): - lines.extend( - [ - "In this phase:", - "- You CAN push state files to git (contracts, drafts, checkpoints)", - "- You CAN create HITL decisions (egg-contract add-decision)", - "- You CAN create feedback requests (egg-contract add-feedback)", - "- You CANNOT push code changes", - "- You CANNOT create PRs (gh pr create)", - "- You CANNOT post comments to the GitHub issue (gh issue comment) — write reviews to `.egg-state/reviews/` instead", - "- You CANNOT edit the GitHub issue (gh issue edit)", - "- You CAN read and modify local files", - "- You CAN run tests", - "- You CAN commit locally", - "", - ] - ) - elif issue_number is None and phase == "implement": - lines.extend( - [ - "In this phase:", - "- You CAN push code changes to git", - "- You CANNOT push .egg-state/ files (except checkpoints)", - "- You CANNOT create PRs (gh pr create)", - "- You CANNOT post comments to the GitHub issue (gh issue comment)", - "- You CANNOT edit the GitHub issue (gh issue edit)", - "- You CAN read and modify local files", - "- You CAN run tests", - "- You CAN commit locally", - "", - ] - ) - else: - if phase in ("refine", "plan"): - lines.extend( - [ - "- You CAN write drafts to `.egg-state/drafts/`", - "- You CAN push draft files (git push)", - "- You CAN create HITL decisions (egg-contract add-decision)", - "- You CAN create feedback requests (egg-contract add-feedback)", - "- You CANNOT post comments to the GitHub issue (gh issue comment) — write reviews to `.egg-state/reviews/` instead", - "- You CANNOT edit the GitHub issue (gh issue edit)", - "- You CANNOT create PRs (gh pr create)", - "", - ] - ) - elif phase == "implement": - lines.extend( - [ - "- You CAN push code (git push)", - "- You CAN link commits to tasks (egg-contract add-commit)", - "- You CANNOT create PRs (the pipeline manages the PR)", - "- You CANNOT post comments to the GitHub issue (gh issue comment)", - "- You CANNOT edit the GitHub issue (gh issue edit)", - "", - ] - ) - # --- Completion --- - lines.append("## Phase Completion\n") - if phase in ("refine", "plan"): - lines.append( - "When your draft is complete, commit and push it. " - "The pipeline will have an internal reviewer evaluate your work. " - "If revisions are needed, you'll be re-invoked with feedback. " - "Only after internal review passes will the output be posted " - "for human approval." - ) - else: - lines.append( - "When you have completed your work for this phase, " - "ensure everything is committed and exit successfully." - ) - - return "\n".join(lines) - - -# --------------------------------------------------------------------------- -# Multi-agent execution helpers -# --------------------------------------------------------------------------- - - -def _contract_enforcer_role_names() -> frozenset[str]: - """Roles whose ACK/CONFIRM is gated on contract-task completeness (#3114). - - Lazy wrapper so the preamble builder keys its enforcer-specific - instructions off the same capability set the orchestrator's signal - gate enforces (``egg_contracts.agent_roles.CONTRACT_ENFORCER_ROLES``) - — prose and enforcement stay in lockstep. - """ - from egg_contracts.agent_roles import CONTRACT_ENFORCER_ROLE_NAMES - - return CONTRACT_ENFORCER_ROLE_NAMES - - -def _build_brc_preamble( - role_value: str, - phase: str, - repo: str | None = None, - branch: str | None = None, - base_branch: str | None = None, -) -> str: - """Build the BRC consensus lifecycle preamble for an agent. - - Returns a formatted string block that can be appended to any agent prompt - to inject BRC protocol instructions. Used by both the coder/refiner path - (which delegates to _build_phase_prompt) and the generic multi-agent path. - - Includes: - - Agent roster showing all active agents and what they produce - - Role-specific proactive preparation instructions - - Full BRC lifecycle steps (including the generic no-op propose path, - #3027, for a producer that finds it has no work in this slice) - """ - try: - from review_graph import get_review_graph_for_phase - - graph = get_review_graph_for_phase(phase, repo=repo) - is_producer = graph.is_producer(role_value) - is_reviewer = graph.is_reviewer(role_value) - reviewers = graph.reviewers_for(role_value) if is_producer else [] - producers = graph.producers_for(role_value) if is_reviewer else [] - wake_only_producers = graph.wake_only_producers_for(role_value) - all_roles = sorted(graph.all_roles()) - graph_available = True - except Exception: - is_producer = role_value in ( - "coder", - "tester", - "documenter", - "refiner", - "architect", - "task_planner", - "risk_analyst", - "simplifier", - ) - is_reviewer = role_value in ( - "reviewer_code", - "reviewer_code_holistic", - "reviewer_contract", - "tester", - "reviewer_refine", - "reviewer_agent_design", - # first_principles_reviewer is a genuine refine-phase reviewer: it - # casts a real ACK verdict on the refiner (CRITICAL edge), so the - # degraded fallback keeps its Reviewer Lifecycle block. It never - # NACKs (redirects go to the operator as HITL decisions), but it - # DOES vote, so — unlike the simplifier — it is a real verdict and - # ``casts_real_verdicts`` (raw ``is_reviewer`` here) stays True. - "first_principles_reviewer", - "reviewer_plan", - # risk_analyst is a genuine dual-role reviewer in the plan graph - # (CRITICAL reviewer of architect + task_planner, #2809) as well as - # a producer of the risk register. Listed here so the degraded - # fallback path keeps its Reviewer Lifecycle / "As a reviewer" - # block instead of stripping it to producer-only — mirroring the - # live plan graph. Unlike the simplifier its edges are real - # verdicts, so ``casts_real_verdicts`` (raw ``is_reviewer`` in the - # degraded path) correctly stays True for it. - "risk_analyst", - # simplifier retains a wake_only advisory edge over the upstream - # refine/plan producer, so the graph reports it as a reviewer — - # but it casts no verdict (#3381) and is rendered PRODUCER-ONLY - # below (the wake_only edge is excluded from the real-reviewer - # determination). Listed here so the degraded fallback path still - # recognizes it; producer-only rendering is handled uniformly. - "simplifier", - ) - reviewers = [] - producers = [] - wake_only_producers = set() - all_roles = [] - graph_available = False - - lines: list[str] = [ - "\n\n## CRITICAL: BRC Consensus Protocol\n", - "You are running in CONCURRENT mode with the Broadcast-Review-Converge " - "(BRC) protocol. Your job is NOT just your task — it is the **full " - "BRC lifecycle**.\n", - ] - - is_dual_role = is_producer and is_reviewer - - # A role whose only reviewed producers are reached via wake_only edges - # (the de-roled simplifier, #3381) casts no verdict on anyone, so it is a - # PRODUCER in every behavioural sense — render it as one. We keep the - # graph-level ``is_dual_role`` flag intact for the banner dispatch below; - # only the rendered role-type label and the "assigned producers" line - # exclude wake_only producers, so the preamble does not contradict the - # producer-only execution banner the simplifier receives. - real_producers = [p for p in producers if p not in wake_only_producers] - if graph_available: - casts_real_verdicts = bool(real_producers) - else: - # Degraded path: the graph load failed, so ``producers == []`` for every - # role and we cannot distinguish wake_only edges from real ones. Fall - # back to raw ``is_reviewer`` so we don't silently strip the Reviewer - # Lifecycle / "As a reviewer" coordination block from a *genuine* - # reviewer (reviewer_code/refine/plan) when the graph is unavailable — - # pre-#3381 this path gated those blocks on raw ``is_reviewer``. The - # simplifier — the only wake_only role — stays producer-only: it is - # excluded here, and is independently rendered producer-only by the - # ``is_dual_role and role_value == "simplifier"`` banner dispatch, which - # still fires in the fallback. - casts_real_verdicts = is_reviewer and role_value != "simplifier" - - if is_producer and casts_real_verdicts: - role_type_desc = "PRODUCER and REVIEWER (dual role)" - elif is_producer: - role_type_desc = "PRODUCER" - elif casts_real_verdicts: - role_type_desc = "REVIEWER" - else: - role_type_desc = "PARTICIPANT" - - lines.append(f"Your role type: **{role_type_desc}**") - if reviewers: - lines.append(f"Your reviewers: {', '.join(reviewers)}") - if real_producers: - lines.append(f"Your assigned producers: {', '.join(real_producers)}") - lines.append("") - - # Agent roster: show all active agents and what they do - if all_roles: - roster = _build_agent_roster(all_roles, role_value, phase) - if roster: - lines.append(roster) - - # Dual-role ordering banner (#2749, updated for coder-owns-tests). A - # dual-role agent (today: only TESTER in the implement graph) receives - # both the Producer and Reviewer Lifecycle blocks below. The coder now - # authors its own tests; the tester's job is to review-and-harden them - # after the coder proposes. So the tester's producer WORK legitimately - # depends on the coder's ``CONSENSUS_PROPOSE`` — it orients up-front, - # exits after ORIENT, and is re-invoked by the event-pump wrapper when - # the coder proposes, at which point it hardens + proposes + ACK/NACKs - # in one pass. This does not reintroduce the f4c7d780 / 8b81ed32 - # self-block (where the tester idled on a reviewer wait-loop before - # proposing its own scaffolded work): the coder proposes independently - # and does not wait on the tester, so the coder's propose is the - # trigger, and the tester proposes right after. The tester therefore - # has TWO reviewer rendezvous points, both surfaced as fresh wrapper - # invocations under the event-pump model: (a) the coder's first - # ``CONSENSUS_PROPOSE`` re-invokes the tester so it has something to - # harden; (b) subsequent re-proposes and peer-producer proposals - # (after the tester has proposed) likewise re-invoke the tester to - # handle the Reviewer Lifecycle for those events. - if is_dual_role and role_value == "simplifier": - # The simplifier is a PRODUCER ONLY (#3381). It is woken to write the - # companion by the ordinary producer propose-arm (it self-gates on the - # upstream draft existing), NOT by its advisory edge over the upstream - # — that edge is wake_only and casts no verdict, so it is inert in - # consensus derivation (see review_graph.ReviewEdge.wake_only). It is - # NOT a reviewer in any behavioural sense: it issues no verdict, casts - # no ACK/NACK, and never critiques the draft. Treating it as a reviewer - # is what made the companion come out as a review/critique memo instead - # of a plain-language summary. So it gets a PRODUCER-ONLY banner here - # and must NOT inherit the tester's review-and-harden banner below. - lines.append( - "### Execution Order (READ FIRST — simplifier)\n\n" - "You are a **producer only**: your single job is to write a " - "plain-language, human-focused companion to the upstream " - "producer's draft. You do **not** review, critique, score, or " - "vote on that draft — you never issue an ACK or a NACK. (An " - "internal wake-wire re-invokes you when the upstream proposes so " - "you know its draft is ready; consensus never waits on a verdict " - "from you, so there is nothing to respond to.)\n\n" - "**Execute in this order:**\n\n" - "1. **ORIENT (FIRST).** Read the contract and orient. Your work " - "depends on the upstream producer's draft existing, so you begin " - "writing only once that producer issues `CONSENSUS_PROPOSE` — the " - "event-pump wrapper re-invokes you carrying that proposal. Do not " - "race ahead before the draft exists.\n" - "2. **On the upstream producer's PROPOSE**, the wrapper re-invokes " - "you with the proposal in your event payload. SYNC the worktree, " - "read the draft, then write and PROPOSE the human-focused " - "companion (see Producer role below). That is the whole job — " - "the companion is a simplified summary written *for humans to " - "read*, never a review of the draft, a list of constraints the " - "draft should satisfy, or an ACK/NACK rationale.\n" - ) - elif is_dual_role: - lines.append( - "### Dual-Role Execution Order (READ FIRST — #2749, updated for " - "coder-owns-tests)\n\n" - "You are both PRODUCER and REVIEWER (TESTER). **The BRC round " - "cannot close until every producer (including you) has issued " - "`mcp__brc__propose` / `egg-orch consensus propose`** — so you " - "MUST eventually propose, and if you never propose your own " - "hardening you self-block the round. But your producer WORK " - "(reviewing and **hardening the coder's tests**) genuinely " - "depends on the coder's proposed tests existing, so unlike a " - "normal producer you start that work at the coder's PROPOSE, " - "not before. This does not deadlock: the coder proposes " - "independently and does **not** wait on you, so its " - "`CONSENSUS_PROPOSE` is the trigger that unblocks your work; " - "the event-pump wrapper re-invokes you carrying that PROPOSE " - "in your event payload, and you propose right after.\n\n" - "**Execute the lifecycles in this strict order:**\n\n" - "1. **Producer ORIENT (step 1) comes FIRST.** Run ORIENT now " - "to load context. **Your role-specific orientation tells you " - "whether Producer WORK (step 2) runs immediately or is gated " - "on an upstream producer's `CONSENSUS_PROPOSE`** — e.g. the " - "implement-phase tester reviews-and-hardens the coder's tests, " - "so its WORK begins after the coder proposes (#2936). Do not " - "race ahead of the role-specific orientation. While you are in " - "ORIENT, you may *opportunistically* do the Reviewer " - "Lifecycle's `1. PREPARE` work — read the contract, scan the " - "upstream producer's commits as they land on the branch — but " - "do NOT start producing artifacts your role-specific " - "orientation gates on an upstream PROPOSE. Do NOT block on a " - "reviewer wait as your scheduling primitive: the event-pump " - "wrapper invokes you again when the upstream producer's " - "`CONSENSUS_PROPOSE` arrives, at which point you handle the " - "review AND (if your WORK was gated on it) start producing.\n" - "2. **On an upstream producer's PROPOSE**, the wrapper " - "re-invokes you with the proposal in your event payload. " - "SYNC the worktree, then do your Producer WORK (read the " - "coder's tests; add the missing regression + adversarial " - "cases yourself — you share the test scope with the coder; " - "run the tests) and **PROPOSE** your hardening. In the same " - "invocation, issue your reviewer verdict on the coder: ACK " - "if coverage is sound, or NACK naming the specific failing " - "test / coverage gap.\n" - "3. **Subsequent invocations** (re-proposes from any " - "producer — `CONSENSUS_PROPOSE` version > 1 — and " - "`CONSENSUS_RE_REVIEW` events) surface as new wrapper " - "invocations. Each one is a fresh review against the new " - "delta; the per-event prompt includes the full " - "`git log {last_reviewed_commit_sha}..HEAD --not " - "origin/{base_branch} -p` so you can audit the change. " - "Fall through to Reviewer Lifecycle step 3 (SYNC) → step 4 " - "(REVIEW) → step 5 (ACK/NACK), then exit. Do NOT skip step " - "4 (REVIEW) — reading the actual referenced files and " - "forming independent judgment from them is what keeps " - "re-reviews from becoming rubber-stamps.\n" - ) - - if is_producer: - producer_lifecycle: list[str] = ["### Producer Lifecycle"] - # The no-op propose path (#3027) is only valid in the implement - # phase. In refine/plan the producer's draft is mandatory and the - # orchestrator rejects no-op explicitly — so don't even surface the - # affordance to refine/plan producers (architect, refiner, - # task_planner, risk_analyst), keeping prose and enforcement in - # lockstep (review feedback on #3029). - propose_line = ( - "3. **PROPOSE**: When done, run: " - '`egg-orch consensus propose --summary "..." --artifacts "file1" "file2" ' - '--files-changed "f1.py" "f2.py" --tests-run "test_a" "test_b" ' - '--tasks "task-1-1" "task-1-2" --commit-sha $(git rev-parse HEAD)`. ' - "The `--summary` must be ≥50 chars of substantive content describing what was " - "built, what was tested, and which contract tasks it satisfies. " - "Boilerplate like 'looks good' or 'approved' will be rejected." - ) - if phase in ("refine", "plan") and role_value in ( - # Keep in lockstep with ``_DECISION_ATTESTING_ROLES`` in - # ``routes/signals/_validation.py`` — the enforcement side of - # this prose (#3390). - "refiner", - "task_planner", - "architect", - "risk_analyst", - ): - propose_line += ( - "\n\n" - " **Attest your decision ledger (#3390 — MANDATORY).** The " - "orchestrator REJECTS your propose unless its attestation " - "carries your HITL decision ledger. Pass " - "`--decisions-registered cq-1 cq-2 ...` listing every decision " - "you registered this phase (via `egg-contract add-decision` / " - "`mcp__sdlc__register_open_question`), or " - '`--no-decisions-rationale "<why>"` when the phase ' - "deliberately raises none — an explicit empty ledger, never an " - "omission. (Via MCP: the `attestation` arg of " - "`mcp__brc__propose`, fields `decisions_registered` / " - "`no_decisions_rationale`.) Attested ids are cross-checked " - "against the contract, and your draft must cite each attested " - "`cq-N` (copying the `--format markdown` output into the " - "draft satisfies this). A decision your draft commits to " - "without a registered `cq-N` is a reviewer NACK — register " - "it or remove the unilateral commitment. The rationale form " - "is not a shortcut (#3462): the operator is asked to confirm " - "it as its own decision before the phase gate, and a rejected " - "'none' re-runs the phase. If the task names decisions to " - "surface — or you believe a decision is already resolved by " - "prior context — register it with your recommended answer as " - "the first option instead of attesting none." - ) - if phase == "implement": - propose_line += ( - "\n\n" - " **Mark your contract tasks complete (#3114).** Record each " - "delivered task with `mcp__task__complete` (link the commit) — " - "the contract reviewer's ACK is gated on your rows being " - "`complete`, so finished-but-unrecorded work blocks the slice. " - "A task waiting on a peer's work: note it in your proposal and " - "deliver after the dependency lands; the gate holds the slice " - "open until then." - ) - propose_line += ( - "\n\n" - " **No work for you in this slice? Submit a no-op propose (#3027).** " - "If after ORIENT you find your role has no assigned task here AND " - "nothing to contribute (e.g. a documenter on a code-only slice, a " - "tester on a doc-only slice, your domain is not impacted by the " - "diff), do NOT skip silently and do NOT invent busywork — run " - "`egg-orch consensus propose --no-changes-needed --no-changes-reason " - '"<why you have no work here>"` (no artifacts or commit-sha needed). ' - "This counts as proposing, so consensus is not blocked waiting on " - "you; reviewers accept it as a non-blocking no-op (they will not " - "NACK it). Then CONFIRM (step 5) as normal once peers have proposed. " - "Reach for a real propose instead the moment you do find work " - "(e.g. the coder's diff turns out to need docs). Rejected while " - "you still own incomplete contract tasks here (#3114)." - ) - producer_lifecycle.extend( - [ - "1. **ORIENT**: Before starting work, " - + _build_producer_orientation( - role_value, - phase, - reviewers, - branch=branch, - ), - "2. **WORK**: Complete your assigned task (see Your Task below).", - propose_line, - "4. **RESPOND TO REVIEWS**: When a reviewer NACKs your " - "proposal you will be re-invoked to address it. Read every " - "NACK in the event payload, fix all named blockers, and " - "re-propose with `--changed-artifacts`. **Aggregation is " - "enforced by the orchestrator (#2142):** when two or more " - "distinct reviewers have NACKed the current version, the " - "re-propose call returns HTTP 409 with the full set " - "(reviewer, reason, artifact_refs) inline in `details`; " - "address every NACK then retry. A single-reviewer NACK " - "does not trigger the barrier — re-propose proceeds " - "normally.\n\n" - " **A NACK naming new findings on your re-propose is " - "legitimate adversarial review, not goalpost-moving.** " - "Reviewers re-review v2+ as a fresh delta; \"that's not " - "what you NACK'd last time\" is not a valid objection. " - "**You can and should push back on a NACK on its merits** — " - "if the reviewer misread the code or the concern does not " - "apply, contest it via a directed message with evidence " - "(file:line, test, doc reference). What is *not* productive " - "is contesting a NACK you know is correct — re-reviews are " - "cheap by design, so when the finding is real, fix it and " - "re-propose.", - "5. **CONFIRM**: When all reviewers ACK, run " - "`egg-orch consensus confirmed` to mark your role's " - "consensus.", - "6. **HANDLE RE-REVIEW**: When you are re-invoked with a " - "`CONSENSUS_RE_REVIEW` event" - + ( - " (or a `CONSENSUS_PROPOSE` for a re-propose — " - "version > 1, after you NACKed a prior version; " - "dual-role agents handle both — see Reviewer " - "Lifecycle step 7 for the adversarial re-review " - "framing)" - if is_dual_role and casts_real_verdicts - else "" - ) - + ", act on it — failure to respond stalls the pipeline. " - + ( - "If you are a reviewer of the re-proposing producer, " - "re-review and ACK/NACK the new proposal (dual-role " - "agents: see Reviewer Lifecycle step 7 below for the " - "adversarial re-review framing that applies to this " - "case). Otherwise, re-confirm via " - "`egg-orch consensus confirmed`." - if casts_real_verdicts - else "Re-confirm via `egg-orch consensus confirmed`." - ), - "7. **RESOLVE OBLIGATIONS YOU SATISFY (#2338)**: If you " - "land a commit that satisfies a *different* producer's " - "conditional-ACK obligation in-cycle — typical pattern: " - "the coder is gateway-blocked from a path under `tests/`, " - "you (as tester) cherry-pick the satisfying commit onto " - "the branch — call `mcp__brc__resolve_obligation " - 'reviewer_role="<reviewer>" producer_role="<other_producer>" ' - "commit_sha=$(git rev-parse HEAD)` after pushing. The " - "matrix keeps the obligation text for audit but stops " - "surfacing it on the PR body and HITL gate. Skip this " - "for obligations that genuinely require a human at " - "merge time (deploys, cross-repo flips) — those should " - "remain visible to the merger. **Resolve before " - "`complete_phase`**: once the HITL gate has fired and " - "written the obligation to `contract.pr.deferred_actions`, " - "calling `resolve_obligation` afterwards does *not* " - "retroactively unpersist the entry — the obligation will " - "still appear in the PR body until the next pipeline run. " - "Resolve early. Producers cannot self-resolve their own " - "obligations (the orchestrator rejects " - "`resolver_role == producer_role`), since that would " - "single-handedly bypass the reviewer's veto.\n", - ] - ) - lines.extend(producer_lifecycle) - - # Gate the Reviewer Lifecycle on ``casts_real_verdicts``, not raw - # ``is_reviewer`` (#3381). A role whose only reviewed producers are - # reached via ``wake_only`` edges (the de-roled simplifier) issues no - # ACK/NACK and must NOT receive the reviewer playbook — REVIEW, ACK/NACK, - # CONFIRM, adversarial re-review — which would directly contradict its - # producer-only execution banner. This mirrors the producer-only invariant - # already asserted for the coder (``test_producer_only_no_sync_step``): a - # producer-only role gets no ``### Reviewer Lifecycle`` at all. A pure - # reviewer (``reviewer_refine``) and the dual-role tester both cast real - # verdicts, so they keep it. - if is_reviewer and casts_real_verdicts: - lines.extend( - [ - "### Reviewer Lifecycle", - "1. **PREPARE** (while waiting): " - + _build_reviewer_preparation( - role_value, - phase, - branch=branch, - base_branch=base_branch, - ), - "2. **INVOKED PER EVENT**: The orchestrator's event-pump " - "wrapper invokes you one-shot per actionable event. When a " - "`CONSENSUS_PROPOSE` arrives for an assigned producer, " - "you are spawned with the proposal in your event payload. " - "Do your preparation work from step 1 on the first " - "invocation; subsequent invocations land you directly at " - "step 3 (SYNC) with the proposal already in context." - + ( - "\n\n **Dual-role agents (you)** — per the " - "*Dual-Role Execution Order* banner above (updated " - "for coder-owns-tests): your first invocation does " - "ORIENT/PREPARE only. On the coder's " - "`CONSENSUS_PROPOSE` the wrapper re-invokes you with " - "the proposal in your event payload; SYNC, do your " - "Producer WORK (review + harden the coder's tests), " - "then PROPOSE your hardening and ACK/NACK the coder " - "in the same invocation (fall through to step 3 " - "(SYNC) → step 4 (REVIEW) → step 5 (ACK/NACK) here). " - "Subsequent invocations (re-proposes — " - "`CONSENSUS_PROPOSE` version > 1 — and peer-producer " - "proposals) are fresh reviews against the new delta, " - "not continuations." - if is_dual_role - else "" - ), - "3. **SYNC**: Before reviewing, sync your worktree so you have the " - "producer's commits: `git fetch origin && git merge " - + _resolve_origin_ref(branch or base_branch) - + " --no-edit`", - "4. **REVIEW**: Once a proposal arrives, form independent judgment from " - "the referenced code artifacts. Read the actual files — do not rely " - "solely on the proposal summary.", - "5. **ACK/NACK**: Your `--reason` IS your review. Put your **full analysis** " - "there — this is what the producer reads and acts on. **Always " - "pass `--ack-version` / `--nack-version`** with the producer's " - "current proposal version (#2142) — read it from the " - "`CONSENSUS_PROPOSE` message that triggered your review (the " - "`version` field). The orchestrator rejects the verdict with " - "`stale_version` if the producer has re-proposed since you " - "started reviewing.\n" - "\n" - " **NACK format** (use when blocking issues exist):\n" - " ```\n" - ' egg-orch consensus nack <role> --files-reviewed "f1" "f2" ' - '--nack-version <N> --reason "\n' - " ### Blocking\n" - " 1. **file.py:123** — Description of the issue. Fix: suggested fix.\n" - " 2. **file.py:456** — Description of the issue. Fix: suggested fix.\n" - " ### Non-blocking\n" - " - **file.py:789** — Suggestion for improvement.\n" - ' "\n' - " ```\n" - "\n" - " **ACK format** (use when no blocking issues):\n" - " ```\n" - ' egg-orch consensus ack <role> --files-reviewed "f1" "f2" ' - '--ack-version <N> --reason "\n' - " Reviewed [N files / specific areas]. Verified [what was checked].\n" - " [Specific observations about correctness, security, etc.]\n" - " ### Non-blocking\n" - " - **file.py:123** — Optional suggestions for improvement.\n" - ' "\n' - " ```\n" - "\n" - " **Conditional ACK (#1998)** — use when the work is " - "correct but a human action is needed at merge time " - "(`git mv`, secret rotation, cross-repo flip): add " - '`--pre-merge-condition "…"` to the ACK. The obligation ' - "renders as a `Pre-merge Obligations` block in the PR " - "body. Do NOT use this to smuggle blocking issues past " - "the producer — if the producer could fix it, NACK " - "instead.\n" - "\n" - " **Drop satisfied obligations on re-ACK (#2338).** When " - "you re-ACK at a new proposal version and the conditioning " - "work has landed in-cycle (the rename is in the diff, the " - "obligation is moot), drop the obligation: re-ACK without " - "`--pre-merge-condition`. Do NOT re-attach it with a " - 'self-contradicting "satisfied" hedge — the PR body ' - "renders obligations verbatim under a `do not merge` " - "banner. To preserve the audit trail instead of dropping, " - "re-ACK with `--pre-merge-condition-resolved-in-diff <sha>` " - "alongside `--pre-merge-condition` so the renderer " - "demotes (not drops) the entry (#2336).\n" - "\n" - " `--reason` must be ≥50 chars of substantive content. " - "Boilerplate like 'lgtm' or 'no issues' will be rejected.\n" - "\n" - " **Stale-version rejection (#2142):** if the producer " - "re-proposed while your verdict was in flight, the ACK / " - "NACK is rejected with HTTP 409 inlining the current " - "proposal snapshot (version, artifacts, commit_sha). " - "`git fetch && git merge`, re-review against the new " - "commit, and re-submit — don't retry the same payload." - + ( - "\n\n" - " **Contract-enforcer gate (#3114) — applies to you.** " - "Your ACK of a producer is structurally gated on the " - "contract: the orchestrator REJECTS it (409 " - "`contract_incomplete`) while any task row owned by that " - "producer in this slice is not `status=complete`. Read " - "the live task records with `mcp__sdlc__show_contract` — " - "the `.egg-state/contracts/` copy in your checkout is an " - "init-time snapshot; do not trust it. When rows are " - "incomplete, NACK the producer citing the exact task " - "ids: either the work is missing (it must deliver) or it " - "landed unrecorded (it must run `mcp__task__complete`). " - "When all rows are complete, your ACK MUST carry " - '`attestation={"tasks_verified": ["task-…", …]}` on ' - "`mcp__brc__ack`, covering every task id the producer " - "owns in this slice — absent or non-covering lists are " - "rejected (`attestation_required` / " - "`attestation_mismatch`). Your CONFIRM is likewise " - "rejected while ANY row in the slice is incomplete. A " - "producer's declared deferral (\"will land in later " - 'proposals") is an open obligation, not an end-state — ' - "hold consensus open until the rows are delivered or a " - "human descopes them." - if phase == "implement" and role_value in _contract_enforcer_role_names() - else "" - ), - "6. **CONFIRM**: When all assigned producers reviewed: " - "`egg-orch consensus confirmed`", - "7. **HANDLE RE-REVIEW**: When you are re-invoked with a " - "`CONSENSUS_RE_REVIEW` event (or a `CONSENSUS_PROPOSE` for " - "a re-propose — version > 1, after you NACKed a prior " - "version), act on it — failure to respond stalls the " - "pipeline. Re-review the re-proposing producer's new " - "proposal and ACK/NACK it, then re-confirm via " - "`egg-orch consensus confirmed`.\n\n" - " **This is adversarial re-review, not blocker-verification.** " - "Your re-review has TWO equal-weight mandates: (1) verify the " - "blockers from your prior NACK were addressed AND (2) audit the " - "delta since your last review — the commits landed since the " - "version you last verdicted (per REVIEWER-SYNC.md: `git log " - "{last_reviewed_commit}..HEAD --not origin/{base_branch} -p`) — " - "as a fresh reviewer with no NACK history, bounded to that " - "delta, NOT the whole accumulated surface. Both must pass to " - "ACK. The orchestrator's adversarial re-prime in the event " - "body carries the full framing; this is a pointer. New issues " - "outside your prior NACK's scope are blocking; **NACK without " - "hesitance** — re-reviews are cheap by design, and the " - "downstream GitHub reviewer should find nothing in your " - "re-reviewed deltas.\n", - ] - ) - - # Directed coordination guidance — role-gated - lines.append("### Directed Coordination") - lines.append( - "In addition to the BRC consensus flow (PROPOSE/ACK/NACK), you can send " - "directed peer-to-peer messages to specific agents using " - "`egg-orch message send --to <role> --type <TYPE>`. These directed messages " - "are **supplementary** to BRC consensus — they do NOT replace the " - "PROPOSE/ACK/NACK lifecycle and are never required for consensus to proceed.\n" - ) - - if is_producer: - lines.extend( - [ - "**As a producer**, use directed messages to coordinate handoffs and " - "broadcast progress:", - "- **HANDOFF**: When your work is ready for a specific peer to act on, " - "send a HANDOFF message so they know to begin. For example, a coder " - "notifying the tester that implementation is complete.", - " ```", - ' egg-orch message send --to tester --type HANDOFF --subject "Auth module ready" ' - '--body "auth.py is complete, tests can begin"', - " ```", - "- **STATUS**: Broadcast progress updates to all agents when you reach " - "significant milestones (e.g., halfway through implementation, blocked " - "on a dependency).", - " ```", - ' egg-orch message send --to all --type STATUS --subject "Implementation 50% complete" ' - '--body "Core logic done, working on edge cases"', - " ```\n", - ] - ) - - # Same gate as the Reviewer Lifecycle above (#3381): a wake-only, - # verdict-free role (de-roled simplifier) gets no reviewer-coordination - # guidance, since it never ACK/NACKs. - if is_reviewer and casts_real_verdicts: - lines.extend( - [ - "**As a reviewer**, when you need clarification before " - "ACK/NACKing, put the question in your NACK `--reason` " - "block under `### Non-blocking`. The producer sees it " - "atomically with the review verdict and the audit " - "trail is preserved. The legacy QUESTION message " - "type was removed in issue #1897; off-protocol chatter " - "is no longer advertised. A follow-up issue will " - "introduce a structured REQUEST/REPLY subsystem that " - "names a target peer and times out.", - "", - ] - ) - - lines.extend( - [ - "**Event-handler contract (#2908):** The orchestrator's " - "event-pump wrapper drives your lifecycle. You are invoked " - "one-shot per actionable BRC event: handle the event per the " - "lifecycle above, update durable BRC memory (writes happen " - "automatically inside `egg-orch consensus ack` / `nack` " - "handlers), then exit naturally. The wrapper polls " - "`egg-orch brc next-action` and re-invokes you with the next " - "event. You do NOT block on `egg-orch message wait-loop` " - "yourself; the wrapper owns the wait and the heartbeat.\n", - "", - ] - ) - - return "\n".join(lines) - - -# Role descriptions for agent roster — maps role names to (short description, -# what artifacts they produce). -_ROLE_DESCRIPTIONS: dict[str, tuple[str, str]] = { - "coder": ( - "Implements code changes", - "commits with source files, tests may be included", - ), - "tester": ( - "Writes comprehensive regression tests AND adversarially probes the " - "coder's implementation for bugs and edge cases (dual role: also " - "reviews coder)", - "test files (including failing tests that demonstrate bugs), check " - "results, gap reports back to the coder", - ), - "documenter": ( - "Documents the current state of the code", - "doc files, README updates, inline documentation", - ), - "refiner": ( - "Refines implementation based on review feedback", - "updated source files addressing review concerns", - ), - "architect": ( - "Designs architecture and component structure", - "architecture analysis, component breakdown", - ), - "task_planner": ( - "Breaks work into implementation tasks", - "task list with acceptance criteria", - ), - "risk_analyst": ( - "Assesses technical risks", - "risk assessment with mitigations", - ), - "reviewer_code": ( - "Reviews code quality, correctness, and security", - "ACK/NACK with file-level feedback", - ), - "reviewer_code_holistic": ( - "Holistic single-pass review for cross-module coherence " - "(use-case end-to-end, doc↔code symmetry, synthetic-key audit, " - "silent-fallback hunt)", - "ACK/NACK with cross-module findings", - ), - "reviewer_contract": ( - "Verifies implementation matches contract/requirements", - "ACK/NACK with task-level verification", - ), - "reviewer_refine": ( - "Reviews refinement changes", - "ACK/NACK on refined implementation", - ), - "first_principles_reviewer": ( - "Adversarially reviews the seed and the refiner's direction from " - "first principles; surfaces significant redirects to the operator as " - "HITL decisions and never NACKs the refiner", - "an ACK on the refiner plus any HITL redirect decisions", - ), - "reviewer_agent_design": ( - "Reviews agent design and architecture decisions", - "ACK/NACK on design choices", - ), - "reviewer_plan": ( - "Reviews plan phase outputs", - "ACK/NACK on architecture, tasks, and risk assessment", - ), - "simplifier": ( - "Distills the producer's draft into a jargon-free, human-focused " - "companion summary (depends on the producer's pushed draft)", - "a simplified `*-human.md` companion to the analysis/plan", - ), -} - - -def _build_agent_roster(all_roles: list[str], current_role: str, phase: str) -> str: - """Build a roster of all active agents for the current phase. - - Shows each agent's role, what they do, and what they produce so that - every agent understands who else is running and what to expect. - """ - roster_lines = ["### Active Agents in This Phase\n"] - roster_lines.append( - "The following agents are running **simultaneously**. " - "Each must complete their task AND reach CONFIRMED via BRC.\n" - ) - for role in all_roles: - desc, artifacts = _ROLE_DESCRIPTIONS.get( - role, ("Executes assigned role", "role-specific artifacts") - ) - marker = " **(you)**" if role == current_role else "" - roster_lines.append(f"- **{role}**{marker}: {desc}. Produces: {artifacts}.") - roster_lines.append("") - return "\n".join(roster_lines) - - -def _build_reviewer_preparation( - role_value: str, - phase: str, - *, - branch: str | None = None, - base_branch: str | None = None, -) -> str: - """Build proactive preparation instructions for reviewer agents. - - Tells reviewers what to do while waiting for proposals — e.g., reading - the contract, familiarizing themselves with the codebase, preparing - review criteria. This avoids idle waiting and produces better reviews. - - Args: - role_value: The reviewer role (e.g. ``reviewer_code``). - phase: Pipeline phase name. - branch: The pipeline's work branch, if any. - base_branch: The resolved base branch for diff/log commands. Falls - back to ``main`` when ``None``. - """ - base_ref = _resolve_origin_ref(base_branch) - - if phase == "implement": - if role_value == "reviewer_code": - return ( - "Start reviewing immediately — do not wait idle for proposals. " - "(a) Read the contract with `egg-contract show` to understand " - "what was planned. " - "(b) Review the issue/PR description for context. " - "(c) Check for commits on the branch: run " - f"`git fetch origin && git log --oneline {base_ref}..origin/{branch or '$(git branch --show-current)'}` " - "and if changes exist, begin reviewing the diff with " - f"`git diff {base_ref}...HEAD`. " - "(d) Note existing test patterns and code conventions. " - "By the time a proposal arrives, you should already have " - "a thorough understanding of the changes and be ready to " - "ACK or NACK with specific, detailed feedback. " - "When reviewing the tester's proposal, check whether tests were " - "actually executed (look for `tests_run` and `tests_execution_blocked` " - "in the attestation). If the tester reports `tests_execution_blocked: true`, " - "this is a blocking concern — NACK unless the limitation is clearly " - "documented and the tests are syntactically valid. " - "Also scrutinize low `tests_run` counts relative to change scope — " - "a multi-file change with only 1 test run warrants investigation. " - "If a producer has no work in this slice it submits a generic " - "no-op propose (`no_changes_needed=true`, #3027): the orchestrator " - "treats that as a non-blocking no-op and will not surface it to " - "you for review — there is nothing to ACK or NACK, and it does " - "not block consensus." - ) - elif role_value == "reviewer_code_holistic": - return ( - "Start preparing immediately — do not wait idle for proposals. " - "(a) Read the contract with `egg-contract show` to extract " - "the primary advertised use case (this is the path you will " - "walk end-to-end once the producer proposes). " - "(b) Review the issue / PR description and any doc files " - "the contract names — collect the doc-claimed behaviours " - "into a checklist for the symmetry pass. " - "(c) Identify the producer / consumer module pairs the plan " - "touches; these are where synthetic-key and silent-fallback " - "asymmetries hide. " - "(d) Once commits land " - f"(`git fetch origin && git log --oneline {base_ref}..origin/{branch or '$(git branch --show-current)'}`), " - f"skim `git diff {base_ref}...HEAD` once with the whole PR " - "in mind — do not verify line-by-line; defer that to " - "`reviewer_code`. Your job is the architectural-coherence " - "question line-by-line review does not own." - ) - elif role_value == "reviewer_contract": - return ( - "While waiting for proposals, prepare by: " - "(a) reading the contract with `egg-contract show` to understand " - "every task and its acceptance criteria, " - "(b) reviewing the issue description for original requirements, " - "(c) noting which tasks are marked as must-have vs nice-to-have. " - "When proposals arrive, you will verify each task's acceptance " - "criteria is met — prepare a checklist now." - ) - elif role_value == "tester": - return ( - "While waiting for the coder's proposal, prepare by: " - "(a) reading the contract with `egg-contract show` to understand " - "what's being implemented, " - "(b) identifying edge cases and boundary conditions from the " - "requirements, " - "(c) checking the existing test infrastructure (test frameworks, " - "fixtures, test utilities). " - "Start writing test scaffolding for known requirements while " - "waiting — you can finalize once you see the actual implementation." - ) - elif phase == "plan": - if role_value == "reviewer_plan": - return ( - "While waiting for proposals, prepare by: " - "(a) reading the issue description to understand the original " - "request, " - "(b) exploring the codebase to understand the current architecture " - "and components that may be affected, " - "(c) identifying potential risks or constraints the planners " - "should address. " - "Form your own mental model of how you would approach this — " - "then compare against the proposals when they arrive. " - "\n\n" - "**#2137 slice-DAG checks (mandatory):** " - "(1) **Forest-violation NACK** — if the contract was " - "rejected at plan ingestion with a " - "``forest_violation`` log discriminator (or the contract's " - "``plan_review_feedback`` carries a 'Plan ingestion REJECTED' " - "block), NACK the architect and cite the structured errors " - "verbatim. Instruct the architect to re-emit the slice " - "scaffold with ``serialized_chain_order`` populated on the " - "downstream slice. The SAME NACK applies to a " - "``slice_overlap_violation`` rejection (#3046 — a 'Plan " - "ingestion REJECTED: slices touch overlapping files' block): " - "two or more slices touch the same file with no dependency " - "ordering, so their branches fork independently off the shared " - "base and collide at integration. Instruct the architect to " - "serialise the overlapping cluster into one linear " - "``dependencies`` chain (or merge the slices) so each later " - "slice's branch is cut from the earlier one. " - "(2) **Slice-sizing NACK (hard, judgment-based — #2809)**: " - "slice composition is owned by the **architect**, not the " - "task_planner. You ARE empowered and required to hard-NACK " - "the architect on ``slice_size`` when a slice is oversized " - "for one BRC cycle. Use judgment — no fixed tasks-per-slice " - "or LOC budget. NACK when a slice bundles more than ~3 " - "distinct file-categories, combines deletion-heavy with " - "new-API-introduction work, would require >3–4 " - "commit-propose-revise cycles, or contains independent " - "task groups with no internal dependency. Name the seam in " - "your NACK so the architect's re-propose is actionable. " - "See criteria §11 for the full rubric and examples." - "\n\n" - "**Human-focused plan companion (the simplifier's " - "``*-plan-human.md``):** the simplifier produces a simplified, " - "plain-language companion to the plan for a **broad audience — " - "engineers, PMs, and managers**. You review it (CRITICAL). " - "**Read it side-by-side with the full plan** and ACK only when " - "it (a) faithfully captures the plan's essence, (b) is " - "materially lighter and more digestible than the full plan — " - "not a near-copy, (c) is readable by a non-engineer, and (d) " - "is free of egg-internal jargon (no " - "BRC/consensus/slice-DAG/contract/role terms). NACK the " - "**simplifier** (not the task_planner) if it misrepresents the " - "plan, leaks pipeline jargon, omits a material point, merely " - "duplicates the full plan, or — critically — reads as a " - "**review/critique** of the plan rather than a summary of it " - '(ACK/NACK language, "should commit to", "anti-pattern to ' - 'reject", constraint lists) or buries the reader in ' - "implementation detail (`file:line` refs, function/struct/field " - "names). A missing or empty companion is a NACK — the companion " - "is mandatory." - ) - elif phase == "refine": - if role_value in ("reviewer_refine", "reviewer_agent_design"): - base = ( - "While waiting for the refiner's proposal, prepare by: " - "(a) reading the prior review feedback that triggered this " - "refinement cycle, " - "(b) checking the current state of the code to understand " - "what was already implemented, " - "(c) verifying which review concerns are still outstanding. " - "When the proposal arrives, focus on whether the specific " - "feedback items were addressed." - ) - if role_value == "reviewer_refine": - base += ( - "\n\n" - "**Human-focused analysis companion (the simplifier's " - "``*-analysis-human.md``):** the simplifier produces a " - "simplified, plain-language companion to the analysis for a " - "**broad audience — engineers, PMs, and managers**. You " - "review it (CRITICAL). **Read it side-by-side with the full " - "analysis** and ACK only when it (a) faithfully captures the " - "analysis's essence, (b) is materially lighter and more " - "digestible than the full draft — not a near-copy, (c) is " - "readable by a non-engineer, and (d) is free of " - "egg-internal jargon. NACK the **simplifier** (not the " - "refiner) if it misrepresents the analysis, leaks pipeline " - "jargon, omits a material point, merely duplicates the full " - "draft, or — critically — reads as a **review/critique** of " - "the analysis rather than a summary of it (ACK/NACK " - 'language, "should commit to", "anti-pattern to reject", ' - "constraint lists) or buries the reader in implementation " - "detail (`file:line` refs, function/struct/field names). A " - "missing or empty companion is a NACK — it is mandatory." - ) - return base - if role_value == "first_principles_reviewer": - return ( - "While waiting for the refiner's proposal, prepare your " - "first-principles pass: (a) read the seed — `egg-contract " - "show` and the linked issue — and restate, in your own words, " - "the problem it claims to solve and why; (b) explore the " - "codebase to test that premise against reality (does the thing " - "already exist? is the problem already handled? is there a far " - "simpler path?); (c) form your own view of whether this is the " - "right direction and what a materially better one would be. " - "When the refiner proposes, you are checking the *premise and " - "direction*, not the analysis quality — surface any concrete " - "redirect as a phase-scoped HITL decision for the operator and " - "ACK the refiner. Never NACK on first-principles grounds." - ) - - # Generic fallback - return ( - "While waiting for proposals, read the contract " - "(`egg-contract show`), explore the codebase for context, " - "and prepare your review criteria. " - "Do NOT inspect producer artifacts before proposals arrive." - ) - - -def _re_review_priming_block( - *, - version: int | None = None, - delta_range: str | None = None, -) -> str: - """Adversarial re-prime injected at the moment of every re-review. - - Counter-anchors the persistent reviewer against the "verify named - blockers got fixed" framing that long-lived context naturally - biases toward (see #2724 post-mortem: slice-1 v2 was ACK'd despite - the v2 delta introducing a non-executable inline `python3 -c` - snippet that a downstream GitHub-bot reviewer caught immediately). - - Three design choices worth flagging: - - - **Delta-scoped, not exploration-forcing.** The block tells the - reviewer to re-read *the delta since their own last review* - adversarially, not to re-traverse the codebase. The amortized - exploration from cycle-1 is the feature; re-Reading every - referenced file on every cycle would throw away BRC's cost - advantage. - - **Per-reviewer delta, not a fixed version pair (#2887).** The - block was originally hardcoded to the v1→v2 transition and took - no arguments, yet was appended verbatim to every re-review (v3, - v4, …). On N>2 cycles the stale "audit the v2 delta as a fresh - reviewer, ignore your v1 NACK history" prose read as "re-audit - the whole accumulated surface," widening scope each cycle and - blocking multi-round convergence. The block is now parameterized - by the current proposal version (``vN`` / its prior ``v(N-1)``) - and, on per-reviewer ``CONSENSUS_RE_REVIEW`` notices, anchored to - that reviewer's own ``<last_reviewed_sha>..HEAD`` ``delta_range`` - (resolved orchestrator-side from the reviewer's last-verdicted - version). When ``delta_range`` is absent (the broadcast - ``CONSENSUS_PROPOSE`` body, ``to_role=all`` — one text for - reviewers sitting at different last-reviewed versions) the block - references the reviewer-self-tracked range from REVIEWER-SYNC.md - (``git log {last_reviewed_commit}..HEAD --not origin/{base} -p``) - instead. - - **Economic framing is explicit.** "Re-reviews are cheap / NACK - without hesitance" is load-bearing — without it, persistent - reviewers naturally optimize for convergence (ACK to end the - cycle) over rigor. The orchestrator absorbs the cost of extra - cycles; the reviewer should not be carrying it. - - The block is appended to ``CONSENSUS_RE_REVIEW`` message bodies - (signals.py, both withdrawal/re-propose and push-after-propose - paths) and to ``CONSENSUS_PROPOSE`` bodies when the producer is - re-proposing (version > 1, ``changed_artifacts`` set). Reviewers - who NACK'd the prior version receive ``CONSENSUS_PROPOSE`` rather - than ``CONSENSUS_RE_REVIEW`` on a re-propose, so both surfaces need - the re-prime to reach every reviewer. - - Args: - version: The current (re-proposed) proposal version ``N``. When - ``None`` (legacy / defensive callers) the block falls back - to generic "current" / "prior" wording without numbered - anchors. - delta_range: A concrete ``<sha>..HEAD`` git range scoping this - reviewer's mandate-2 audit to the commits landed since their - own last verdict. Only available on the per-reviewer - ``CONSENSUS_RE_REVIEW`` path; omitted on the broadcast - ``CONSENSUS_PROPOSE`` body. - """ - # Adjective placed before "review"/"verdict" ("Your v6 review" / - # "Your current review"); and the prior-version qualifier placed - # before "blockers"/"NACK history" ("named v5 blockers" / "named - # prior blockers"). Both read naturally with or without a version. - vN = f"v{version}" if version is not None else "current" - vNm1 = f"v{version - 1}" if version is not None and version >= 2 else "prior" - # Mandate-2's delta anchor. On the per-reviewer path we have an - # authoritative range; on the broadcast path we point at the - # reviewer-self-tracked range REVIEWER-SYNC.md already defines, so - # each reviewer scopes to the commits since *their* last review - # rather than the whole accumulated surface. - if delta_range: - delta_clause = ( - f"the delta since your last review (`git log {delta_range} " - "--not origin/<base> -p` — the commits landed since the " - "version you last verdicted)" - ) - delta_short = f"this delta (`{delta_range}`)" - else: - # NOTE: `{last_reviewed_commit}` and `{base_branch}` here are - # *literal* braces, deliberately matching the placeholder names - # the reviewer agent already learned from REVIEWER-SYNC.md - # (shared/prompts/REVIEWER-SYNC.md:110) — the agent substitutes - # them at read-time from its own bookkeeping. Do NOT convert this - # string to an f-string: there are no Python locals named - # `last_reviewed_commit` / `base_branch` here, so f-stringifying - # would raise `NameError` at call time. The per-reviewer branch - # above uses `<base>` instead because that path embeds a - # concrete, orchestrator-resolved range — only `<base>` remains - # for the reviewer to fill in, so the angle-bracket convention - # makes the (already-resolved vs. still-to-resolve) distinction - # visible at a glance. - delta_clause = ( - "the delta since your last review (per REVIEWER-SYNC.md: " - "`git log {last_reviewed_commit}..HEAD --not " - "origin/{base_branch} -p` — the commits landed since the " - "version you last verdicted, NOT the whole accumulated " - "proposal surface)" - ) - delta_short = "this delta (the commits since your last review)" - return ( - "\n\n**Adversarial re-review**\n\n" - f"**Your {vN} review has TWO equal-weight mandates:**\n\n" - f"1. **Verify named {vNm1} blockers were addressed** — confirm " - "the producer fixed what you NACK'd.\n" - f"2. **Audit {delta_clause} as a fresh reviewer** — ignore your " - f"{vNm1} NACK history. Read that diff as if you'd never seen the " - "prior version. Apply your lens (security threat-model, " - "concurrency races, contract AC, line-by-line bugs, " - "silent-fallback shapes — whichever your role owns) to the " - "delta itself, not to whether your previous concerns were " - "satisfied. **Mandate 2 is bounded to this delta** — it does " - "NOT ask you to re-traverse the whole accumulated surface from " - "earlier cycles; that work was amortized when you first " - "reviewed those commits.\n\n" - "Both mandates have equal weight. If (1) passes but (2) finds new " - "issues, you NACK. ACK requires both pass.\n\n" - "**The named-blockers anchor is a known trap. Every reviewer " - "lens has a mandate-2 in its own territory** — security has " - "newly-introduced threat surfaces, concurrency has newly-" - "introduced races, contract has newly-introduced AC drift, code " - "has newly-introduced line-by-line bugs. The four issues that " - "escaped PR #2724 to the GitHub bot were all of code-lens shape " - "(`${ANSWER}` as bare Python, deprecated `datetime.utcnow()`, " - "non-atomic write, bare `except: pass`) — the persistent " - 'reviewer correctly answered mandate 1 ("did prior issues get ' - 'fixed? yes") and skipped mandate 2 ("does this delta introduce ' - 'new issues? actually yes"). The shape generalizes: whatever ' - "your lens, this delta can introduce issues your prior NACK " - "didn't name. Watching the producer deliver a targeted fix " - 'pulls strongly toward "verify my fix-request landed → ACK." ' - "Recognize the pull and do mandate 2 anyway.\n\n" - "**How to execute mandate 2:**\n\n" - "- Read each new hunk as an operator who's about to copy-paste / " - "run / integrate it. Would this code execute as written? Would " - "these docs send a copy-paster down a working path?\n" - "- Apply every rubric pass to the new hunks. New issues outside " - "the scope of your prior NACK are blocking; your prior NACK does " - "not bound this re-review.\n" - "- **Fresh-reviewer simulation.** Before issuing your " - f"{vN} verdict, ask: would a reviewer who has only seen " - f"{delta_short} with no NACK history ACK this? If you can't " - "argue yes from that diff alone, NACK.\n" - "- **External-bot anchor.** Imagine `egg-reviewer[bot]` reads " - f"only {delta_short} with no NACK context. What would it flag? " - "Anything it'd flag, you should NACK first.\n\n" - f"**Your {vN} verdict must enumerate both halves** so mandate 2 " - "doesn't silently disappear from the record:\n\n" - f"- (a) Which {vNm1} blockers you verified-fixed (mandate 1).\n" - "- (b) What new issues you audited-and-did-not-find (mandate 2). " - 'Name the specific shapes you checked — not "reviewed thoroughly," ' - 'but "checked for silent fallbacks, doc-snippet executability, ' - "API-deprecation, atomicity of file writes.\" If you can't " - "enumerate (b), you haven't done mandate 2.\n\n" - "**Re-reviews are cheap by design.** Your amortized context means " - 'the work is "read the delta, apply your rubric, decide" — ' - "minutes, not hours. NACK without hesitance; the orchestrator " - "absorbs cycles. Two NACKs on the same producer where the second " - "names new findings is the correct trajectory, not " - "goalpost-moving. The downstream GitHub reviewer should find " - "nothing in this delta. Anything it catches that lives in this " - "cycle's diff is a miss attributable to this re-review." - ) - - -def _build_producer_orientation( - role_value: str, - phase: str, - reviewers: list[str], - branch: str | None = None, -) -> str: - """Build orientation instructions for producer agents. - - Tells producers what to research before starting work — understanding - context, knowing what reviewers will check, and checking existing code - patterns. This produces higher-quality first proposals and fewer NACKs. - - A producer that orients and finds it has no work in this slice takes the - generic no-op propose path described in the Producer Lifecycle (#3027) — - no special orientation text is needed. - - Args: - role_value: Producer role (e.g. ``coder``). - phase: Pipeline phase name. - reviewers: Names of reviewers that will review this producer. - branch: The pipeline's working branch, used for sync instructions. - """ - reviewer_awareness = "" - if reviewers: - reviewer_names = ", ".join(reviewers) - reviewer_awareness = ( - f" Your work will be reviewed by **{reviewer_names}** — " - "keep their review criteria in mind as you work." - ) - - # The simplifier runs in both the refine and plan phases as a PRODUCER - # ONLY (the human-focused companion). It carries an advisory review edge - # over the upstream producer purely as the event-pump wake-wire — that is - # what re-invokes it on the upstream's PROPOSE — but it issues no verdict - # and never reviews the draft (#3381). Its work depends on the upstream - # producer's draft existing, so — like the implement-phase tester — it - # orients up-front and starts producing only once the upstream proposes. - if role_value == "simplifier": - if phase == "plan": - upstream, draft_desc = "task_planner", "the implementation plan" - else: # refine - upstream, draft_desc = "refiner", "the refine analysis" - sync_note = "" - if branch: - sync_note = ( - f" When re-invoked on the PROPOSE, sync your worktree first: " - f"`git fetch origin && git merge origin/{branch} --no-edit`." - ) - return ( - f"your WORK depends on **{upstream}**'s draft of {draft_desc} " - "existing — do NOT write your companion before it is pushed. ORIENT " - "now (read the contract and the issue/task description so you " - "understand the subject), then exit; the event pump re-invokes you " - f"when **{upstream}** issues `CONSENSUS_PROPOSE`, carrying that " - "proposal in your event payload. On that invocation: read the " - "upstream draft, then write a simplified, higher-level companion " - "that captures its essence in plain, jargon-free language for a " - "broad audience (engineers, PMs, and managers) — a summary, NOT a " - "review of the draft — and PROPOSE it. That is the whole job: you " - f"do NOT review **{upstream}**'s draft and you issue no ACK or NACK " - "on it." + sync_note + reviewer_awareness - ) - - if phase == "implement": - if role_value == "coder": - return ( - "read the contract (`egg-contract show`) to understand all tasks " - "and acceptance criteria. Explore the codebase to find existing " - "patterns, conventions, and the files you will modify. Check for " - "existing tests that cover the areas you will change — do not " - "break them." + reviewer_awareness - ) - elif role_value == "tester": - sync_note = "" - if branch: - sync_note = ( - f" Before starting work, sync your worktree: " - f"`git fetch origin && git merge origin/{branch} --no-edit`." - ) - return ( - "read the contract (`egg-contract show`) to understand what is " - "being implemented. Check the existing test infrastructure — " - "test frameworks, fixtures, conftest files, and naming conventions. " - "Identify edge cases from the requirements before writing tests. " - "**Your mandate is two-fold**: comprehensive regression " - "coverage AND adversarial probing for bugs the coder missed " - "— see the *Your Task* → mandate block for the full " - "instruction (including the failing-test → NACK → HANDOFF " - "workflow when you catch a coder-side bug). " - "**Scaffold-first while the coder is producing**: draft test " - "scaffolding from the plan alone — test file paths from " - "`tasks[].files`, function signatures from each task's acceptance " - "criteria, fixture imports, and mock-input scenarios from the YAML. " - "Leave assertion bodies as TODOs. Do NOT call `wait-loop` for the " - "coder's CONSENSUS_PROPOSE before drafting these scaffolds — the " - "scaffold work does not depend on coder output and recovers " - "downstream-producer time. Your propose-ready iteration should " - "start at the coder's first commit, not their first propose. " - "**You MUST propose** even when the slice warrants no new tests " - "(pure refactor / doc-only / symbol moves with no behavior " - "change): the BRC consensus blocks until every producer has " - "proposed. For that case, submit a generic no-op propose " - "(#3027) — `egg-orch consensus propose --no-changes-needed " - "--no-changes-reason '<why: e.g. pure refactor, existing tests " - "cover>'`. It is accepted as a non-blocking no-op (reviewers do " - "not review or NACK it). Do NOT just heartbeat indefinitely " - "waiting for test work that isn't there — that deadlocks the " - "slice." + sync_note + reviewer_awareness - ) - elif role_value == "documenter": - sync_note = "" - if branch: - sync_note = ( - f" Before starting work, sync your worktree: " - f"`git fetch origin && git merge origin/{branch} --no-edit`." - ) - return ( - "read the contract (`egg-contract show`) to understand what is " - "being implemented. Check existing documentation structure — " - "README files, doc directories, inline documentation patterns. " - "Identify which docs describe the surfaces this work touches, so " - "you can fold the resulting state into them as a snapshot of " - "current behavior once the implementation is complete. " - "**You MUST propose** even when the slice warrants no doc " - "updates (pure refactor / test-only / internal-only with no " - "documented-surface impact): the BRC consensus blocks until " - "every producer has proposed. For that case, submit a generic " - "no-op propose (#3027) — `egg-orch consensus propose " - "--no-changes-needed --no-changes-reason '<why: e.g. no " - "documented surface impacted by the coder's diff>'`. It is " - "accepted as a non-blocking no-op (reviewers do not review or " - "NACK it). Do NOT just heartbeat indefinitely waiting for doc " - "work that isn't there — that deadlocks the slice." + sync_note + reviewer_awareness - ) - elif phase == "plan": - if role_value == "architect": - return ( - "read the issue/task description carefully. Explore the codebase " - "to understand the current architecture, component boundaries, " - "and dependencies. Identify the areas that will be affected by " - "the proposed changes." + reviewer_awareness - ) - elif role_value == "task_planner": - return ( - "read the issue/task description carefully. Review the codebase " - "structure to understand the scope of work. Break the work into " - "tasks with clear acceptance criteria that reviewers can verify." - + reviewer_awareness - ) - elif role_value == "risk_analyst": - return ( - "read the issue/task description carefully. Research the affected " - "areas of the codebase for potential risks — security, " - "performance, backwards compatibility, and third-party " - "dependencies." + reviewer_awareness - ) - elif phase == "refine": - if role_value == "refiner": - return ( - "read the prior review feedback carefully. Understand exactly " - "what concerns were raised and what changes are expected. Check " - "the current state of the code before making modifications. " - "When the draft you are refining is an analysis or plan, " - "surface every runtime-primitive assumption explicitly at the " - "phase_gate (see #2594) — name each class, function, route, " - "env var, ConfigMap key, fixture, CLI flag, or decorator the " - "downstream plan will depend on, with `file:line` evidence " - "and execution-context scope (in-sandbox-agent vs " - "trusted-CI-runner vs human-operator). This makes the " - "plan-phase Primitive-Existence and Trust-Boundary audits " - "cheap." + reviewer_awareness - ) - - # Generic fallback - return ( - "read the contract (`egg-contract show`) and explore the codebase " - "to understand context, patterns, and conventions before starting." + reviewer_awareness - ) - - -def _build_file_boundary_section(role_value: str, repo: str | None = None) -> str: - """Build a file boundary section for an agent prompt. - - Sources the role's allowed/blocked patterns from - ``egg_restrictions.patterns.build_agent_patterns`` so the prompt - matches what the gateway will actually enforce on push — including - per-repo ``role_patterns:`` overrides from ``repositories.yaml`` - (#2528). The legacy ``egg_contracts.agent_roles`` patterns were - Python-only and didn't honour the per-repo knobs, which created a - contradictory message for non-Python repos: the gateway would - enforce Go conventions while the prompt told the agent the boundary - was Python. - - Returns an empty string when no patterns are defined for the role. - """ - try: - from egg_restrictions.patterns import get_agent_pattern_for_repo - except ImportError: - return "" - - pattern = get_agent_pattern_for_repo(role_value, repo=repo) - if pattern is None: - return "" - - if ( - not pattern.allowed_patterns - and not pattern.blocked_patterns - and not pattern.hard_blocked_patterns - ): - return "" - - lines = [ - "## File Boundaries (Gateway-Enforced)\n", - f"Your role ({role_value.upper()}) can only push changes to files " - "matching these patterns. The gateway will **reject your push** if it " - "includes files outside your boundaries. Only create and modify files " - "you are allowed to push.\n", - ] - if pattern.allowed_patterns: - lines.append("**Allowed:** " + ", ".join(f"`{p}`" for p in pattern.allowed_patterns)) - if pattern.blocked_patterns: - lines.append("**Blocked:** " + ", ".join(f"`{p}`" for p in pattern.blocked_patterns)) - # Hard blocks are a stricter tier: they are rejected even when they would - # otherwise match your allow list or a docs/fixture exemption (#3396). The - # agent must see them, or it will author a hard-blocked path (e.g. - # `.egg-state/contracts/fixtures/x.json`, `.github/actions/x/testdata/`), - # hit a gateway 403, and have no way to understand why. - if pattern.hard_blocked_patterns: - hard_line = "**Hard-blocked (never pushable, no exemption applies):** " + ", ".join( - f"`{p}`" for p in pattern.hard_blocked_patterns - ) - if pattern.hard_block_exempt_patterns: - hard_line += " — except " + ", ".join( - f"`{p}`" for p in pattern.hard_block_exempt_patterns - ) - lines.append(hard_line) - - # `.github/` staging-dir convention (issue #2508). Surfaced for the - # coder role specifically because it's the producer that's expected - # to initiate `.github/` work. The role-pattern check - # (``startswith(".github/")``) doesn't match `.github-staging/`, so - # autofixer / conflict_resolver allowlists technically reach the - # staging path too — but those roles are reactive and aren't asked - # to plan new `.github/` changes, so the convention's planning-time - # guidance only needs to land for coder. - if role_value == "coder": - lines.append("") - lines.append( - "**`.github/` changes**: `.github/` is blocked above. If your " - "task requires modifying CI workflows, CODEOWNERS, dependabot " - "config, or anything else under `.github/`, write the proposed " - "end-state to top-level `.github-staging/` instead, mirroring " - "the `.github/` structure (e.g. stage " - "`.github/workflows/test-e2e.yml` as " - "`.github-staging/workflows/test-e2e.yml`). Call out the " - "staged files explicitly in your PR body so the human reviewer " - "knows to move them into `.github/` before merge — see issue " - "#2508." - ) - lines.append("") - return "\n".join(lines) - - -def _build_agent_prompt( - role_value: str, - phase: str, - pipeline_id: str, - pipeline_mode: str, - prompt: str | None = None, - issue_number: int | None = None, - repo: str | None = None, - branch: str | None = None, - base_branch: str | None = None, - review_feedback: str | None = None, - review_cycle: int = 0, - repo_path: str | None = None, - phase_obj=None, - all_phases=None, - concurrent: bool = False, - network_mode: str | None = None, - operator_directives: list[OperatorDirective] | None = None, - iteration_history: list[IterationSummary] | None = None, -) -> str: - """Build a role-specific prompt for multi-agent execution. - - For the CODER role, delegates to the existing _build_phase_prompt(). - Other roles (TESTER, DOCUMENTER, ARCHITECT, etc.) get - role-specific instructions. - - Execution roles (tester, documenter) receive a summarized - background with structured task information instead of the full issue - body. Analysis roles (architect, task_planner, risk_analyst) receive - the full issue body. - - Note: Handoff data is passed via the EGG_HANDOFF_DATA environment - variable, not via the prompt — prompts are built once before - execution starts. - - Args: - role_value: Agent role string (e.g. "coder", "tester") - phase: Pipeline phase name - pipeline_id: Pipeline ID - pipeline_mode: "issue" or "local" - prompt: Original task prompt - issue_number: GitHub issue number - repo: Repository name - branch: Branch name - review_feedback: Feedback from prior review cycle - review_cycle: Current review cycle number - repo_path: Filesystem path to repository (for user override lookup) - phase_obj: Current plan phase object (optional) - all_phases: All contract phases (optional) - concurrent: Whether agent runs in concurrent multi-agent mode. - When True, adds consensus lifecycle preamble instructing the - agent to stay alive, poll messages, and participate in consensus. - network_mode: Pipeline network mode ("public", "private", or None). - When "private", injects warnings about blocked package downloads. - - Returns: - Complete prompt string for the agent - """ - # CODER and REFINER use the existing phase prompt (phase-specific - # instructions are already tailored for refine vs implement etc.) - if role_value in ("coder", "refiner"): - base_prompt = _build_phase_prompt( - phase=phase, - pipeline_id=pipeline_id, - pipeline_mode=pipeline_mode, - prompt=prompt, - issue_number=issue_number, - repo=repo, - branch=branch, - review_feedback=review_feedback, - review_cycle=review_cycle, - repo_path=repo_path, - operator_directives=operator_directives, - iteration_history=iteration_history, - ) - # Surface file boundaries so agent knows what it can push (#1431). - # Pass repo so the rendered patterns match per-repo overrides - # (#2528) the gateway will enforce on push. - boundary_section = _build_file_boundary_section(role_value, repo=repo) - if boundary_section: - base_prompt += "\n" + boundary_section - # Producer escape hatch (#2529) — coder is one of the impassing - # producer roles, so it must see the actionable - # check_file_restriction / report_impasse guidance instead of - # inventing workarounds. Refiner runs in the refine phase and - # never owns implement-phase tasks, so it doesn't need this. - if role_value == "coder": - base_prompt += "\n" + _build_impasse_escape_hatch_section() - # In concurrent mode, inject BRC consensus preamble so the coder/refiner - # knows to propose, respond to reviews, confirm, and stay alive. - if concurrent: - base_prompt += _build_brc_preamble( - role_value, - phase, - repo=repo, - branch=branch, - base_branch=base_branch, - ) - return base_prompt - - if role_value.startswith("reviewer_"): - # Reviewer prompts are fully built by _build_review_prompt with its - # own criteria/verdict format + iteration-context wiring; we don't - # accumulate the role-shared ``lines`` block for them. Dispatching - # here (rather than mid-function with an early return) prevents - # future drift where a "must always be included" line is added to - # the accumulation and silently never reaches reviewers (#2795). - reviewer_type = role_value.replace("reviewer_", "", 1).replace("_", "-") - review_prompt = _build_review_prompt( - phase=phase, - pipeline_id=pipeline_id, - pipeline_mode=pipeline_mode, - reviewer_type=reviewer_type, - issue_number=issue_number, - review_cycle=review_cycle + 1, - prior_feedback=review_feedback, - repo_path=repo_path, - base_branch=base_branch, - concurrent=concurrent, - operator_directives=operator_directives, - iteration_history=iteration_history, - ) - if concurrent: - review_prompt += "\n" + _build_brc_preamble( - role_value, - phase, - repo=repo, - branch=branch, - base_branch=base_branch, - ) - return review_prompt - - # Build context header (shared across all roles) - lines = [f"You are the **{role_value.upper()}** agent in the **{phase}** phase.\n"] - lines.append("## Context\n") - lines.append(f"Pipeline ID: {pipeline_id}") - lines.append(f"Phase: {phase}") - lines.append(f"Mode: {pipeline_mode}") - lines.append(f"Agent Role: {role_value}") - if repo: - lines.append(f"Repository: {repo}") - if branch: - lines.append(f"Branch: {branch}") - if issue_number is not None: - lines.append(f"Issue: #{issue_number}") - lines.append("") - - # Concurrent mode: add BRC consensus lifecycle preamble so agents understand - # they must stay alive and participate in Broadcast-Review-Converge consensus. - if concurrent: - lines.append( - _build_brc_preamble( - role_value, - phase, - repo=repo, - branch=branch, - base_branch=base_branch, - ) - ) - - # Include role-appropriate context instead of the raw issue body. - # Analysis roles (architect, task_planner, risk_analyst) receive the full - # issue body. Execution roles (tester, documenter) receive a - # brief summary with structured task information and context pointers. - role_context = _build_role_context( - role_value=role_value, - prompt=prompt, - issue_number=issue_number, - phase_obj=phase_obj, - all_phases=all_phases, - base_branch=base_branch, - ) - if role_context: - lines.append(role_context) - - # Phase iteration context: operator directives + prior iteration history. - # Rendered for all roles (producers AND reviewers) so reviewers cannot - # NACK a directive-driven change against a stale default rubric (#2795). - iteration_context = _build_phase_iteration_context(operator_directives, iteration_history) - if iteration_context: - lines.append(iteration_context) - - # Review feedback from prior agentic cycles (scoped to agentic NACKs - # since #2795 — HITL kickbacks render via the iteration context above). - if review_feedback: - lines.append("## Review Feedback\n") - lines.append(review_feedback) - lines.append("") - - # Derive the pipeline identifier for namespaced output filenames. - _identifier = _pipeline_identifier(issue_number, pipeline_id) - - # Spec-driven agent-output paths (#3077 slice-3): resolve each path - # via the artifact registry so the prompt prose, the propose-time - # validator (signals._validate_producer_artifacts), and the gateway - # artifact-read endpoint (slice-4) all share one source of truth. - # The slice-2 mandatory consistency test - # (TestConsistencyC in shared/egg_contracts/tests/test_artifact_spec.py) - # pins these call sites to the registry; a future row rename - # surfaces here as a missing prompt path instead of as #3016-style - # drift between spec and rendered prose. - from egg_contracts.artifact_spec import resolve_artifact_path as _resolve_artifact_path - - _architect_output_path = _resolve_artifact_path("architect-output", _identifier) - _architect_slices_path = _resolve_artifact_path("architect-slices", _identifier) - _risk_analyst_output_path = _resolve_artifact_path("risk-analyst-output", _identifier) - # Human-focused companion drafts the simplifier produces (one per phase). - _analysis_human_path = _resolve_artifact_path("analysis-draft-human", _identifier) - _plan_human_path = _resolve_artifact_path("plan-draft-human", _identifier) - - # Role-specific instructions - lines.append("## Your Task\n") - - if role_value == "tester": - # Look up per-repo check commands from repositories.yaml - repo_checks: list[dict[str, str]] = [] - if repo: - try: - repo_checks = get_repo_checks(repo) - except FileNotFoundError: - repo_checks = [] - - lines.extend( - [ - "**ROLE BOUNDARY: You are the TESTER, not the CODER.** " - "Do NOT implement application logic, create source files, write configuration, " - "or set up project infrastructure. Your job is to write tests for the CODER's " - "implementation, run checks, and report gaps. If the coder hasn't committed yet, " - "wait — do not implement the solution yourself.", - "", - "**Your mandate is two-fold**:", - "", - "1. **Comprehensive coverage** — write tests that prevent " - "regressions, covering the happy path and realistic alternative " - "paths through every changed area. New behavior gets new tests; " - "modified behavior gets updated tests; nothing the coder changed " - "should silently lose coverage.", - "2. **Adversarial probing** — actively probe the coder's " - "implementation for bugs and edge cases they missed. Treat the " - "implementation as suspect until you have tried to break it. " - "Write tests that target suspected weaknesses. When a test " - "fails because of a coder-side bug, **the committed failing " - "test is evidence — the NACK is the bug report**. Pair every " - "failing test with an explicit NACK on the coder's proposal " - "that names the failing test in its rationale; otherwise the " - "bug is easy for the coder to miss. Also list the bug in " - "`gaps_found` and HANDOFF to coder with the failure output. " - "The coder owns the fix; you own surfacing the bug.", - "", - "You are also responsible for **lint/type-check validation**.", - "", - "### When the slice warrants no new tests (#3027)", - "", - "Pure refactors (symbol moves, decompositions with no behavior " - "change), doc-only slices, and other no-test-work slices still " - "require you to **propose** — BRC consensus blocks until every " - "producer has proposed at least once. **Don't just heartbeat " - "and wait for work that isn't coming.** Instead submit a " - "generic no-op propose:", - "", - "1. (Optional but encouraged) run the configured checks against " - "the coder's diff (`make lint`, `make test`, etc.) to confirm " - "the slice really is behavior-preserving.", - "2. Propose a no-op: `egg-orch consensus propose " - "--no-changes-needed --no-changes-reason '<concrete reason, " - "e.g. slice-3 is a pure decomposition: symbol moves between " - "submodules, no behavior change; existing suite covers the " - "re-exported barrel>'`. No artifacts or commit-sha are needed.", - "", - "The no-op counts as proposing (so consensus is not blocked on " - "you) and is accepted as a non-blocking no-op — reviewers do not " - "review or NACK it. If the slice **does** have new test work " - "(real behavior changes, new edge cases, modified contracts), do " - "NOT use the no-op path — author tests and propose as usual.", - "", - "### Testing", - "", - "1. Review the changed files (available in handoff data or via git diff)", - "2. Build coverage tests for the happy path and realistic " - "alternative paths in every changed area", - "3. **Adversarially probe** the implementation: identify " - "suspected bugs and untested edge cases, then write tests that " - "target them", - "4. Run all tests. Tests that pass demonstrate coverage; " - "**tests that fail demonstrate bugs you have found** — keep them", - "5. For every failing test caused by a coder-side bug: " - "commit the failing test AND **NACK the coder's proposal, " - "explicitly naming the failing test in the NACK rationale**. " - "The committed test alone is not sufficient — the NACK is " - "what surfaces the bug to the coder. Also list the bug in " - "`gaps_found` and HANDOFF to the coder with the failure " - "output. Your `test` configured check will fail until the " - "coder pushes a fix — that is expected; do NOT propose " - "consensus until every configured check passes per the " - "*Configured Checks* section below", - "6. Commit all test files with descriptive messages", - "", - "Adversarial probing — actively try to break the implementation:", - "- Missing error handling and input validation", - "- Boundary conditions, off-by-one, empty/null/oversized inputs", - "- Uncovered code paths and branches (especially error paths)", - "- Concurrency: races, partial failures, retry behavior, ordering assumptions", - "- Contract violations: does the code actually match the " - "acceptance criteria, or just the happy path of them?", - "- Integration gaps between components and unstated interface assumptions", - "", - "Gap-finding focus (still report these in `gaps_found` even " - "when you cannot write a test for them):", - "- Logic errors that would require design changes to fix", - "- Inconsistencies between the implementation and the plan/contract", - "- Missing test infrastructure that prevents adequate coverage", - "", - "### Configured Checks (MANDATORY)", - "", - "You MUST run **ALL** configured checks below and fix any failures " - "before proposing consensus. Skipping checks (e.g., running tests but " - "not lint) is a common failure mode — do not skip any.", - "", - ] - ) - - if repo_checks: - # Inject explicit check commands from repositories.yaml - lines.extend( - [ - "The following check commands are configured for this repository. " - "Run **every one** of them **in order**:", - "", - ] - ) - for i, check in enumerate(repo_checks, 1): - name = check["name"].replace("\n", " ").strip() - cmd = check["command"].replace("\n", " ").strip() - lines.append(f"{i}. **{name}**: `{cmd}`") - lines.extend( - [ - "", - "If ANY check fails in test files you wrote, fix the issue and re-run. " - "If failures are in source code, do NOT fix them — report them to the coder.", - "", - "After running all checks:", - ] - ) - else: - # Fall back to auto-discovery - lines.extend( - [ - "1. **Discover commands**: Look for Makefile, pyproject.toml, package.json, " - "setup.cfg, tox.ini, or similar build/test configuration files", - "2. **Run linters**: Execute linters (ruff, eslint, golangci-lint, etc.)", - "3. **Run type checkers**: Execute type checkers (mypy, pyright, tsc, etc.)", - "", - "After running all checks:", - ] - ) - - lines.extend( - [ - "- **Auto-fix test files only**: Fix auto-fixable issues in test files you wrote " - "(formatting, import order, simple type errors)", - "- **Repeat**: Re-run checks to verify fixes. Repeat up to 3 times.", - "- **Commit test fixes**: Commit all test-file fixes together with a descriptive message", - "", - "Auto-fixable (in test files only — commit fixes directly):", - "- Lint errors in test files (formatting, import order, code style)", - "- Type errors in test files with clear fixes", - "", - "Report only (do NOT modify source code — NACK the coder and explain what's needed):", - "- Lint or type errors in source code — tell the coder to fix these", - "- Test failures caused by bugs in the coder's implementation — tell the coder to fix", - "- Complex logic errors requiring design decisions", - "- Security issues requiring architectural changes", - "", - "When testing third-party library integrations or unfamiliar frameworks, " - "use WebSearch and WebFetch (when available) to look up testing patterns, " - "known edge cases, and recommended test approaches for those libraries.", - "", - "## Parallel Execution with Subagents\n", - "If the changes span multiple independent components or modules, you can use " - "Claude Code's **Agent tool** to parallelize test writing. Launch one subagent " - "per component to write and run tests concurrently. Each subagent should work " - "on non-overlapping test files. Subagents should only write files — do NOT " - "stage or commit from subagents. After all subagents complete, run the full " - "test suite to verify everything passes together, then stage and commit yourself.", - "", - *_EXPLORATION_SUBAGENT_GUIDANCE, - ] - ) - - # Test execution verification — prevents proposing consensus with - # unverified tests (issue #1359). - test_verify_lines = [ - "### Test Execution Verification (CRITICAL)\n", - "You MUST actually execute the test suite (`go test`, `pytest`, `jest`, etc.). " - "Passing gofmt, syntax checks, or linting alone does NOT count as tests run.\n", - "If tests cannot run (e.g., dependency downloads blocked in private network mode, " - "missing build tools), you MUST:", - "1. Set `tests_execution_blocked: true` and provide `tests_execution_blocked_reason` " - "in your attestation when proposing consensus", - '2. Include an explicit **"TESTS UNVERIFIED"** warning in your proposal summary', - '3. Do NOT claim your work is "complete" — state that tests are written but unverified', - "", - "**Distinguish `tests_execution_blocked` from a no-op propose** " - "(see the no-op section above): set `tests_execution_blocked=true` " - "when you DID author / intend tests but the configured checks could " - "not run (blocked downloads, missing tools) — that is a real " - "proposal with lower confidence. Use the generic no-op propose " - "(`--no-changes-needed`) only when the slice genuinely warrants no " - "new tests at all. Don't conflate the two.", - "", - ] - if network_mode == "private": - test_verify_lines.extend( - [ - "**WARNING: Private network mode is active** — external package downloads " - "(go mod download, npm install, pip install, etc.) may be blocked. " - "If dependency installation fails, you cannot verify tests. " - "Follow the instructions above to flag tests as unverified.", - "", - ] - ) - lines.extend(test_verify_lines) - - # Check execution verification — prevents proposing consensus without - # running all configured checks (issue #1414). - check_verify_lines = [ - "### Check Execution Verification (CRITICAL)\n", - "You MUST run **every** configured check command and ensure they **pass** " - "before proposing consensus. Running tests alone is NOT sufficient — " - "lint, type-check, and security checks must also pass. If you skip a " - "check or propose with a failing check, the server will reject your " - "proposal.\n", - "Before proposing, verify:", - "- [ ] All configured check commands have been executed", - "- [ ] All checks pass (or failures have been auto-fixed and re-verified)", - "- [ ] Any auto-fix commits have been pushed", - "", - # Source-failure handling — without this, agents have rationalised - # inventing ad-hoc check names so their attestation passes, masking - # red CI on the initial push (issue #1966). - "### When Source-Code Checks Fail (CRITICAL)\n", - "If a configured check fails because of the **coder's source code** " - "(not test files you wrote), you have a binding choice: " - "**do NOT propose consensus**. The role boundary above forbids you " - "from fixing source code, and the rules below forbid you from " - "papering over the failure. Instead:\n", - "1. **Do NOT fix it yourself** — that crosses the tester role boundary.", - "2. **Do NOT invent a narrower or renamed check** " - "(e.g. `pytest-<your-suite>`, `ruff-check-tester-files`) and attest to " - "*that* in `checks_passed`. Only the literal names from " - "`repositories.yaml` (`lint`, `test`, `security`, etc.) are valid; " - "the server will reject anything else, and substituting narrower names " - "hides real CI failures from reviewers.", - "3. **Send a HANDOFF message to the coder** describing the failing " - "check, the command, and the diagnostic output, e.g.:", - " ```", - " egg-orch message send --to coder --type HANDOFF \\", - ' --subject "lint failing on src/foo.py" \\', - ' --body "make lint exits 1: mypy errors in src/foo.py:42 ' - '(incompatible types). Please fix and push; I will re-run lint."', - " ```", - " If you are also reviewing the coder's own consensus proposal, " - "NACK it for the same reason — the two channels reinforce each other.", - "4. **Wait** for the coder to push a fix, then **re-run every " - "configured check** from scratch. Use `egg-orch message wait-loop` " - "(see Producer Lifecycle) — do not spin in a shell `for` loop or " - "prefix with `sleep`.", - "5. **Only propose consensus once every configured check passes " - "literally**, with the configured names in `checks_passed`.", - "", - "If the coder is unresponsive or the failure genuinely cannot be " - "fixed within this phase, document it in `gaps_found` and let the " - "orchestrator escalate via `OVERSEER_ALERT`. Do NOT work around the " - "block by proposing with a partial or renamed `checks_passed` list.", - "", - "### Attestation: `checks_passed` (REQUIRED)\n", - "When proposing consensus, your attestation MUST include a `checks_passed` " - "list containing the **name** of every configured check that **passed**. " - "Do NOT include checks that failed, and do NOT invent ad-hoc names " - "(e.g. `pytest-<scope>`, `ruff-check-tester-files`) — only the literal " - "names from `repositories.yaml`. " - "For example, if the repo has `lint` and `test` checks and both pass, " - 'your attestation must include `"checks_passed": ["lint", "test"]`. ' - "The server will reject your proposal if any configured check is missing " - "from this list (i.e. did not pass).", - "", - ] - lines.extend(check_verify_lines) - - elif role_value == "documenter": - lines.extend( - [ - "Document the CURRENT STATE of the code after this change. " - "Write as if the code has always worked this way — the " - "slice/pipeline machinery that produced the change does not " - "belong in the documentation:", - "", - "1. Review the changed files (available in handoff data or via git diff)", - "2. Update relevant documentation (READMEs, docstrings, API docs) so it " - "describes how the system works now", - "3. Add or update inline code comments where they clarify current behavior", - "4. Commit documentation changes with descriptive messages", - "", - "Write snapshots, not changelogs:", - "- Describe what the code does now, not what changed or when it changed.", - "- NEVER reference SDLC artifacts — slice numbers, TASK-N ids, phase or " - "HITL iteration numbers — in any doc, docstring, or inline comment you write.", - '- Include historical context (issue links, "previously X" rationale, ' - "migration notes) ONLY when it is tangibly valuable to a reader of the " - 'current system, and prefer rationale ("why it is this way") over ' - 'chronology ("what it used to be / when it changed").', - "- When updating an existing doc, fold the new state into the snapshot and " - "REMOVE now-stale ledger or historical entries rather than appending " - "another layer.", - "", - "When documenting third-party integrations or external APIs, use WebSearch " - "and WebFetch (when available) to verify current API signatures, link to " - "official documentation, and confirm usage examples are up to date.", - "", - "### When the slice warrants no doc updates (#3027)", - "", - "Pure refactors (symbol moves, decompositions with no " - "surfaced API change), test-only slices, and internal-only " - "slices that don't touch any documented surface still " - "require you to **propose** — BRC consensus blocks until " - "every producer has proposed at least once. **Don't just " - "heartbeat and wait for work that isn't coming.** Instead " - "submit a generic no-op propose:", - "", - "1. Walk the coder's diff and confirm there is no " - "documented-surface impact: no public API signature " - "changes, no behavior changes a user-facing doc describes, " - "no new feature or flag mentioned in README / docs/, no " - "docstring contracts that drift.", - "2. Propose a no-op: `egg-orch consensus propose " - "--no-changes-needed --no-changes-reason '<concrete reason, " - "e.g. a pure decomposition: symbol moves between " - "submodules, no surfaced API change; no README / docs/ / " - "docstring surface impacted>'`. No artifacts or commit-sha " - "are needed.", - "", - "The no-op counts as proposing (so consensus is not blocked " - "on you) and is accepted as a non-blocking no-op — reviewers " - "do not review or NACK it. If the slice **does** have doc " - "impact (any of the bullets above), do NOT use the no-op " - "path — author doc changes and propose as usual.", - "", - *_EXPLORATION_SUBAGENT_GUIDANCE, - ] - ) - elif role_value == "architect": - lines.extend( - [ - "Analyze the task and produce an architecture analysis:", - "", - "1. Understand the problem or feature request from the issue", - "2. Research the current codebase to understand existing patterns", - "3. Research externally when the task involves third-party libraries, APIs, " - "or frameworks — use WebSearch and WebFetch (when available) to verify " - "assumptions, check current documentation, review architectural patterns, " - "and look up current best practices. Skip external research for purely " - "internal changes.", - "4. Identify key files, constraints, and dependencies", - "5. Consider multiple implementation approaches", - "6. Recommend an approach with justification and document technical decisions", - "7. **Surface runtime-primitive assumptions explicitly (see #2594).** " - "When your analysis mentions a class, function, HTTP route, env var, " - "ConfigMap key, test fixture, CLI flag, or decorator, cite it with " - "`file:line` evidence (`grep -rn` is enough). Call out scope on " - "**both** of the following orthogonal axes when either matters: " - "(a) **purpose** — is the primitive unit-test-only (e.g. a test " - "double like `ScriptedProvider`) vs deployed-pod / production " - "code; (b) **execution context** — does the consumer run as " - "`in-sandbox-agent` (agent pod, reaches gateway via `GATEWAY_URL`) " - "vs `trusted-CI-runner` (pytest from outside the cluster, sees " - "`orchestrator_url` / lifecycle-secret-gated routes / kubectl). A " - "primitive can be unit-test-only but invoked from either runner, " - "or deployed-pod-only but called from either runner — these are " - "independent dimensions, so spell out whichever applies. Buried " - "runtime assumptions are the dominant cause of expensive " - "implement-phase NACKs; surfacing them here makes the plan-phase " - "audit cheap.", - "", - f"Write your analysis to `{_architect_output_path}`.", - "", - # ---------------------------------------------------- - # #2809 — architect owns slice composition - # ---------------------------------------------------- - "## Slice composition authority (#2809)", - "", - "**You are the sole authority for slice composition in the " - "plan phase.** ``task_planner`` enumerates tasks within the " - "slices you define; ``risk_analyst`` surfaces risks that " - "feed your design. Neither owns slice shape — you do. " - "Specifically, you own:", - "", - "- **Slice count.** Treat the operator's ``cq-1`` (or " - "equivalent refine-phase complexity answer) as a coarse " - "top-level hint, not a literal slice count. Subdivide " - "further when the natural slice DAG calls for it.", - "- **Slice boundaries.** Which work goes into which slice, " - "anchored on design seams.", - "- **Slice DAG shape.** Parent/child dependencies between " - "slices. The forest constraint (every slice has at most " - "ONE DAG parent) is HARD — multi-parent slices break the " - "stacked-PR invariant. If a slice would naturally have >1 " - "parents, serialise the upstream slices into a linear " - "chain and record the chosen ordering on the downstream " - "slice's ``serialized_chain_order`` field. See " - "``docs/architecture/slice-dag.md``.", - "- **File-overlap ⇒ dependency edge (HARD — #3046).** Any two " - "slices that touch the same file MUST be ordered on one " - "dependency chain (express the order in ``dependencies`` — a " - "single-parent id per slice — not just in " - "``serialized_chain_order``, which the scheduler does not read " - "for branch topology). Slices that edit a shared file but are " - "left as parallel roots/siblings fork independently off the " - "shared base and collide at integration — plan ingestion " - "hard-rejects this. A slice that deletes or retires a file " - "must depend on every slice that modifies it. Keep slices with " - "disjoint file sets parallel so they still run concurrently.", - "- **Test co-location (HARD — #3411).** A slice that removes, " - "renames, or rewrites code must carry the matching updates to " - "the tests exercising that code — skip-guards, deletions, " - "rewrites — in the SAME slice, never a later one. Every " - "cumulative slice tip must be independently green: the " - "per-slice green gate (#3398) runs the repo's checks at each " - "slice tip and blocks the PR while any check is red, so a " - "plan that parks test obsolescence in a later slice " - "guarantees a blocked slice and repair-loop churn on slices " - "whose only sin is plan topology. In repos that ship the " - "changeset-aware selector (this repo's " - "``scripts/select_tests``), the affected tests are " - "statically discoverable with the same import graph ``make " - "test`` narrowing uses: ``python3 " - "scripts/select_tests/__main__.py --impacted-tests " - "<file>...`` prints every test file that transitively " - "imports the named files (exit 2 = closure unavailable — " - "fall back to grepping the removed symbols in the test " - "trees). Write the removing slice's ``goal`` so it " - "explicitly includes those test updates; ``task_planner`` " - "enumerates them as tasks in that slice.", - "- **Sub-slicing.** When one slice would be too coarse, " - "subdivide it. Right-size slices for a single BRC cycle: " - "avoid bundling distinct file-category groups (e.g. " - "orchestrator + gateway + schema + tests + docs all in " - "one slice), avoid bundling deletion-heavy work with " - "new-API-introduction work, and avoid bundling task " - "groups that have no internal dependency — those are " - "natural seams for parallel sub-slices. If a slice would " - "require the implementing producer to " - "commit-propose-revise more than 3–4 times to converge, " - "subdivide it.", - "", - "Emit the slice scaffold as a YAML file alongside your " - "JSON analysis. ``task_planner`` will copy this scaffold " - "**verbatim** into the plan document's ``# yaml-tasks`` " - "appendix and fill in ``tasks:`` under each slice — the " - "scaffold is binding. If ``reviewer_plan`` NACKs on " - "``slice_size`` or the structural lens calls a " - "sub-division, you re-propose with the updated scaffold; " - "task_planner re-consumes the new scaffold on the next " - "BRC cycle.", - "", - f"Write the slice scaffold to `{_architect_slices_path}`:", - "", - "```yaml", - "slices:", - " - id: 1", - " name: |-", - " <slice name>", - " goal: |-", - " <what this slice achieves>", - " # root slice — omit ``dependencies``", - " - id: 2", - " name: |-", - " <slice name>", - " goal: |-", - " <what this slice achieves>", - " dependencies: slice-1", - "```", - "", - "Omit ``dependencies`` for root slices; for every non-root " - "slice set ``dependencies`` to its single parent's " - "``slice-<id>`` (e.g. ``slice-1``). ``dependencies`` is the " - "canonical ordering key the plan parser reads (per " - "`.egg/schemas/yaml-tasks.schema.json`) — the slice DAG is a " - "forest, so each slice has at most one parent (one id, not a " - "list). Do NOT include ``tasks:`` in the scaffold — that is " - "task_planner's job. Keep ``name`` and ``goal`` concise " - "enough that task_planner can copy them without rewording.", - "", - "### File Restrictions", - "", - "You MUST only write to:", - f"- `{_architect_output_path}`", - f"- `{_architect_slices_path}`", - "", - "Do NOT create or modify any other files. Specifically:", - "- Do NOT modify analysis drafts (`.egg-state/drafts/*-analysis.md`) — " - "these are finalized in the refine phase and are read-only", - "- Do NOT create or modify contracts (`.egg-state/contracts/`)", - "- Do NOT create or modify reviews (`.egg-state/reviews/`)", - "- Do NOT create or modify plan drafts (`.egg-state/drafts/*-plan.md`)", - "", - *_EXPLORATION_SUBAGENT_GUIDANCE, - ] - ) - elif role_value == "task_planner": - draft_path = _get_draft_path("plan", issue_number=issue_number, pipeline_id=pipeline_id) - # Spec-driven (#3077 slice-3) — reuses the helper-resolved path above - # so the task_planner prose and the architect prompt cannot drift. - architect_slices_path = _architect_slices_path - lines.extend( - [ - "Decompose the architecture analysis into a slice-DAG implementation " - "plan. The implement-phase pipeline ships each slice as its own " - "stacked PR.", - "", - "**Slice composition is NOT your call (#2809).** ``architect`` owns " - "slice count, slice boundaries, slice DAG shape, and sub-slicing — " - f"and emits the binding scaffold at `{architect_slices_path}`. Your job " - "is to enumerate ``tasks:`` within those slices, **not to re-shape " - "them**. Copy the architect's scaffold verbatim into the " - "``# yaml-tasks`` appendix (preserving slice ``id``, ``name``, " - "``goal``, and ``dependencies``) and add ``tasks:`` under each " - "slice with task IDs of the form ``TASK-<slice_id>-<n>``.", - "", - "If a slice has too many tasks for one BRC cycle, or you discover a " - "natural sub-seam the architect missed, that is a **slicing problem " - "the architect must fix** — surface it as NACK pressure (your peer " - "reviewer ``risk_analyst`` and the structural reviewer " - "``reviewer_plan`` will NACK ``architect`` on ``slice_size`` when " - "evidence supports it; you can also flag the concern in your plan " - "prose so the reviewers pick it up). **Do NOT silently re-shape " - "slices.** Re-propose against the architect's revised scaffold " - "once it lands.", - "", - "**Test co-location (HARD — #3411).** When a slice removes, " - "renames, or rewrites code, enumerate the matching test " - "updates (skip-guard, deletion, rewrite) as tasks IN THAT " - "SLICE — never in a later slice — and list the test files in " - "those tasks' ``files:``. Every cumulative slice tip must be " - "independently green: the per-slice green gate (#3398) " - "blocks a slice PR while any repo check is red at its tip, " - "so a test that still imports a symbol removed two slices " - "earlier blocks the whole stack. Discover the affected " - "tests with the same import graph ``make test`` narrowing " - "uses, where the repo ships it (this repo: ``python3 " - "scripts/select_tests/__main__.py --impacted-tests " - "<file>...``; exit 2 = closure unavailable — fall back to " - "grepping the removed symbols in the test trees).", - "", - "Steps:", - f"1. Read the architecture analysis AND the slice scaffold at `{architect_slices_path}`", - "2. Copy the architect's slice scaffold verbatim into the " - "``# yaml-tasks`` appendix (same ``id`` / ``name`` / ``goal`` / " - "``dependencies`` values, in the same order)", - "3. Enumerate ``tasks:`` under each slice — discrete, " - "actionable, with clear acceptance criteria and dependency ordering " - "between tasks", - "4. Identify the test strategy — what automated tests cover the " - "changes, and what manual verification is needed", - "5. Identify any manual pre-merge or post-merge steps " - "(migrations, config changes, deployments)", - "", - "## Output Format", - "", - "Write a markdown plan document with a **yaml-tasks** structured", - "appendix at the end. The prose section should explain the approach;", - "the appendix is machine-parsed for contract population.", - "", - *_PR_DESCRIPTION_GUIDANCE, - "", - "End your document with a fenced YAML block like this:", - "", - "````", - "```yaml", - "# yaml-tasks", - "pr:", - ' title: "Short imperative summary (≤70 chars)"', - " description: |", - *_PR_DESCRIPTION_YAML_EXAMPLE, - " test_plan: |", - " - Automated: describe which tests cover the changes", - " - Manual: specific steps a reviewer should take to verify", - " manual_steps: |", - " Pre-merge: any required steps before merging", - " Post-merge: any required steps after merging", - "slices:", - " - id: 1", - " name: |-", - " Slice Name", - " goal: |-", - " What this slice achieves, written for a reviewer of the", - " target repo. This text is rendered verbatim as the lead", - " paragraph of the slice's PR body (#3115), so keep it 1-3", - " plain-language sentences with no plan-internal", - " cross-references (reviewer codes, section numbers, draft", - " version markers).", - " tasks:", - " - id: TASK-1-1", - " description: |-", - " What to do — safe to include `code: type` snippets,", - " URLs, and other punctuation inside a block scalar.", - " acceptance: |-", - " How to verify it is done", - " role: coder # optional: coder (default), tester, or documenter", - " files:", - " - path/to/file.py", - "```", - "````", - "", - *_YAML_TASKS_SAFETY_GUIDANCE, - "", - "Do NOT use a `pr_plan` key — slice packaging is owned by the " - "slice-DAG section below, not by an ad-hoc PR list.", - "", - "The `test_plan` field is **required** — describe both automated test " - "coverage and any manual verification steps. The `manual_steps` field " - "should list any pre-merge or post-merge actions required by the reviewer " - "or deployer; use an empty string if none.", - "", - # ---------------------------------------------------- - # #2594 — primitives audit (cheap plan-phase NACK) - # ---------------------------------------------------- - "## Primitives audit (#2594)", - "", - "Plan-phase NACKs are cheap; implement-phase NACKs on missing " - "primitives are expensive (8+ pod spawns per slice, ~60–90 min " - "per cycle). Make the audit cheap by **pre-citing every " - "primitive your tasks depend on**. For each named class, " - "function, HTTP route, env var, ConfigMap key, test fixture, " - "CLI flag, or decorator your plan references:", - "", - "1. **Cite existence** with `file:line` (use `grep -rn` to " - "verify *before* writing the task). If the primitive does not " - "exist yet because the task itself will create it, mark it " - "`(NEW — task TASK-X-Y)` so the plan reviewer doesn't NACK on " - "missing-primitive evidence. When you mark a primitive " - "`(NEW — task TASK-X-Y)`, you MUST also: (a) ensure the " - "referenced task's acceptance criteria actually produce that " - "primitive in the form the plan uses (right kind, right " - 'module, right scope — not just "adds the feature"), and ' - "(b) order downstream tasks that consume the primitive " - "**after** the creating task in the slice DAG. The plan " - "reviewer's §9 exception verifies both; mismatches NACK.", - "2. **Cite trust-boundary scope.** Some primitives exist but " - "are unavailable in the execution context the task assumes. " - "Canonical example: `ScriptedProvider` is unit-test-only; " - "deployed agent pods run the real provider. Likewise the " - "`integration_tests/` fixture tiering — the only " - "`gateway_url` pytest fixture lives at " - "`integration_tests/local_pipeline/conftest.py:261` and is " - "kubectl-gated via `local_pipeline_stack`. The parent " - "`integration_tests/conftest.py` does **not** expose " - "`gateway_url` as a fixture; it exposes `gateway_url` as an " - "attribute on the `EggStack` dataclass " - "(`integration_tests/conftest.py:78`), accessed as " - "`egg_stack.gateway_url`, not as a fixture-injectable " - "parameter. `orchestrator_url` and lifecycle-secret-gated " - "routes are also `local_pipeline/`-only. **No pytest fixture " - "in `integration_tests/` is `in-sandbox-agent`-runnable " - "today** — every fixture transitively depends on `egg_stack` " - "or `local_pipeline_stack`, both of which `pytest.skip` when " - "`_kubectl_available()` returns `False`. Tasks that need any " - "of `gateway_url` / `orchestrator_url` as a pytest fixture " - "MUST live under (or below) `local_pipeline/` or an " - "equivalent trusted directory. Verify with " - "`grep -rn 'def gateway_url' integration_tests/` — exactly " - "one hit. The agent-runtime `GATEWAY_URL` env is a " - "**separate surface** from pytest fixtures; production code " - "an agent writes can reach the gateway sidecar through it, " - "but that is not a pytest test. See " - "`docs/architecture/integration-test-trust-boundary.md`.", - "", - "Recommended shape: a short `## Primitives` section in the " - "prose with one row per primitive (name, `file:line`, " - "execution-context scope). The plan reviewer will run the " - "Primitive-Existence Audit (criteria §9) and Trust-Boundary " - "Audit (criteria §10) against this table; both are hard " - "NACKs when a named primitive has no grep hit or is used " - "outside its scope.", - "", - # ---------------------------------------------------- - # #2137 — slice-DAG planner guidance - # ---------------------------------------------------- - "## Slice-DAG guidance (#2137)", - "", - "The implement-phase pipeline now ships each plan **slice** " - "(formerly **phase**) as its own stacked PR. The plan you " - "emit drives that DAG; the planner rules below are mandatory.", - "", - "**Yaml key swap**: prefer the canonical ``slices:`` key in " - "your ``# yaml-tasks`` block (the parser also accepts " - "``phases:`` for backward compatibility with already-shipped " - "planner prompts). New plans should use ``slices:``.", - "", - "**Slice sizing is the architect's call (#2809).** Slice " - "count, boundaries, and DAG shape come from the architect's " - "scaffold — copy them verbatim. ``reviewer_plan`` will hard " - "NACK ``architect`` on ``slice_size`` when a slice is " - "oversized for one BRC cycle (judgment-based — see the " - "reviewer's §11 rubric); do NOT silently re-shape slices " - "to dodge a size concern. Raise it as NACK pressure on " - "architect instead (see the surfacing guidance above).", - "", - "**Forest constraint (HARD, enforced at plan ingestion)**: " - "every slice must have at most ONE DAG parent — the " - "implement-phase pipeline ships every slice as a stacked " - "PR with exactly one base branch. The architect's scaffold " - "encodes this via a single-parent ``dependencies`` id " - "(``slice-<N>``); preserve it.", - "", - "**Auto-serialization for would-be multi-parent slices**: " - "the architect is responsible for serialising would-be " - "multi-parent slices and populating " - "``serialized_chain_order`` on the downstream slice. " - "Preserve that field verbatim from the scaffold.", - "", - "**File-overlap ⇒ ordering (HARD — #3046)**: you fill in each " - "slice's tasks and their ``files_affected``, so you see the " - "file sets first. If you find yourself assigning the SAME file " - "to two slices that the architect left unordered (parallel " - "roots or siblings), do NOT silently proceed — plan ingestion " - "hard-rejects overlapping slices with no dependency edge, " - "because their branches fork independently off the shared base " - "and collide at integration. Raise NACK pressure on the " - "architect (via the plan prose) to serialise the overlapping " - "cluster into one ``dependencies`` chain — or to merge the " - "slices. Do not re-shape the slice DAG yourself.", - "", - "Worked example: if ``slice-3`` would naturally have " - "parents ``[slice-1, slice-2]``, instead emit:", - "", - "```yaml", - " - id: 1", - " name: |-", - " Foundations", - " # ... (root)", - " - id: 2", - " name: |-", - " Middle", - " dependencies:", - " - slice-1", - " - id: 3", - " name: |-", - " Downstream", - " dependencies:", - " - slice-2 # serialised — slice-2 is the only DAG parent", - " serialized_chain_order:", - " - slice-1", - " - slice-2 # records that you deliberately picked", - " # slice-1 → slice-2 → slice-3", - "```", - "", - "Your judgement is the source of truth. The fallback " - "heuristic when you have no preference is: cluster " - "would-be parents by ``files_affected`` Jaccard overlap " - "(>0.3), then order by descending downstream fan-out.", - "", - f"Write your plan to `{draft_path}`.", - "", - *_EXPLORATION_SUBAGENT_GUIDANCE, - ] - ) - # Append role file restriction info so the planner assigns tasks correctly. - # Pass the pipeline's repo so per-repo role_patterns from - # repositories.yaml are rendered (#2528) — keeps planner-prompt - # boundaries in sync with the gateway's push-time enforcement. - lines.append(_build_role_restrictions_section(repo=repo or None)) - elif role_value == "risk_analyst": - lines.extend( - [ - "**You are dual-role (producer AND reviewer) in this phase " - "(#2809).** You produce the risk register AND you review " - "``architect`` and ``task_planner`` through the risk lens — " - "your NACK blocks plan-phase consensus until the upstream " - "producer re-proposes addressing the concern. This mirrors " - "the implement-phase ``tester`` dual-role pattern (#2749); " - "the *Dual-Role Execution Order* banner in your BRC " - "preamble is the authoritative ordering — read it first.", - "", - "## Producer role (risk register)", - "", - "Assess technical risks for the proposed implementation:", - "", - "1. Review the architecture analysis from the ARCHITECT agent", - "2. Identify technical risks (security, performance, compatibility)", - "3. Research externally when the change involves third-party dependencies — " - "use WebSearch and WebFetch (when available) to check for known " - "vulnerabilities, deprecation notices, and compatibility issues. " - "Skip external research for purely internal changes.", - "4. Assess impact and likelihood of each risk", - "5. Propose mitigation strategies and rollback plans", - "6. Flag areas that need human review", - "7. **Flag runtime-primitive and trust-boundary risks (see " - "#2594).** Plans that depend on classes, fixtures, routes, " - "or env vars which don't exist in the form the plan assumes " - "— or which exist but only in a different execution context " - "than the task uses (e.g. unit-test-only `ScriptedProvider` " - "vs deployed agent pods; `orchestrator_url` fixture defined " - "only in `integration_tests/local_pipeline/conftest.py` vs " - "in-sandbox-agent tests) — are a recurring high-impact " - "failure mode (see #2474). Call these out explicitly so the " - "plan reviewer can audit them.", - "", - f"Write your risk assessment to `{_risk_analyst_output_path}`.", - "", - "## Reviewer role (risk lens on architect + task_planner)", - "", - "When ``architect`` or ``task_planner`` proposes (their " - "``CONSENSUS_PROPOSE`` will wake you via the dual-role " - "augmentation on your producer waits — see the banner), " - "review their work through the risk lens and emit ACK or " - "NACK. ``blocking_concerns`` are NACK-shaped: they block " - "plan-phase consensus and force the upstream producer to " - "re-propose addressing them.", - "", - "Use this verdict shape in your producer artifact " - "(risk-register JSON) **and** mirror the verdict / " - "feedback in your ``egg-orch consensus ack`` / " - "``egg-orch consensus nack`` ``--reason`` body so the " - "upstream producer can act on it:", - "", - "```json", - "{", - ' "verdict": "ACK" | "NACK",', - ' "risks": [...],', - ' "top_3_risks": [...],', - ' "blocking_concerns": [...],', - ' "feedback": "concrete revision instructions for architect / task_planner (empty on ACK)"', - "}", - "```", - "", - "NACK when a risk is severe enough that shipping the plan " - "as-proposed would invite a known-class failure (security " - "regression, data loss, compliance break, runtime-primitive " - "or trust-boundary mismatch that would surface as an " - "expensive implement-phase NACK). ACK when risks are real " - "but mitigated, or low enough that the plan can ship and " - "the risks belong in the register as forward-looking " - "notes. Be specific in ``feedback`` — name the file, " - "the slice, the missing mitigation — so the upstream " - "producer's re-propose is actionable.", - "", - *_EXPLORATION_SUBAGENT_GUIDANCE, - ] - ) - elif role_value == "simplifier": - if phase == "plan": - _upstream = "task_planner" - _upstream_draft = "the implementation plan" - _human_path = _plan_human_path - _essence = ( - "what will be built, the major steps/phases, the test strategy " - "in brief, and the key risks" - ) - else: # refine - _upstream = "refiner" - _upstream_draft = "the refine analysis" - _human_path = _analysis_human_path - _essence = "the problem, the recommended approach, and the key trade-offs" - lines.extend( - [ - "**You are a producer only in this phase.** You produce a " - f"human-focused companion to {_upstream_draft}. You do NOT " - f"review **{_upstream}**'s draft, and you issue no ACK or NACK " - "on it: an internal wake-wire re-invokes you when it proposes " - "so you know its draft is ready, and consensus never waits on a " - "verdict from you. The *Execution Order* banner in your BRC " - "preamble is the authoritative ordering — read it first.", - "", - "## Producer role (human-focused companion)", - "", - f"Your WORK depends on **{_upstream}**'s draft existing. ORIENT " - f"now, then start producing only once **{_upstream}** issues " - "`CONSENSUS_PROPOSE` (the event pump re-invokes you carrying that " - "proposal). On that invocation:", - "", - f"1. Read **{_upstream}**'s draft of {_upstream_draft}.", - f"2. Write a HUMAN-FOCUSED companion to `{_human_path}`. This is a " - "simplified, higher-level summary for a **broad audience — " - "engineers, PMs, and managers** — not a peer review. Capture the " - f"essence: {_essence}.", - "", - " Rules:", - " - **Broad, mixed audience.** Write so a non-engineer " - "(PM, manager) can follow *what is changing and why it matters*, " - "while staying accurate enough for an engineer. Explain any " - "unavoidable technical term in plain language.", - " - **No egg-internal jargon.** Do not mention BRC, consensus, " - "propose/ACK/NACK, slices / slice-DAG, contracts, phases, " - "`serialized_chain_order`, Jaccard, or agent-role names. Describe " - "independently-shippable pieces in plain terms if you must " - "reference them at all.", - " - **No implementation minutiae.** No `file:line` references, " - "no function / struct / field / type names or other code " - "identifiers, no per-field enumerations. Describe behaviour and " - "impact, not the code.", - " - **This is NOT a review.** Do not critique, score, or gate " - 'the upstream draft. No ACK/NACK language, no "the draft should ' - 'commit to …", no "anti-pattern to reject", no constraint ' - "lists. You have no critique to record anywhere — your only " - "output is this plain-language summary.", - f" - **Exactly one file.** Commit ONLY `{_human_path}`. Do " - "NOT create any other `.egg-state/drafts/` file — no separate " - "`*-simplifier-*.md` constraints/guardrails/verification " - "companion. Any review reasoning goes in the BRC channel " - "(your verdict), never a second persisted document. A " - "proposal that introduces a second draft is rejected at " - "propose time.", - " - **Much shorter and more digestible** than the upstream " - "draft — plain prose and short lists, not exhaustive enumeration.", - " - **Faithful** — reflect the upstream draft accurately; " - "introduce no new scope, claims, or recommendations.", - "", - f"3. Commit and push `{_human_path}`, then PROPOSE it via " - "`egg-orch consensus propose`. The companion is **mandatory** — " - "always write at least a one-paragraph summary; do NOT take the " - "no-op propose path. That completes your work for this phase.", - "", - *_EXPLORATION_SUBAGENT_GUIDANCE, - ] - ) - else: - lines.extend( - [ - f"Execute your role as {role_value} for this phase.", - "", - ] - ) - - # Phase restrictions - _recovery_base_ref = _resolve_origin_ref(base_branch) - lines.append("## Phase Restrictions\n") - if phase == "implement": - lines.extend( - [ - "- You CAN push code changes to git (git push)", - "- You CAN link commits to tasks (egg-contract add-commit)", - "- You CANNOT push .egg-state/ files (except checkpoints)", - "- You CANNOT create PRs (the pipeline manages the PR)", - "", - "### Push Recovery", - "", - "If your push is rejected due to restricted files on the branch, " - f"create a clean branch from {_recovery_base_ref} and cherry-pick " - "only your code commits:", - "```", - f"git checkout -b egg/<new-branch> {_recovery_base_ref}", - "git cherry-pick <your-commit-hash>", - "git push origin egg/<new-branch>", - "```", - "Do NOT retry the same push — fix the branch first.", - "After pushing to the new branch, use `egg-contract add-commit` to " - "link your commits so the pipeline can track them on the new branch.", - "", - ] - ) - elif phase in ("refine", "plan"): - lines.extend( - [ - "- You CAN write to `.egg-state/drafts/` and `.egg-state/agent-outputs/`", - "- You CAN push these state files to git (git push)", - "- You CAN create HITL decisions (egg-contract add-decision)", - "- You CAN create feedback requests (egg-contract add-feedback)", - "- You CANNOT modify production code (src/, lib/, gateway/, sandbox/, " - "action/, docs/, tests/, test/)", - "- You CANNOT modify contracts (.egg-state/contracts/) or CI config (.github/)", - "- You CANNOT create PRs (gh pr create)", - "", - "### Push Recovery", - "", - "If your push is rejected due to restricted files on the branch, " - f"create a clean branch from {_recovery_base_ref} and cherry-pick " - "only your state file commits:", - "```", - f"git checkout -b egg/<new-branch> {_recovery_base_ref}", - "git cherry-pick <your-commit-hash>", - "git push origin egg/<new-branch>", - "```", - "Do NOT retry the same push — fix the branch first.", - "After pushing to the new branch, use `egg-contract add-commit` to " - "link your commits so the pipeline can track them on the new branch.", - "", - ] - ) - - # File boundaries (#1431) — surface allowed/blocked patterns so - # the agent avoids creating files the gateway will reject on push. - # Pass repo so the rendered patterns match per-repo overrides - # (#2528) the gateway will enforce on push. - boundary_section = _build_file_boundary_section(role_value, repo=repo) - if boundary_section: - lines.append(boundary_section) - - # Producer escape hatch (#2529) — tester/documenter are the other - # two impassing producer roles (coder is handled in the early-return - # branch above). They need the actionable - # check_file_restriction / report_impasse guidance so they don't - # invent workarounds when their assigned task is structurally - # impossible. - if role_value in ("tester", "documenter"): - lines.append(_build_impasse_escape_hatch_section()) - - lines.append("## Phase Completion\n") - if concurrent: - lines.extend( - [ - "When you have completed your primary work:\n", - "1. Commit all changes", - '2. Run: `egg-orch signal readiness --state READY --reason "Work complete"`', - "3. Enter an **event-driven** stay-alive wait (issue #1897). " - "Do NOT wrap `egg-orch` in a shell `for i in 1..N` loop, " - "and do NOT `sleep N` — use the server-side blocking primitive:", - "```bash", - "egg-orch message wait-loop \\", - " --for CONSENSUS_CONFIRMED \\", - " --for CONSENSUS_RE_REVIEW \\", - " --for OVERSEER_ALERT \\", - " --timeout 60", - "```", - "`wait-loop` blocks server-side and loops forever until a " - "NEW matching BRC event arrives (exit 0) or a permanent error " - "occurs (exit 1). There is no outer timeout — the wrapper " - "owns the 0/1 contract. Events that predate the call " - "(including your own just-sent CONSENSUS_CONFIRMED) are " - "skipped (issue #1925); if you need zero-drop semantics " - "across a send→wait boundary, capture the ID of your " - "send and pass `--since <id>`. See " - "`docs/reference/agent-wait-patterns.md` for the full " - "exit-code contract and the five anti-patterns to avoid.", - "4. If `wait-loop` returns with a message that affects your work, " - "transition back to WORKING, address it, then signal READY again. " - "**In particular, if you receive a `CONSENSUS_RE_REVIEW` message, " - "you MUST re-confirm via `egg-orch consensus confirmed` (or " - "re-review and ACK/NACK if you are a reviewer of the re-proposing " - "producer). Ignoring this message will stall the pipeline.**", - "5. **Do NOT exit.** The orchestrator will stop your container when consensus " - "is reached.", - ] - ) - else: - lines.append( - "When you have completed your work, ensure everything is committed and exit successfully." - ) - - return "\n".join(lines) - - -def _format_nack_summary(nack_details: list[dict]) -> str: - """Format unresolved NACK details into a human-readable summary string.""" - return "; ".join( - f"{n['reviewer']} NACKed {n['producer']}: {n.get('reason') or 'no reason given'}" - for n in nack_details - ) - - -def _incomplete_consensus_decision_text( - final_consensus: dict, - container_failure_count: int, - orchestrator_mode: bool = False, -) -> tuple[str, str]: - """Build (question, log_suffix) for incomplete-consensus HITL escalation. - - Distinguishes the two failure modes — unresolved NACKs vs. agents that - never confirmed — so the operator sees actionable detail in `/sdlc`. - - ``orchestrator_mode`` selects a mode-aware prefix: when the orchestrator - owns the event loop, no up-front containers ever ran, so the terminal - here is the consensus timeout, not container exit — the "All containers - exited" prefix would mislead an operator reading `/sdlc`. - """ - nacks = final_consensus.get("unresolved_nacks", []) or [] - blocking = final_consensus.get("blocking_agents", []) or [] - if container_failure_count: - prefix = f"{container_failure_count} container(s) exited with non-zero code; " - elif orchestrator_mode: - prefix = "Consensus timed out; " - else: - prefix = "All containers exited; " - # Retry semantics must match what "Retry phase" actually executes on - # resolve — the restart_phase route (#3421 dispatch, #3080 preservation - # semantics): fresh worktrees re-fork from the shared work branch tip, - # and unpushed per-role commits survive only via best-effort salvage. - retry_copy = ( - "'Retry phase' re-runs the phase from the shared work branch tip " - "(work pushed to the shared branch is preserved; unpushed per-role " - "commits are salvaged best-effort to egg/recovered/*)." - ) - if nacks: - summary = _format_nack_summary(nacks) - question = ( - f"{prefix}consensus incomplete with {len(nacks)} unresolved NACK(s): " - f"{summary}. {retry_copy} How to proceed?" - ) - log_suffix = f"\n--- INCOMPLETE CONSENSUS / UNRESOLVED NACKs ({len(nacks)}) ---\n{summary}" - else: - agent_list = ", ".join(blocking) if blocking else "unknown" - question = ( - f"{prefix}consensus incomplete; agents never confirmed: {agent_list}. " - f"{retry_copy} How to proceed?" - ) - log_suffix = ( - f"\n--- INCOMPLETE CONSENSUS / NO CONFIRMATION ---\nblocking_agents={agent_list}" - ) - return question, log_suffix - - -def _persist_hitl_decision( - pipeline_id: str, - pipeline: Pipeline, - store: StateStore, - *, - question: str, - options: list[str], - phase: PipelinePhase | None = None, - context: str | None = None, -): - """Create and persist an HITL decision under the pipeline state lock. - - `pipeline.add_decision()` only mutates an in-memory object. The caller - of `_run_concurrent_phase` reloads the pipeline fresh from disk before - writing FAILED, so any in-memory decision is silently dropped — the - on-disk state (which `/sdlc` reads via `pipeline.get_pending_decisions()`) - never sees it. This helper mirrors the *persistence half* of - `DecisionQueue.queue_decision()` and the HITL-gate write at - pipelines.py:13080-13089: load → mutate → save under the reentrant - pipeline state lock. Note: it intentionally does **not** invoke - `_notify_handlers` — no production code currently registers a - `DecisionHandler` and `/sdlc` reads from disk on each request, so - notifications are not needed for the issue-2203 path. The in-memory - `pipeline` argument is also synced so callers observe consistent state. - - ``context`` is set on the persisted decision before save so dispatch - handlers in :mod:`routes.decisions` can route on a stable string - discriminator rather than the prose-y ``question`` text (see the - ``failed_role:`` pattern). - - Returns the created decision, or None if persistence failed (logged; - callers should not raise — losing an HITL decision is bad but losing - the rest of the cleanup path is worse). - """ - try: - with get_pipeline_state_lock(pipeline_id): - disk_pipeline = store.load_pipeline(pipeline_id) - decision = disk_pipeline.add_decision( - question=question, - options=options, - phase=phase or disk_pipeline.current_phase, - ) - if context is not None: - decision.context = context - store.save_pipeline(disk_pipeline) - # Defensive copy: avoid sharing the list reference with the - # disk-loaded copy, which is local and goes out of scope. - pipeline.decisions = list(disk_pipeline.decisions) - return decision - except Exception: - logger.warning( - "Failed to persist HITL decision", - pipeline_id=pipeline_id, - question=question[:100], - exc_info=True, - ) - return None - - -def _cancel_consensus_timeout_decisions(pipeline: Pipeline) -> int: - """Cancel any pending consensus-timeout HITL on ``pipeline`` (#3315 facet c). - - Pure mutator (no lock / load / save): marks every pending - ``consensus_timeout_incomplete`` decision ``CANCELLED`` with an - auto-withdrawal note and returns how many it cancelled. Called from the - consensus-success path (under the pipeline state lock, on the freshly - loaded pipeline that is about to be saved) so a stale forced-choice a - *superseded* thread opened before the phase converged is withdrawn in the - same write that marks the agents COMPLETE — the operator is never left - disposing of a decision the system already obsoleted by converging. - """ - withdrawn = 0 - for decision in pipeline.get_pending_decisions(): - if decision.context != _CONSENSUS_TIMEOUT_HITL_CONTEXT: - continue - decision.status = DecisionStatus.CANCELLED - decision.resolution = "auto-withdrawn: consensus subsequently converged" - decision.resolved_at = datetime.now(UTC) - withdrawn += 1 - return withdrawn - - -def _withdraw_arms_exhausted_decisions(pipeline_id: str, store: StateStore) -> int: - """Cancel any pending arms-exhausted HITL on ``pipeline_id`` (#3496 review). - - The symmetric counterpart to :func:`_persist_hitl_decision` on the - arms-exhausted path: when the wedge clears (the blocked arms recovered by - a route other than the operator resolving this decision — a fresh key - derived, a spawn succeeded, an unrelated decision re-keyed the arms) the - pending ``event_arms_exhausted`` decision is obsolete, so this withdraws it - rather than leaving the operator to dispose of a decision the system - already resolved for them (mirrors :func:`_cancel_consensus_timeout_decisions` - on the convergence-success path). - - Unlike ``_cancel_consensus_timeout_decisions`` — a pure mutator that - piggybacks on the convergence-success write already under the state lock — - the wedge-clear path has no ambient lock/load/save, so this does its own - load → cancel → save under ``get_pipeline_state_lock``. Returns how many - decisions were withdrawn (0 when none were pending, so the caller can skip - logging on the common no-op). - """ - from concurrent_executor import ARMS_EXHAUSTED_HITL_CONTEXT - - with get_pipeline_state_lock(pipeline_id): - disk_pipeline = store.load_pipeline(pipeline_id) - withdrawn = 0 - for decision in disk_pipeline.get_pending_decisions(): - if decision.context != ARMS_EXHAUSTED_HITL_CONTEXT: - continue - decision.status = DecisionStatus.CANCELLED - decision.resolution = ( - "auto-withdrawn: the wedge cleared (blocked arms recovered) " - "before this decision was resolved" - ) - decision.resolved_at = datetime.now(UTC) - withdrawn += 1 - if withdrawn: - store.save_pipeline(disk_pipeline) - return withdrawn - - -# Bound on how many times the worktree-divergence reconcile will pause -# for the operator before giving up and failing the pipeline (#2979). -# A small budget guards against an operator repeatedly choosing -# "Reconciled — resume" without actually reconciling the worktree, which -# would otherwise re-pause forever. -_MAX_DIVERGENCE_RECONCILE_PAUSES = 3 - -_DIVERGENCE_RECONCILE_RESUME = "Reconciled — resume" -_DIVERGENCE_RECONCILE_ABORT = "Abort pipeline" -_DIVERGENCE_RECONCILE_HITL_OPTIONS = [ - _DIVERGENCE_RECONCILE_RESUME, - _DIVERGENCE_RECONCILE_ABORT, -] - -# Stable string discriminator set on persisted reconcile HITLs (#2979). The -# non-blocking ``populate_contract`` route uses this to dedupe: when an -# operator re-POSTs against an already-paused pipeline (e.g. an automated -# retry or a refresh through ``/sdlc`` before resolving the prior HITL), the -# route surfaces the existing pending decision rather than appending a fresh -# one. -_DIVERGENCE_RECONCILE_HITL_CONTEXT = "divergence_reconcile_unacked" - -# Stable string discriminator set on the consensus-timeout / incomplete- -# consensus HITL the orchestrator opens when a phase times out without -# converging (the "Consensus timed out; consensus incomplete …" -# Retry/Accept/Abort decision). The convergence-success path uses this to -# auto-withdraw the decision once the phase reaches genuine consensus, so an -# operator is never left disposing of a decision the system already obsoleted -# (#3315 facet c — happens when a superseded thread opens the decision and a -# restarted thread then converges). -_CONSENSUS_TIMEOUT_HITL_CONTEXT = "consensus_timeout_incomplete" - - -def _find_pending_divergence_reconcile_decision(pipeline: Pipeline): - """Return the oldest pending reconcile HITL on ``pipeline`` (or None). - - Used by the non-blocking ``populate_contract`` route to dedupe re-POSTs - against a pipeline already paused on a reconcile HITL — without this, - every retry would append a fresh decision and bloat ``pipeline.decisions`` - (the abort path still works on the most recent decision; this is a UX / - cleanliness fix, not a correctness fix). - """ - for decision in pipeline.get_pending_decisions(): - if decision.context == _DIVERGENCE_RECONCILE_HITL_CONTEXT: - return decision - return None - - -def _divergence_reconcile_is_abort(resolution: str) -> bool: - """True when a reconcile-HITL resolution selects abort (#2979). - - Accepts the canonical ``Abort pipeline`` label, a couple of forgiving - synonyms, and the JSON ``{"action": ...}`` envelope the collaborator - UI sends. Any *other* resolution — the resume label, free text, an - empty string — is treated as "Reconciled — resume", so an ambiguous - resolution errs toward re-attempting the (now non-destructive) sync - rather than failing the pipeline. - """ - r = resolution.strip() - if not r: - return False - try: - payload = json.loads(r) - if isinstance(payload, dict) and "action" in payload: - r = str(payload["action"]) - except json.JSONDecodeError, TypeError: - pass - return r.strip().lower() in { - _DIVERGENCE_RECONCILE_ABORT.lower(), - "abort", - "cancel", - } - - -def _divergence_reconcile_hitl_question( - *, - pipeline_id: str, - phase: PipelinePhase | None, - backup_ref: str | None, - local_only_commit_shas: tuple[str, ...] | list[str], - rebase_category: str | None = None, - rebase_detail: str | None = None, -) -> str: - """Build the HITL question for the non-destructive divergence pause (#2979). - - The worktree diverged from origin and the rebase autoresolve could - not reconcile it. Nothing has been discarded — the autoresolve - aborted back to the clean local HEAD, so the orchestrator's committed - work is intact — and the pipeline is paused (AWAITING_HUMAN, not - FAILED). The operator reconciles the orchestrator-side worktree - manually, then either resumes (the sync re-runs and the phase's - post-processing continues from where it paused) or aborts. - - ``rebase_category`` / ``rebase_detail`` name the actual autoresolve - failure (conflicting paths, rebase argv, git output excerpt) so the - operator can judge the pause from the decision alone (#3416). - """ - phase_label = phase.value if phase is not None else "current phase" - if rebase_category or rebase_detail: - failure_label = rebase_category or "unknown failure" - failure_line = ( - f"({failure_label}: {rebase_detail})" if rebase_detail else f"({failure_label})" - ) - else: - failure_line = ( - "(failure detail unavailable — see the divergence_rebase_failed " - "log line for the rebase output)" - ) - backup_line = ( - f"A backup ref pins the current tip: {backup_ref} (inspect with `git log {backup_ref}`)." - if backup_ref - else "Backup ref write failed — see the WARN log for the inlined commit SHAs." - ) - if local_only_commit_shas: - commits_block = "Local-only commits preserved on the worktree HEAD:\n - " + "\n - ".join( - local_only_commit_shas - ) - else: - commits_block = ( - "The local-only commit list could not be enumerated; check the " - "WARN log and the backup ref for the exact set." - ) - return ( - f"Pipeline {pipeline_id}: the worktree diverged from origin at the " - f"{phase_label} boundary and the rebase autoresolve could not " - f"reconcile it {failure_line}. " - f"Nothing was discarded — the worktree is left at the local HEAD " - f"with the orchestrator's committed work intact, and the pipeline " - f"is paused (not failed) for a manual reconcile (#2979). " - f"{backup_line}\n{commits_block}\n\n" - f"Reconcile the orchestrator-side worktree manually (e.g. rebase the " - f"local commits onto origin/<branch> and resolve the conflict), then " - f"choose:\n" - f"- '{_DIVERGENCE_RECONCILE_RESUME}' — re-run the worktree sync and " - f"resume the {phase_label} phase's post-processing from where it " - f"paused (no full phase re-run).\n" - f"- '{_DIVERGENCE_RECONCILE_ABORT}' — fail the pipeline; the backup " - f"ref preserves the commits for offline inspection." - ) - - -def _emit_divergence_reconcile_hitl( - pipeline_id: str, - store, # noqa: ANN001 — StateStore (avoid import cycle) - *, - phase: PipelinePhase | None, - backup_ref: str | None, - local_only_commit_shas: tuple[str, ...] | list[str], - rebase_category: str | None = None, - rebase_detail: str | None = None, -): - """Pin pipeline+phase to AWAITING_HUMAN and persist the reconcile HITL (#2979). - - Used by the non-blocking ``populate_contract`` route, which cannot - block on the operator the way the in-loop phase-boundary callers do. - Sets the pipeline + phase to ``AWAITING_HUMAN`` (NOT ``FAILED`` — the - divergence is recoverable and nothing was discarded) and persists the - reconcile HITL under the same lock so a reader never observes - ``AWAITING_HUMAN`` without the pending decision, then broadcasts a - ``decision.created`` event. - - Returns the persisted decision (or None on persistence failure). The - operator reconciles the worktree, resolves this decision, and re-runs - ``populate_contract`` against the now-reconciled worktree. - """ - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - if phase is not None: - phase_execution = pipeline.get_phase_execution(phase) - if phase_execution is not None: - phase_execution.status = PipelineStatus.AWAITING_HUMAN - pipeline.status = PipelineStatus.AWAITING_HUMAN - store.save_pipeline(pipeline) - decision = _persist_hitl_decision( - pipeline_id, - pipeline, - store, - question=_divergence_reconcile_hitl_question( - pipeline_id=pipeline_id, - phase=phase, - backup_ref=backup_ref, - local_only_commit_shas=tuple(local_only_commit_shas), - rebase_category=rebase_category, - rebase_detail=rebase_detail, - ), - options=list(_DIVERGENCE_RECONCILE_HITL_OPTIONS), - phase=phase, - context=_DIVERGENCE_RECONCILE_HITL_CONTEXT, - ) - report_pipeline_status( - pipeline, - event_type="decision.created", - message=( - f"Awaiting manual worktree reconcile for " - f"{phase.value if phase else 'current phase'} phase" - ), - ) - _emit_pipeline_event(pipeline, "decision.created") - return decision - - -def _fail_pipeline_after_divergence_abort( - pipeline_id: str, - store, # noqa: ANN001 — StateStore (avoid import cycle) - *, - phase: PipelinePhase | None, - backup_ref: str | None, - local_only_commit_shas: tuple[str, ...] | list[str], - budget_exhausted: bool = False, - pre_event_hook: Callable[[], None] | None = None, -) -> None: - """Pin pipeline+phase to FAILED after an aborted divergence reconcile (#2979). - - Reached when the operator resolved the reconcile HITL with - ``Abort pipeline`` (or the reconcile pause budget was exhausted). No - HITL is emitted here — the reconcile decision was already surfaced and - resolved. Mirrors the FAILED-write + ``pipeline.failed`` broadcast of - the old destructive-recovery helper, minus the discard: the committed - work is still on HEAD and pinned under ``backup_ref`` for offline - recovery. - - ``pre_event_hook`` runs after the FAILED-write but before the public - ``pipeline.failed`` broadcast (the post-phase site uses it to tear down - the per-phase overseer container). - """ - phase_label = phase.value if phase is not None else "current phase" - reason = ( - "the reconcile pause budget was exhausted" - if budget_exhausted - else "the operator chose to abort" - ) - error_message = ( - f"Worktree diverged from origin at {phase_label} and could not be " - f"auto-reconciled; {reason} (#2979). Local-only commits are " - f"preserved under {backup_ref or '(backup ref write failed)'} " - f"({len(local_only_commit_shas)} commit(s))." - ) - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - if phase is not None: - phase_execution = pipeline.get_phase_execution(phase) - if phase_execution is not None: - phase_execution.status = PipelineStatus.FAILED - phase_execution.error = error_message - phase_execution.completed_at = datetime.now(UTC) - pipeline.status = PipelineStatus.FAILED - pipeline.error = error_message - store.save_pipeline(pipeline) - if pre_event_hook is not None: - pre_event_hook() - report_pipeline_status( - pipeline, - event_type="pipeline.failed", - message=f"Pipeline failed: {error_message[:100]}", - ) - _emit_pipeline_event(pipeline, "pipeline.failed") - - -def _sync_worktree_reconciling_divergence( - spawner: "ContainerSpawner", # noqa: UP037 - pipeline_id: str, - store, # noqa: ANN001 — StateStore (avoid import cycle) - repo_path: Path, - *, - worktree_repo_path: Path, - phase: PipelinePhase | None, - gateway_mode: Literal["public", "private"] = "public", - base_branch: str | None = None, - pipeline_branch: str | None = None, - prior_phase_succeeded: bool = True, - max_reconcile_pauses: int = _MAX_DIVERGENCE_RECONCILE_PAUSES, -) -> tuple[WorktreeSyncOutcome, bool]: - """Sync the worktree, pausing for a manual reconcile on divergence (#2979). - - Runs :func:`_sync_worktree_with_remote`. When the helper reports an - unreconciled divergence (``diverged_unreconciled``), the worktree is - left non-destructively at the local HEAD; this function pauses the - pipeline (``AWAITING_HUMAN``) on a reconcile HITL and **blocks** the - ``_run_pipeline`` thread on ``wait_for_decision`` — the same proven - pause primitive the phase-approval gate uses. When the operator - resolves the HITL with "Reconciled — resume", the pipeline returns to - ``RUNNING`` and the sync re-runs; the caller then continues the same - phase's post-processing from where it paused, with no full re-run and - nothing discarded. - - Returns ``(outcome, aborted)``. ``aborted`` is True when the operator - chose "Abort pipeline" or the reconcile-pause budget was exhausted; the - caller should fail the pipeline via - :func:`_fail_pipeline_after_divergence_abort`. When ``aborted`` is - False the worktree is reconciled (or never diverged) and the caller - proceeds normally. - - Only call this from inside the ``_run_pipeline`` loop thread, which is - allowed to block; route handlers that cannot block use - :func:`_emit_divergence_reconcile_hitl` instead. - """ - dq = get_decision_queue(pipeline_id, repo_path) - phase_label = phase.value if phase is not None else "current phase" - - outcome = _sync_worktree_with_remote( - spawner, - pipeline_id, - worktree_repo_path, - prior_phase_succeeded=prior_phase_succeeded, - gateway_mode=gateway_mode, - base_branch=base_branch, - pipeline_branch=pipeline_branch, - ) - - pauses = 0 - while outcome.diverged_unreconciled: - if pauses >= max_reconcile_pauses: - logger.error( - "OVERSEER_ALERT worktree_divergence_reconcile_budget_exhausted", - pipeline_id=pipeline_id, - phase=phase_label, - pauses=pauses, - backup_ref=outcome.backup_ref, - ) - return outcome, True - pauses += 1 - - # Persist the reconcile HITL and flip to AWAITING_HUMAN under the - # (reentrant) state lock so a reader never sees AWAITING_HUMAN - # without the pending decision. - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - pipeline.status = PipelineStatus.AWAITING_HUMAN - if phase is not None: - phase_execution = pipeline.get_phase_execution(phase) - if phase_execution is not None: - phase_execution.status = PipelineStatus.AWAITING_HUMAN - store.save_pipeline(pipeline) - decision = _persist_hitl_decision( - pipeline_id, - pipeline, - store, - question=_divergence_reconcile_hitl_question( - pipeline_id=pipeline_id, - phase=phase, - backup_ref=outcome.backup_ref, - local_only_commit_shas=outcome.local_only_commit_shas, - rebase_category=outcome.rebase_category, - rebase_detail=outcome.rebase_detail, - ), - options=list(_DIVERGENCE_RECONCILE_HITL_OPTIONS), - phase=phase, - context=_DIVERGENCE_RECONCILE_HITL_CONTEXT, - ) - if decision is None: - # Could not persist the HITL — fail closed rather than spin on - # a pause the operator can never see. - logger.error( - "worktree_divergence_reconcile_hitl_persist_failed", - pipeline_id=pipeline_id, - phase=phase_label, - ) - return outcome, True - - logger.error( - "OVERSEER_ALERT worktree_divergence_reconcile_pause", - pipeline_id=pipeline_id, - phase=phase_label, - backup_ref=outcome.backup_ref, - local_only_commit_count=len(outcome.local_only_commit_shas), - rebase_category=outcome.rebase_category, - rebase_detail=outcome.rebase_detail, - pause_attempt=pauses, - ) - - # Once AWAITING_HUMAN is persisted, an unexpected exception - # between here and the resume-write (e.g. a broadcast IO error, - # a decision-queue runtime error, a transient store failure on - # ``get_decision``) would leave the pipeline pinned to - # AWAITING_HUMAN on disk while ``_run_pipeline``'s outer - # ``try/except`` catches the error and moves on — stranding the - # operator with no waiter ever returning. Guard the - # wait-and-resolve span: on unexpected error revert to RUNNING - # before re-raising so the caller still observes the failure - # but the pipeline is not left in an unrecoverable paused state. - # The abort path (operator chose ``Abort pipeline``) returns - # normally with ``aborted=True`` so the caller can flip to - # FAILED — that's not an exception and skips the revert. - try: - report_pipeline_status( - pipeline, - event_type="decision.created", - message=f"Awaiting manual worktree reconcile for {phase_label} phase", - ) - _emit_pipeline_event(pipeline, "decision.created") - - dq.wait_for_decision(decision.id) - - resolved = dq.get_decision(decision.id) - resolution = (resolved.resolution or "") if resolved is not None else "" - if _divergence_reconcile_is_abort(resolution): - logger.warning( - "worktree_divergence_reconcile_aborted_by_operator", - pipeline_id=pipeline_id, - phase=phase_label, - ) - return outcome, True - - # Operator reconciled the worktree — return to RUNNING and re-run - # the sync. If it still diverges, loop and re-pause (bounded). - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - pipeline.status = PipelineStatus.RUNNING - if phase is not None: - phase_execution = pipeline.get_phase_execution(phase) - if phase_execution is not None: - phase_execution.status = PipelineStatus.RUNNING - store.save_pipeline(pipeline) - except Exception: - # Best-effort revert: load fresh, flip AWAITING_HUMAN→RUNNING - # only if still pinned, then re-raise. Swallow secondary - # errors from the revert itself — losing the revert is bad, - # but masking the original failure with a save error is - # worse. The operator can still recover via the pending - # decision (the decision queue may have replayed it on - # restart) or via the backup ref. - try: - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - if pipeline.status == PipelineStatus.AWAITING_HUMAN: - pipeline.status = PipelineStatus.RUNNING - if phase is not None: - phase_execution = pipeline.get_phase_execution(phase) - if ( - phase_execution is not None - and phase_execution.status == PipelineStatus.AWAITING_HUMAN - ): - phase_execution.status = PipelineStatus.RUNNING - store.save_pipeline(pipeline) - except Exception: - logger.warning( - "worktree_divergence_reconcile_revert_failed", - pipeline_id=pipeline_id, - phase=phase_label, - exc_info=True, - ) - raise - logger.info( - "worktree_divergence_reconcile_resume", - pipeline_id=pipeline_id, - phase=phase_label, - pause_attempt=pauses, - ) - outcome = _sync_worktree_with_remote( - spawner, - pipeline_id, - worktree_repo_path, - prior_phase_succeeded=prior_phase_succeeded, - gateway_mode=gateway_mode, - base_branch=base_branch, - pipeline_branch=pipeline_branch, - ) - - return outcome, False - - -def _emit_empty_contract_hitl( - pipeline_id: str, - pipeline: Pipeline, - store: StateStore, - *, - reason: str, - draft_slice_count: int | None, - gate: Literal[ - "slice_gate", - "start_phase_implement_safety_net", - "plan_complete", - ], - phase: PipelinePhase | None = None, -): - """Persist a dedicated HITL naming the empty-contract divergence (#2627). - - Built on top of :func:`_persist_hitl_decision` so it inherits the - "load → mutate → save under lock" persistence semantics that make - the decision survive the FAILED-write the calling block does next. - Best-effort: a persistence failure logs and returns None so the - surrounding FAILED-cleanup is not blocked. - - Returns the persisted decision (or None on persistence failure). - - Plain "Retry phase" against this HITL would respawn the implement - phase into the same empty-contract state, so the option set is - distinct from the generic phase-failure decision: callers are - expected to wire each option to its concrete recovery action - (see :data:`_EMPTY_CONTRACT_HITL_OPTIONS` for the mapping). - """ - # ``_empty_contract_hitl_question`` is defined further down the - # module alongside the other #2627 follow-up helpers; importing - # the symbol here keeps the call-site test isolated from module - # top-level ordering. - return _persist_hitl_decision( - pipeline_id, - pipeline, - store, - question=_empty_contract_hitl_question( - pipeline_id=pipeline_id, - reason=reason, - draft_slice_count=draft_slice_count, - gate=gate, - ), - options=list(_EMPTY_CONTRACT_HITL_OPTIONS), - phase=phase, - ) - - -def _check_brc_progress_gate( - pipeline_id: str, - slice_id: str | None, - active_role_names: list[str], - gate_seconds: float, -) -> tuple[bool, str | None]: - """Return (defer, reason) for the BRC consensus-timeout progress gate (#2243). - - Defers the consensus-timeout ``OVERSEER_ALERT`` (#2264; previously - an auto-``choice`` HITL decision) when *any* of the following has - fired within ``gate_seconds``: - - * The BRC tracker's most recent ``CONSENSUS_PROPOSE`` (producer - proposal) timestamp. - * The most recent ACK/NACK timestamp on the approval matrix. - * The most recent container heartbeat for any role in - ``active_role_names`` (filters out cross-phase pollution in the - shared :class:`HealthMonitor` singleton). - - The gate is the operator-friendly half of the issue-2243 fix: at - :data:`consensus_timeout_minutes` we previously opened a `choice` - decision unconditionally, even when producers were minutes from - their first commit. With the gate, the polling loop keeps polling - while signals are alive; the alert is only published once the bus - and containers have both gone quiet for ``gate_seconds``. - - ``gate_seconds <= 0`` disables the gate (returns ``(False, None)``). - Failures in any signal source are logged at WARNING and treated as - "no signal from that source" — never as a gate defer, since a - crashed signal collector must not silently keep us off the alert - surface. - - Heartbeat-cadence contract: the coder-mid-merge-conflict path - (no ``CONSENSUS_PROPOSE`` yet, only container heartbeats — the - original incident's ``decision-17`` flavour, pre-#2264) relies on - container heartbeats firing at least every ``gate_seconds``. Sandbox - heartbeats (see ``shared/egg_agent`` heartbeat scheduler and - ``orchestrator/health_monitor.py``) cadence today is well under - 300s, but a long uninterruptible subprocess (e.g. ``git rebase`` - blocked on a merge driver) could starve them; once that happens - the gate falls open and the pre-fix behaviour returns. Tracked as - a follow-up under #2243. - - TODO(#2243 step 2): same-role cross-phase pollution. The role-name - filter handles different-role ghosts (refiner heartbeat lingering - during a coder phase) but not same-role ghosts: ``coder`` reappears - across implement / implement-fix / fix-on-PR phases and - ``HealthMonitor._last_heartbeat['coder']`` is only popped on - ``clear_agent_state``. A phase boundary clear (or stamping the - heartbeat key with the phase) would close it; per-phase timeouts - in step 2 of the issue plan will likely subsume it. - """ - if gate_seconds <= 0: - return False, None - - # Two clocks, deliberately. ``now_dt`` is used for tracker - # timestamps (datetime in UTC). ``now_wallclock`` is the float - # epoch ``time.time()`` returns, matching the wall-clock values - # ``HealthMonitor._last_heartbeat`` is populated with. Despite the - # earlier name ``now_mono``, these are NOT monotonic — an NTP step - # on the orchestrator host can make ``(now - latest_hb)`` negative - # or skip the gate window. Acceptable today; revisit alongside the - # per-phase-timeout follow-up. - now_dt = datetime.now(UTC) - now_wallclock = time.time() - - # 1. BRC bus signals (proposal + ACK/NACK timestamps). - try: - try: - from peer_consensus import get_peer_consensus_tracker - except ImportError: - from ..peer_consensus import ( - get_peer_consensus_tracker, # type: ignore[no-redef] - ) - tracker = get_peer_consensus_tracker(pipeline_id, slice_id) - if tracker is not None: - ts = tracker.get_latest_progress_timestamp() - if ts is not None and (now_dt - ts).total_seconds() < gate_seconds: - age = (now_dt - ts).total_seconds() - return True, f"BRC bus active {age:.0f}s ago" - except Exception as e: - logger.warning( - "BRC progress-gate tracker check failed", - pipeline_id=pipeline_id, - error=str(e), - ) - - # 2. Container heartbeats. Filter by active roles so a stale - # heartbeat from a prior phase in the singleton HealthMonitor - # doesn't keep us out of the HITL surface forever. An empty - # ``active_role_names`` means the caller has no live containers - # to gate on, so match nothing rather than every stale heartbeat. - if not active_role_names: - return False, None - try: - from health_monitor import get_health_monitor - - hm = get_health_monitor() - if hm is not None: - active_set = set(active_role_names) - latest_hb: float | None = None - with hm._lock: # noqa: SLF001 — read-only snapshot - hb_snapshot = dict(hm._last_heartbeat) # noqa: SLF001 - for agent_id, hb_time in hb_snapshot.items(): - if agent_id not in active_set: - continue - if latest_hb is None or hb_time > latest_hb: - latest_hb = hb_time - if latest_hb is not None and (now_wallclock - latest_hb) < gate_seconds: - age = now_wallclock - latest_hb - return True, f"container heartbeat {age:.0f}s ago" - except Exception as e: - logger.warning( - "BRC progress-gate heartbeat check failed", - pipeline_id=pipeline_id, - error=str(e), - ) - - return False, None - - -def _latest_active_role_heartbeat(active_role_names: list[str]) -> datetime | None: - """Return the most recent heartbeat timestamp across ``active_role_names``. - - Mirrors the heartbeat half of :func:`_check_brc_progress_gate` so the - consensus-timeout ``OVERSEER_ALERT`` carries a meaningful - ``latest_heartbeat_at`` value. Filters by active role to avoid - pollution from stale entries in the singleton ``HealthMonitor``. - - Returns ``None`` when no live heartbeat is available (no roles - given, no health monitor, or any failure in the lookup — failures - are logged at WARNING and treated as "no signal", consistent with - the gate). - """ - if not active_role_names: - return None - try: - from health_monitor import get_health_monitor - - hm = get_health_monitor() - if hm is None: - return None - active_set = set(active_role_names) - latest_hb: float | None = None - with hm._lock: # noqa: SLF001 — read-only snapshot - hb_snapshot = dict(hm._last_heartbeat) # noqa: SLF001 - for agent_id, hb_time in hb_snapshot.items(): - if agent_id not in active_set: - continue - if latest_hb is None or hb_time > latest_hb: - latest_hb = hb_time - if latest_hb is None: - return None - return datetime.fromtimestamp(latest_hb, tz=UTC) - except Exception as e: - logger.warning( - "Consensus-timeout alert heartbeat lookup failed", - error=str(e), - exc_info=True, - ) - return None - - -def _unresolved_contract_hitl_ids( - pipeline_id: str, - pipeline: Pipeline, - phase_str: str, -) -> list[str]: - """Return ids of unresolved contract HITL (``cq-N``) decisions gating ``phase_str``. - - Feeds the consensus-timeout HITL gate (#3426): while an agent-registered - contract question (``register_open_question`` / impasse escalation) for - the running phase awaits an operator answer, the slice is *operator-gated* - — a reviewer correctly withholding its ACK pending the ruling is not a - convergence failure, so the consensus-timeout clock must not expire the - phase. Scoped to decisions whose ``phase`` matches the running phase; - phase-less decisions are skipped, mirroring - ``_collect_unresolved_phase_decisions`` (we cannot prove they gate this - phase, and an eternally-unanswered legacy entry must not suspend the - timeout forever). - - Contract decisions have no slice tag, so during a sliced implement phase - any unresolved implement-tagged question suspends every slice's timeout. - That errs toward parking rather than failing — acceptable, since the - overseer's "wedged on HITL" alert stays sticky and the operator's answer - releases the gate. - - The gate keys on the *existence* of an operator-facing HITL decision - tagged to this phase, not on causal proof that decision is what a - reviewer is withholding an ACK for — decisions carry no link to the - ACK they block. An unrelated implement-tagged HITL therefore suspends - the timeout too; that is the conservative "park rather than fail" - direction, self-corrected by the clock reset on release (a genuine - stall times out on the fresh window) and by the overseer's other - health checks. - - Fail-open: any failure (missing worktree, unloadable contract) returns - ``[]`` so a broken scan degrades to the pre-#3426 timeout behaviour - rather than suspending the clock indefinitely. Matching the sibling - ``_collect_unresolved_phase_decisions``, the except set is narrowed to - the IO/validation failures a real scan can hit and logged at - ``warning`` (so a broken scan is observable, not a silent no-op), - while programming errors (``AttributeError``/``TypeError``/``NameError``) - are left to propagate so they surface during development. - """ - try: - import contract_store - from egg_contracts import load_contract - from egg_contracts.loader import ( - ContractNotFoundError, - ContractValidationError, - ) - except ImportError: - logger.warning( - "Consensus-timeout HITL gate: egg_contracts unavailable, cannot scan", - pipeline_id=pipeline_id, - exc_info=True, - ) - return [] - - try: - worktree = contract_store.resolve_pipeline_worktree(pipeline_id) - if worktree is None: - return [] - identifier = _pipeline_identifier(getattr(pipeline, "issue_number", None), pipeline_id) - contract = load_contract(identifier, worktree) - except OSError, ValueError, ContractNotFoundError, ContractValidationError: - # OSError: filesystem failures resolving the worktree / reading the - # contract. ValueError: identifier / path-resolution failures from - # ``_pipeline_identifier`` (``load_contract`` wraps pydantic-V2 - # validation errors as ContractValidationError, so a raw ValueError - # here does not come from schema validation). Contract*: missing or - # corrupt contract JSON. All fail open to ``[]``. - logger.warning( - "Consensus-timeout HITL gate contract scan failed", - pipeline_id=pipeline_id, - exc_info=True, - ) - return [] - - ids: list[str] = [] - for d in contract.decisions or []: - if d.resolved: - continue - if getattr(d.type, "value", d.type) != "hitl": - continue - if getattr(d.phase, "value", d.phase) != phase_str: - continue - ids.append(d.id) - return ids - - -def _publish_consensus_timeout_alert( - pipeline: Pipeline, - pipeline_id: str, - consensus_timeout: float, - blocking_agents: list[str], - *, - priority: str, - latest_proposal_at: datetime | None, - latest_heartbeat_at: datetime | None, - slice_id: str | None, -) -> None: - """Publish a consensus-timeout ``OVERSEER_ALERT`` (#2264). - - Replaces the old auto-``choice`` HITL decision the orchestrator - used to open at ``consensus_timeout_minutes``. The SDLC skill's - existing ``OVERSEER_ALERT`` flow surfaces this as a non-blocking - notification (Check agent logs / Acknowledge / Cancel pipeline) - rather than gating the pipeline on a binary choice. - - Best-effort: if the message store import or write fails, log at - WARNING and return — the orchestrator log is the always-on - fallback (mirrors the slice-cascade alert path). - """ - timeout_minutes = int(consensus_timeout / 60) - phase_value = ( - pipeline.current_phase.value - if hasattr(pipeline.current_phase, "value") - else str(pipeline.current_phase) - ) - # Subject role slot follows the SDLC skill convention - # ``<anomaly_type>: <agent_role> [<priority>]`` (skills/sdlc/SKILL.md - # §"Overseer Alert Detection") so "Check agent logs" extracts a role - # the host can pass to ``get_container_logs``. Fall back to the phase - # only when no blocking role is reported — the phase still appears in - # ``metadata.phase`` regardless. - subject_role = blocking_agents[0] if blocking_agents else phase_value - subject = f"consensus-timeout: {subject_role} [{priority}]" - blockers_render = ", ".join(blocking_agents) if blocking_agents else "(none reported)" - proposal_render = ( - latest_proposal_at.isoformat() if latest_proposal_at is not None else "no proposals seen" - ) - heartbeat_render = ( - latest_heartbeat_at.isoformat() - if latest_heartbeat_at is not None - else "no recent heartbeat" - ) - body = ( - f"BRC consensus has not converged after {timeout_minutes} minutes " - f"in phase '{phase_value}'.\n" - f"Blocking agents: {blockers_render}\n" - f"Latest proposal: {proposal_render}\n" - f"Latest heartbeat (active roles): {heartbeat_render}\n\n" - "The pipeline continues to poll for convergence (up to ~60 min " - "before still-running containers are force-killed). If you want " - "to intervene, use `cancel_task` to stop the pipeline or " - "`restart_phase` to retry." - ) - metadata: dict[str, Any] = { - "anomaly_type": "consensus-timeout", - "phase": phase_value, - "blocking_agents": list(blocking_agents), - "latest_proposal_at": ( - latest_proposal_at.isoformat() if latest_proposal_at is not None else None - ), - "latest_heartbeat_at": ( - latest_heartbeat_at.isoformat() if latest_heartbeat_at is not None else None - ), - "consensus_timeout_minutes": timeout_minutes, - "priority": priority, - } - if slice_id is not None: - metadata["slice_id"] = slice_id - - try: - try: - from message_store import Message, MessageType - except ImportError: - from ..message_store import ( # type: ignore[no-redef] - Message, - MessageType, - ) - store_fn = _get_message_store() - if store_fn is None: - logger.warning( - "Consensus-timeout alert: message store unavailable", - pipeline_id=pipeline_id, - ) - return - msg_store = store_fn() - msg_store.add_message( - Message( - pipeline_id=pipeline_id, - from_role="orchestrator", - to_role="all", - message_type=MessageType.OVERSEER_ALERT, - subject=subject, - body=body, - metadata=metadata, - phase=phase_value, - ) - ) - except Exception as e: - logger.warning( - "Failed to publish consensus-timeout OVERSEER_ALERT", - pipeline_id=pipeline_id, - error=str(e), - exc_info=True, - ) - - -def _emit_producer_death_alert( - *, - pipeline_id: str, - role: str, - phase: str, - slice_id: str | None, - exit_code: int, -) -> None: - """Publish a high-priority ``OVERSEER_ALERT`` for permanent producer death (#2806). - - Fires from ``_run_concurrent_phase`` when a producer's - consensus-wrapper container exits with a non-clean code after - exhausting its retry budget. The pipeline (or slice) is about to - transition to FAILED — the alert is what makes the operator notice - rather than waiting for the consensus-timeout / overseer - ``stuck-phase-transition`` alert to fire 30+ minutes later. - - Best-effort: failures to write to the message store degrade to a - WARNING log, mirroring ``_publish_consensus_timeout_alert``. - """ - phase_value = phase if isinstance(phase, str) else getattr(phase, "value", str(phase)) - # ``is not None`` (not truthy) so subject and metadata agree on edge - # values like ``slice_id == ""``: metadata at 15349 also uses ``is - # not None`` (#2811 round 3 item 1). In practice ``slice_id`` is - # validated to ``slice-<N>`` upstream, so the asymmetry can't fire - # today — keeping the two checks aligned avoids a future footgun. - subject_slice = f" slice={slice_id}" if slice_id is not None else "" - subject = f"producer-permanent-death: {role} exit={exit_code}{subject_slice} [high]" - slice_render = f" (slice {slice_id})" if slice_id is not None else "" - body = ( - f"Producer '{role}'{slice_render} died permanently in phase " - f"'{phase_value}': container exited with code {exit_code} after the " - f"consensus-wrapper exhausted its retry budget.\n\n" - "The slice/pipeline state machine cannot replace a permanently " - "dead producer, so the pipeline is being transitioned to FAILED " - "(Option A, issue #2806). The agent's committed work — if any — " - "is still on the per-role branch; use `restart_phase` to resume " - "from the prior known-good state, or `cancel_task` to abort." - ) - metadata: dict[str, Any] = { - "anomaly_type": "producer-permanent-death", - "phase": phase_value, - "role": role, - "exit_code": exit_code, - "priority": "high", - } - if slice_id is not None: - metadata["slice_id"] = slice_id - - try: - try: - from message_store import Message, MessageType - except ImportError: - from ..message_store import ( # type: ignore[no-redef] - Message, - MessageType, - ) - store_fn = _get_message_store() - if store_fn is None: - logger.warning( - "Producer-death alert: message store unavailable", - pipeline_id=pipeline_id, - role=role, - ) - return - msg_store = store_fn() - msg_store.add_message( - Message( - pipeline_id=pipeline_id, - from_role="orchestrator", - to_role="all", - message_type=MessageType.OVERSEER_ALERT, - subject=subject, - body=body, - metadata=metadata, - phase=phase_value, - ) - ) - except Exception as e: - logger.warning( - "Failed to publish producer-permanent-death OVERSEER_ALERT", - pipeline_id=pipeline_id, - role=role, - error=str(e), - exc_info=True, - ) - - -# Pipeline-branch divergence alert (#2224 PR 3; #2270 §2 calibration). -# -# Watches ``origin/<pipeline_branch>`` for the contamination shape from -# #2222: the branch has absorbed already-merged main commits (a bad -# rebase / merge re-introduces commits that already live in -# ``origin/<base>``). The original detector keyed on a ``(#NNNN)`` -# subject regex, which both *false-positives* (an agent legitimately -# references a PR number in a commit subject) and *false-negatives* (a -# reabsorbed commit whose subject was rewritten). The #2270 calibration -# replaces that brittle heuristic with a git-history signal: an -# ahead-commit is contamination when its **patch-id matches a commit -# already in ``origin/<base>``** (it is a reabsorbed merged-main commit), -# or — at branch granularity — the branch is neither an ancestor of base -# nor patch-id-equivalent to it. The scan window is capped -# (``_BRANCH_DIVERGENCE_SCAN_CAP``) so a long-lived branch / deep base -# history cannot make the tick unbounded. -# -# Detection latency: the polling thread checks every 30 s, but the -# orchestrator's local ``origin/<pipeline_branch>`` only refreshes -# when it fetches — which happens at pipeline start, phase -# boundaries, and a few resume / signal paths (the polling thread -# itself does not fetch). Contamination introduced mid-phase is -# therefore detected at the next phase boundary's fetch, not within -# 30 s. This is **phase-boundary granularity, not real time** — -# strictly better than detecting at PR open, but defense-in-depth -# only; PR 1 (#2282) remains the primary gate. -BRANCH_DIVERGENCE_THRESHOLD = 20 -# Cap on how many commits we patch-id on each side of the comparison. The -# contamination we care about is recent (a bad rebase during this pipeline), -# so bounding the window keeps the per-tick git work flat regardless of how -# far the branch / base have grown. -_BRANCH_DIVERGENCE_SCAN_CAP = 200 - - -def detect_branch_divergence(snapshot: Any) -> Any | None: - """Calibration detector for the ``branch_divergence`` corpus rows (#2222/#2224). - - Keys on the git-history signal in ``snapshot.git_state`` rather than the - brittle PR-subject regex: the branch is genuinely diverged only when it is - **neither** an ancestor of base **nor** patch-id-equivalent to the merged - commit. A branch that is an ancestor of base, or whose patch-id matches the - merged commit, is NOT diverged — even if its PR-style subject would have - tripped the old regex. Deterministic and cheap → ``requires_adjudication= - False``. - """ - from health_checks.types import Finding, FindingClass, Severity - - git_state = getattr(snapshot, "git_state", {}) or {} - if not isinstance(git_state, dict): - return None - - is_ancestor = bool(git_state.get("is_ancestor_of_base")) - patch_id_matches = bool(git_state.get("patch_id_matches")) - # An ancestor-of-base branch (or a patch-id match against the merged commit) - # is fully accounted for in main — not divergence. - if is_ancestor or patch_id_matches: - return None - - return Finding( - finding_class=FindingClass.BRANCH_DIVERGENCE, - severity=Severity.MEDIUM, - evidence={ - "branch": git_state.get("branch"), - "is_ancestor_of_base": is_ancestor, - "patch_id_matches": patch_id_matches, - "pr_subject_divergence": bool(git_state.get("pr_subject_divergence")), - }, - recommended_action=( - "Pipeline branch is neither an ancestor of base nor patch-id-" - "equivalent to the merged commit — it has genuinely diverged " - "(see #2222 recovery: rebase --onto the correct base)." - ), - requires_adjudication=False, - detector_key="branch_divergence", - ) - - -def _check_branch_divergence_for_alert( - pipeline_id: str, - worktree_repo_path: Path, - pipeline_branch: str, - base_branch: str, - threshold: int = BRANCH_DIVERGENCE_THRESHOLD, - scan_cap: int = _BRANCH_DIVERGENCE_SCAN_CAP, -) -> tuple[int, list[tuple[str, str]]]: - """Return ``(ahead_count, offenders)``. - - ``offenders`` is the list of ahead-commits that are **reabsorbed merged-main - commits** — an ahead-commit whose patch-id matches a commit already present - in ``origin/<base>`` (within the capped scan window) — when the pipeline - branch is more than ``threshold`` commits ahead of base. This replaces the - old ``(#NNNN)`` subject regex with a patch-id signal that neither - false-positives on legitimate PR references nor false-negatives on rewritten - subjects. Returns ``(ahead, [])`` when the branch is not far enough ahead, - nothing reabsorbed matches, or any git invocation fails (best-effort — - observability must never block the pipeline). - """ - if not pipeline_branch or not base_branch or pipeline_branch == base_branch: - return 0, [] - - git_base = [ - "git", - "-c", - "core.hooksPath=/dev/null", - "-c", - f"safe.directory={worktree_repo_path}", - "-C", - str(worktree_repo_path), - ] - - def _run(args: list[str]) -> subprocess.CompletedProcess[str] | None: - try: - return subprocess.run( - [*git_base, *args], - capture_output=True, - text=True, - timeout=15, - check=False, - ) - except (subprocess.TimeoutExpired, OSError) as exc: - logger.debug( - "branch-divergence: git command failed", - pipeline_id=pipeline_id, - git_args=args, - error=str(exc), - ) - return None - - def _patch_id_to_sha(rev_range: str) -> dict[str, str]: - """Map ``patch_id -> sha`` for up to ``scan_cap`` commits in ``rev_range``. - - Runs ``git log -p | git patch-id --stable``. Best-effort: any failure - yields an empty map (the caller degrades to "no offenders"). - """ - log_p = _run( - [ - "log", - "-p", - "--no-merges", - f"--max-count={scan_cap}", - rev_range, - ] - ) - if log_p is None or log_p.returncode != 0 or not log_p.stdout: - return {} - try: - pid = subprocess.run( - [*git_base, "patch-id", "--stable"], - input=log_p.stdout, - capture_output=True, - text=True, - timeout=15, - check=False, - ) - except subprocess.TimeoutExpired, OSError: - return {} - if pid.returncode != 0: - return {} - mapping: dict[str, str] = {} - for line in (pid.stdout or "").splitlines(): - parts = line.split() - if len(parts) >= 2: - mapping[parts[0]] = parts[1] - return mapping - - count = _run( - [ - "rev-list", - "--count", - f"origin/{base_branch}..origin/{pipeline_branch}", - ] - ) - if count is None or count.returncode != 0: - return 0, [] - try: - ahead = int((count.stdout or "0").strip() or "0") - except ValueError: - return 0, [] - if ahead <= threshold: - return ahead, [] - - # Patch-ids present in recent base history — the set an ahead-commit must - # collide with to count as a reabsorbed merged-main commit. - base_patch_ids = set(_patch_id_to_sha(f"origin/{base_branch}").keys()) - if not base_patch_ids: - return ahead, [] - ahead_sha_by_patch_id = _patch_id_to_sha(f"origin/{base_branch}..origin/{pipeline_branch}") - contaminated_shas = {sha for pid, sha in ahead_sha_by_patch_id.items() if pid in base_patch_ids} - if not contaminated_shas: - return ahead, [] - - # Re-read subjects (capped, ordered newest-first) for the alert body. - log = _run( - [ - "log", - "--no-merges", - "--pretty=format:%H%x09%s", - f"--max-count={scan_cap}", - f"origin/{base_branch}..origin/{pipeline_branch}", - ] - ) - if log is None or log.returncode != 0: - return ahead, [] - - offenders: list[tuple[str, str]] = [] - for line in (log.stdout or "").splitlines(): - line = line.strip() - if not line: - continue - sha, _, subject = line.partition("\t") - if not sha: - continue - if sha in contaminated_shas: - offenders.append((sha, subject or "(no subject)")) - return ahead, offenders - - -def _publish_branch_divergence_alert( - pipeline: Pipeline, - pipeline_id: str, - *, - pipeline_branch: str, - base_branch: str, - ahead_count: int, - offenders: list[tuple[str, str]], -) -> None: - """Publish an ``OVERSEER_ALERT`` for branch-divergence contamination. - - Best-effort: import or write failures are logged at WARNING and - swallowed — the orchestrator log is the always-on fallback. - """ - phase_value = ( - pipeline.current_phase.value - if hasattr(pipeline.current_phase, "value") - else str(pipeline.current_phase) - ) - subject = f"branch-divergence: {pipeline_branch} contains merged-main commits" - offender_render = "\n".join(f" {sha[:12]} {subj}" for sha, subj in offenders[:10]) - if len(offenders) > 10: - offender_render += f"\n ... and {len(offenders) - 10} more" - body = ( - f"Pipeline branch ``origin/{pipeline_branch}`` is {ahead_count} commits " - f"ahead of ``origin/{base_branch}`` and contains {len(offenders)} " - f"commit(s) whose **patch-id matches a commit already merged into " - f"base** — i.e. reabsorbed merged-main commits. This is the " - f"contamination shape investigated in #2222 (Phase 4 / #2224 " - f"detector; #2270 §2 patch-id calibration).\n\n" - f"Offending commits:\n{offender_render}\n\n" - f"If this is real contamination, the resulting PR will show a " - f"borked diff against current main — see #2222 recovery procedure " - f"(rebase ``--onto`` the right base)." - ) - metadata: dict[str, Any] = { - "anomaly_type": "branch-divergence", - "phase": phase_value, - "pipeline_branch": pipeline_branch, - "base_branch": base_branch, - "ahead_count": ahead_count, - "offending_shas": [sha for sha, _ in offenders], - } - - try: - try: - from message_store import Message, MessageType - except ImportError: - from ..message_store import ( # type: ignore[no-redef] - Message, - MessageType, - ) - store_fn = _get_message_store() - if store_fn is None: - logger.warning( - "Branch-divergence alert: message store unavailable", - pipeline_id=pipeline_id, - ) - return - msg_store = store_fn() - msg_store.add_message( - Message( - pipeline_id=pipeline_id, - from_role="orchestrator", - to_role="all", - message_type=MessageType.OVERSEER_ALERT, - subject=subject, - body=body, - metadata=metadata, - phase=phase_value, - ) - ) - except Exception as e: - logger.warning( - "Failed to publish branch-divergence OVERSEER_ALERT", - pipeline_id=pipeline_id, - error=str(e), - exc_info=True, - ) - - -def _branch_divergence_tick( - pipeline_id: str, - worktree_repo_path: Path, - store: StateStore, - alerted_shas: set[str], -) -> None: - """One iteration of the branch-divergence detector. - - Extracted from the ``_health_monitor_poll`` closure so the - dedupe + reset behavior is unit-testable. Mutates ``alerted_shas`` - in place: adds newly-fired SHAs, and clears the set when the - contamination window goes empty so re-introduction (same SHA, - e.g. agent re-runs a bad rebase) re-fires per the issue's - "rather over-alert than miss" stance. - - All errors are logged-and-swallowed — observability must never - block the pipeline. - """ - try: - pipeline = store.load_pipeline(pipeline_id) - branch = pipeline.branch - base = pipeline.base_branch - if not branch or not base: - return - ahead, offenders = _check_branch_divergence_for_alert( - pipeline_id=pipeline_id, - worktree_repo_path=worktree_repo_path, - pipeline_branch=branch, - base_branch=base, - ) - if not offenders and alerted_shas: - # Note: transient git errors in ``_check_branch_divergence_for_alert`` - # also surface as ``offenders == []`` and therefore flush the dedupe - # set; this is intentional per #2224's "rather over-alert than miss" - # posture — a flaky git tick will re-fire on the next clean tick. - alerted_shas.clear() - new_offenders = [(sha, subj) for sha, subj in offenders if sha not in alerted_shas] - if new_offenders: - _publish_branch_divergence_alert( - pipeline, - pipeline_id, - pipeline_branch=branch, - base_branch=base, - ahead_count=ahead, - offenders=new_offenders, - ) - alerted_shas.update(sha for sha, _ in new_offenders) - except Exception as div_err: - logger.debug( - "Branch-divergence check failed", - pipeline_id=pipeline_id, - error=str(div_err), - ) - - -def _handle_brc_consensus_timeout( - pipeline: Pipeline, - pipeline_id: str, - consensus_timeout: float, - blocking_agents: list[str], - store: StateStore, # noqa: ARG001 — kept for call-site compatibility (#2264) - slice_id: str | None = None, - active_role_names: list[str] | None = None, -) -> None: - # Extracted from _run_concurrent_phase so k3s-style top-level-module - # layouts (and tests) can exercise this path in isolation — issue #1783. - # ``slice_id`` is propagated so per-slice trackers (#2137) are looked - # up under the nested ``{pipeline_id}/{slice_id}`` key. - # - # Issue #2264: the auto-``choice`` HITL decision this used to open - # was the wrong protocol shape — the platform should not gate the - # pipeline on a binary choice when the operator already has the - # levers (`cancel_task`, `restart_phase`, `provide_input`). The - # two former decision paths now publish ``OVERSEER_ALERT`` messages - # so the SDLC skill's existing alert flow surfaces them as - # notifications rather than a blocking decision. - _brc_handled = False - _brc_timeout_result: dict | None = None - _brc_tracker = None - try: - try: - from peer_consensus import get_peer_consensus_tracker - except ImportError: - from ..peer_consensus import ( - get_peer_consensus_tracker, # type: ignore[no-redef] - ) - - _brc_tracker = get_peer_consensus_tracker(pipeline_id, slice_id) - if _brc_tracker is not None: - _brc_timeout_result = _brc_tracker.handle_timeout() - _brc_handled = _brc_tracker.is_timeout_handled() - logger.info( - "BRC timeout handler result", - pipeline_id=pipeline_id, - action=(_brc_timeout_result.get("action") if _brc_timeout_result else None), - brc_handled=_brc_handled, - ) - except Exception as e: - logger.warning( - "BRC timeout check failed, falling back to OVERSEER_ALERT", - pipeline_id=pipeline_id, - error=str(e), - ) - - latest_proposal_at: datetime | None = None - if _brc_tracker is not None: - try: - latest_proposal_at = _brc_tracker.get_latest_proposal_timestamp() - except Exception as e: - logger.warning( - "Consensus-timeout alert proposal lookup failed", - pipeline_id=pipeline_id, - error=str(e), - exc_info=True, - ) - latest_heartbeat_at = _latest_active_role_heartbeat(active_role_names or []) - - if ( - _brc_handled - and _brc_timeout_result is not None - and _brc_timeout_result.get("action") == "escalate" - ): - # Narrow the alert's blocking_agents to the *critical* blockers - # the tracker just escalated on. The caller-supplied - # ``blocking_agents`` is the full unconfirmed-roles set - # (advisory + critical) from ``evaluate()`` — surfacing - # advisory roles on a high-priority alert dilutes the signal. - critical_entries = _brc_timeout_result.get("critical_blockers") or [] - critical_role_names: list[str] = [] - for entry in critical_entries: - for role in (entry.get("reviewer_role"), entry.get("producer_role")): - if role and role not in critical_role_names: - critical_role_names.append(role) - escalate_blocking = critical_role_names or blocking_agents - _publish_consensus_timeout_alert( - pipeline, - pipeline_id, - consensus_timeout, - escalate_blocking, - priority="high", - latest_proposal_at=latest_proposal_at, - latest_heartbeat_at=latest_heartbeat_at, - slice_id=slice_id, - ) - elif not _brc_handled: - if _emit_event is not None: - _emit_event( - EventType.CONSENSUS_TIMEOUT, - pipeline_id, - data={ - "timeout_minutes": consensus_timeout / 60, - "blocking_agents": blocking_agents, - }, - ) - _publish_consensus_timeout_alert( - pipeline, - pipeline_id, - consensus_timeout, - blocking_agents, - priority="medium", - latest_proposal_at=latest_proposal_at, - latest_heartbeat_at=latest_heartbeat_at, - slice_id=slice_id, - ) - - -def _start_stacked_pr_reconciler( - pipeline_id: str, - contract_loader: Callable[[], Any], - gateway, - pipeline, - *, - interval_seconds: float | None = None, - worktree_repo_path: Path | None = None, - repo: str | None = None, -) -> tuple[threading.Thread, threading.Event]: - """Start the periodic stacked-PR reconciler as a daemon thread (#2137 TASK-5-3). - - Returns ``(thread, stop_event)``: caller calls ``stop_event.set()`` - when the implement phase is shutting down so the daemon exits - cleanly. The daemon loops on the configured interval and invokes - :func:`stacked_pr_reconciler.reconcile_once` with callables that - decouple it from the gateway client. - - The list-callables (``list_open_prs`` and ``list_remote_branches``) - forward to ``GatewayClient.list_open_prs`` / - ``GatewayClient.list_remote_branches``. ``list_open_prs`` routes - through the launcher-authed control-plane route - ``/api/v1/gh/list_open_prs`` (#2925); ``list_remote_branches`` routes - through the existing per-agent ``git ls-remote`` allowlist. The rebase - callable forwards to - ``GatewayClient.rebase_onto``, which performs the full local - rebase + ``--force-with-lease`` push + ``gh api PATCH base=…`` - retarget so an orphaned child PR is fully healed on origin - rather than just locally rewritten. - """ - try: - from orchestrator.env_config import get_stacked_pr_reconciler_interval_seconds - except ImportError: - from env_config import ( # type: ignore[no-redef] - get_stacked_pr_reconciler_interval_seconds, - ) - try: - from orchestrator.stacked_pr_reconciler import reconcile_once - except ImportError: - from stacked_pr_reconciler import reconcile_once # type: ignore[no-redef] - # #3393 slice-5: the cross-repo merge-sequencing gate rides the SAME - # reconciler cadence (no new scheduler subsystem). Imported here (not - # top-level) to keep this helper's import surface minimal, mirroring - # the ``reconcile_once`` import above. - try: - import orchestrator.cross_repo_merge_gate as cross_repo_merge_gate - except ImportError: - import cross_repo_merge_gate # type: ignore[no-redef] - try: - from orchestrator.env_config import get_cross_repo_merge_gate_max_attempts - except ImportError: - from env_config import ( # type: ignore[no-redef] - get_cross_repo_merge_gate_max_attempts, - ) - try: - from orchestrator.models import resolve_slice_repo - except ImportError: - from models import resolve_slice_repo # type: ignore[no-redef] - - if interval_seconds is None: - try: - interval_seconds = float(get_stacked_pr_reconciler_interval_seconds()) - except Exception: # noqa: BLE001 - interval_seconds = 30.0 - - stop_event = threading.Event() - - # ``repo_path`` must be a filesystem path the gateway's - # ``validate_repo_path`` accepts (``/home/egg/repos/``, - # ``/home/egg/.egg-worktrees/``, etc.) — NOT the git branch - # name. Use the orchestrator-side worktree path that the - # implement loop already owns. - repo_path_str = str(worktree_repo_path) if worktree_repo_path is not None else "" - pr_repo = repo or str(getattr(pipeline, "repo", "") or "") - - # #3393 slice-5: only multi-repo pipelines can have cross-repo - # dependency edges, so the merge gate is a strict no-op for N=1 — - # skip it entirely rather than burning a contract scan per tick. - _gate_enabled = len(getattr(pipeline, "repos", None) or []) > 1 - # Per-run gate bookkeeping (attempts / hold-registered / resolved), - # keyed by dependent slice id; persists across reconciler ticks. - _gate_state: dict[str, Any] = {} - try: - _gate_max_attempts = int(get_cross_repo_merge_gate_max_attempts()) - except Exception: # noqa: BLE001 - _gate_max_attempts = cross_repo_merge_gate.DEFAULT_MAX_POLL_ATTEMPTS - _gate_current_phase = getattr(pipeline, "current_phase", None) - - def _poll_cross_repo_merge_gate(contract: Any) -> None: - # Drive one cross-repo merge-sequencing pass on the reconciler - # cadence (#3393 slice-5, task-5-1 / task-5-2). Reads upstream PR - # merge-state and auto-readies a dependent draft PR on merge - # (Tier A); registers a HITL hold on the closed-unmerged / timeout - # terminals and for plan-declared beyond-merge-state edges (Tier - # B). All gateway/contract effects are funnelled through the - # injected callables so the gate logic stays pure + unit-tested. - if not _gate_enabled: - return - cross_repo_merge_gate.poll_once( - contract, - resolve_repo=lambda s: resolve_slice_repo(s, pipeline), - get_merge_state=lambda repo_slug, pr_num: gateway.get_pr_merge_state( - pipeline_id, repo_slug, pr_number=pr_num - ), - mark_ready=lambda repo_slug, pr_num: bool( - gateway.mark_pr_ready(pipeline_id, repo_slug, pr_number=pr_num) - ), - register_hold=lambda gate, reason: _register_cross_repo_hold( - pipeline_id=pipeline_id, - slice_id=gate.slice_id, - repo=gate.repo, - pr_number=gate.pr_number, - reason=reason, - worktree_repo_path=worktree_repo_path, - current_phase=_gate_current_phase, - ), - hold_resolution=lambda gate: _cross_repo_hold_resolution(contract, gate.slice_id), - state=_gate_state, - max_attempts=_gate_max_attempts, - ) - - def _list_open_prs() -> list[dict[str, Any]]: - # Lists open PRs in ``pr_repo`` so ``find_orphaned_child_prs`` - # can detect children whose base branch was deleted (parent - # merged through the GitHub UI). Routes through the launcher-authed - # control-plane endpoint ``/api/v1/gh/list_open_prs`` — the - # orchestrator is the server that manages pipelines, not an agent, - # so it does not register a synthetic agent session or impersonate - # a role (#2922 / #2925). - if not pr_repo: - return [] - try: - return list(gateway.list_open_prs(pipeline_id, pr_repo)) - except Exception as exc: # noqa: BLE001 - logger.debug( - "stacked_pr_reconciler: list_open_prs raised — treating as empty", - pipeline_id=pipeline_id, - error=str(exc), - ) - return [] - - def _list_extant_branches() -> set[str]: - # Lists remote branches via ``git ls-remote --heads origin`` - # so the reconciler can detect deleted parents. Routes through - # the existing per-agent ``git ls-remote`` allowlist. The - # synthetic session uses ``agent_role="orchestrator"`` so this - # orchestrator-driven ls-remote is attributed to the orchestrator - # in the audit log instead of a phantom coder (#2919). - if not repo_path_str: - return set() - try: - return set( - gateway.list_remote_branches( - pipeline_id, - repo_path_str, - agent_role="orchestrator", - ) - ) - except Exception as exc: # noqa: BLE001 - logger.debug( - "stacked_pr_reconciler: list_remote_branches raised — treating as empty", - pipeline_id=pipeline_id, - error=str(exc), - ) - return set() - - def _rebase_onto(orphan: Any) -> bool: - # ``orphan`` is a ``stacked_pr_reconciler.OrphanedChildPR``; - # avoid the import here so this module stays a pure consumer - # of the reconciler's typed interface (the type checker at - # the reconciler boundary already validates the shape). - try: - return bool( - gateway.rebase_onto( - pipeline_id, - repo_path_str, - branch=orphan.branch, - new_base=orphan.intended_new_base, - old_base=orphan.deleted_base, - pr_number=orphan.pr_number, - repo=pr_repo or None, - # Orchestrator-driven heal (rebase + force-push + - # pr-edit); attribute to the orchestrator, not a - # phantom coder (#2919). The force-push targets the - # slice integration branch on a synthetic session, so - # the slice-integration exemption admits it regardless - # of role. - agent_role="orchestrator", - ) - ) - except Exception: # noqa: BLE001 - logger.debug( - "stacked_pr_reconciler: rebase_onto raised — counted as failure", - pipeline_id=pipeline_id, - branch=getattr(orphan, "branch", "?"), - ) - return False - - def _loop() -> None: - # Defensive: a slow tick must not pin this thread on a stale - # sleep — Event.wait returns True the moment ``stop_event`` is - # set, so shutdown is bounded by the configured interval. - while not stop_event.wait(interval_seconds): - try: - contract = contract_loader() - if contract is None: - continue - reconcile_once( - contract, - list_open_prs=_list_open_prs, - list_extant_branches=_list_extant_branches, - rebase_onto=_rebase_onto, - ) - # #3393 slice-5: drive the cross-repo merge-sequencing - # gate on the same tick + same freshly-loaded contract. - # No-op for N=1 pipelines. Wrapped in its own try so a - # gate failure never disrupts stacked-PR reconciliation. - try: - _poll_cross_repo_merge_gate(contract) - except Exception as gate_exc: # noqa: BLE001 - logger.debug( - "cross_repo_merge_gate tick raised — continuing", - pipeline_id=pipeline_id, - error=str(gate_exc), - ) - except Exception as exc: # noqa: BLE001 - logger.debug( - "stacked_pr_reconciler tick raised — continuing", - pipeline_id=pipeline_id, - error=str(exc), - ) - - thread = threading.Thread( - target=_loop, - name=f"stacked-pr-reconciler-{pipeline_id}", - daemon=True, - ) - thread.start() - return thread, stop_event - - -def _run_implement_phase_slices( - pipeline_id: str, - pipeline: Pipeline, - spawner, - repo_volumes: dict[str, str], - gateway_mode: str, - repos: list[str], - sandbox_env: dict[str, str], - store, - certs_volume: str | None, - worktree_repo_path: Path, - run_epoch: datetime | None = None, -) -> tuple[int, str]: - """Drive the implement phase as a DAG of independent slices (#2137). - - For each wave produced by :class:`SliceScheduler`, spawns a fresh - BRC team per slice and waits for that slice's consensus before - advancing the scheduler. Each slice runs through the existing - :func:`_run_concurrent_phase` machinery with a slice-scoped tracker - namespace (``{pipeline_id}/{slice_id}``) and slice-scoped per-role - branches (``egg/issue-N/{slice_id}/{role}/work``). - - Per-slice PRs are opened via ``GatewayClient.create_slice_pr`` after - each slice reaches CONSENSUS_CONFIRMED — root slices target the - pipeline branch; child slices target their parent slice's - integration branch. The stacked-PR reconciler runs in parallel as a - daemon thread for the lifetime of this call. - - Returns ``(exit_code, logs)`` where ``exit_code == 0`` means every - slice reached CONFIRMED; non-zero means at least one slice failed. - """ - try: - from orchestrator.slice_scheduler import SliceScheduler - except ImportError: - from slice_scheduler import SliceScheduler - - try: - from egg_contracts.loader import load_contract, save_contract - except ImportError as exc: - logger.error( - "Slice loop: egg_contracts.loader unavailable — falling back", - pipeline_id=pipeline_id, - error=str(exc), - ) - return 1, "slice loop bootstrap failed" - - contract = load_contract(pipeline_id, worktree_repo_path) - slices = list(getattr(contract, "slices", []) or []) - if not slices: - logger.warning( - "Slice loop: contract has no slices, falling back to monolithic implement", - pipeline_id=pipeline_id, - ) - return 1, "no slices in contract" - - pipeline_branch = pipeline.branch or ( - f"egg/issue-{pipeline.issue_number}/work" - if pipeline.issue_number is not None - else f"egg/{pipeline_id}/work" - ) - issue_number = pipeline.issue_number - # Slice integration branches stack as siblings of the pipeline tip - # under ``egg/<id>/`` (see :func:`_ensure_pipeline_work_ref` for the - # ``/work`` namespace decision in #2399). The namespace root drops the - # trailing ``/work`` so slice paths build to ``<root>/slice-M`` rather - # than ``<root>/work/slice-M``. The qualifier suffix (``-v3``, - # ``-backend``) is preserved through ``pipeline.branch`` so two - # qualified pipelines for the same issue do not collide on - # ``egg/issue-N/slice-M`` (#2368). - issue_branch = _slice_namespace_root(pipeline_branch) - - # Wrap scheduler construction so the run loop doesn't crash if the - # contract bypassed plan-ingestion validation and reaches the - # scheduler with a multi-parent / cyclic forest. ``SliceScheduler`` - # raises ``ValueError`` with the structured forest errors; surface - # them to the operator via the existing return path so the run - # loop can route to HITL escalation rather than wedge the pipeline. - try: - scheduler = SliceScheduler( - contract, - max_parallel_slices=pipeline.config.max_parallel_slices, - ) - except ValueError as exc: - logger.error( - "Slice loop: scheduler refused to start (forest validation failed)", - pipeline_id=pipeline_id, - error=str(exc), - ) - return 1, f"slice scheduler validation failed: {exc}" - - # Defensive idempotent context-PR opener (#2777 cq-4). The - # canonical advance_phase REST path enforces hard-required, but - # the runner-driven entries (auto-advance, implement-entry, - # HITL-resume, this slice-loop entry) must also fire it to avoid - # silent strands on ``egg/<id>/work``. Soft-fail on transient - # gateway errors here — the canonical site already enforces the - # 422 contract. - try: - # Pass the main repo path (``store.repo_path``) — not - # ``worktree_repo_path`` — so all four opener call sites of - # ``_open_context_pr_at_implement_start`` read identically. - # The opener rederives its own per-pipeline worktree internally - # via ``resolve_worktree_path(pipeline_id, store.repo_path)``. - _open_context_pr_at_implement_start(pipeline_id, repo_path=Path(store.repo_path)) - except ContextPrCreationError as ctx_err: - logger.warning( - "Context PR opener: slice-loop entry safety net failed " - "(continuing — hard-require enforced at advance_phase and " - "the implement-start plan pre-flight gate) (#2777, #3100)", - pipeline_id=pipeline_id, - reason=ctx_err.reason, - error=str(ctx_err), - ) - except Exception as safety_err: # noqa: BLE001 - # Defence in depth: import / lookup failures must not strand - # the slice loop. - logger.warning( - "Context PR opener: slice-loop entry safety net outer " - "wrapper raised (continuing) (#2777)", - pipeline_id=pipeline_id, - error=str(safety_err), - ) - - def _contract_loader() -> Any: - try: - return load_contract(pipeline_id, worktree_repo_path) - except Exception: # noqa: BLE001 - # Best-effort loader for callers that just need "current - # contract or None". Catches loader validation errors, - # OSError on the contract file read, and any pydantic - # re-serialisation failure. - return None - - # Stacked-PR reconciler starts after the bootstrap pass below so - # an unhandled bootstrap exception cannot leak its daemon thread - # (the ``finally`` at the bottom of the run loop owns teardown). - aggregate_logs: list[str] = [] - overall_exit = 0 - poll_interval = 5.0 - - from egg_contracts.models import SliceStatus - - try: - from orchestrator import global_slice_admit - except ImportError: - import global_slice_admit # type: ignore[no-redef] - try: - from orchestrator.peer_consensus import remove_peer_consensus_tracker - except ImportError: - from peer_consensus import remove_peer_consensus_tracker # type: ignore[no-redef] - try: - from orchestrator.state_store import get_pipeline_state_lock - except ImportError: - from state_store import get_pipeline_state_lock # type: ignore[no-redef] - - def _commit_and_push_slice_statefiles(message: str) -> None: - """Commit + push pipeline-scoped ``.egg-state/`` writes to the work branch. - - Contract mutations — agent task-record updates via - ``mutate_contract`` and the ``slice.status`` flips below — land - on the shared pipeline worktree's disk copy only. Without a - slice-boundary commit, the work branch's contract file stays - frozen at the init-time "Initialize SDLC contract" commit for - the entire implement phase, and a mid-phase orchestrator crash - or worktree prune loses every accumulated task record (#3117). - The phase-boundary commit at the end of the run loop is too - coarse for multi-slice phases. - - Scope (per #3117): this closes durability for the post-prune - audit record, operator/PR-side review of mid-phase contract - state, and orchestrator-restart resume at slice granularity. - It is deliberately NOT the read path for live agents — agents - read the contract via ``mcp__sdlc__show_contract`` against the - orchestrator's in-memory state, never from their checkout's - ``.egg-state/contracts/`` file (#3077). - - Best-effort: slice completion must not block on statefile - durability; failures are logged and the next boundary (later - slice close or phase completion) carries the writes. The commit - runs under the per-pipeline state lock to serialise concurrent - slice-close threads against the shared worktree's git index; - the push runs outside the lock. The expected case is a linear - fast-forward (lock-serialised commits stack), and a no-op FF - of the same SHA from two threads is harmless. The residual - hazard is ``_reconcile_and_retry_push`` on a non-FF rejection - (``gateway_client.py:1361``): two threads both fetching+rebasing - in the shared worktree can interleave ``.git/index.lock``. - Within the implement phase no other writer pushes to - ``pipeline.branch`` so non-FF shouldn't fire in normal - operation; an external push (operator hand-fix, stale - concurrent orchestrator) is the only known trigger. - """ - try: - with get_pipeline_state_lock(pipeline_id): - committed = _commit_statefiles_to_worktree( - worktree_repo_path, - message, - _pipeline_identifier(issue_number, pipeline_id), - pipeline_id=pipeline_id, - ) - except Exception as commit_err: # noqa: BLE001 - # The helper raises CalledProcessError / TimeoutExpired - # from subprocess.run and OSError from glob (#2219 family). - logger.warning( - "Failed to commit slice statefiles to work branch (continuing) (#3117)", - pipeline_id=pipeline_id, - commit_message=message, - error=str(commit_err), - ) - return - if not committed or not pipeline.branch or worktree_repo_path == store.repo_path: - return - try: - spawner.gateway.push_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(worktree_repo_path), - branch=pipeline.branch, - mode=gateway_mode, # type: ignore[arg-type] - base_branch=pipeline.base_branch, - ) - except Exception as push_err: # noqa: BLE001 - # Gateway HTTP push (GatewayError / OSError). The commit is - # already on the local work branch; the next successful - # push carries it. - logger.warning( - "Failed to push slice statefiles to work branch (continuing) (#3117)", - pipeline_id=pipeline_id, - commit_message=message, - error=str(push_err), - ) - - def _persist_slice_status_complete( - slice_id: str, - *, - pr_number: int | None = None, - pr_url: str | None = None, - basis: str | None = None, - commit_to_branch: bool = True, - ) -> None: - """Mark ``slice_id`` as ``SliceStatus.COMPLETE`` on the contract. - - Durable signal so the bootstrap reconciliation pass below and - the ``restart_agent`` parent-complete fallback can skip the - slice without a GitHub round-trip (#2549, #2470). Best-effort: - on save failure the in-memory scheduler state still reflects - completion for this pass and the next ``start_pipeline`` - re-detects via the merged-detection helper. - - With *commit_to_branch* (the default), the saved contract — - along with any other uncommitted pipeline statefiles, e.g. - agent task-record mutations made during the slice — is - committed and pushed to the pipeline work branch so the durable - copy tracks the live one (#3117). The bootstrap reconciliation - passes set it to ``False`` and batch a single commit after the - loop instead of one per reconciled slice. - - Called only after a slice successfully closes (BRC consensus - reached + PR opened, or merged-skip / bootstrap-COMPLETE - reconciliation). Failed slices — ``exit_code_inner != 0`` - (#16410) or ``pr_created == False`` (#16588) — return early - without calling this helper, so their accumulated task-record - mutations remain uncommitted in the worktree until the next - successful slice's commit (the pipeline-scoped glob picks them - up) or the phase-boundary commit, whichever fires first. - - ``basis`` lets a caller declare *why* the slice is complete when - not every task is marked COMPLETE on the contract: ``"merged"`` - (integration branch ancestry-verified merged into its parent) or - ``"consensus_complete"`` (BRC consensus reached pre-restart, PR - not yet opened). The PR-open caller passes ``pr_number`` instead. - Absent any of these — and with tasks still pending — the write is - a #3214 false-complete and :func:`_validate_slice_completion_basis` - raises :class:`SliceCompletionInvariantError` rather than persist - a slice as done that never ran. - - When the caller just opened the slice's PR it passes - ``pr_number`` / ``pr_url`` so the linkage lands in the same - contract write (#3122) — the context-PR body refresh and any - later stack consumer read them from ``Slice.pr_number``. - ``None`` (the merged-skip and bootstrap callers) leaves any - previously recorded linkage untouched. - - TODO(#3122): the three ``None`` callers — bootstrap layer-A - (contract-recorded COMPLETE), bootstrap layer-B (merged on - origin), and the run-loop merged-skip — do not recover the - slice PR number from GitHub (`gh pr list --head … --state - merged`), so on a resume past those points the slice-table - entries for merged slices stay unlinked. Acceptable for v1 - because the per-slice ``— #N`` link is most useful while the - stack is live, but worth backfilling if reviewers ask for - complete cross-linkage on archived stacks. - """ - try: - with get_pipeline_state_lock(pipeline_id): - contract_local = load_contract(pipeline_id, worktree_repo_path) - for s in contract_local.slices: - if s.id == slice_id: - # #3214 — refuse to persist a contradictory COMPLETE. - # An interior forest node marked COMPLETE without a - # valid basis (tasks pending, no PR, no verified - # merge/consensus) skips a slice that never ran and - # wedges the chain a phase later. Fail loud here, at - # the source of the bad write, instead. - invalid = _validate_slice_completion_basis( - s, pr_number=pr_number, basis=basis - ) - if invalid is not None: - logger.error( - "Refusing to persist slice.status=COMPLETE — " - "invalid completion basis (#3214)", - pipeline_id=pipeline_id, - slice_id=slice_id, - reason=invalid, - ) - raise SliceCompletionInvariantError(invalid) - s.status = SliceStatus.COMPLETE - if pr_number is not None: - s.pr_number = pr_number - if pr_url is not None: - s.pr_url = pr_url - logger.info( - "Slice marked COMPLETE", - pipeline_id=pipeline_id, - slice_id=slice_id, - basis=( - basis - or ( - "pr" - if (pr_number is not None or s.pr_number is not None) - else "tasks_complete" - ) - ), - pr_number=pr_number if pr_number is not None else s.pr_number, - ) - break - save_contract(contract_local, worktree_repo_path) - except SliceCompletionInvariantError: - # Fail loud — never swallow the completion invariant into the - # best-effort save handler below (#3214). - raise - except Exception as save_err: # noqa: BLE001 - # Contract load/save under per-pipeline state lock. - # Catches loader validation errors, atomic-rename / fdopen - # I/O failures, and pydantic re-serialisation errors. - # Best-effort: the in-memory scheduler still reflects - # COMPLETE for this pass; next start_pipeline re-detects. - logger.warning( - "Failed to persist slice.status=COMPLETE", - pipeline_id=pipeline_id, - slice_id=slice_id, - error=str(save_err), - ) - return - if commit_to_branch: - _commit_and_push_slice_statefiles( - f"Persist contract after slice {slice_id} completion (#3117)" - ) - - # Bootstrap reconciliation pass (#2549). Before the run loop ticks, - # fold in two sources of "this slice is already done" state that - # the scheduler (a pure rebuild from ``contract.slices``) cannot - # see on its own: - # - # (A) Slices the contract already records as - # ``SliceStatus.COMPLETE`` — trusted directly, no I/O. - # (B) Slices whose integration branch on origin is reachable from - # their parent's tip (PR merged). On a hit, also persist (A) - # so subsequent restarts skip the GitHub round-trip. - # - # Without this pass the scheduler would re-yield merged slices as - # READY and ``create_slice_integration_branch`` would - # non-fast-forward-reject. Best-effort: failure falls through to - # the run loop. - bootstrap_complete: list[str] = [] - bootstrap_merged: list[str] = [] - - # Layer (A): cheap, no I/O. Trust contract-recorded COMPLETE status — - # but verify the recorded COMPLETE is not itself a #3214 false-complete - # (an interior forest node persisted COMPLETE with pending tasks, no - # PR, no merge). Blindly trusting a corrupt contract here is how the - # false-complete propagated into the scheduler and wedged the chain. - # On an invalid record, alert and decline to trust it — route the - # slice through Layer-B/C so it is re-evaluated and (re-)run rather - # than silently skipped. - # - # Note: a COMPLETE slice that recorded *no* durable evidence (no - # pr_number, no integration_base_sha — e.g. a legacy pre-#2871 - # contract from before integration_base_sha existed) is distrusted - # here on every restart, even when it was genuinely merged. That is - # intentional, not a bug: such a slice falls through to Layer-B, - # where origin-side merge detection re-confirms it and re-marks it - # COMPLETE. The outcome stays correct; the only cost is one extra - # GitHub round-trip per restart. A slice that forked under current - # code *usually* records integration_base_sha and is trusted here - # directly — but that write is best-effort (the get_remote_branch_sha - # call at slice spawn swallows failures and degrades to ancestor-only - # detection), so a current-code slice whose base-SHA write failed also - # falls through to Layer-B and self-corrects identically to the legacy - # case above. - # - # Known limitation (#3253): a slice that pre-fix code *already* persisted - # COMPLETE basis="merged" with a stale ``integration_base_sha`` and no - # produced commits / PR is still trusted here — Layer-A validates with no - # ``basis`` (the #3253 merged-empty guard keys on ``basis == "merged"``, - # which Layer-A never supplies), so the ``forked`` free-pass below accepts - # the stale fork base. This is deliberately *not* fixed by broadening the - # guard to the basis-less path: a legitimate ``basis="consensus_complete"`` - # slice can also have no PR and no recorded task commit (best-effort agent - # recording + ``pr_number`` None on an unparseable PR URL, #3122), so - # re-running on "no commit + no PR" alone here would re-run genuinely - # completed work. The #3253 fix prevents the corrupt write going forward; - # a pipeline already wedged by this exact bug *before* the upgrade needs a - # manual contract touch-up (clear the slice's COMPLETE status) rather than - # self-healing on restart. - layer_b_candidates = [] - for s in slices: - if s.status == SliceStatus.COMPLETE: - invalid = _validate_slice_completion_basis(s, pr_number=s.pr_number) - if invalid is not None: - logger.error( - "Contract records slice COMPLETE but the completion basis is " - "invalid — NOT trusting it; re-evaluating the slice (#3214)", - pipeline_id=pipeline_id, - slice_id=s.id, - reason=invalid, - ) - layer_b_candidates.append(s) - continue - scheduler.record_complete(s.id) - bootstrap_complete.append(s.id) - continue - layer_b_candidates.append(s) - - # Layer (B): origin-side detection for slices not yet recorded as - # COMPLETE on the contract. Each helper call uses its own synthetic - # gateway session, so we parallelise across slices to keep startup - # latency bounded as forests grow. Cap workers so a large forest - # doesn't burst against the gateway. - if pipeline.repo and layer_b_candidates: - - def _bootstrap_check_one(slice_obj: Any) -> tuple[str, bool]: - # Prefer the parent branch the slice was actually forked - # off of (recorded by ``_run_one_slice_inner``). Falls back - # to the dependency-derived parent for slices that never - # made it through ``_run_one_slice_inner`` (e.g. fresh - # contract on first run). Both should agree today, but a - # future re-plan that mutates ``dependencies`` post-creation - # would diverge — preferring the recorded value future- - # proofs the check. - if slice_obj.parent_branch_at_creation: - parent_branch_for_check = slice_obj.parent_branch_at_creation - elif slice_obj.dependencies: - parent_branch_for_check = f"{issue_branch}/{slice_obj.dependencies[0]}" - else: - parent_branch_for_check = pipeline_branch - integration_branch_for_check = f"{issue_branch}/{slice_obj.id}" - try: - merged = spawner.gateway.is_slice_branch_merged_into_parent( - pipeline_id, - str(worktree_repo_path), - integration_branch=integration_branch_for_check, - parent_branch=parent_branch_for_check, - # #2871 — pass the recorded fork base so an empty - # (un-started) slice branch whose tip is still at its - # creation base is not mistaken for merged work. - integration_base_sha=slice_obj.integration_base_sha, - # Read-only ancestry check run by the orchestrator's - # slice-loop scheduler; attribute to the orchestrator - # in the audit log, not a phantom coder (#2919). - agent_role="orchestrator", - mode=gateway_mode, # type: ignore[arg-type] - ) - except Exception as detect_err: # noqa: BLE001 - # Gateway `is_slice_branch_merged_into_parent` call. - # Catches gateway HTTP/timeout errors (GatewayError), - # low-level socket / DNS errors (OSError), and any - # rare argument-shape errors. Default to "not merged" - # so the slice can still spawn fresh. - logger.warning( - "Bootstrap merged-detection raised; treating slice as not-merged", - pipeline_id=pipeline_id, - slice_id=slice_obj.id, - error=str(detect_err), - ) - return slice_obj.id, False - # #3253 — guard against a false-positive merged result. A slice - # whose producers never committed (no produced task commit) and - # that has no slice PR has an empty integration branch: its tip - # is still the fork base, so it is trivially an ancestor of an - # advanced parent and the origin ancestry check reports it - # merged. Marking it COMPLETE basis=merged silently drops the - # slice and lets the pipeline complete with its work missing — - # the restart-to-retry failure mode (#3138 producer exhaustion → - # operator restart → false-complete). Override to not-merged so - # the slice falls through to Layer-C and re-runs. A genuine merge - # has produced commits or a recorded PR, so this never overrides - # a real merge. - if ( - merged - and not _slice_produced_commits(slice_obj) - and getattr(slice_obj, "pr_number", None) is None - ): - logger.warning( - "Bootstrap merged-detection overridden: origin ancestry " - "reports merged but the slice has no produced task commit " - "and no PR — empty/un-started branch, re-running rather than " - "false-completing as merged (#3253)", - pipeline_id=pipeline_id, - slice_id=slice_obj.id, - ) - return slice_obj.id, False - return slice_obj.id, bool(merged) - - max_workers = min(len(layer_b_candidates), 8) - with concurrent.futures.ThreadPoolExecutor( - max_workers=max_workers, - thread_name_prefix=f"slice-bootstrap-{pipeline_id}", - ) as bootstrap_pool: - results = list(bootstrap_pool.map(_bootstrap_check_one, layer_b_candidates)) - - for slice_id, already_merged in results: - if already_merged: - scheduler.record_complete(slice_id) - _persist_slice_status_complete(slice_id, basis="merged", commit_to_branch=False) - bootstrap_merged.append(slice_id) - - # Layer (C): non-COMPLETE slice classification (slice-4 TASK-4-4). - # After layers A (contract-recorded COMPLETE) and B (merged on - # origin), classify the remaining slices per the 5-way matrix - # so crash recovery does not respawn agents for a slice that is - # already running, silently advance a slice whose HITL is still - # pending, or treat a corrupt status enum as a benign default: - # - # (1) IN_PROGRESS, no commits on integration branch → no Layer-C - # action; the scheduler will re-yield the slice as READY and - # the run loop spawns fresh agents. - # (2) IN_PROGRESS, commits on integration branch, consensus - # NOT reached → call ``scheduler.mark_spawned`` so the run - # loop does NOT respawn. Per-slice tracker reconstruction - # is handled at orchestrator boot by - # startup_reconciliation.py (slice-4 TASK-4-5); the - # producer pods (if alive) or the lazy spawn-on-need path - # carry the slice forward. - # (3) IN_PROGRESS, commits on integration branch, consensus - # REACHED, slice PR NOT opened → mark COMPLETE so the - # slice-PR opener path (with TASK-3-2 idempotency - # pre-flight) fires on the next loop iteration; do not - # respawn agents. - # (4) BLOCKED (HITL pending) → preserve the BLOCKED status. - # Verify the HITL decision is still on the contract; if - # not, surface an OVERSEER_ALERT so a human investigates. - # (5) Unknown / corrupt state (impossible status enum value) - # → surface an OVERSEER_ALERT instead of silently - # re-yielding as READY. - bootstrap_resumed: list[str] = [] - bootstrap_consensus_complete: list[str] = [] - bootstrap_blocked: list[str] = [] - bootstrap_corrupt: list[str] = [] - bootstrap_reclassified_fresh: list[str] = [] # resume-but-dead → fresh (#2914) - layer_b_marked_complete = set(bootstrap_merged) - for s in layer_b_candidates: - if s.id in layer_b_marked_complete: - continue - classification = _classify_non_complete_slice( - pipeline_id=pipeline_id, - slice_obj=s, - issue_branch=issue_branch, - pipeline_repo=pipeline.repo, - worktree_repo_path=worktree_repo_path, - gateway=spawner.gateway, - gateway_mode=gateway_mode, - consensus_tracker_lookup=_lookup_peer_consensus_tracker_or_none, - ) - if classification == "consensus_complete": - # Case 3 — louder than fresh-spawn but quieter than - # case-4/5 HITL. A warning here makes the non-trivial - # recovery (consensus reached pre-crash, PR not opened) - # auditable in operator logs without paging anyone - # (reviewer_code v1 non-blocking). - logger.warning( - "Layer-C case 3 — slice consensus reached pre-restart but " - "slice PR was never opened; marking COMPLETE so the next " - "loop iteration runs the slice-PR opener (slice-4 TASK-4-4)", - pipeline_id=pipeline_id, - slice_id=s.id, - ) - scheduler.record_complete(s.id) - _persist_slice_status_complete(s.id, basis="consensus_complete", commit_to_branch=False) - bootstrap_consensus_complete.append(s.id) - continue - if classification == "resume": - # Verify agents are actually live before marking as spawned (#2914). - # On restart_phase, agents were torn down but contract still shows - # IN_PROGRESS with commits — we must not mark_spawned when cohort - # is absent, or the pipeline wedges with no agents running. - if _slice_agents_alive(spawner, pipeline_id, s.id): - scheduler.mark_spawned(s.id) - bootstrap_resumed.append(s.id) - else: - logger.warning( - "Layer-C resume classification but no live agents; " - "treating as fresh to force re-spawn (#2914)", - pipeline_id=pipeline_id, - slice_id=s.id, - ) - bootstrap_reclassified_fresh.append(s.id) - continue - if classification == "blocked": - bootstrap_blocked.append(s.id) - continue - if classification == "corrupt": - bootstrap_corrupt.append(s.id) - continue - # "fresh" → no Layer-C action, scheduler re-yields READY. - - # The bootstrap passes above persist with ``commit_to_branch=False`` - # — one batched commit+push here covers every reconciled slice - # (Layer B merged-detection + Layer-C case 3) instead of a commit - # per slice (#3117). - if bootstrap_merged or bootstrap_consensus_complete: - _commit_and_push_slice_statefiles( - "Persist slice completion statuses after bootstrap reconciliation (#3117)" - ) - - if bootstrap_complete or bootstrap_merged: - logger.info( - "Slice bootstrap reconciliation marked slices complete", - pipeline_id=pipeline_id, - already_complete_on_contract=bootstrap_complete, - detected_merged_on_origin=bootstrap_merged, - ) - if ( - bootstrap_resumed - or bootstrap_consensus_complete - or bootstrap_blocked - or bootstrap_corrupt - or bootstrap_reclassified_fresh - ): - # NOTE: include ``bootstrap_blocked`` in the gate (reviewer_code - # v3 NACK fix) — a bootstrap pass whose only Layer-C activity is - # BLOCKED slices was previously suppressing the audit-trail line - # entirely. Case-4 escalation still fires, but operators need - # the structured "we saw a blocked slice" log to spot - # pending-HITL backlogs without grepping for the side-effect. - # - # Also include ``bootstrap_reclassified_fresh`` (#2914) — resume- - # classified slices that were re-verified against k8s and found - # to have no live agents. Surfacing the reclassification here - # gives operators a structured audit trail for the - # ``restart_phase``-recovery path. - logger.info( - "Slice bootstrap reconciliation classified non-COMPLETE slices (slice-4 TASK-4-4)", - pipeline_id=pipeline_id, - resumed=bootstrap_resumed, - consensus_complete_unrecorded=bootstrap_consensus_complete, - blocked=bootstrap_blocked, - corrupt=bootstrap_corrupt, - reclassified_fresh=bootstrap_reclassified_fresh, - ) - # Case 5 — escalate via HITL so the pipeline pauses until the - # operator picks an option (reviewer_contract / reviewer_code v1 - # blocker). OVERSEER_ALERT alone is too weak — it surfaces but - # does not gate progress. The Decision lands on the contract via - # ``_escalate_corrupt_slice_to_hitl`` so ``/sdlc`` reads it on - # the next poll. - _current_phase = getattr(pipeline, "current_phase", None) - for _corrupt_slice_id in bootstrap_corrupt: - try: - _escalate_corrupt_slice_to_hitl( - pipeline_id=pipeline_id, - slice_id=_corrupt_slice_id, - worktree_repo_path=worktree_repo_path, - current_phase=_current_phase, - ) - except Exception as escalate_err: # noqa: BLE001 - logger.warning( - "Failed to escalate corrupt-state slice to HITL during " - "bootstrap (slice-4 TASK-4-4 case 5)", - pipeline_id=pipeline_id, - slice_id=_corrupt_slice_id, - error=str(escalate_err), - ) - # Case 4 — symmetric HITL escalation for BLOCKED-without-HITL. - for _blocked_slice_id, _escalate_reason in [ - (sid, "no pending HITL decision found on contract") - for sid in bootstrap_blocked - if not _slice_has_pending_decision(sid, getattr(contract, "decisions", None) or []) - ]: - try: - _escalate_blocked_slice_to_hitl( - pipeline_id=pipeline_id, - slice_id=_blocked_slice_id, - reason=_escalate_reason, - worktree_repo_path=worktree_repo_path, - current_phase=_current_phase, - ) - except Exception as escalate_err: # noqa: BLE001 - logger.warning( - "Failed to escalate blocked-without-HITL slice to HITL " - "during bootstrap (slice-4 TASK-4-4 case 4)", - pipeline_id=pipeline_id, - slice_id=_blocked_slice_id, - error=str(escalate_err), - ) - - reconciler_thread, reconciler_stop = _start_stacked_pr_reconciler( - pipeline_id, - _contract_loader, - spawner.gateway, - pipeline, - worktree_repo_path=worktree_repo_path, - repo=getattr(pipeline, "repo", None), - ) - - try: - while not scheduler.all_done(): - # 1. Snapshot ready slices for this tick. - ready_batch = list(scheduler.iter_ready()) - if not ready_batch: - # 2. Drain cascades whose grace window expired so the - # descendants are visibly BLOCKED in the runtime view - # and we don't busy-spin. - events = scheduler.poll_cascades() - for event in events: - logger.warning( - "Slice cascade fired", - pipeline_id=pipeline_id, - failed_slice=event.failed_slice_id, - blocked=event.blocked_subtree, - ) - try: - from orchestrator.gateway_client import ( - get_gateway_client as _get_gateway_client, - ) - - _ = _get_gateway_client # noqa: F841 — kept for symmetry - except ImportError: - # Symmetry-only import; the module not being - # available means the cascade alert path can't - # call the gateway, but the warning above is - # the always-on fallback. - pass - if scheduler.all_done(): - break - time.sleep(poll_interval) - continue - - # Run every ready slice in this wave in parallel - # (#2137 TASK-4-4 + decision-5: unbounded). The - # ``max_parallel_slices`` cap from ``iter_ready`` already - # bounds ``ready_batch`` so the executor's worker pool - # mirrors that cap. Each slice runs through the existing - # ``_run_concurrent_phase`` machinery in its own thread. - # Per-slice failure / completion events are recorded back - # on the scheduler from inside ``_run_one_slice`` so the - # cascade machinery sees the same wall-clock as the run - # loop. - - def _run_one_slice(slice_id: str, parent_slice_id: str | None) -> tuple[int, str]: - # Release the global-admission slot when the slice - # exits, regardless of how (consensus, failure, raised - # exception). Idempotent — safe even if a future - # codepath calls release() somewhere else (#2241 gap 1). - try: - return _run_one_slice_inner(slice_id, parent_slice_id) - finally: - global_slice_admit.release(pipeline_id, slice_id) - - def _run_one_slice_inner( - slice_id: str, - parent_slice_id: str | None, # noqa: ARG001 — kept for caller compat; resolver reads contract - ) -> tuple[int, str]: - # Resolve parent branch for stacking via - # :func:`_resolve_slice_base_branch` (#2777, cq-2 / cq-4 / - # cq-9 / cq-10). The helper handles both: - # - # * eager-persisted ``parent_branch_at_creation`` (the - # primary path post-slice-4 TASK-4-2), and - # * fresh-pipeline derivation from - # ``slice.dependencies[0]`` (the path #2777's slice-2 - # takes before slice-4 lands). - # - # The legacy ``egg/<id>/context`` branch was removed in - # cq-4 so slice-1 (the root) now stacks on - # ``pipeline_branch`` like every other root slice — the - # work-branch context PR's diff already encompasses the - # slice-1 integration branch via ancestry. - # #2928: wire a parent-branch-existence probe so the - # resolver can tell a FRESH non-root slice (whose - # dependency parent branch is still on origin → stack - # on it) apart from an orphaned one (parent merged - # into ``work`` and cascade-deleted → base on - # ``pipeline_branch``). This replaces the pre-#2928 - # merge-base probe, which probed the slice's OWN - # integration branch — non-existent on a first run — - # and so mis-routed every fresh non-root slice onto - # ``work`` whenever ``work`` had advanced ahead of the - # parent. Repoless test scaffolds short-circuit to - # ``True`` (no origin to check; the derived parent is - # the correct DAG target), mirroring the resolver's - # conservative "assume parent exists" default. - # - # IMPORTANT: this wrapper calls the STRICT ls-remote - # variant (``ls_remote_branch_strict``) so a gateway / - # network / policy failure RAISES into the resolver's - # ``try/except`` instead of being collapsed to - # ``False``. The lenient ``ls_remote_branch`` / - # ``get_remote_branch_sha`` helpers swallow all - # exceptions and return ``False`` / ``None`` for both - # "branch absent" AND "gateway error" — using either - # here would silently route a real slice onto - # ``pipeline_branch`` on a flaky gateway, re-creating - # the #2928 wedge that this PR claims to fix. - def _probe_parent_branch_exists(parent_branch: str) -> bool: - if not pipeline.repo: - return True - return spawner.gateway.ls_remote_branch_strict( - pipeline_id, - str(worktree_repo_path), - f"refs/heads/{parent_branch}", - mode=gateway_mode, # type: ignore[arg-type] - ) - - parent_branch = _resolve_slice_base_branch( - contract, - slice_id, - pipeline_id=pipeline_id, - pipeline_branch=pipeline_branch, - parent_branch_exists=_probe_parent_branch_exists, - ) - integration_branch = f"{issue_branch}/{slice_id}" - - # Persist the parent-branch reference on the contract - # under the per-pipeline state lock so a concurrent - # tester / documenter contract write doesn't race with - # ours (reviewer_code v4 #5). While we hold the contract, - # also read back any integration_base_sha recorded on a - # prior run (#2871) — on a restart this lets the race - # check below tell an empty branch apart from a merged - # one. It is ``None`` on a slice's first run (recorded - # only after the branch is created, just below). - recorded_base_sha: str | None = None - # #3253 — capture whether the slice has any produced task - # commit / PR while we hold the contract, so the race-merged - # skip below cannot mistake an empty / un-started branch for - # a merged one (see the merged-acceptance guard below). - slice_produced_work = False - try: - with get_pipeline_state_lock(pipeline_id): - contract_local = load_contract(pipeline_id, worktree_repo_path) - for s in contract_local.slices: - if s.id == slice_id: - s.parent_branch_at_creation = parent_branch - # Slice-4 TASK-4-2: flip PENDING → - # IN_PROGRESS in the SAME contract write - # that persists parent_branch_at_creation - # (cq-9). Crash recovery (TASK-4-4 - # Layer C) now has a single signal to - # distinguish a fresh slice from one - # whose run was interrupted between - # status flip and branch creation. - # Idempotent on re-entry (e.g. orphan - # reconciler): only PENDING is flipped; - # COMPLETE / BLOCKED / IN_PROGRESS are - # left untouched. - if s.status == SliceStatus.PENDING: - s.status = SliceStatus.IN_PROGRESS - recorded_base_sha = s.integration_base_sha - slice_produced_work = ( - _slice_produced_commits(s) or s.pr_number is not None - ) - break - save_contract(contract_local, worktree_repo_path) - except Exception as save_err: # noqa: BLE001 - # Contract load/save under per-pipeline state lock. - # Same exception surface as the COMPLETE-persist - # site above (loader validation, atomic-rename - # I/O, pydantic re-serialisation). Best-effort. - logger.warning( - "Failed to persist parent_branch_at_creation", - pipeline_id=pipeline_id, - slice_id=slice_id, - error=str(save_err), - ) - - # Race protection: a slice's PR can be merged between - # bootstrap reconciliation and this spawn. Detect and - # skip to COMPLETE so the create-branch push below - # doesn't non-fast-forward (#2549). - if pipeline.repo: - try: - already_merged = spawner.gateway.is_slice_branch_merged_into_parent( - pipeline_id, - str(worktree_repo_path), - integration_branch=integration_branch, - parent_branch=parent_branch, - integration_base_sha=recorded_base_sha, - # Read-only ancestry check run by the - # orchestrator's slice-loop scheduler; attribute - # to the orchestrator, not a phantom coder (#2919). - agent_role="orchestrator", - mode=gateway_mode, # type: ignore[arg-type] - ) - except Exception as detect_err: # noqa: BLE001 - # Same `is_slice_branch_merged_into_parent` - # surface as the bootstrap pass above - # (GatewayError + OSError). Default to "not - # merged" so the slice can still spawn. - logger.warning( - "Slice merged-detection raised; treating as not-merged", - pipeline_id=pipeline_id, - slice_id=slice_id, - error=str(detect_err), - ) - already_merged = False - # #3253 — a slice with no produced task commit and no PR - # has an empty integration branch (tip still at the fork - # base); origin ancestry reports it merged because that - # base is trivially an ancestor of an advanced parent. - # Don't skip it as merged — spawn so it actually runs. - if already_merged and not slice_produced_work: - logger.info( - "Slice merged-detection ignored: no produced task commit " - "and no PR — empty/un-started branch, spawning instead of " - "skipping as merged (#3253)", - pipeline_id=pipeline_id, - slice_id=slice_id, - integration_branch=integration_branch, - parent_branch=parent_branch, - ) - already_merged = False - if already_merged: - logger.info( - "Slice already merged into parent on origin — skipping spawn (#2549)", - pipeline_id=pipeline_id, - slice_id=slice_id, - integration_branch=integration_branch, - parent_branch=parent_branch, - ) - scheduler.record_complete(slice_id) - _persist_slice_status_complete(slice_id, basis="merged") - try: - remove_peer_consensus_tracker(pipeline_id, slice_id) - except Exception: # noqa: BLE001 - # In-memory dict pop under a lock; only - # programming errors (KeyError, AttributeError) - # could fire. Bare-except keeps the slice - # COMPLETE/return path crash-proof. - pass - return 0, ( - f"slice {slice_id}: already merged into " - f"{parent_branch} on origin — skipped" - ) - - # #2137 TASK-4-2: create the slice integration branch - # on origin BEFORE spawning containers. Push - # ``parent_branch:refs/heads/integration_branch`` - # through the existing per-agent push allowlist. Agents - # then push their commits directly to the slice's - # integration branch (``egg/issue-N/slice-M``) so the - # slice PR's diff is non-empty when ``gh pr create`` - # runs. On failure, mark the slice failed so the - # cascade machinery can surface the missing-parent - # error to the operator instead of silently spawning - # agents that would push to a missing parent. - if pipeline.repo: - try: - # #3185 — the helper now returns the fork-base - # SHA it pushed the integration branch at (the - # parent tip resolved inside the call), or None - # on failure. Recording that SHA directly here - # replaces a prior best-effort - # ``get_remote_branch_sha`` re-fetch that could - # silently fail (no ``retry_transient``) and - # leave ``integration_base_sha`` unset — arming - # the empty-pre-created-branch trap on the next - # restart. - created_base_sha = spawner.gateway.create_slice_integration_branch( - pipeline_id, - str(worktree_repo_path), - integration_branch=integration_branch, - parent_branch=parent_branch, - # #2947 — hand the slice's recorded fork - # base to the gateway so a crash/restart - # over a branch that already carries this - # slice's commits (with an additively - # advanced parent) resumes in place - # instead of non-fast-forward-failing. - integration_base_sha=recorded_base_sha, - # Orchestrator pre-creates the slice - # integration branch on a synthetic session - # before agents spawn; attribute to the - # orchestrator, not a phantom coder (#2919). - # The push rides the slice-integration - # exemption (synthetic + branch shape), not a - # role gate. - agent_role="orchestrator", - mode=gateway_mode, # type: ignore[arg-type] - ) - except Exception as branch_err: # noqa: BLE001 - # Gateway `create_slice_integration_branch` - # call. Catches GatewayError (HTTP/timeout) - # and OSError (DNS / socket). Treat as failure - # so the cascade machinery surfaces a - # missing-parent error. - logger.error( - "Slice integration branch creation raised", - pipeline_id=pipeline_id, - slice_id=slice_id, - error=str(branch_err), - ) - created_base_sha = None - if created_base_sha is None: - logger.error( - "Slice integration branch creation failed; " - "marking slice failed (agents not spawned)", - pipeline_id=pipeline_id, - slice_id=slice_id, - parent_branch=parent_branch, - integration_branch=integration_branch, - ) - scheduler.record_failure(slice_id) - return 1, ( - f"slice {slice_id}: integration branch " - f"{integration_branch} could not be created from " - f"{parent_branch}" - ) - - # #2871 / #3185 — record the integration branch's fork - # base exactly once, on first creation. The branch was - # just pushed at the parent's tip and no agent has been - # spawned yet, so its origin tip still equals its base. - # Persisting it now lets a later restart's bootstrap - # reconciliation (and the race check above) tell an - # *empty* slice branch — tip still at this base, hence - # a trivial ancestor of an advanced parent — apart from - # a genuinely *merged* one whose tip moved past it. We - # only write it when unset so a restart over a branch - # that already carries slice commits (#2512 recovery) - # keeps its original base rather than the advanced tip. - # ``created_base_sha`` is the SHA the create call - # returned (no extra round-trip); it is an empty string - # on the unreachable no-op path - # (``integration_branch == parent_branch``), which we - # skip here. - if recorded_base_sha is None and created_base_sha: - try: - with get_pipeline_state_lock(pipeline_id): - contract_local = load_contract(pipeline_id, worktree_repo_path) - for s in contract_local.slices: - if s.id == slice_id: - s.integration_base_sha = created_base_sha - break - save_contract(contract_local, worktree_repo_path) - recorded_base_sha = created_base_sha - except Exception as base_err: # noqa: BLE001 - # Contract load/save under per-pipeline state - # lock. Catches loader validation, atomic- - # rename I/O, and pydantic re-serialisation - # errors. Best-effort: the fork base is no - # longer a round-trip failure (the SHA came - # from the create call itself), so this now - # only fires on a contract-write failure — a - # transient the next run repairs on the same - # create path. - logger.warning( - "Failed to persist slice integration_base_sha " - "(#2871); a future restart re-records it on the " - "create path", - pipeline_id=pipeline_id, - slice_id=slice_id, - integration_branch=integration_branch, - error=str(base_err), - ) - - logger.info( - "Slice spawn", - pipeline_id=pipeline_id, - slice_id=slice_id, - parent_branch=parent_branch, - integration_branch=integration_branch, - ) - - exit_code_inner, logs_inner = _run_concurrent_phase_with_impasse_retry( - pipeline_id=pipeline_id, - pipeline=pipeline, - phase="implement", - spawner=spawner, - repo_volumes=repo_volumes, - gateway_mode=gateway_mode, - repos=repos, - sandbox_env=sandbox_env, - store=store, - certs_volume=certs_volume, - worktree_repo_path=worktree_repo_path, - slice_id=slice_id, - run_epoch=run_epoch, - ) - - if exit_code_inner != 0: - scheduler.record_failure(slice_id) - logger.warning( - "Slice failed", - pipeline_id=pipeline_id, - slice_id=slice_id, - exit_code=exit_code_inner, - ) - return exit_code_inner, logs_inner - - # Slice consensus reached — load the contract ONCE - # under the per-pipeline state lock and reuse the same - # snapshot for the #3125 evidence-reachability gate - # AND the slice's PR data snapshot below. Both readers - # previously took the lock independently; collapsing - # them eliminates one file read + lock acquire per - # slice close (#3125 review). - # - # The slice_pr_data block below originally documented - # the lock as covering only the contract read so the - # gateway HTTP round-trip wouldn't serialise other - # writers — the same posture applies here: we release - # the lock before the gateway call inside the gate. - contract_post: Any | None = None - try: - with get_pipeline_state_lock(pipeline_id): - contract_post = load_contract(pipeline_id, worktree_repo_path) - except Exception as load_err: # noqa: BLE001 - logger.warning( - "Slice close: contract load failed (continuing) (#3125)", - pipeline_id=pipeline_id, - slice_id=slice_id, - error=str(load_err), - ) - - # #3125 — evidence-reachability gate: every commit SHA - # cited by this slice's contract task records must be - # an ancestor of the integration branch tip, or the - # slice PR would ship without a deliverable the task - # record claims is done (the post-confirmation - # ``complete-task --commit`` unblock flow, #3124). - # Fails the slice BEFORE any close side effect so the - # cascade + HITL machinery surfaces the gap loudly. - # ``contract_post`` may be None if the load above - # raised — the gate falls back to its own load in that - # case (and skips gracefully if that fails too). - if pipeline.repo: - evidence_failure = _check_slice_evidence_reachability( - pipeline_id, - spawner, - worktree_repo_path, - slice_id, - integration_branch, - gateway_mode=gateway_mode, # type: ignore[arg-type] - contract=contract_post, - ) - if evidence_failure is not None: - scheduler.record_failure(slice_id) - return 1, evidence_failure - - # #3398 — per-slice green gate: execute the repo's - # configured checks (repositories.yaml, via - # get_repo_checks) against the integration-branch tip - # in a sandboxed one-shot runner, and refuse to open - # the slice PR while any check is red. Closes the - # trust-vs-verify gap in the propose-time - # checks_passed self-report. Same posture as the - # evidence gate above: fail-open on infra errors, - # fail-closed only on a definitive red verdict; - # EGG_SLICE_GREEN_GATE is the operator switch - # (off during rollout / log / on). - if pipeline.repo: - try: - import slice_green_gate as _green_gate - except ImportError: - from .. import slice_green_gate as _green_gate # type: ignore[no-redef] - - green_gate_failure = _green_gate.run_slice_green_gate( - pipeline_id, - spawner, - slice_id, - integration_branch, - pipeline.repo, - gateway_mode=gateway_mode, # type: ignore[arg-type] - ) - if green_gate_failure is not None: - scheduler.record_failure(slice_id) - return 1, green_gate_failure - - # Snapshot the slice's PR data from the same loaded - # contract — no second lock acquire, no second file - # read. - slice_pr_data: dict[str, Any] | None = None - try: - if contract_post is not None: - slice_obj = next( - (s for s in contract_post.slices if s.id == slice_id), - None, - ) - if slice_obj is not None and pipeline.repo: - # #2538: every slice carries the - # planner-authored narrative on its PR so - # reviewers see context on whichever slice - # they open first. Pre-#2777 cq-6 the - # terminal slice additionally carried a - # program-level rollup (test plan + manual - # steps + pre-merge obligations) and a - # ``[merge-gate]`` title marker. Under cq-4 - # the merge gate is the up-front context - # PR (``egg/<id>/work → main``) opened by - # ``_open_context_pr_at_implement_start``, - # so every slice PR — terminal or not — - # now uses the same lean shape and the - # terminal-slice computation is gone. - program_pr = contract_post.pr - # #2745: derive 1-based slice position + - # total slice count from declared contract - # order so the slice PR title can carry - # ``[slice-N/M]``. - slice_count = len(contract_post.slices) - slice_index_lookup = next( - ( - i + 1 - for i, s in enumerate(contract_post.slices) - if s.id == slice_id - ), - None, - ) - # Union of ``task.files_affected`` across the - # slice's tasks; rendered under - # ``## This slice`` so reviewers see what - # this slice actually touches without - # opening the diff (#2745). - slice_files_affected_list: list[str] = [] - seen_paths: set[str] = set() - for t in slice_obj.tasks or []: - for path in t.files_affected or []: - if path and path not in seen_paths: - seen_paths.add(path) - slice_files_affected_list.append(path) - # #3393 slice-4 / task-4-1: route this slice's - # PR to its OWN repo (``resolve_slice_repo`` → - # ``slice.repo`` else the pipeline primary) and - # gather CROSS-repo coordination references for - # the PR body. Same-repo relationships are left - # to ``## Stack``, so for an N=1 pipeline - # ``slice_repo`` is the single repo and both - # ref sets are empty — behaviour is unchanged. - try: - from models import ( # type: ignore[no-redef] - resolve_slice_repo, - ) - except ImportError: - from ..models import ( # type: ignore[no-redef] - resolve_slice_repo, - ) - slice_repo = resolve_slice_repo(slice_obj, pipeline) or pipeline.repo - sibling_pr_refs: list[dict[str, Any]] = [] - for other in contract_post.slices: - if other.id == slice_id: - continue - other_repo = resolve_slice_repo(other, pipeline) or pipeline.repo - if other_repo and other_repo != slice_repo and other.pr_number: - sibling_pr_refs.append( - {"repo": other_repo, "number": other.pr_number} - ) - # Dependent-slice upstream PR — surfaced only - # when the upstream slice is in a DIFFERENT repo - # (a same-repo parent is the stack base already - # rendered by ``## Stack``). - upstream_pr_ref: dict[str, Any] | None = None - upstream_ids = slice_obj.dependencies or [] - if upstream_ids: - upstream = next( - (s for s in contract_post.slices if s.id == upstream_ids[0]), - None, - ) - if upstream is not None and upstream.pr_number: - upstream_repo = ( - resolve_slice_repo(upstream, pipeline) or pipeline.repo - ) - if upstream_repo and upstream_repo != slice_repo: - upstream_pr_ref = { - "repo": upstream_repo, - "number": upstream.pr_number, - } - # #3393 slice-5 / task-5-1: a slice with a - # CROSS-repo dependency opens its PR as a DRAFT - # — cross-repo edges can't stack, so the - # dependent slice is developed in parallel and - # only its PR *ready* transition waits on the - # merge gate (auto draft→ready when the upstream - # merges, else a HITL hold). A dep is cross-repo - # iff the upstream slice resolves to a DIFFERENT - # repo; same-repo-only deps and N=1 pipelines - # stay non-draft (behaviour unchanged). Checks - # ALL deps so any cross-repo upstream holds it. - cross_repo_draft = False - for _dep_id in slice_obj.dependencies or []: - _dep = next( - (s for s in contract_post.slices if s.id == _dep_id), - None, - ) - if _dep is None: - continue - _dep_repo = resolve_slice_repo(_dep, pipeline) or pipeline.repo - if _dep_repo and _dep_repo != slice_repo: - cross_repo_draft = True - break - slice_pr_data = { - # #3393 slice-4: the repo this slice's PR is - # opened in + its cross-repo coordination - # references (empty for N=1). - "slice_repo": slice_repo, - # #3393 slice-5: open draft when this slice - # has a cross-repo dependency (see above). - "cross_repo_draft": cross_repo_draft, - "sibling_pr_refs": sibling_pr_refs, - "upstream_pr_ref": upstream_pr_ref, - "slice_name": slice_obj.name or slice_id, - # Planner's reviewer-facing summary — - # rendered as the slice PR body's lead - # paragraph (#3115). Empty for - # pre-#3115 contracts. - "slice_goal": getattr(slice_obj, "goal", "") or None, - "slice_tasks": [ - { - "id": t.id, - "description": t.description, - "acceptance_criteria": t.acceptance_criteria, - } - for t in (slice_obj.tasks or []) - ], - "slice_index": slice_index_lookup, - "slice_count": slice_count, - "slice_files_affected": slice_files_affected_list or None, - # ``context_pr_number`` is populated by - # ``_open_context_pr_at_implement_start`` - # at the plan→implement boundary (#2777 - # cq-4). When the contract linkage is - # missing (e.g. ``contract.pr`` is None - # on an implement-start resume, #3100), - # fall back to ``pipeline.pr_number`` — - # the pipeline-level mirror written by - # ``_persist_context_pr_number`` whose - # sole post-#2777 writer is the same - # opener — so the slice PR still links - # its base PR (#3115). When both are - # None — should be unreachable under - # the hard-required opener but kept as - # defence-in-depth — ``create_slice_pr`` - # falls back to the pre-#2745 inline- - # narrative body so the slice PR stays - # reviewable as a standalone diff - # against ``/work``. - "context_pr_number": ( - (program_pr.context_pr_number if program_pr else None) - or pipeline.pr_number - ), - "program_title": (program_pr.title if program_pr else None), - "program_description": ( - program_pr.description if program_pr else None - ), - "program_test_plan": (program_pr.test_plan if program_pr else None), - "program_manual_steps": ( - program_pr.manual_steps if program_pr else None - ), - } - except Exception as attr_err: # noqa: BLE001 - # Nested attribute traversal on slice/program PR - # objects (the contract load was lifted out to the - # block above). Surface is AttributeError / - # KeyError on partially-populated PR rollup - # fields. Continue without slice_pr_data (the - # gateway PR creation just below is gated on it - # being non-None). - logger.warning( - "Slice PR pre-load failed (continuing)", - pipeline_id=pipeline_id, - slice_id=slice_id, - error=str(attr_err), - ) - - # Persist this slice's per-slice BRC consensus history - # onto its integration branch as the final - # orchestrator-authored commit before the slice PR is - # opened, so reviewers see the consensus transcript in - # the PR diff (#2548). Best-effort + idempotent on - # retry; per-slice files live ONLY on the integration - # branch. - if pipeline.repo: - try: - _commit_slice_brc_history_to_integration_branch( - pipeline, - spawner, - worktree_repo_path, - slice_id, - integration_branch, - gateway_mode=gateway_mode, # type: ignore[arg-type] - ) - except Exception as brc_commit_err: # noqa: BLE001 - # Per-slice BRC commit helper calls into the - # full git/gateway/message-store machinery - # — the exception surface is unbounded - # (gateway push failures, git plumbing - # errors, message-store reads, file I/O). - # Best-effort: the BRC transcript commit is - # non-essential to slice consensus. - logger.warning( - "Per-slice BRC commit raised (continuing) (#2548)", - pipeline_id=pipeline_id, - slice_id=slice_id, - error=str(brc_commit_err), - ) - - pr_created = True - slice_pr_url: str | None = None - slice_pr_number: int | None = None - if slice_pr_data is not None and pipeline.repo: - # Best-effort real-diff summary for the PR body - # (#3115) — commit subjects + diffstat from the - # pushed integration branch. (None, None) on any - # failure; the PR opens without the section. - commit_subjects, diffstat = _build_slice_diff_summary( - pipeline, - spawner, - worktree_repo_path, - integration_branch, - parent_branch, - gateway_mode=gateway_mode, # type: ignore[arg-type] - ) - try: - slice_pr_url = spawner.gateway.create_slice_pr( - pipeline_id=pipeline_id, - # #3393 slice-4 / task-4-1: route to the slice's - # own repo (falls back to the pipeline primary - # when ``slice.repo`` is absent — the N=1 case). - repo=slice_pr_data["slice_repo"] or pipeline.repo, - slice_id=slice_id, - slice_name=slice_pr_data["slice_name"], - slice_tasks=slice_pr_data["slice_tasks"], - head=integration_branch, - base=parent_branch, - issue_number=issue_number, - agent_role="orchestrator", - mode=gateway_mode, # type: ignore[arg-type] - # #3393 slice-5 / task-5-1: draft when this - # slice has a cross-repo dependency; the merge - # gate marks it ready on upstream merge (or a - # HITL hold releases it). False for N=1. - draft=slice_pr_data["cross_repo_draft"], - program_title=slice_pr_data["program_title"], - program_description=slice_pr_data["program_description"], - program_test_plan=slice_pr_data["program_test_plan"], - program_manual_steps=slice_pr_data["program_manual_steps"], - slice_index=slice_pr_data["slice_index"], - slice_count=slice_pr_data["slice_count"], - slice_files_affected=slice_pr_data["slice_files_affected"], - context_pr_number=slice_pr_data["context_pr_number"], - slice_goal=slice_pr_data["slice_goal"], - diffstat=diffstat, - commit_subjects=commit_subjects, - sibling_pr_refs=slice_pr_data["sibling_pr_refs"], - upstream_pr_ref=slice_pr_data["upstream_pr_ref"], - ) - except Exception as pr_err: # noqa: BLE001 - # Single `gateway.create_slice_pr` HTTP call. - # Catches GatewayError (HTTP) and OSError - # (DNS / socket). Mark pr_created=False so - # the cascade machinery fires. - logger.error( - "Slice PR creation failed", - pipeline_id=pipeline_id, - slice_id=slice_id, - error=str(pr_err), - ) - pr_created = False - - if not pr_created: - scheduler.record_failure(slice_id) - return 1, ( - f"slice {slice_id}: PR creation failed (head={integration_branch}, " - f"base={parent_branch})" - ) - - # Parse the slice PR number from the returned URL - # (#3122) — same trailing-boundary pattern the context- - # PR opener uses, narrowed to ``[1-9]\d*`` so a - # malformed ``/pull/0/...`` URL doesn't make it as far - # as ``Slice.pr_number``'s ``ge=1`` validator (which - # would silently downgrade to a warning log via the - # save try/except in ``_persist_slice_status_complete``). - # Best-effort: an unparseable URL just means the - # linkage isn't recorded this pass; the idempotent - # ``create_slice_pr`` re-yields it on a resume. - if slice_pr_url: - pr_match = re.search(r"/pull/([1-9]\d*)(?:[/?#]|$)", slice_pr_url) - if pr_match: - slice_pr_number = int(pr_match.group(1)) - - # Hold the per-pipeline state lock across both the - # contract-write (``_persist_slice_status_complete`` - # itself reacquires this RLock) and the context-PR - # body refresh (load + compose + push). Without the - # outer lock, two slices in the same wave could - # interleave between persist and push so the slice - # whose refresh starts earlier but lands later - # clobbers the body that already included both links - # — and because no later slice fires a refresh, the - # final slice's ``— #N`` link would stay missing - # forever. Serializing here bounds the per-slice tail - # latency by one gateway PATCH per concurrent slice - # rather than racing them. - with get_pipeline_state_lock(pipeline_id): - scheduler.record_complete(slice_id) - # Reaching here means ``_run_concurrent_phase`` returned - # success (BRC consensus) AND ``pr_created`` gated above — - # a verified completion independent of whether the PR URL - # parsed to a number (#3122 stub URLs leave - # ``slice_pr_number`` None). Declare the consensus basis so - # the #3214 invariant accepts it; ``pr_number`` is still - # passed for the slice-table linkage. - _persist_slice_status_complete( - slice_id, - pr_number=slice_pr_number, - pr_url=slice_pr_url if slice_pr_number else None, - basis="consensus_complete", - ) - - # Refresh the context PR body so its slice table - # links the PR that just opened (#3122). Strictly - # cosmetic and best-effort: every failure path - # inside logs + returns False without raising, and - # the slice outcome below never depends on it. - if slice_pr_number: - _refresh_context_pr_body( - pipeline_id, - pipeline=pipeline, - spawner=spawner, - worktree_repo_path=worktree_repo_path, - identifier=_pipeline_identifier(pipeline.issue_number, pipeline_id), - gateway_mode=gateway_mode, - ) - - try: - remove_peer_consensus_tracker(pipeline_id, slice_id) - except Exception: # noqa: BLE001 - # In-memory dict pop under a lock; same crash-proof - # defence-in-depth as the merged-skip branch above. - pass - return exit_code_inner, logs_inner - - # Gate every ready slice through the orchestrator-process-wide - # admission counter (#2241 gap 1). Slices the global cap - # rejects stay in READY and re-yield next tick — the per- - # pipeline ``iter_ready`` accounting is unaffected because - # we admit BEFORE ``mark_spawned``. If the entire batch is - # rejected, sleep one poll interval before re-checking so - # we don't burn CPU spinning on iter_ready. - admitted_batch: list[tuple[str, str | None]] = [ - (slice_id, parent_slice_id) - for slice_id, parent_slice_id in ready_batch - if global_slice_admit.try_admit(pipeline_id, slice_id) - ] - if not admitted_batch: - logger.info( - "Slice wave deferred behind global cap", - pipeline_id=pipeline_id, - ready=[s for s, _ in ready_batch], - admit=global_slice_admit.snapshot(), - ) - time.sleep(poll_interval) - continue - - # Mark admitted slices as spawned BEFORE submitting them to - # the executor so a subsequent ``iter_ready`` from any other - # thread sees the in-flight count correctly. - for slice_id, _parent in admitted_batch: - scheduler.mark_spawned(slice_id) - - max_workers = max(1, len(admitted_batch)) - with concurrent.futures.ThreadPoolExecutor( - max_workers=max_workers, - thread_name_prefix=f"slice-wave-{pipeline_id}", - ) as wave_pool: - futures: dict[concurrent.futures.Future, str] = {} - for slice_id, parent_slice_id in admitted_batch: - fut = wave_pool.submit(_run_one_slice, slice_id, parent_slice_id) - futures[fut] = slice_id - - for fut in concurrent.futures.as_completed(futures): - slice_id_done = futures[fut] - try: - exit_code, logs = fut.result() - except Exception as exc: # noqa: BLE001 - # fut.result() re-raises whatever the slice - # worker raised. Workers call into the full - # implement-phase machinery (gateway, contract, - # spawner, message store, docker) so the - # exception surface is unbounded; mark the - # slice failed and continue rather than tearing - # down the whole wave. - scheduler.record_failure(slice_id_done) - exit_code = 1 - logs = f"slice {slice_id_done} raised: {exc!r}" - logger.error( - "Slice worker raised", - pipeline_id=pipeline_id, - slice_id=slice_id_done, - error=str(exc), - ) - aggregate_logs.append(f"--- slice {slice_id_done} ---\n{logs}") - if exit_code != 0: - overall_exit = exit_code - - # Drain cascades after each wave so descendants of a - # failed slice are visibly BLOCKED before the next - # iteration computes ready slices. Emit an - # OVERSEER_ALERT per cascade so the human operator sees - # the blocked subtree (#2137 TASK-3-4 emission path). - events = scheduler.poll_cascades() - for event in events: - logger.warning( - "Slice cascade fired", - pipeline_id=pipeline_id, - failed_slice=event.failed_slice_id, - blocked=event.blocked_subtree, - ) - # Emit OVERSEER_ALERT directly through the in-process - # message store so the human operator's overseer - # surface picks up the cascade-block event (TASK-3-4 - # emission path). - try: - try: - from message_store import Message, get_message_store - except ImportError: - from ..message_store import ( # type: ignore[no-redef] - Message, - get_message_store, - ) - - msg = Message( - pipeline_id=pipeline_id, - from_role="orchestrator", - to_role="all", - message_type="OVERSEER_ALERT", - subject=f"slice-cascade-block: {event.failed_slice_id}", - body=( - f"Slice {event.failed_slice_id} failed; " - f"downstream subtree {event.blocked_subtree} marked " - "BLOCKED_ON_FAILED_DEPENDENCY (60 s grace expired). " - "HITL resolution required to restart the failed slice." - ), - metadata={ - "anomaly": "slice-cascade-block", - "priority": "high", - "failed_slice_id": event.failed_slice_id, - "blocked_subtree": list(event.blocked_subtree), - }, - phase="implement", - ) - get_message_store().add_message(msg) - except Exception: # noqa: BLE001 - # Best-effort: the log line above is the - # always-on fallback so the operator still sees - # the cascade in the orchestrator log. - pass - finally: - reconciler_stop.set() - try: - reconciler_thread.join(timeout=5.0) - except RuntimeError: - # Thread.join only raises RuntimeError (e.g. joining the - # current thread). Other failures are silent timeouts. - pass - - aggregated = "\n".join(aggregate_logs) if aggregate_logs else "Slice loop completed." - return overall_exit, aggregated - - -def _clear_stale_impasses_for_producers( - repo_path: Path, - pipeline_id: str, - producer_roles: "list[ContractAgentRole]", # noqa: UP037 - *, - cleanup_reason: str, -) -> None: - """Drop the ``impasse`` field from each producer's per-pipeline - agent-output file before the next BRC cycle. - - ``save_agent_output`` writes with ``mode="w"`` so a producer that - respawns and reaches its handoff write will overwrite the stale - impasse on its own. But if a producer crashes before writing in the - next iteration (or if the implement roster ever becomes - contract-task-driven, in which case a producer with no remaining - tasks won't spawn at all), the iter-N impasse file would persist - into iter-N+1's ``collect_impasses`` scan and re-trigger routing on - a stale signal — which the ``delegation_attempts`` counter would - then translate into a spurious "second impasse on same task" HITL - escalation. - - Pre-clearing the field keeps ``collect_impasses`` honest about what - came out of the *current* iteration only. Other top-level fields on - the agent output (``handoff_data``, ``role``, anything else) are - preserved. - """ - for role_enum in producer_roles: - try: - existing = load_agent_output(repo_path, role_enum, identifier=pipeline_id) - except Exception as exc: # noqa: BLE001 - # Best-effort agent-output file read. Catches OSError on - # the file read, json.JSONDecodeError on parse, and - # pydantic.ValidationError on the role-specific shape. - # Continue (no impasse to clear if the file is unreadable). - logger.debug( - "Could not pre-load agent output to clear stale impasse", - pipeline_id=pipeline_id, - role=role_enum.value, - error=str(exc), - ) - continue - if not isinstance(existing, dict) or "impasse" not in existing: - continue - cleaned = {k: v for k, v in existing.items() if k != "impasse"} - try: - save_agent_output( - repo_path, - role_enum, - cleaned, - identifier=pipeline_id, - ) - except Exception as exc: # noqa: BLE001 - # Atomic file write of JSON-serialisable dict. Catches - # OSError (write/rename), TypeError/ValueError (non- - # serialisable value sneaking in). Continue — the stale - # impasse will re-trigger routing but the delegation - # counter still bounds the retry. - logger.warning( - "Failed to clear stale impasse from agent output", - pipeline_id=pipeline_id, - role=role_enum.value, - error=str(exc), - ) - continue - logger.info( - "Cleared stale impasse from agent output", - pipeline_id=pipeline_id, - role=role_enum.value, - cleanup_reason=cleanup_reason, - ) - - -def _pipeline_superseded_by_restart(store, pipeline_id: str, run_epoch: datetime | None) -> bool: - """True if a newer ``run_epoch`` means another thread now owns this pipeline. - - Reloads pipeline state and compares its ``run_epoch`` against the epoch the - caller runs under (#3315 facet a). Best-effort: a missing epoch or a load - failure returns ``False`` so a transient store hiccup never tears down a - legitimately-running phase. Shared by the ``_run_concurrent_phase`` poll - loop and the slice-path impasse-retry wrapper so the "no escalation when - superseded" property holds on both routes. - """ - if store is None or run_epoch is None: - return False - try: - _epoch_pip = store.load_pipeline(pipeline_id) - except Exception as _epoch_err: # noqa: BLE001 — never wedge the caller - logger.debug( - "Epoch supersession check failed; continuing", - pipeline_id=pipeline_id, - error=str(_epoch_err), - ) - return False - current_epoch = _epoch_pip.run_epoch or _epoch_pip.created_at - return current_epoch != run_epoch - - -def _run_concurrent_phase_with_impasse_retry( - pipeline_id: str, - pipeline: Pipeline, - phase: str, - spawner, - repo_volumes: dict[str, str], - gateway_mode: str, - repos: list[str], - sandbox_env: dict[str, str], - store, - certs_volume: str | None, - worktree_repo_path: Path, - review_feedback: str | None = None, - slice_id: str | None = None, - operator_directives: list[OperatorDirective] | None = None, - iteration_history: list[IterationSummary] | None = None, - run_epoch: datetime | None = None, -) -> tuple[int, str]: - """Run a concurrent phase, auto-delegating impasses once before HITL. - - Wraps :func:`_run_concurrent_phase` with the runtime escape-hatch - introduced in #2529: - - 1. Run the BRC cycle as usual. - 2. After it exits, scan each producer's ``AgentOutput`` for a typed - :class:`egg_contracts.Impasse`. - 3. For ``WRONG_ROLE`` impasses with a single eligible alternative - producer role and ``task.delegation_attempts == 0``, mutate - ``task.role`` to the suggested role and re-run the BRC cycle - once. The new spawn picks up the role flip when - ``_build_agent_prompt`` re-reads the contract. - 4. For everything else (second impasse, non-WRONG_ROLE category, - no eligible alternative role, unresolvable task_id) the helper - creates a HITL decision on the contract and the slice exits - so the operator can choose between cancel / re-plan / manual - resolution. ``feedback_no_auto_hitl.md``: the orchestrator - creates the decision; surfacing to the user is the operator - layer's job. - - Pipeline-level (non-sliced) callers can pass ``slice_id=None``; - the routing helper falls back to a contract-wide search for the - impassed task. - """ - try: - from orchestrator.impasse_routing import ( - ImpasseAction, - collect_impasses, - route_impasses, - ) - except ImportError: - from impasse_routing import ( # type: ignore[no-redef] - ImpasseAction, - collect_impasses, - route_impasses, - ) - - try: - from egg_contracts.agent_roles import AgentRole as ContractAgentRoleEnum - except ImportError: # pragma: no cover - import seam parity - from shared.egg_contracts.agent_roles import ( # type: ignore[no-redef] - AgentRole as ContractAgentRoleEnum, - ) - # Two attempts max: original + at most one delegated retry. The - # ``delegation_attempts`` counter on the contract task enforces the - # same bound when the slice is restarted out-of-band by an - # operator, so a long-lived pipeline can never escape this gate. - MAX_IMPASSE_ATTEMPTS = 2 - - # Producer roles only — impasses are a producer concept; reviewers - # don't author tasks. Mirrors the producer trio in - # ``shared/egg_restrictions/patterns.py``. - producer_roles = [ - ContractAgentRoleEnum.CODER, - ContractAgentRoleEnum.TESTER, - ContractAgentRoleEnum.DOCUMENTER, - ] - - last_exit = 0 - last_logs = "" - for attempt in range(MAX_IMPASSE_ATTEMPTS): - is_terminal = attempt + 1 == MAX_IMPASSE_ATTEMPTS - - last_exit, last_logs = _run_concurrent_phase( - pipeline_id=pipeline_id, - pipeline=pipeline, - phase=phase, - spawner=spawner, - repo_volumes=repo_volumes, - gateway_mode=gateway_mode, - repos=repos, - sandbox_env=sandbox_env, - store=store, - certs_volume=certs_volume, - worktree_repo_path=worktree_repo_path, - review_feedback=review_feedback, - slice_id=slice_id, - operator_directives=operator_directives, - iteration_history=iteration_history, - run_epoch=run_epoch, - ) - - try: - impasses = collect_impasses( - Path(worktree_repo_path), - pipeline_id, - producer_roles, - ) - except Exception as scan_err: # noqa: BLE001 - logger.warning( - "Impasse scan raised; continuing without delegation", - pipeline_id=pipeline_id, - slice_id=slice_id, - error=str(scan_err), - ) - return last_exit, last_logs - - if not impasses: - return last_exit, last_logs - - # Defense-in-depth (#3315 facet a, slice path): if a restart bumped - # ``run_epoch`` while this thread was running, a stale producer-written - # impasse file could otherwise drive ``route_impasses`` into a HITL - # against the freshly-restarted phase. The poll loop in - # ``_run_concurrent_phase`` already bails on supersession before any - # escalation; mirror that here so the "no escalation when superseded" - # property holds on the slice path too — return the (superseded) result - # without routing. - if _pipeline_superseded_by_restart(store, pipeline_id, run_epoch): - logger.info( - "Restart superseded this thread before impasse routing; " - "skipping route_impasses to avoid escalating against a " - "freshly-restarted phase", - pipeline_id=pipeline_id, - slice_id=slice_id, - ) - return last_exit, last_logs - - try: - # On the terminal iteration we have no remaining BRC cycle - # to respawn with a new role, so a delegation made here - # would silently dangle (review feedback #2 on PR #2553). - # Force every impasse onto the escalate path instead. - decisions = route_impasses( - repo_path=Path(worktree_repo_path), - pipeline_id=pipeline_id, - contract_identifier=pipeline_id, - impasses=impasses, - slice_id=slice_id, - force_escalate=is_terminal, - ) - except Exception as route_err: # noqa: BLE001 - logger.error( - "Impasse routing raised; surfacing slice failure", - pipeline_id=pipeline_id, - slice_id=slice_id, - error=str(route_err), - ) - return last_exit, last_logs - - all_delegated = decisions and all(d.action == ImpasseAction.DELEGATE for d in decisions) - if not all_delegated: - # Any escalation, or an empty decision list, means the - # operator gates the next move. Don't auto-retry. - for d in decisions: - logger.info( - "Impasse decision", - pipeline_id=pipeline_id, - slice_id=slice_id, - action=d.action.value, - role=d.role, - task_id=d.task_id, - new_role=d.new_role, - reason=d.reason, - hitl_decision_id=d.hitl_decision_id, - ) - return last_exit, last_logs - - # All impasses delegated cleanly — the contract has been - # mutated, log the swap and let the loop respawn with the new - # roles. Last attempt falls through and returns whatever the - # second BRC cycle produced. - for d in decisions: - logger.info( - "Impasse delegated; retrying slice with new role", - pipeline_id=pipeline_id, - slice_id=slice_id, - attempt=attempt + 1, - from_role=d.role, - to_role=d.new_role, - task_id=d.task_id, - ) - - # Drop the now-routed impasse signals before the next BRC - # cycle, so a producer that crashes pre-handoff in iter-N+1 - # cannot resurrect this iteration's impasse via a stale file. - _clear_stale_impasses_for_producers( - Path(worktree_repo_path), - pipeline_id, - producer_roles, - cleanup_reason="post-delegation cleanup", - ) - - return last_exit, last_logs - - -def _run_concurrent_phase( - pipeline_id: str, - pipeline: Pipeline, - phase: str, - spawner, - repo_volumes: dict[str, str], - gateway_mode: str, - repos: list[str], - sandbox_env: dict[str, str], - store, - certs_volume: str | None, - worktree_repo_path: Path, - review_feedback: str | None = None, - slice_id: str | None = None, - operator_directives: list[OperatorDirective] | None = None, - iteration_history: list[IterationSummary] | None = None, - run_epoch: datetime | None = None, -) -> tuple[int, str]: - """Run a phase using concurrent all-agents-at-once execution. - - Creates a ConcurrentPhaseExecutor that spawns all agents simultaneously, - all sharing the pipeline branch. Each container receives a role-specific - prompt built via ``_build_agent_prompt``. After spawning, waits for all - containers to exit and records their state in the pipeline store. - - Returns: - (exit_code, logs) — 0 on success. - - Raises: - SpawnFailureError: If any agent fails to spawn. Survivors are stopped - and their pipeline-state records are marked FAILED before the - exception propagates. Distinguishes spawn failures from container - exits so the outer caller's ``pipeline.error`` is accurate. - """ - from models import ( - AgentExecution as StateAgentExecution, - ) - from models import ( - AgentExecutionStatus as StateAgentStatus, - ) - from models import ( - ContainerInfo, - ContainerStatus, - PipelinePhase, - resolve_consensus_timeout_minutes, - ) - - try: - from concurrent_executor import ( - ConcurrentPhaseExecutor, - _is_transient_agent_error, - ) - except ImportError: - from ..concurrent_executor import ( # type: ignore - ConcurrentPhaseExecutor, - _is_transient_agent_error, - ) - - phase_str = phase if isinstance(phase, str) else phase.value - pipeline_mode = "issue" if pipeline.issue_number is not None else "prompt" - - # Slice-aware sandbox env (#2137 TASK-4-3 / #2403): when running a - # per-slice team, the spawner exposes the slice id via - # ``EGG_SLICE_ID`` and leaves ``EGG_PIPELINE_ID`` as the bare - # pipeline id. An earlier shape encoded the slice into - # ``EGG_PIPELINE_ID`` itself (``{pipeline_id}/{slice_id}``) so the - # orchestrator's ``_tracker_key`` would route CONSENSUS_* to the - # slice tracker without an extra signal-level field. That broke - # every agent → orchestrator round-trip: - # - # * the orchestrator-side ``PIPELINE_ID_PATTERN`` and the agent - # handler validator (``[a-zA-Z0-9_-]+``) both reject the slash, - # * Flask's default URL converter doesn't allow ``/``, so every - # ``POST /api/v1/pipelines/{pid}/...`` route 404s — i.e. all - # of progress, BRC, heartbeat, message, phase, decision, etc. - # - # Slice routing is plumbed explicitly instead: the BRC handlers - # pull ``EGG_SLICE_ID`` and forward it on the signal payload, and - # the orchestrator's signal handlers feed it into - # ``get_peer_consensus_tracker(pipeline_id, slice_id)``. CONSENSUS_* - # isolation is preserved; HEARTBEAT and OVERSEER_ALERT are not - # tracker-scoped at all — ``handle_heartbeat_signal`` is a no-op - # ACK with no tracker lookup, and OVERSEER_ALERT flows through the - # message bus (``MessageType.OVERSEER_ALERT``) rather than the - # consensus tracker. So per-slice scoping doesn't apply to either, - # and operator telemetry stays pipeline-wide as before. The - # pipeline-level fan-out for OVERSEER_ALERT mentioned in earlier - # comments here is tracked alongside the per-slice MCP control - # verbs in #2199. - # - # Single source of truth (#2410 v2 review): ``EGG_SLICE_ID`` is - # injected by ``KubernetesSpawner.spawn_agent_job`` from the same - # ``slice_id`` parameter that drives Job naming and worktree id, so - # there is no need to also stuff it into ``sandbox_env`` here. The - # key is in ``_PROTECTED_ENV_KEYS`` so any future caller that does - # supply a value via ``extra_env`` is logged and overridden. - - # Build per-role prompts for concurrent phase execution. - from egg_contracts.agent_roles import get_roles_for_phase as _get_roles_for_phase - - roles: list[AgentRole] = [] - for r in _get_roles_for_phase( - phase_str, - include_reviewers=True, - repo=pipeline.repo, - has_contract=getattr(pipeline, "has_contract", True), - ): - try: - roles.append(AgentRole(r.value)) - except ValueError: - # New roles not yet in orchestrator AgentRole — skip - continue - - # Build a review graph filtered to only active roles so consensus - # tracking doesn't wait for unspawned agents. - from review_graph import ReviewGraph - from review_graph import get_review_graph_for_phase as _get_graph - - full_graph = _get_graph(phase_str, repo=pipeline.repo) - active_role_names = {r.value for r in roles} - filtered_edges = [ - e - for e in full_graph.edges - if e.reviewer_role in active_role_names and e.producer_role in active_role_names - ] - filtered_graph = ReviewGraph(filtered_edges) - - # Scope the per-slice team to the slice's repo (#3393 task-6-1). - # - # Every slice maps to exactly one repo (slice ↔ repo, 1:1). For a - # multi-repo pipeline the slice's work, worktree, test gate, reviewer - # diff and PR all live in ITS repo — not necessarily the pipeline - # primary. We resolve the slice's repo via ``resolve_slice_repo`` and - # thread the slice-scoped repo / worktree / base-branch into the agent - # prompts (which drive ``get_repo_checks`` for the tester's configured - # checks, the file-boundary patterns, and the reviewer's - # ``git diff origin/<base>...HEAD``) and the spawn (via ``base_branch`` - # → ``EGG_BASE_BRANCH`` and a slice-primary-first ``repos`` ordering so - # the spawner sets the agent cwd / ``EGG_REPO_PATH`` to the slice's - # repo worktree). - # - # N=1 stays byte-identical: a single-repo pipeline has one RepoSpec, so - # the block below is skipped entirely (``len(pipeline.repos) <= 1``), - # leaving ``slice_repo == pipeline.repo``, ``worktree_repo_path``, and - # the pipeline base branch exactly as before — no extra contract read. - slice_repo = pipeline.repo - slice_repo_path = worktree_repo_path - slice_repos = repos - slice_base_branch: str | None = None - if slice_id and len(getattr(pipeline, "repos", None) or []) > 1: - from egg_contracts.loader import load_contract - - slice_obj = None - try: - _contract = load_contract(pipeline_id, worktree_repo_path) - slice_obj = next((s for s in _contract.slices if s.id == slice_id), None) - except Exception as contract_err: # noqa: BLE001 - # Best-effort: a contract load/parse failure degrades to the - # pipeline-primary repo (today's behaviour), it does not block - # the spawn. The slice still runs, just against the primary. - logger.warning( - "Slice-repo scoping: contract load failed; using pipeline primary repo (#3393)", - pipeline_id=pipeline_id, - slice_id=slice_id, - error=str(contract_err), - ) - - # Single gate-repo accessor (shared with the tester's task-6-2 - # TestSliceGateRepoAccessor): the repo the whole slice team scopes to. - resolved = _resolve_slice_gate_repo(slice_obj, pipeline) if slice_obj else None - if resolved and resolved != pipeline.repo: - slice_repo = resolved - slice_repo_path = _resolve_slice_worktree_path(pipeline, resolved, worktree_repo_path) - # Per-repo base branch from the pipeline's RepoSpec list. - for spec in pipeline.repos or []: - if getattr(spec, "repo", None) == resolved: - slice_base_branch = getattr(spec, "base_branch", None) - break - # Order the slice's repo first so the spawner treats it as the - # effective repo for this per-slice team (cwd / EGG_REPO_PATH). - # ``repo_volumes`` already carries every repo owner/repo-keyed - # (slice-3), so only the ordering changes here. - slice_repos = [resolved, *[r for r in repos if r != resolved]] - logger.info( - "Slice scoped to secondary repo (#3393 task-6-1)", - pipeline_id=pipeline_id, - slice_id=slice_id, - slice_repo=slice_repo, - slice_worktree=str(slice_repo_path), - ) - - # Resolve base branch for diff commands in agent prompts. Prefer the - # slice repo's own base (its RepoSpec.base_branch) over the pipeline - # singleton, then fall back to auto-detecting the default branch in the - # slice's worktree (#3393 task-6-1). For N=1 this is the pipeline base / - # pipeline worktree exactly as before. - _resolved_base_branch = slice_base_branch or pipeline.base_branch - if not _resolved_base_branch: - try: - _resolved_base_branch = get_default_branch(slice_repo_path) - except Exception: - _resolved_base_branch = None - - # A producer with no work in this slice is no longer pre-seeded (#3027 - # retired the #2581 pre-seed). It stays spawned and, if it finds it has - # nothing to contribute, submits a generic no-op propose - # (``no_changes_needed=true``) — the prompts below tell every producer - # about that path. The consensus protocol accepts the no-op durably, so - # no orchestrator-side roster pre-classification is needed. - agent_prompts: dict[AgentRole, str] = {} - for role in roles: - prompt = _build_agent_prompt( - role_value=role.value, - phase=phase_str, - pipeline_id=pipeline_id, - pipeline_mode=pipeline_mode, - prompt=pipeline.prompt, - issue_number=pipeline.issue_number, - # Slice-scoped repo / worktree (#3393 task-6-1): drives the - # tester's ``get_repo_checks`` (per-repo configured checks), - # the role file-boundary patterns, and the reviewer diff base — - # all resolve from the slice's repo, not the pipeline primary. - # N=1 ⇒ these equal ``pipeline.repo`` / ``worktree_repo_path``. - repo=slice_repo, - branch=pipeline.branch, - base_branch=_resolved_base_branch, - repo_path=str(slice_repo_path), - concurrent=True, - review_feedback=review_feedback, - network_mode=gateway_mode, - operator_directives=operator_directives, - iteration_history=iteration_history, - ) - agent_prompts[role] = prompt - - # Create spawn function and executor. - spawn_fn = spawner.create_concurrent_spawn_fn( - pipeline_id=pipeline_id, - issue_number=pipeline.issue_number, - repo_volumes=repo_volumes, - mode=gateway_mode, - # Slice's repo first (#3393 task-6-1): the spawner derives the agent - # cwd / EGG_REPO_PATH from the primary (first) repo, so ordering the - # slice's repo first sets the working directory to that repo's - # worktree. N=1 / primary-repo slices leave ``repos`` unchanged. - repos=slice_repos, - phase=phase_str, - sandbox_env=sandbox_env, - certs_volume=certs_volume, - # Pass the *resolved* base branch (above) rather than the raw - # ``pipeline.base_branch`` so a ``None`` (auto-detect) base still - # reaches the spawner as a concrete branch name. The spawner exports - # it as ``EGG_BASE_BRANCH`` for the BRC event-pump's per-producer - # ``git log --not origin/<base>`` delta (#2967); without a concrete - # value the wrapper + composer fall back to ``origin/main`` and the - # delta errors out on every non-``main`` repo. Worktree creation is - # unaffected: the gateway resolves the same default branch when handed - # ``None``, so resolving one layer up here is equivalent. - base_branch=_resolved_base_branch, - spawn_max_retries=pipeline.config.spawn_max_retries, - spawn_retry_initial_backoff_seconds=pipeline.config.spawn_retry_initial_backoff_seconds, - slice_id=slice_id, - ) - - max_concurrent = getattr(pipeline.config, "max_concurrent_agents", 6) - # #3064 slice-3: in orchestrator-ownership mode the event loop watches - # one-shot Job termination to drive failure supervision (backoff / - # respawn / OVERSEER_ALERT). Hand it a Job-status observer when the - # spawner can provide one (the kubernetes spawner); spawners without it - # leave supervision observation dormant (pod mode is unaffected either way). - event_status_view = None - _make_status_view = getattr(spawner, "create_event_job_status_view", None) - if callable(_make_status_view): - event_status_view = _make_status_view() - executor = ConcurrentPhaseExecutor( - pipeline=pipeline, - spawn_fn=spawn_fn, - max_concurrent=max_concurrent, - review_graph=filtered_graph, - roles=roles, - slice_id=slice_id, - event_status_view=event_status_view, - ) - - # Spawn all agents with their prompts. - executions = executor.spawn_all(agent_prompts=agent_prompts) - - # Phase-level retry for transient spawn failures (#1879). Per-role - # retries in kubernetes_spawner handle short blips (~7s budget); this - # outer budget bridges longer outages like a gateway cold start by - # respawning only the failed roles while survivors wait idle. BRC can - # not start without the full cohort anyway, so leaving survivors alone - # during the retry window does not risk correctness. - phase_max_retries = getattr(pipeline.config, "phase_spawn_max_retries", 2) - phase_initial_backoff = getattr( - pipeline.config, "phase_spawn_retry_initial_backoff_seconds", 30.0 - ) - _PHASE_RETRY_BACKOFF_MULTIPLIER = 3.0 - for attempt in range(phase_max_retries): - failed = [e for e in executions if e.status.value == "failed"] - if not failed: - break - transient_failed = [e for e in failed if _is_transient_agent_error(e.error)] - if not transient_failed: - # All remaining failures are permanent — retrying would just - # burn the budget for no benefit. - break - - delay = phase_initial_backoff * (_PHASE_RETRY_BACKOFF_MULTIPLIER**attempt) - failed_roles = [e.role for e in failed] - logger.warning( - "Phase-level spawn retry scheduled", - pipeline_id=pipeline_id, - phase=phase_str, - attempt=attempt + 1, - max_attempts=phase_max_retries, - delay_seconds=delay, - failed_roles=[r.value for r in failed_roles], - transient_roles=[e.role.value for e in transient_failed], - ) - time.sleep(delay) - - # Clear any half-created gateway worktree state for failed roles - # so the retry sees a clean slate. Survivors' worktrees use - # different container_ids and are untouched. - for role in failed_roles: - agent_worktree_id = f"{pipeline_id}-{role.value}" - try: - spawner.gateway.delete_worktrees( - container_id=agent_worktree_id, - force=True, - ) - except Exception as clear_err: - logger.warning( - "Failed to clear partial worktree before retry", - pipeline_id=pipeline_id, - agent_worktree_id=agent_worktree_id, - error=str(clear_err), - ) - - retry_executions = executor.spawn_specific_roles(failed_roles, agent_prompts=agent_prompts) - by_role = {e.role: e for e in retry_executions} - executions = [ - by_role.get(e.role, e) if e.status.value == "failed" else e for e in executions - ] - - still_failed = [e for e in executions if e.status.value == "failed"] - logger.info( - "Phase-level spawn retry outcome", - pipeline_id=pipeline_id, - phase=phase_str, - attempt=attempt + 1, - recovered_roles=[ - r.value for r in failed_roles if r not in {e.role for e in still_failed} - ], - still_failed_roles=[e.role.value for e in still_failed], - ) - - # Record spawned containers/agents in pipeline state. - if store is not None: - try: - with get_pipeline_state_lock(pipeline_id): - pip = store.load_pipeline(pipeline_id) - phase_execution = pip.get_phase_execution(PipelinePhase(phase_str)) - for exec_info in executions: - if exec_info.container_id: - spawn_info = exec_info.container_info - if spawn_info is not None: - # Preserve backend-specific fields (pod_name, - # namespace, job_name on k8s) from the spawner - # while overriding the live bookkeeping fields. - container_info = spawn_info.model_copy( - update={ - "status": ContainerStatus.RUNNING, - "started_at": datetime.now(UTC), - "agent_role": exec_info.role, - } - ) - else: - container_info = ContainerInfo( - container_id=exec_info.container_id, - container_name=f"{pipeline_id}-{exec_info.role.value}", - status=ContainerStatus.RUNNING, - started_at=datetime.now(UTC), - agent_role=exec_info.role, - ) - phase_execution.containers.append(container_info) - - agent_state = StateAgentExecution( - role=exec_info.role, - status=( - StateAgentStatus.RUNNING - if exec_info.status == StateAgentStatus.RUNNING - else StateAgentStatus.FAILED - ), - container_id=exec_info.container_id, - started_at=datetime.now(UTC), - slice_id=slice_id, - # Carry the per-agent resolved model through the - # reconstruction (#3174). ``_spawn_agent`` stamps this on - # the in-memory execution, but the persisted record is - # rebuilt from scratch here — without this copy the field - # dead-ends at None and both operator confirmation - # channels (get_status, list_containers), which read from - # persisted state, surface ``resolved_model: null`` for - # every concurrent-phase agent (initial spawn and - # restart_phase respawn alike). - resolved_model=exec_info.resolved_model, - ) - phase_execution.agents.append(agent_state) - store.save_pipeline(pip) - except Exception as track_err: - logger.warning( - "Failed to record concurrent agents in pipeline state", - pipeline_id=pipeline_id, - error=str(track_err), - ) - - # Check for spawn failures before waiting. Stop successfully-spawned - # containers so they don't continue running after the phase is aborted, - # then write their terminal status back to pipeline state so get_status - # agrees with list_containers (kubernetes_monitor won't reconcile a - # non-RUNNING pipeline, so we must finalize here). - spawn_failures = [e for e in executions if e.status.value == "failed"] - if spawn_failures: - survivor_container_ids: set[str] = set() - for e in executions: - if e.container_id and e.status.value != "failed": - survivor_container_ids.add(e.container_id) - try: - spawner.backend.stop_container(e.container_id, timeout=10) - except Exception: - pass - - if store is not None: - try: - with get_pipeline_state_lock(pipeline_id): - pip = store.load_pipeline(pipeline_id) - phase_execution = pip.get_phase_execution(PipelinePhase(phase_str)) - abort_error = "Aborted during spawn-failure cleanup" - now = datetime.now(UTC) - for agent_state in phase_execution.agents: - if ( - agent_state.container_id in survivor_container_ids - and agent_state.status == StateAgentStatus.RUNNING - ): - agent_state.status = StateAgentStatus.FAILED - agent_state.error = abort_error - agent_state.completed_at = now - for container_info in phase_execution.containers: - if ( - container_info.container_id in survivor_container_ids - and container_info.status == ContainerStatus.RUNNING - ): - container_info.status = ContainerStatus.FAILED - container_info.exited_at = now - store.save_pipeline(pip) - except Exception as cleanup_err: - logger.warning( - "Failed to record spawn-failure cleanup in pipeline state", - pipeline_id=pipeline_id, - error=str(cleanup_err), - ) - - raise SpawnFailureError([(e.role.value, e.error) for e in spawn_failures]) - - # Consensus-driven polling loop with container-exit fallback. - # - # The loop periodically checks consensus via executor.check_consensus(). - # When all agents signal READY, the phase completes immediately without - # waiting for containers to exit. If consensus is never reached (timeout - # or all containers exit first), fall back to exit-code-based completion. - active_executions = [e for e in executions if e.container_id] - docker_client = spawner.backend - all_logs: list[str] = [] - has_failures = [False] # Mutable container for closure access - # Lock kept for forward-compat; the polling loop is single-threaded - # after the #1921 refactor but _record_container_exit uses the lock - # and is called from multiple code paths. - _logs_lock = threading.Lock() - - poll_interval = 5 # seconds - raw_timeout = resolve_consensus_timeout_minutes(pipeline.config, phase_str) - consensus_timeout = max(raw_timeout, 1) * 60 # minimum 1 minute - start_time = time.monotonic() - objection_decision_created = False - - # ``run_epoch`` is the authoritative epoch the owning ``_run_pipeline`` - # thread captured at start (#1638). The poll loop uses it to detect a - # ``restart_phase`` (or any restart that bumps ``run_epoch``) that - # superseded this thread (#3315). ``start_time`` is a fresh monotonic - # clock per call, but a parked-then-restarted phase leaves the *old* - # ``_run_concurrent_phase`` thread alive in its poll loop with a - # ``start_time`` from the original phase start; once its ``elapsed`` - # crosses ``consensus_timeout`` it would fire a spurious consensus-timeout - # OVERSEER_ALERT + HITL decision against the freshly-restarted phase. The - # new ``_run_pipeline`` thread owns the pipeline now, so this stale thread - # must bail before escalating. When ``run_epoch`` is not supplied (legacy - # / direct-call callers) the guard is dormant — behaviour is unchanged. - - def _superseded_by_restart() -> bool: - """True if a newer run_epoch means another thread owns this pipeline. - - Reloads pipeline state and compares its ``run_epoch`` against the - epoch this thread runs under. Mirrors the post-return epoch check - (#1638) but runs *inside* the poll loop so a superseded thread stops - polling before it can fire stale escalations. Best-effort: a load - failure returns ``False`` so a transient store hiccup never tears - down a legitimately-running phase. - """ - return _pipeline_superseded_by_restart(store, pipeline_id, run_epoch) - - # Track which containers have exited and their results. - exited_containers: dict[str, ContainerInfo] = {} - - def _record_container_exit(exec_info: StateAgentExecution, final_info: ContainerInfo) -> None: - """Capture logs and update pipeline state for an exited container.""" - container_logs = "" - if final_info.exit_code != 0: - try: - container_logs = docker_client.get_container_logs( - exec_info.container_id, - tail=200, - ) - except Exception: - pass - - with _logs_lock: - # 143 (SIGTERM) is orchestrator-initiated teardown, not a - # failure — match the K8s monitor's classifier (#2210) so - # the two layers don't disagree about what 143 means. - if final_info.exit_code not in (0, 143): - has_failures[0] = True - all_logs.append( - f"--- {exec_info.role.value} (exit={final_info.exit_code}) ---\n{container_logs}" - ) - - if store is not None: - try: - with get_pipeline_state_lock(pipeline_id): - pip = store.load_pipeline(pipeline_id) - pe = pip.get_phase_execution(PipelinePhase(phase_str)) - - for ci in pe.containers: - if ci.container_id == exec_info.container_id: - ci.status = final_info.status - ci.exited_at = final_info.exited_at - ci.exit_code = final_info.exit_code - break - - for agent in pe.agents: - if agent.container_id == exec_info.container_id: - agent.completed_at = datetime.now(UTC) - if final_info.exit_code in (0, 143): - agent.status = StateAgentStatus.COMPLETE - else: - agent.status = StateAgentStatus.FAILED - agent.error = f"Container exited with code {final_info.exit_code}" - break - - # Cap each tail line at 4096 chars: containers that print - # large JSON blobs on one line could otherwise persist - # multi-MB lines into pipeline state on every chatty exit. - last_lines = ( - [ln[:4096] for ln in container_logs.splitlines()[-200:]] - if container_logs - else [] - ) - pe.agent_exits.append( - AgentExitInfo( - role=exec_info.role, - exit_code=final_info.exit_code, - last_lines=last_lines, - terminated_at=datetime.now(UTC), - container_id=exec_info.container_id, - ) - ) - - store.save_pipeline(pip) - except Exception as track_err: - logger.warning( - "Failed to update concurrent agent state", - container_id=exec_info.container_id, - error=str(track_err), - ) - - def _stop_running_containers() -> None: - """Gracefully stop all containers that haven't exited yet.""" - for e in active_executions: - if e.container_id not in exited_containers: - try: - docker_client.stop_container(e.container_id, timeout=30) - except Exception: - pass - - # Import peer_consensus tracker once at function scope so we don't - # re-run import machinery inside _update_agents_complete under the lock. - _get_brc_tracker = None - try: - from peer_consensus import get_peer_consensus_tracker as _get_brc_tracker - except ImportError: - from ..peer_consensus import ( - get_peer_consensus_tracker as _get_brc_tracker, # type: ignore[no-redef] - ) - - def _latest_proposal_ts(_pid: str, _sid: str | None) -> datetime | None: - """Return the latest CONSENSUS_PROPOSE timestamp from the BRC tracker. - - Used by the post-consensus-timeout poll loop (#2245) to rebaseline - the per-iteration budget on producer progress. Returns ``None`` if - the tracker is unavailable, has no proposals, or any lookup raises — - callers treat ``None`` as "no progress signal yet" and proceed - without a rebaseline. - """ - if _get_brc_tracker is None: - return None - try: - _t = _get_brc_tracker(_pid, _sid) - except Exception: - return None - if _t is None: - return None - try: - return _t.get_latest_proposal_timestamp() - except Exception: - return None - - def _update_agents_complete() -> None: - """Mark all running agents as COMPLETE in pipeline state (consensus path).""" - if store is None: - return - try: - with get_pipeline_state_lock(pipeline_id): - pip = store.load_pipeline(pipeline_id) - pe = pip.get_phase_execution(PipelinePhase(phase_str)) - completed_container_ids: set[str] = set() - - # Look up proposal commit SHAs from the BRC tracker so we can - # populate agent.commit (issue #1691). The lookup is slice- - # aware (#2137) — when ``slice_id`` is set the tracker key - # is the nested ``{pipeline_id}/{slice_id}`` form. - _brc = None - if _get_brc_tracker is not None: - try: - _brc = _get_brc_tracker(pipeline_id, slice_id) - except TypeError: - # Older tracker import-shim without slice_id support. - try: - _brc = _get_brc_tracker(pipeline_id) - except Exception: - pass - except Exception: - pass - - # Filter to this slice's agents — without the filter, slice-2 - # BRC completing flips slice-3's still-running agents to - # COMPLETE because they share ``pe.agents`` (#2422). For - # pipeline-level (non-sliced) phases ``slice_id`` is ``None`` - # and we still match all agents whose ``slice_id`` is ``None``. - for agent in pe.agents: - if getattr(agent, "slice_id", None) != slice_id: - continue - if agent.status in (StateAgentStatus.RUNNING, StateAgentStatus.FAILED): - agent.status = StateAgentStatus.COMPLETE - agent.completed_at = datetime.now(UTC) - if agent.container_id: - completed_container_ids.add(agent.container_id) - # Populate commit SHA from the consensus tracker's proposal - # records. Only producers have SHAs; reviewers get "". - if _brc is not None and not agent.commit: - sha = _brc.get_proposal_commit_sha(agent.role.value) - if sha and sha != "RECONSTRUCTED_NO_SHA": - agent.commit = sha - elif sha is None or sha == "RECONSTRUCTED_NO_SHA": - # Diagnostic only (#1911): log when the BRC - # tracker returns null or the - # RECONSTRUCTED_NO_SHA sentinel for a role - # so we can see on real runs whether the - # three-role implement phase - # (coder/tester/documenter) wiring misses - # SHAs. Deliberately no auto-fallback — - # that would mask the real bug. Empty - # string is the expected reviewer default - # (reviewers never propose) — do NOT warn - # for that case or the signal drowns in - # noise. - logger.warning( - "BRC tracker returned no commit sha for completed agent", - pipeline_id=pipeline_id, - phase=phase_str, - role=agent.role.value, - brc_value=sha, - ) - - # Also mark containers as exited so the container monitor - # doesn't find stale RUNNING entries and mark pipeline FAILED. - # See issue #1294. - for ci in pe.containers: - if ( - ci.container_id in completed_container_ids - and ci.status == ContainerStatus.RUNNING - ): - ci.status = ContainerStatus.EXITED - # Synthetic: container will be stopped next, but 0 - # reflects successful consensus completion. - ci.exit_code = 0 - ci.exited_at = datetime.now(UTC) - - # Auto-withdraw any stale consensus-timeout HITL a superseded - # thread opened before this phase converged (#3315 facet c). - # Folded into this already-locked load→save so it costs no - # extra lock and rides every consensus-success path. - _withdrawn = _cancel_consensus_timeout_decisions(pip) - if _withdrawn: - logger.info( - "Auto-withdrew stale consensus-timeout HITL decision(s) on convergence", - pipeline_id=pipeline_id, - phase=phase_str, - withdrawn=_withdrawn, - ) - - store.save_pipeline(pip) - except Exception as track_err: - logger.warning( - "Failed to update agents to COMPLETE after consensus", - pipeline_id=pipeline_id, - error=str(track_err), - ) - - _demoted_agents: set[str] = set() - - # #2243 progress-gate state: log on first defer + first un-defer only - # so the polling loop doesn't spam at every iteration once we cross - # ``consensus_timeout``. - _progress_gate_deferring = False - - # #3426 HITL-gate state: same log-once discipline for the - # operator-gated suspension of the consensus timeout. - _hitl_gate_deferring = False - - while True: - elapsed = time.monotonic() - start_time - - # 0. Bail if a restart superseded this thread (#3315). A parked phase - # that is restarted after the consensus-timeout budget elapsed - # leaves this old thread polling with a stale ``start_time``; the - # new ``_run_pipeline`` thread already owns the pipeline. Exit - # cleanly — stop this executor's event loop so it stops requesting - # one-shot spawns — WITHOUT firing the timeout escalation. Return a - # NON-zero exit so the caller never mistakes this for success and - # advances the phase; the post-return epoch check (#1638) at the - # call site re-confirms the restart and exits the old thread without - # marking the phase FAILED. - if _superseded_by_restart(): - logger.info( - "Phase superseded by restart (run_epoch changed) — exiting stale " - "_run_concurrent_phase thread without escalation", - pipeline_id=pipeline_id, - phase=phase, - slice_id=slice_id, - ) - executor.stop_event_loop() - return 1, "Phase superseded by restart; stale monitor thread exited." - - # 1. Check consensus - try: - consensus = executor.check_consensus() - except Exception as e: - logger.warning( - "Consensus check failed, continuing poll", - pipeline_id=pipeline_id, - error=str(e), - ) - consensus = {"is_complete": False, "has_objections": False, "blocking_agents": []} - - # 2. Consensus reached — stop containers and return - if consensus.get("is_complete"): - # Recover pipeline if externally marked FAILED (issue #1273). - # The container_monitor reconciliation thread may have marked the - # pipeline FAILED while we were polling. Now that consensus is - # confirmed complete, restore the pipeline to RUNNING so stored - # state matches the successful outcome. - # - # NOTE: consensus staleness is acceptable here. The `consensus` - # dict was fetched earlier in this loop iteration and is not - # re-evaluated under the lock. If consensus regressed between - # the outer check and lock acquisition (extremely unlikely), the - # next iteration of this monitoring loop will re-evaluate and - # self-correct. - if store is not None: - try: - _current_pip = store.load_pipeline(pipeline_id) - if _current_pip.status == PipelineStatus.FAILED: - logger.warning( - "Pipeline externally marked FAILED but consensus is complete — recovering", - pipeline_id=pipeline_id, - ) - with get_pipeline_state_lock(pipeline_id): - _current_pip = store.load_pipeline(pipeline_id) - if _current_pip.status == PipelineStatus.FAILED: - _current_pip.status = PipelineStatus.RUNNING - _current_pip.error = None - store.save_pipeline(_current_pip) - except Exception as recovery_err: - logger.warning( - "External FAILED recovery check failed", - pipeline_id=pipeline_id, - error=str(recovery_err), - ) - - if _emit_event is not None: - _emit_event( - EventType.CONSENSUS_REACHED, - pipeline_id, - data={"elapsed_seconds": elapsed}, - ) - logger.info( - "Consensus reached, stopping containers", - pipeline_id=pipeline_id, - elapsed_seconds=round(elapsed, 1), - has_failures=has_failures[0], - ) - _update_agents_complete() - _stop_running_containers() - combined_logs = ( - "\n".join(all_logs) if all_logs else "Consensus reached; phase complete." - ) - # Consensus is the authoritative success signal. When all agents - # have confirmed (is_complete=True), container-level failures - # (e.g. OOM kills that happened *before* the surviving agents - # reached agreement) should not override the consensus result. - # Any pending HITL decisions from handle_agent_failure remain - # active for human review, but the pipeline itself succeeds. - if has_failures[0]: - logger.warning( - "Container failures detected but consensus is complete — treating as success", - pipeline_id=pipeline_id, - has_failures=has_failures[0], - ) - # Orchestrator mode (#3064): tear down the BRC event loop now that - # the slice has converged so it stops requesting one-shot spawns. - # No-op in pod mode. - executor.stop_event_loop() - return 0, combined_logs - - # 3. Handle objections (create HITL decision once). - # The decision is fire-and-forget: resolution is processed by the - # orchestrator's decision queue (outside this function). If the - # human selects "Override objections", the orchestrator updates - # agent readiness, which is picked up by check_consensus() on - # the next poll iteration. "Abort phase" triggers pipeline - # cancellation via a separate control path. - if consensus.get("has_objections") and not objection_decision_created: - decision = _persist_hitl_decision( - pipeline_id, - pipeline, - store, - question="Agent(s) objecting to phase completion. How to proceed?", - options=["Override objections", "Wait for resolution", "Abort phase"], - phase=pipeline.current_phase, - ) - if decision is not None: - objection_decision_created = True - logger.info( - "Objection detected, HITL decision created", - pipeline_id=pipeline_id, - blocking_agents=consensus.get("blocking_agents", []), - ) - - # 3b. RC3: Stall demotion for dual-role agents. - # If a dual-role agent has missed heartbeats for 5+ minutes, - # demote its reviewer edges to ADVISORY so other agents can proceed. - try: - from health_monitor import get_health_monitor - - _hm = get_health_monitor() - if _hm is not None: - try: - from peer_consensus import get_peer_consensus_tracker - except ImportError: - from ..peer_consensus import ( - get_peer_consensus_tracker, # type: ignore[no-redef] - ) - - # Slice-aware tracker lookup (#2137): per-slice trackers - # are namespaced ``{pipeline_id}/{slice_id}`` so the - # stall-demotion check fires against the correct scope. - try: - _brc_tracker = get_peer_consensus_tracker(pipeline_id, slice_id) - except TypeError: - _brc_tracker = get_peer_consensus_tracker(pipeline_id) - if _brc_tracker is not None: - heartbeat_actions = _hm.check_heartbeats() - for hb_action in heartbeat_actions: - stalled_agent = hb_action.get("agent_id", "") - stall_elapsed = hb_action.get("elapsed_seconds", 0) - if ( - stall_elapsed >= 300 - and stalled_agent not in _demoted_agents - and _brc_tracker.graph.is_dual_role(stalled_agent) - ): - try: - _brc_tracker.handle_stall_demotion( - stalled_agent, - reason=f"Missed heartbeats for {stall_elapsed}s", - ) - _demoted_agents.add(stalled_agent) - except Exception as demote_err: - logger.debug( - "Stall demotion skipped", - agent=stalled_agent, - error=str(demote_err), - ) - except Exception as stall_err: - logger.debug( - "Stall demotion check failed", - pipeline_id=pipeline_id, - error=str(stall_err), - ) - - # 4. Non-blocking check for exited containers - for exec_info in active_executions: - if exec_info.container_id in exited_containers: - continue - try: - info = docker_client.get_container_info(exec_info.container_id) - except ( - ContainerNotFoundError, - ContainerOperationError, - PodNotFoundError, - JobOperationError, - ) as e: - logger.warning( - "Container lost during poll", - container_id=exec_info.container_id, - role=exec_info.role.value, - error=str(e), - ) - info = ContainerInfo( - container_id=exec_info.container_id, - container_name=f"{pipeline_id}-{exec_info.role.value}", - status=ContainerStatus.FAILED, - exit_code=-1, - exited_at=datetime.now(UTC), - ) - - if info.status in ( - ContainerStatus.EXITED, - ContainerStatus.FAILED, - ContainerStatus.REMOVED, - ): - exited_containers[exec_info.container_id] = info - _record_container_exit(exec_info, info) - - # Handle non-clean exit as agent failure. 0 = normal, - # 143 = orchestrator-initiated SIGTERM (#2210) — both - # are classified as clean here to match the K8s monitor's - # _classify_exit, so the two layers can't race to write - # contradictory agent.status values. - if info.exit_code not in (0, 143): - # Issue #2806 (Option A): a producer's consensus-wrapper - # exhausting its retry budget is unrecoverable — the - # slice state machine cannot replace a permanently dead - # producer, and the surviving reviewers will heartbeat - # forever waiting on a proposal that will never come. - # Detect this case and short-circuit the polling loop - # with a non-zero return so the caller transitions the - # pipeline (or slice) to FAILED. Reviewer-only deaths - # still flow through ``handle_agent_failure`` because - # peer-review redistribution can recover them. - role_value = exec_info.role.value - if filtered_graph.is_producer(role_value): - # Race window guard: a producer can legitimately - # exit non-zero after CONFIRMED (wrapper cleanup - # crash) — between step 1 (consensus check) and - # step 4 (exit detection) the producer could have - # written CONFIRMED and then died. Re-query - # consensus before hard-failing; if it has - # completed, fall through and let the next - # iteration's step 1/2 return success. - try: - recheck = executor.check_consensus() - except Exception as recheck_err: - logger.warning( - "Producer-death consensus recheck failed", - pipeline_id=pipeline_id, - role=role_value, - error=str(recheck_err), - ) - recheck = {"is_complete": False} - if recheck.get("is_complete"): - logger.info( - "Producer container exited non-zero but consensus already complete — skipping hard-fail", - pipeline_id=pipeline_id, - role=role_value, - exit_code=info.exit_code, - ) - # Consensus completed in the race window before - # the producer's wrapper-cleanup crash. Step 5 - # (or the next iteration's step 1/2) will return - # success; skip handle_agent_failure (reviewer - # recovery path, not applicable to producers). - continue - _emit_producer_death_alert( - pipeline_id=pipeline_id, - role=role_value, - phase=phase_str, - slice_id=slice_id, - exit_code=info.exit_code, - ) - logger.error( - "Producer agent died permanently — failing phase", - pipeline_id=pipeline_id, - phase=phase_str, - slice_id=slice_id, - role=role_value, - exit_code=info.exit_code, - ) - _stop_running_containers() - combined_logs = "\n".join( - all_logs - + [ - "--- PRODUCER PERMANENT DEATH ---", - ( - f"Producer '{role_value}' container exited with code " - f"{info.exit_code} after the consensus-wrapper exhausted " - f"its retry budget. Pipeline failing (issue #2806)." - ), - ] - ) - return 1, combined_logs - try: - executor.handle_agent_failure( - role=role_value, - error=f"Container exited with code {info.exit_code}", - ) - except Exception as e: - logger.warning( - "handle_agent_failure error", - role=role_value, - error=str(e), - ) - else: - # Clean exit (0 or 143): the consensus wrapper inside - # the container handles restarts if the agent didn't - # signal READY. We do NOT auto-register READY here — - # agents must explicitly participate in consensus. - logger.info( - "Container exited cleanly, wrapper handles consensus", - pipeline_id=pipeline_id, - role=exec_info.role.value, - exit_code=info.exit_code, - ) - - # 5. All containers exited — fall back to exit-code-based result. - # - # Guarded on a non-empty ``active_executions`` so an empty set is - # never misread as "everything exited" (``0 >= 0``). In orchestrator - # mode (#3064) ``spawn_all`` returns ``[]`` by design — the - # orchestrator owns the BRC loop and spawns one-shot pods per event, - # so there are no up-front containers to track. Completion is driven - # purely off ``check_consensus()`` (step 2) and the consensus timeout - # (step 6); a zero-container fallback here would otherwise fail the - # phase on the first poll, before any event-driven pod ran. - if active_executions and len(exited_containers) >= len(active_executions): - combined_logs = "\n".join(all_logs) - if has_failures[0]: - # Final consensus recheck: consensus may have completed between - # the step-2 check and now (race window while containers were - # shutting down). Re-query before giving up. - try: - final_consensus = executor.check_consensus() - except Exception as e: - logger.warning( - "Final consensus recheck failed, treating as incomplete", - pipeline_id=pipeline_id, - error=str(e), - ) - final_consensus = {"is_complete": False} - - if final_consensus.get("is_complete"): - # Guard: consensus may be "complete" by quorum but still - # have unresolved NACKs — mirror the step 5 no-failure - # NACK check and the timeout path NACK check. - if final_consensus.get("has_unresolved_nacks"): - nack_details = final_consensus.get("unresolved_nacks", []) - nack_summary = _format_nack_summary(nack_details) - logger.warning( - "Consensus complete on final recheck but unresolved NACKs remain (has_failures path)", - pipeline_id=pipeline_id, - nack_count=len(nack_details), - nack_summary=nack_summary, - ) - # Tag with the consensus-timeout context so "Retry - # phase" dispatches through restart_phase on resolve - # (#3421), for symmetry with the incomplete-consensus - # sites below. This question is hand-built and does not - # promise restart copy, but restart_phase is the correct - # "Retry phase" action regardless. Like its siblings - # this pod-mode path is unreachable today (spawn_all - # returns [] post-#3164, so active_executions is always - # empty); tagging keeps the dispatch honest if pod mode - # is ever revived. - _persist_hitl_decision( - pipeline_id, - pipeline, - store, - question=( - f"Consensus reached but {len(nack_details)} NACK(s) " - f"remain unresolved: {nack_summary}. How to proceed?" - ), - options=["Retry phase", "Accept current state", "Abort phase"], - phase=pipeline.current_phase, - context=_CONSENSUS_TIMEOUT_HITL_CONTEXT, - ) - combined_logs += ( - f"\n--- UNRESOLVED NACKs ({len(nack_details)}) ---\n{nack_summary}" - ) - return 1, combined_logs - - # Consensus reached after all — recover pipeline if needed - if store is not None: - try: - _current_pip = store.load_pipeline(pipeline_id) - if _current_pip.status == PipelineStatus.FAILED: - logger.warning( - "Pipeline externally marked FAILED but consensus is complete — recovering", - pipeline_id=pipeline_id, - ) - with get_pipeline_state_lock(pipeline_id): - _current_pip = store.load_pipeline(pipeline_id) - if _current_pip.status == PipelineStatus.FAILED: - _current_pip.status = PipelineStatus.RUNNING - _current_pip.error = None - store.save_pipeline(_current_pip) - except Exception as recovery_err: - logger.warning( - "External FAILED recovery check failed", - pipeline_id=pipeline_id, - error=str(recovery_err), - ) - - _elapsed_final = time.monotonic() - start_time - if _emit_event is not None: - _emit_event( - EventType.CONSENSUS_REACHED, - pipeline_id, - data={"elapsed_seconds": _elapsed_final}, - ) - logger.info( - "Consensus reached on final recheck, stopping containers", - pipeline_id=pipeline_id, - elapsed_seconds=round(_elapsed_final, 1), - has_failures=has_failures[0], - ) - _update_agents_complete() - _stop_running_containers() - return 0, combined_logs - - # Incomplete consensus + container failures: surface an HITL - # decision so the operator can drive recovery (issue #2203). - # Without this, the phase fails terminally with no signal — - # the agent's committed work is still on the per-role branch - # and `restart_phase` would recover, but the operator has no - # way to know that without out-of-band investigation. - # - # If an objection HITL was created earlier in the polling loop - # this is intentionally a *second* pending decision: it - # carries different options ("Retry phase" / "Accept current - # state" / "Abort phase" vs the objection set) and conveys a - # different operator action. The test - # `test_objection_dedup_distinct_from_incomplete_consensus_hitl` - # locks in the two-decision UX. - failure_count = sum(1 for info in exited_containers.values() if info.exit_code != 0) - question, log_suffix = _incomplete_consensus_decision_text( - final_consensus, container_failure_count=failure_count - ) - logger.warning( - "Incomplete consensus with container failures — escalating to HITL", - pipeline_id=pipeline_id, - failure_count=failure_count, - blocking_agents=final_consensus.get("blocking_agents", []), - nack_count=len(final_consensus.get("unresolved_nacks", []) or []), - ) - # Tag with the consensus-timeout context so "Retry phase" - # dispatches through restart_phase on resolve (#3421), matching - # the restart semantics `_incomplete_consensus_decision_text` - # promises. This pod-mode container-exit path is unreachable - # today (spawn_all returns [] post-#3164, so active_executions - # is always empty), but tagging keeps the copy honest if pod - # mode is ever revived. - _persist_hitl_decision( - pipeline_id, - pipeline, - store, - question=question, - options=["Retry phase", "Accept current state", "Abort phase"], - phase=pipeline.current_phase, - context=_CONSENSUS_TIMEOUT_HITL_CONTEXT, - ) - combined_logs += log_suffix - return 1, combined_logs - - # Before returning success, check the BRC approval matrix for - # unresolved NACKs. If reviewers NACKed but producers exited - # without iterating, we must NOT report success — escalate to - # HITL so a human can decide how to proceed. - if consensus.get("has_unresolved_nacks"): - nack_details = consensus.get("unresolved_nacks", []) - nack_summary = _format_nack_summary(nack_details) - logger.warning( - "All containers exited with unresolved NACKs", - pipeline_id=pipeline_id, - nack_count=len(nack_details), - nack_summary=nack_summary, - ) - # Same as the unresolved-NACK site above: tag with the - # consensus-timeout context so "Retry phase" dispatches through - # restart_phase (#3421) for symmetry. Hand-built question, no - # restart copy, but restart_phase is the right action here too. - # Dead pod-mode path today; tagging is cheap insurance. - _persist_hitl_decision( - pipeline_id, - pipeline, - store, - question=( - f"All agents exited but {len(nack_details)} NACK(s) remain " - f"unresolved: {nack_summary}. How to proceed?" - ), - options=["Retry phase", "Accept current state", "Abort phase"], - phase=pipeline.current_phase, - context=_CONSENSUS_TIMEOUT_HITL_CONTEXT, - ) - combined_logs += f"\n--- UNRESOLVED NACKs ({len(nack_details)}) ---\n{nack_summary}" - return 1, combined_logs - - # Final consensus completeness check: all containers exited - # cleanly (no failures, no NACKs) but consensus may not have - # been reached. Mirror the has_failures branch pattern to - # prevent advancing without confirmed BRC consensus. - try: - final_consensus = executor.check_consensus() - except Exception as e: - logger.warning( - "Final consensus recheck failed on clean exit, treating as incomplete", - pipeline_id=pipeline_id, - error=str(e), - ) - final_consensus = {"is_complete": False} - - if not final_consensus.get("is_complete"): - # Symmetric to the has_failures path: clean exits with no - # consensus also need an HITL decision so the operator can - # drive recovery (issue #2203). - question, log_suffix = _incomplete_consensus_decision_text( - final_consensus, container_failure_count=0 - ) - logger.warning( - "All containers exited cleanly but consensus not reached — escalating to HITL", - pipeline_id=pipeline_id, - elapsed_seconds=round(elapsed, 1), - blocking_agents=final_consensus.get("blocking_agents", []), - ) - # Same as the container-failure path above: tag with the - # consensus-timeout context so "Retry phase" dispatches through - # restart_phase (#3421) and honors the restart copy. Also a - # dead pod-mode path today; tagging is cheap insurance. - _persist_hitl_decision( - pipeline_id, - pipeline, - store, - question=question, - options=["Retry phase", "Accept current state", "Abort phase"], - phase=pipeline.current_phase, - context=_CONSENSUS_TIMEOUT_HITL_CONTEXT, - ) - combined_logs += log_suffix - return 1, combined_logs - - # Consensus confirmed on clean exit — mirror the has_failures - # success path: emit event, update agent state, stop containers. - if _emit_event is not None: - _emit_event( - EventType.CONSENSUS_REACHED, - pipeline_id, - data={"elapsed_seconds": elapsed}, - ) - logger.info( - "Consensus reached on final recheck, stopping containers", - pipeline_id=pipeline_id, - elapsed_seconds=round(elapsed, 1), - has_failures=has_failures[0], - ) - _update_agents_complete() - _stop_running_containers() - return 0, combined_logs - - # 6. Consensus timeout - if elapsed >= consensus_timeout: - # #3426 HITL gate: while an unresolved operator HITL decision - # (contract ``cq-N``) gates the running phase, the slice is - # provably operator-gated — a reviewer withholding its ACK - # pending a human ruling is the system working as designed, not - # a convergence failure. Suspend the timeout (keep polling, no - # alert, no failure) until the operator answers. On release, - # reset the convergence clock so the agents folding in the - # resolution get a full fresh window instead of a clock that - # already expired while the human was thinking. - _hitl_ids = _unresolved_contract_hitl_ids(pipeline_id, pipeline, phase_str) - if _hitl_ids: - if not _hitl_gate_deferring: - logger.info( - "Consensus timeout suspended — phase is operator-gated " - "on unresolved HITL decision(s)", - pipeline_id=pipeline_id, - slice_id=slice_id, - elapsed_seconds=round(elapsed, 1), - decision_ids=_hitl_ids, - ) - _hitl_gate_deferring = True - time.sleep(poll_interval) - continue - if _hitl_gate_deferring: - _hitl_gate_deferring = False - start_time = time.monotonic() - logger.info( - "Consensus timeout clock reset — operator HITL decision(s) resolved", - pipeline_id=pipeline_id, - slice_id=slice_id, - suspended_after_seconds=round(elapsed, 1), - ) - continue - - # #2243 progress gate: keep polling instead of publishing - # the consensus-timeout alert while producer/reviewer - # activity is still live on the BRC bus or in container - # heartbeats. Without this gate, the historical decision-15 - # / decision-17 misfires on ``issue-1557-v2`` (now - # ``OVERSEER_ALERT`` post-#2264) fired minutes before the - # next commit landed. - _gate_seconds = max( - 0, - int(getattr(pipeline.config, "brc_consensus_progress_gate_seconds", 300)), - ) - _gate_defer, _gate_reason = _check_brc_progress_gate( - pipeline_id, - slice_id, - [e.role.value for e in active_executions], - _gate_seconds, - ) - if _gate_defer: - if not _progress_gate_deferring: - logger.info( - "Consensus timeout deferred by progress gate", - pipeline_id=pipeline_id, - elapsed_seconds=round(elapsed, 1), - gate_seconds=_gate_seconds, - reason=_gate_reason, - ) - _progress_gate_deferring = True - time.sleep(poll_interval) - continue - if _progress_gate_deferring: - logger.info( - "Consensus timeout proceeding — progress gate window elapsed", - pipeline_id=pipeline_id, - elapsed_seconds=round(elapsed, 1), - gate_seconds=_gate_seconds, - ) - _progress_gate_deferring = False - - # #3490 live-widening gate: re-resolve the budget from freshly - # loaded config so a PATCH /config update of - # ``consensus_timeout_minutes*`` takes effect any time before the - # wall fires; an operator watching a giant slice can widen the - # window without letting the slice fail and restarting. Checked - # here, after the HITL and progress gates, so the load only - # happens once per firing rather than on every deferred poll. A - # load failure keeps the current budget: a transient store hiccup - # must never widen or shrink the window on its own. - if store is not None: - _fresh_minutes: int | None = None - try: - _fresh_config = store.load_pipeline(pipeline_id).config - _fresh_minutes = resolve_consensus_timeout_minutes(_fresh_config, phase_str) - except Exception as _reresolve_err: - logger.warning( - "Consensus-timeout config re-resolve failed; keeping current budget", - pipeline_id=pipeline_id, - error=str(_reresolve_err), - ) - # The isinstance guard keeps a malformed store payload (or a - # test double) from replacing the numeric budget. - if isinstance(_fresh_minutes, int) and not isinstance(_fresh_minutes, bool): - _fresh_timeout = max(_fresh_minutes, 1) * 60 - if _fresh_timeout != consensus_timeout: - logger.info( - "Consensus timeout budget updated from live config", - pipeline_id=pipeline_id, - slice_id=slice_id, - old_timeout_minutes=consensus_timeout / 60, - new_timeout_minutes=_fresh_timeout / 60, - ) - consensus_timeout = _fresh_timeout - if elapsed < consensus_timeout: - time.sleep(poll_interval) - continue - - logger.warning( - "Consensus timeout reached, falling back to container exit", - pipeline_id=pipeline_id, - timeout_minutes=consensus_timeout / 60, - ) - # Orchestrator mode (#3064): we are giving up on convergence, so - # stop the BRC event loop before the fallback wait so it does not - # keep spawning one-shot pods past the deadline. No-op in pod - # mode. (The progress-gate ``continue`` above is taken before - # this point, so a deferral never reaches here and the loop keeps - # running across the deferral window.) - executor.stop_event_loop() - _handle_brc_consensus_timeout( - pipeline, - pipeline_id, - consensus_timeout, - consensus.get("blocking_agents", []), - store, - slice_id=slice_id, - active_role_names=[e.role.value for e in active_executions], - ) - - # Fall back: event-driven wait for remaining containers. - # - # Issue #1921: the previous implementation used a - # ThreadPoolExecutor with a blocking - # wait_for_container(timeout=3600) per container. During - # that hour the polling loop was blind to BRC progress — - # a NACK → re-propose → ACK cycle completing in the final - # minute could still be force-killed. Now we poll - # container status in short steps and re-check consensus - # between steps, early-returning on completion before - # force-killing anything. - # - # Issue #2245: the per-iteration budget rebaselines on - # producer progress. Each new CONSENSUS_PROPOSE (initial - # or NACK→re-propose) resets ``last_progress_at`` so the - # producer's next iteration gets a clean clock instead of - # inheriting the prior iterations' wall-clock spend. An - # absolute cap (``post_consensus_max_total_seconds``) - # bounds the total wait so an unbounded propose churn - # can't stall the pipeline indefinitely. - remaining = [e for e in active_executions if e.container_id not in exited_containers] - if remaining: - post_timeout_iteration_budget = ( - pipeline.config.post_consensus_iteration_budget_seconds - ) - post_timeout_max_total = pipeline.config.post_consensus_max_total_seconds - post_timeout_poll_interval = 30 # seconds between checks - post_timeout_start = time.monotonic() - last_progress_at = post_timeout_start - - # Snapshot the latest proposal timestamp at entry so we - # only count *new* proposals as progress signals. ``None`` - # is fine: the rebaseline check at the bottom of the loop - # short-circuits on ``last_seen_proposal_ts is None`` - # before any datetime comparison runs. - last_seen_proposal_ts = _latest_proposal_ts(pipeline_id, slice_id) - - while remaining: - now_monotonic = time.monotonic() - total_elapsed = now_monotonic - post_timeout_start - iteration_elapsed = now_monotonic - last_progress_at - if total_elapsed >= post_timeout_max_total: - logger.warning( - "Post-consensus-timeout absolute cap reached", - pipeline_id=pipeline_id, - total_elapsed_seconds=round(total_elapsed, 1), - max_total_seconds=post_timeout_max_total, - ) - break - if iteration_elapsed >= post_timeout_iteration_budget: - logger.warning( - "Post-consensus-timeout iteration budget exhausted", - pipeline_id=pipeline_id, - iteration_elapsed_seconds=round(iteration_elapsed, 1), - iteration_budget_seconds=post_timeout_iteration_budget, - total_elapsed_seconds=round(total_elapsed, 1), - ) - break - - # A. Re-check consensus; if agents converged during - # the wait, stop containers and return success - # before force-killing them. - try: - _wait_consensus = executor.check_consensus() - except Exception as _wait_consensus_err: - logger.warning( - "Consensus recheck during post-timeout wait failed", - pipeline_id=pipeline_id, - error=str(_wait_consensus_err), - ) - _wait_consensus = None - - if ( - _wait_consensus - and _wait_consensus.get("is_complete") - and not _wait_consensus.get("has_unresolved_nacks") - ): - combined_logs = "\n".join(all_logs) - _total_elapsed = time.monotonic() - start_time - if _emit_event is not None: - _emit_event( - EventType.CONSENSUS_REACHED, - pipeline_id, - data={"elapsed_seconds": _total_elapsed}, - ) - logger.info( - "Consensus reached during post-timeout wait", - pipeline_id=pipeline_id, - elapsed_post_timeout_seconds=round(total_elapsed, 1), - total_elapsed_seconds=round(_total_elapsed, 1), - ) - _update_agents_complete() - _stop_running_containers() - return 0, combined_logs - - # A'. Rebaseline the iteration clock on producer - # progress (#2245). A fresh CONSENSUS_PROPOSE - # timestamp means a producer just landed work - # (initial propose or NACK→re-propose) — the next - # round of reviews deserves its own iteration - # budget, not whatever's left of the prior round's. - current_proposal_ts = _latest_proposal_ts(pipeline_id, slice_id) - if current_proposal_ts is not None and ( - last_seen_proposal_ts is None or current_proposal_ts > last_seen_proposal_ts - ): - logger.info( - "Post-consensus-timeout clock rebaselined on producer progress", - pipeline_id=pipeline_id, - iteration_elapsed_seconds=round(iteration_elapsed, 1), - total_elapsed_seconds=round(total_elapsed, 1), - proposal_timestamp=current_proposal_ts.isoformat(), - ) - last_seen_proposal_ts = current_proposal_ts - last_progress_at = time.monotonic() - - # B. Non-blocking container status check; record - # any that have exited naturally. - still_running = [] - for exec_info in remaining: - try: - info = docker_client.get_container_info(exec_info.container_id) - except ( - ContainerNotFoundError, - ContainerOperationError, - PodNotFoundError, - JobOperationError, - ) as _wait_status_err: - logger.warning( - "Container lost during post-timeout wait", - container_id=exec_info.container_id, - role=exec_info.role.value, - error=str(_wait_status_err), - ) - info = ContainerInfo( - container_id=exec_info.container_id, - container_name=f"{pipeline_id}-{exec_info.role.value}", - status=ContainerStatus.FAILED, - exit_code=-1, - exited_at=datetime.now(UTC), - ) - - if info.status in ( - ContainerStatus.EXITED, - ContainerStatus.FAILED, - ContainerStatus.REMOVED, - ): - exited_containers[exec_info.container_id] = info - _record_container_exit(exec_info, info) - else: - still_running.append(exec_info) - - remaining = still_running - if not remaining: - break - - time.sleep(post_timeout_poll_interval) - - # Budget exhausted with containers still running — - # force-kill so they don't orphan (issue #1691). - for exec_info in remaining: - try: - docker_client.stop_container(exec_info.container_id, timeout=30) - except Exception: - pass - final_info = ContainerInfo( - container_id=exec_info.container_id, - container_name=f"{pipeline_id}-{exec_info.role.value}", - status=ContainerStatus.FAILED, - exit_code=-1, - exited_at=datetime.now(UTC), - ) - exited_containers[exec_info.container_id] = final_info - _record_container_exit(exec_info, final_info) - - combined_logs = "\n".join(all_logs) - if has_failures[0]: - # Consensus recheck: consensus may have completed right as the - # post-timeout budget elapsed and containers were force-killed - # (issue #1691). The in-loop consensus check covers the common - # case; this recheck catches the narrow race where consensus - # completed between the last in-loop check and force-kill. - try: - _timeout_consensus = executor.check_consensus() - except Exception as e: - logger.warning( - "Consensus recheck after timeout failed, treating as incomplete", - pipeline_id=pipeline_id, - error=str(e), - ) - _timeout_consensus = {"is_complete": False} - - if _timeout_consensus.get("is_complete"): - # Guard: consensus may be "complete" by quorum but still - # have unresolved NACKs — mirror the step 5 NACK check. - if _timeout_consensus.get("has_unresolved_nacks"): - nack_details = _timeout_consensus.get("unresolved_nacks", []) - nack_summary = _format_nack_summary(nack_details) - logger.warning( - "Consensus complete on timeout recheck but unresolved NACKs remain", - pipeline_id=pipeline_id, - nack_count=len(nack_details), - nack_summary=nack_summary, - ) - # Tag with the consensus-timeout context so "Retry - # phase" dispatches through restart_phase on resolve - # (#3421), for symmetry with the incomplete-consensus - # sites above. This question is hand-built and does not - # promise restart copy, but restart_phase is the correct - # "Retry phase" action regardless. Like its siblings - # this pod-mode path is unreachable today: has_failures[0] - # is only set in _record_container_exit, called solely for - # active_executions / remaining members, which are always - # empty in orchestrator mode (spawn_all returns [] - # post-#3164). Tagging keeps the dispatch honest if pod - # mode is ever revived. - _persist_hitl_decision( - pipeline_id, - pipeline, - store, - question=( - f"Consensus reached after timeout but {len(nack_details)} NACK(s) " - f"remain unresolved: {nack_summary}. How to proceed?" - ), - options=["Retry phase", "Accept current state", "Abort phase"], - phase=pipeline.current_phase, - context=_CONSENSUS_TIMEOUT_HITL_CONTEXT, - ) - combined_logs += ( - f"\n--- UNRESOLVED NACKs ({len(nack_details)}) ---\n{nack_summary}" - ) - return 1, combined_logs - - # Consensus reached during the wait — recover pipeline - if store is not None: - try: - _current_pip = store.load_pipeline(pipeline_id) - if _current_pip.status == PipelineStatus.FAILED: - logger.warning( - "Pipeline externally marked FAILED but consensus is complete — recovering (timeout path)", - pipeline_id=pipeline_id, - ) - with get_pipeline_state_lock(pipeline_id): - _current_pip = store.load_pipeline(pipeline_id) - if _current_pip.status == PipelineStatus.FAILED: - _current_pip.status = PipelineStatus.RUNNING - _current_pip.error = None - store.save_pipeline(_current_pip) - except Exception as recovery_err: - logger.warning( - "External FAILED recovery check failed (timeout path)", - pipeline_id=pipeline_id, - error=str(recovery_err), - ) - - _elapsed_timeout = time.monotonic() - start_time - if _emit_event is not None: - _emit_event( - EventType.CONSENSUS_REACHED, - pipeline_id, - data={"elapsed_seconds": _elapsed_timeout}, - ) - logger.info( - "Consensus reached on recheck after timeout, treating as success", - pipeline_id=pipeline_id, - elapsed_seconds=round(_elapsed_timeout, 1), - has_failures=has_failures[0], - ) - _update_agents_complete() - _stop_running_containers() - return 0, combined_logs - - # Consensus not complete on recheck. Mirror the non-failure - # branch's NACK summary so operators see which reviewer edges - # are still blocking, even when containers had non-zero exits. - if _timeout_consensus.get("has_unresolved_nacks"): - nack_details = _timeout_consensus.get("unresolved_nacks", []) - nack_summary = _format_nack_summary(nack_details) - logger.warning( - "Timeout with unresolved NACKs (has_failures path)", - pipeline_id=pipeline_id, - nack_count=len(nack_details), - ) - combined_logs += ( - f"\n--- UNRESOLVED NACKs ({len(nack_details)}) ---\n{nack_summary}" - ) - return 1, combined_logs - - # After timeout, check the BRC approval matrix for unresolved - # NACKs before declaring success. Producers that exited without - # addressing reviewer feedback should not be treated as passing. - try: - _final_consensus = executor.check_consensus() - except Exception: - logger.warning("Failed to check consensus at timeout", exc_info=True) - _final_consensus = {} - if _final_consensus.get("has_unresolved_nacks"): - nack_details = _final_consensus.get("unresolved_nacks", []) - nack_summary = _format_nack_summary(nack_details) - logger.warning( - "Timeout with unresolved NACKs — returning failure", - pipeline_id=pipeline_id, - nack_count=len(nack_details), - ) - combined_logs += f"\n--- UNRESOLVED NACKs ({len(nack_details)}) ---\n{nack_summary}" - return 1, combined_logs - - # Orchestrator-owned event loop: this timeout fallthrough is the - # dominant non-convergence terminal. spawn_all returns [] by - # design, so step 5's "all containers exited" path is guarded off - # (it requires a non-empty active set) and a slice that never - # converged — producer never proposed, a reviewer pod failed to - # ACK, reviews pending with no NACK — lands here with no NACKs. - # Unlike pod mode, where a clean all-exited phase already routed - # through step 5's is_complete check, nothing upstream has verified - # consensus completeness on this path. Mirror step 5: when the - # orchestrator owns the loop and consensus is incomplete, escalate - # an HITL and fail rather than reporting a non-converged slice as - # success (a bare `return 0` here would advance the phase toward PR - # creation past the BRC consensus gate). - if executor.owns_event_loop() and not _final_consensus.get("is_complete"): - question, log_suffix = _incomplete_consensus_decision_text( - _final_consensus, container_failure_count=0, orchestrator_mode=True - ) - logger.warning( - "Consensus timed out and is incomplete (orchestrator-owned loop) — escalating to HITL", - pipeline_id=pipeline_id, - blocking_agents=_final_consensus.get("blocking_agents", []), - ) - _persist_hitl_decision( - pipeline_id, - pipeline, - store, - question=question, - options=["Retry phase", "Accept current state", "Abort phase"], - phase=pipeline.current_phase, - context=_CONSENSUS_TIMEOUT_HITL_CONTEXT, - ) - combined_logs += log_suffix - return 1, combined_logs - - return 0, combined_logs - - # 7. Sleep before next poll - time.sleep(poll_interval) - - -def _spawn_and_wait( - spawner, - pipeline_id: str, - agent_role: AgentRole, - issue_number: int | None, - repo_volumes: dict[str, str], - gateway_mode: str, - repos: list[str], - phase: str, - sandbox_env: dict[str, str], - sandbox_command: list[str], - timeout: int = 3600, - store=None, - certs_volume: str | None = None, - branch: str | None = None, - extra_mounts: list["MountSpec"] | None = None, # noqa: UP037 - spawn_max_retries: int | None = None, - spawn_retry_initial_backoff_seconds: float | None = None, -) -> tuple[int, str]: - """Spawn a container, wait for it to exit, clean up, return (exit_code, logs). - - If ``store`` is provided, the container is recorded in the phase execution - state so that the status endpoint can report it while it runs. - - The container is launched via the shared ``build_sandbox_config()`` path, - which handles GATEWAY_URL, proxy vars, DNS lockdown, extra_hosts, and - .git shadow mounts automatically. - - Args: - repo_volumes: Mapping of repo_name -> host_path for volume mounts. - Each entry is mounted at /home/egg/repos/<name> in the container, - with .git shadowed by /dev/null bind mounts to force gateway git operations. - certs_volume: Docker named volume for gateway CA certs (mounted at - /shared/certs read-only). If None, certs are not mounted. - spawn_max_retries: Override for spawn retry attempts (None uses spawner default). - spawn_retry_initial_backoff_seconds: Override for initial backoff (None uses spawner default). - - Returns: - (exit_code, container_logs) — logs are captured before cleanup on failure. - """ - from models import ContainerInfo, ContainerStatus, PipelinePhase - - try: - from agent_model_resolution import DEFAULT_AGENT_MODEL - except ImportError: - from ..agent_model_resolution import ( # type: ignore[import-not-found, no-redef] - DEFAULT_AGENT_MODEL, - ) - - retry_kwargs: dict = {} - if spawn_max_retries is not None: - retry_kwargs["spawn_max_retries"] = spawn_max_retries - if spawn_retry_initial_backoff_seconds is not None: - retry_kwargs["spawn_retry_initial_backoff_seconds"] = spawn_retry_initial_backoff_seconds - - # NOTE: this helper only supports the default Anthropic auth path. It - # does not forward ``upstream``/``upstream_model``, so ``spawn_agent_job`` - # falls back to the Anthropic branch and injects the session-token - # placeholder into ``CLAUDE_CODE_OAUTH_TOKEN`` (#2817). It has no - # production callers today (only test references). If this path is ever - # revived for a LiteLLM agent, plumb ``upstream``/``upstream_model`` - # through here — otherwise Claude Code would send ``x-api-key`` (api_key - # auth) while the placeholder lands in the OAuth header, leaving the - # credential header empty and the session unresolvable. - spawned = spawner.spawn_agent_job( - pipeline_id=pipeline_id, - agent_role=agent_role, - issue_number=issue_number, - mode=gateway_mode, - wait_for_gateway=False, - repos=repos, - phase=phase, - extra_env=sandbox_env, - command=sandbox_command, - repo_volumes=repo_volumes, - branch=branch, - extra_mounts=extra_mounts, - jira_ticket=(sandbox_env.get("EGG_JIRA_TICKET") or None), - **retry_kwargs, - ) - - # Record container and agent in phase execution state - if store is not None: - try: - from models import AgentExecution, AgentExecutionStatus - - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - phase_execution = pipeline.get_phase_execution(PipelinePhase(phase)) - - # Track container — preserve backend-specific fields - # (pod_name, namespace, job_name on K8s) from the spawner. - container_info = spawned.container_info.model_copy( - update={ - "status": ContainerStatus.RUNNING, - "started_at": datetime.now(UTC), - "agent_role": agent_role, - } - ) - phase_execution.containers.append(container_info) - - # Track agent execution. - # - # ``slice_id`` is explicitly ``None`` because this helper has - # no production callers today and is reachable only from - # tests that mock-patch it. If a future change resurrects - # this path for a sliced spawn, the caller MUST plumb a - # ``slice_id`` through here — otherwise the new - # ``(role, slice_id)`` walks added in #2422 will not see - # the record. See PR #2435 review thread. - # This helper hard-codes the default Anthropic auth path (see - # the NOTE above ``spawn_agent_job``), so the resolved model is - # always the built-in default alias. Stamp it for parity with - # ``_run_concurrent_phase`` / ``restart_agent`` (#3174) — if this - # test-only path is ever resurrected for production it will not - # silently regress resolved-model visibility. - agent_execution = AgentExecution( - role=agent_role, - status=AgentExecutionStatus.RUNNING, - container_id=spawned.container_info.container_id, - slice_id=None, - started_at=datetime.now(UTC), - resolved_model=DEFAULT_AGENT_MODEL, - ) - phase_execution.agents.append(agent_execution) - - store.save_pipeline(pipeline) - except Exception as track_err: - logger.warning( - "Failed to record container/agent in pipeline state", - container_id=spawned.container_info.container_id[:12], - error=str(track_err), - ) - - backend = spawner.backend - try: - final_info = backend.wait_for_container( - spawned.container_info.container_id, - timeout=timeout, - ) - except ( - ContainerNotFoundError, - ContainerOperationError, - PodNotFoundError, - JobOperationError, - ) as e: - logger.warning( - "Container lost during wait, marking failed", - container_id=spawned.container_info.container_id, - error=str(e), - ) - final_info = ContainerInfo( - container_id=spawned.container_info.container_id, - container_name=spawned.container_info.container_name, - status=ContainerStatus.FAILED, - exit_code=-1, - exited_at=datetime.now(UTC), - ) - - container_logs = "" - if final_info.exit_code != 0: - try: - container_logs = backend.get_container_logs( - spawned.container_info.container_id, - tail=200, - ) - except Exception: - pass - - # Update container and agent status in phase execution - if store is not None: - try: - from models import AgentExecutionStatus - - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - phase_execution = pipeline.get_phase_execution(PipelinePhase(phase)) - - # Update container status - for ci in phase_execution.containers: - if ci.container_id == spawned.container_info.container_id: - ci.status = final_info.status - ci.exited_at = final_info.exited_at - ci.exit_code = final_info.exit_code - break - - # Update agent status - for agent in phase_execution.agents: - if agent.container_id == spawned.container_info.container_id: - agent.completed_at = datetime.now(UTC) - if final_info.exit_code == 0: - agent.status = AgentExecutionStatus.COMPLETE - else: - agent.status = AgentExecutionStatus.FAILED - agent.error = f"Container exited with code {final_info.exit_code}" - break - - store.save_pipeline(pipeline) - except Exception as track_err: - logger.warning( - "Failed to update container/agent status in pipeline state", - container_id=spawned.container_info.container_id[:12], - error=str(track_err), - ) - - # Always clean up the container - try: - spawner.remove_agent_container( - spawned.container_info.container_id, - force=True, - cleanup_session=True, - ) - except Exception as cleanup_err: - logger.warning( - "Failed to clean up container", - container_id=spawned.container_info.container_id[:12], - error=str(cleanup_err), - ) - - return final_info.exit_code, container_logs - - -# Phases that pause for human approval before advancing (HITL gates) -_HITL_GATE_PHASES = {"refine", "plan"} - -# Keywords that indicate human approval at HITL gates -_APPROVE_KEYWORDS = {"approved", "approve", "lgtm", "yes", ""} - -# Bare option labels that indicate "request changes" without actionable feedback -_BARE_OPTION_LABELS = {"request changes", "request_changes"} - - -def _parse_resolution(resolution: str | None) -> tuple[bool, str | None]: - """Parse a HITL phase_gate resolution into (is_approved, feedback). - - Handles both JSON-structured resolutions and legacy bare-string formats. - Used by the AWAITING_HUMAN recovery path in start_pipeline. - - Returns: - (is_approved, feedback): is_approved is True for approve/select/submit_feedback - actions, False for request_changes/change_approach. feedback contains the - revision feedback text (if any) for non-approved resolutions. - """ - if not resolution: - return True, None - - resolution = resolution.strip() - - # JSON-first: try structured payload - try: - payload = json.loads(resolution) - if isinstance(payload, dict) and "action" in payload: - action = payload["action"] - feedback_text = payload.get("feedback", "") or None - - if action in ("approve", "select", "submit_feedback"): - return True, None - elif action in ("request_changes", "change_approach"): - return False, feedback_text - # Unknown action — fall through to legacy matching - except json.JSONDecodeError, TypeError, AttributeError: - pass - - # Legacy bare-string resolution - if resolution.lower() in _APPROVE_KEYWORDS: - return True, None - elif resolution.lower() in _BARE_OPTION_LABELS: - return False, None - elif resolution: - # Free-text feedback — treat as request_changes - return False, resolution - - return True, None - - -# Minimum characters of non-heading content required for a synthesized plan -# draft to be written. This prevents writing near-empty drafts that contain -# only section headings (e.g. when agents produced no meaningful output). -# A short but valid single-section output like "No architectural risks -# identified." is ~40 chars, so 50 provides a small buffer while still -# catching truly empty drafts. -_MIN_PLAN_DRAFT_CONTENT_LENGTH = 50 - - -def _synthesize_plan_draft( - repo_path: Path, - pipeline_id: str, - pipeline_mode: str = "issue", - issue_number: int | None = None, -) -> None: - """Synthesize a plan draft from multi-agent plan outputs. - - In multi-agent plan mode, ARCHITECT and RISK_ANALYST write to - .egg-state/agent-outputs/. TASK_PLANNER writes the plan draft - directly to .egg-state/drafts/{id}-plan.md. This function combines - the remaining agent outputs into the plan draft (if the task_planner - has not already written one) so that _populate_contract_from_plan() - and the HITL gate can find it. - """ - draft_rel = _get_draft_path("plan", issue_number=issue_number, pipeline_id=pipeline_id) - if not draft_rel: - logger.debug( - "No draft path for plan phase, skipping synthesis", - pipeline_id=pipeline_id, - ) - return - - draft_path = repo_path / draft_rel - if draft_path.exists(): - # Draft already written (e.g. by a single-agent run) — don't overwrite. - return - - outputs_dir = repo_path / ".egg-state" / "agent-outputs" - if not outputs_dir.is_dir(): - logger.warning( - "No agent-outputs directory, cannot synthesize plan draft", - pipeline_id=pipeline_id, - ) - return - - # Derive the pipeline identifier for namespaced output filenames. - _synth_id = _pipeline_identifier(issue_number, pipeline_id) - - from egg_contracts.artifact_spec import resolve_artifact_path - - sections: list[str] = [] - # Spec *names* — not bare filenames — so the agent-output path knowledge - # lives only in egg_contracts.artifact_spec (the slice-2 single-source-of- - # truth ratchet covers this reader, not just the prompt builder). - # ``resolve_artifact_path("<name>", id)`` yields the namespaced - # ``.egg-state/agent-outputs/{id}-<file>`` path; the old un-namespaced - # global filename (basename minus the ``{id}-`` prefix) stays the fallback. - agent_specs = [ - ("architect-output", "Architecture Analysis"), - ("architect-slices", "Slice Scaffold"), - ("risk-analyst-output", "Risk Assessment"), - ] - - for spec_name, heading in agent_specs: - prefixed_rel = resolve_artifact_path(spec_name, _synth_id) - global_filename = Path(prefixed_rel).name.removeprefix(f"{_synth_id}-") - # Try prefixed filename first, fall back to old global filename - prefixed_file = repo_path / prefixed_rel - if prefixed_file.exists(): - output_file = prefixed_file - else: - output_file = outputs_dir / global_filename - if not output_file.exists(): - continue - try: - raw = output_file.read_text() - data = json.loads(raw) - # Agent outputs may contain a "content" or "output" key with - # the main text, or may be the full JSON blob. - content = data.get("content") or data.get("output") or json.dumps(data, indent=2) - except json.JSONDecodeError: - # Fall back to raw text if not valid JSON - content = raw - except Exception as e: - logger.warning( - "Failed to read agent output for plan draft", - pipeline_id=pipeline_id, - file=global_filename, - error=str(e), - ) - continue - - # Skip empty or whitespace-only outputs - if not content or not content.strip(): - logger.warning( - "Agent output is empty, skipping from plan draft", - pipeline_id=pipeline_id, - file=global_filename, - ) - continue - - sections.append(f"## {heading}\n\n{content}") - - if not sections: - logger.warning( - "No agent outputs found to synthesize plan draft", - pipeline_id=pipeline_id, - ) - return - - draft_content = "\n\n".join(sections) + "\n" - - # Guard against a draft that has section headings but no real content. - stripped = draft_content - for _, heading in agent_specs: - stripped = stripped.replace(f"## {heading}", "") - if len(stripped.strip()) < _MIN_PLAN_DRAFT_CONTENT_LENGTH: - logger.warning( - "Synthesized plan draft has insufficient content, not writing", - pipeline_id=pipeline_id, - content_length=len(stripped.strip()), - ) - return - - draft_path.parent.mkdir(parents=True, exist_ok=True) - draft_path.write_text(draft_content, encoding="utf-8") - logger.info( - "Synthesized plan draft from agent outputs", - pipeline_id=pipeline_id, - path=str(draft_path), - sections=len(sections), - ) - - -def _slice_gate_block_monolithic_demotion( - worktree_repo_path: Path, - pipeline_id: str, - issue_number: int | None, -) -> "SliceGateMonolithicBlock | None": # noqa: UP037 — forward ref; see docstring - """#2337 defensive recheck for the slice-loop gate. - - Called only when ``contract.slices`` is empty at implement-phase entry. - Returns a :class:`SliceGateMonolithicBlock` when the on-disk plan - draft parses to N>1 slices — the exact contract+plan mismatch that - demoted issue-2261's 15-slice plan to a monolithic slice-1 PR - (#2337). When this fires the implement phase should be marked - FAILED rather than silently routed through ``_run_concurrent_phase``. - - The returned tuple carries the human-readable ``message`` plus the - parsed ``draft_slice_count`` so the caller can emit a dedicated HITL - naming the divergence inline without having to re-parse the message - (#2627 follow-up). The annotation is quoted because - ``SliceGateMonolithicBlock`` is declared further down the module to - keep it grouped with the other #2627 follow-up types. - - Returns ``None`` when: - * The plan draft is missing on local — there's nothing to parse, and - the populator's own ``plan_draft_missing`` warning already covers - that case (with ``source="plan_complete"`` it raises so we wouldn't - reach this gate at all). - * The plan parses to 0 or 1 slice — single-slice/no-slice contracts - legitimately use the monolithic path. - * Plan parsing fails — defensive: don't block on a parser regression, - just log and let the gate fall through to monolithic. - """ - draft_rel = _get_draft_path("plan", issue_number=issue_number, pipeline_id=pipeline_id) - if not draft_rel: - return None - draft_path = worktree_repo_path / draft_rel - if not draft_path.exists(): - return None - try: - from egg_contracts.plan_parser import parse_plan as _parse_plan_for_gate - - plan_text = draft_path.read_text() - parsed = _parse_plan_for_gate(plan_text) - if not parsed.success: - return None - draft_slice_count = len(parsed.to_contract_slices()) - except Exception as parse_err: # noqa: BLE001 - logger.debug( - "Slice-loop gate: draft re-parse failed", - pipeline_id=pipeline_id, - error=str(parse_err), - ) - return None - if draft_slice_count <= 1: - return None - return SliceGateMonolithicBlock( - message=( - f"plan draft parses to {draft_slice_count} slices but contract.slices " - f"is empty — populator silently failed earlier (#2337); refusing to " - f"demote to monolithic implement" - ), - draft_slice_count=draft_slice_count, - ) - - -class PlanDraftMissingOnLocalError(RuntimeError): - """Raised by the natural plan-completion populator path when the plan - draft is missing from the local worktree but present on origin. - - This is the silent-failure mode behind #2337: a multi-slice plan-phase - pipeline whose populator returned without slices because - ``_sync_worktree_with_remote`` left agents' plan-phase commits on - origin. Surfacing as an exception lets the natural call site mark - the pipeline FAILED instead of silently demoting to monolithic - implement. The force-advance call site (#1941) keeps swallowing. - """ - - -class PlanDraftMissingOnLocalAndOriginError(RuntimeError): - """Raised by the natural plan-completion populator path when the plan - draft is missing from BOTH the local worktree and origin. - - Symmetric to :class:`PlanDraftMissingOnLocalError` (#2337) for the - case where the draft was deleted-and-not-replaced rather than left - on origin only. Observed in the wild on issue-1557-v2 (#2627): the - orchestrator's pre-sync state-write commit deleted the draft and - the consolidated-write step never replaced it, leaving the pipeline - to advance to implement with an empty contract and 8 agents - spinning ``WAITING_FOR_EVENT`` for ~45 min. Surfacing as an - exception lets the natural call site mark the pipeline FAILED so - the operator can intervene. The force-advance call site (#1941) - keeps swallowing. - """ - - -class PopulateOutcome(StrEnum): - """Structured discriminator for :func:`_populate_contract_from_plan` outcomes. - - Added in #2627 follow-up: previously the populator returned ``None`` - on every branch (success, draft-missing, parse-failed, etc.), so - callers couldn't tell "populated N>0 tasks" from "silently produced - an empty contract" without re-loading the contract and counting. - The slice-gate guard at implement-phase entry catches the empty - contract case after the orchestrator has already transitioned to - implement, leaving a generic Retry/Accept/Abort HITL that respawns - into the same broken state. A structured outcome lets the - plan-complete and start_phase=implement call sites fail-fast at - the boundary with an actionable HITL inline. - """ - - POPULATED = "populated" - DRAFT_MISSING = "draft_missing" - NO_DRAFT_PATH = "no_draft_path" - PARSE_FAILED = "parse_failed" - EMPTY_RESULT = "empty_result" - CONTRACT_LOAD_FAILED = "contract_load_failed" - EGG_CONTRACTS_UNAVAILABLE = "egg_contracts_unavailable" - FOREST_VIOLATION = "forest_violation" - # #3046 — two slices touch overlapping files with no dependency edge - # between them; rejected at ingestion like a forest violation. - SLICE_OVERLAP_VIOLATION = "slice_overlap_violation" - UNEXPECTED_EXCEPTION = "unexpected_exception" - - -class PopulateProducedEmptyContractError(RuntimeError): - """Raised at the natural plan-completion call site when - :func:`_populate_contract_from_plan_safe` returns a ``PopulateResult`` - whose outcome indicates the populate step did not produce a contract - with tasks the implement-phase agents can act on. - - Two shapes: - - * ``outcome != POPULATED`` — the populator returned a non-success - outcome (``EMPTY_RESULT``, ``PARSE_FAILED``, ``CONTRACT_LOAD_FAILED``, - ``EGG_CONTRACTS_UNAVAILABLE``, ``FOREST_VIOLATION``, - ``UNEXPECTED_EXCEPTION``, ``NO_DRAFT_PATH``). ``DRAFT_MISSING`` at - ``source="plan_complete"`` is pre-empted by - :class:`PlanDraftMissingOnLocalError` / - :class:`PlanDraftMissingOnLocalAndOriginError` so it never reaches - this exception via that path. - * ``outcome == POPULATED`` with ``slice_count == 0`` — the populator - considered itself "changed" (PR metadata populated, or - ``current_phase`` advanced) but produced no slices/tasks, so - implement-phase agents would have nothing to do. This is the - orthogonal silent-corruption shape flagged in #2627's "Additionally - — and orthogonally" paragraph (#2627 review). - - Orthogonal to :class:`PlanDraftMissingOnLocalError` / - :class:`PlanDraftMissingOnLocalAndOriginError` (which fire when the - draft is missing from one or both refs). The slice-gate at - implement-phase entry would catch most of these later, but failing at - the boundary lets the same dedicated HITL fire for both paths. - Force-advance call sites (#1941) keep swallowing — they inspect the - return value but never raise. - """ - - def __init__(self, outcome: PopulateOutcome, slice_count: int = 0) -> None: - if outcome == PopulateOutcome.POPULATED: - # Populator returned "changed=True" but produced no slices/tasks - # (only PR metadata or current_phase advance changed). #2627 - # review's "POPULATED with slice_count == 0" case. - message = ( - "plan populate completed but produced 0 slices/tasks — " - "refusing to advance plan phase with empty contract" - ) - else: - message = ( - f"plan populate produced {outcome.value} outcome — refusing to " - f"advance plan phase with empty contract" - ) - super().__init__(message) - self.outcome = outcome - self.slice_count = slice_count - - -class PopulateResult(NamedTuple): - """Return type of :func:`_populate_contract_from_plan` and its safe wrapper. - - ``slice_count`` and ``task_count`` are populated only on - ``POPULATED`` (zero on every failure outcome). ``FOREST_VIOLATION`` - is observed at the wrapper after catching the inner raise — the - inner function continues to ``raise ForestValidationError`` so - HTTP callers keep their 422 contract. - """ - - outcome: PopulateOutcome - slice_count: int = 0 - task_count: int = 0 - - -class SliceGateMonolithicBlock(NamedTuple): - """Return type of :func:`_slice_gate_block_monolithic_demotion`. - - Carries the human-readable failure message plus the parsed slice - count so callers can emit a structured HITL naming the divergence - inline (#2627 follow-up). Previously the helper returned a bare - ``str`` and the slice count had to be re-parsed from the message, - making the dedicated HITL payload awkward to build. - """ - - message: str - draft_slice_count: int - - -def _populate_result_is_empty_contract(result: PopulateResult) -> bool: - """Return True if a ``PopulateResult`` means the contract is empty/broken. - - Centralizes the fail-fast condition used by the natural plan-complete - handler and the ``start_phase=implement`` safety net. The two - branches it discriminates: - - * ``outcome != POPULATED`` — the populator reported any non-success - outcome (``EMPTY_RESULT``, ``PARSE_FAILED``, ``CONTRACT_LOAD_FAILED``, - ``EGG_CONTRACTS_UNAVAILABLE``, ``FOREST_VIOLATION``, - ``UNEXPECTED_EXCEPTION``, ``DRAFT_MISSING``, ``NO_DRAFT_PATH``). - ``DRAFT_MISSING`` at ``source="plan_complete"`` is pre-empted by - the ``PlanDraftMissing*`` raises in the safe wrapper so it does - not reach this check via that path; the safety net (which calls - the inner directly with no source) does see it here. - * ``outcome == POPULATED`` with ``slice_count == 0`` — the populator - considered itself "changed" (PR metadata populated, or - ``current_phase`` advanced) but produced no slices/tasks, so the - implement-phase agents would have nothing to do. Flagged in the - "Additionally — and orthogonally" paragraph on #2627 and the - review's "POPULATED with slice_count == 0 still silently advances" - observation. - - Extracted so the call-site check is unit-testable without standing - up the full ``_run_pipeline`` integration setup, and so the two - call sites can't drift out of agreement. Re #2627 review. - """ - return result.outcome != PopulateOutcome.POPULATED or result.slice_count == 0 - - -# Recovery options offered by the dedicated empty-contract HITL emitted -# from the slice-gate, start_phase=implement safety net, and plan-complete -# paths. Plain "Retry phase" would respawn into the same empty-contract -# state (#2627 incident); these options map each choice to a concrete -# operator action that actually changes state. -_EMPTY_CONTRACT_HITL_OPTIONS = [ - "Repopulate contract from plan draft and retry", - "Restart plan phase", - "Abort pipeline", -] - - -# Per-reason divergence prose used by :func:`_empty_contract_hitl_question` -# when no parsed-slice count is available. The generic fallback wording -# ("draft is missing, unparseable, or yielded no tasks") was written when -# only ``EMPTY_RESULT`` / ``PARSE_FAILED`` / ``DRAFT_MISSING`` / ``NO_DRAFT_PATH`` -# routed through this HITL. The widened -# :func:`_populate_result_is_empty_contract` check now also routes -# ``FOREST_VIOLATION`` / ``CONTRACT_LOAD_FAILED`` / -# ``EGG_CONTRACTS_UNAVAILABLE`` / ``UNEXPECTED_EXCEPTION`` plus the -# orthogonal ``populated_but_empty_slices`` case through here, where -# the operator would otherwise read a contradictory message: the -# prose says "draft missing/unparseable/yielded no tasks" while -# ``reason=forest_violation`` says the draft parsed fine but the slice -# DAG was rejected (#2627 review). Reasons NOT in this dict -# (``empty_result``, ``parse_failed``, ``draft_missing``, -# ``no_draft_path``, ``plan_draft_missing_on_local``, -# ``plan_draft_missing_on_local_and_origin``) fall through to the -# generic line, which describes them accurately. -_DIVERGENCE_LINE_BY_REASON: dict[str, str] = { - "forest_violation": ( - "contract.slices is empty because the plan slice DAG was rejected as not a forest" - ), - "slice_overlap_violation": ( - "contract.slices is empty because the plan slice DAG was rejected: two or more " - "slices touch overlapping files with no dependency ordering between them (#3046)" - ), - "contract_load_failed": ( - "contract.slices is empty because the parsed contract on disk failed to deserialize" - ), - "egg_contracts_unavailable": ( - "contract.slices is empty because the egg-contracts library could " - "not be imported during populate" - ), - "unexpected_exception": ( - "contract.slices is empty because the populator raised an unexpected exception" - ), - "populated_but_empty_slices": ( - "contract.slices is empty because the populator ran but produced 0 slices/tasks" - ), -} - - -def _empty_contract_hitl_question( - *, - pipeline_id: str, - reason: str, - draft_slice_count: int | None, - gate: str, -) -> str: - """Build the HITL question text naming the empty-contract root cause inline. - - ``pipeline_id`` is interpolated into the recovery URL so operators can - copy it verbatim instead of substituting a literal ``{id}`` placeholder - by hand (#2627 review). ``reason`` is the operator-visible identifier - (typically a :class:`PopulateOutcome` value or the slice-gate's own - discriminator). ``draft_slice_count`` is None when the plan draft - itself could not be parsed (so we can't quote a count). ``gate`` names - the call site that detected the divergence — ``slice_gate`` / - ``start_phase_implement_safety_net`` / ``plan_complete`` — so the - operator sees which guard fired. - - The opening phrase is "Pipeline blocked at {gate}" rather than - "Implement-phase blocked at {gate}": ``gate=plan_complete`` fires while - the *plan* phase is being marked FAILED, before the implement phase is - spawned, so the implement-specific phrasing would read oddly against - ``pipeline.error`` and the phase-execution status (#2627 review). - """ - if draft_slice_count is not None: - divergence_line = ( - f"contract.slices is empty but the on-disk plan draft parses " - f"to {draft_slice_count} slices" - ) - elif reason in _DIVERGENCE_LINE_BY_REASON: - # Reason-aware wording for outcomes whose root cause isn't - # "draft missing/unparseable/empty" — the widened - # :func:`_populate_result_is_empty_contract` check now routes - # ``FOREST_VIOLATION`` / ``CONTRACT_LOAD_FAILED`` / - # ``EGG_CONTRACTS_UNAVAILABLE`` / ``UNEXPECTED_EXCEPTION`` / - # ``POPULATED``-with-zero-slices through this same HITL, where - # the generic "draft missing, unparseable, or yielded no tasks" - # prose would contradict the ``reason=`` field (#2627 review). - divergence_line = _DIVERGENCE_LINE_BY_REASON[reason] - else: - divergence_line = ( - "contract.slices is empty and the plan draft is missing, " - "unparseable, or yielded no tasks" - ) - return ( - f"Pipeline blocked at {gate}: {divergence_line} " - f"(reason={reason}). The sync helper's auto-reconcile path " - f"(#2792) tried to bring the worktree forward before the " - f"populator ran; if you're seeing this, that reconcile either " - f"didn't fire or didn't restore the draft, so pipeline state " - f"and the contract have diverged. Plain restart_phase implement " - f"will respawn into the same broken state. How to proceed?\n" - f"- 'Repopulate contract from plan draft and retry' — run " - f"POST /pipelines/{pipeline_id}/phase/populate-contract, then " - f"restart_phase implement.\n" - f"- 'Restart plan phase' — restart_phase plan to regenerate the " - f"draft from scratch.\n" - f"- 'Abort pipeline' — cancel_task." - ) - - -def _populate_outcome_to_hitl_reason(outcome: PopulateOutcome) -> str: - """Return the empty-contract HITL ``reason`` for a populate outcome. - - Maps a :class:`PopulateOutcome` to the operator-visible ``reason`` - string used by the dedicated empty-contract HITL: - - * ``POPULATED`` → ``"populated_but_empty_slices"`` — the populator - ran but yielded 0 slices/tasks (the orthogonal "draft existed, - populator ran, but produced nothing" case so the HITL doesn't - claim a bare ``"populated"`` reason that contradicts the empty - contract — #2627 review). - * every other outcome → ``outcome.value`` (e.g. ``forest_violation``, - ``contract_load_failed``, ``empty_result``). - - Extracted so both empty-contract call sites — the plan-complete - handler (via :func:`_empty_contract_hitl_reason`) and the - ``start_phase=implement`` safety net — share a single dispatch and - can't drift if a new outcome needs special-cased reason handling - (#2627 review follow-up). - """ - if outcome == PopulateOutcome.POPULATED: - return "populated_but_empty_slices" - return outcome.value - - -# Single source of truth for ForestValidationError.reason → PopulateOutcome -# mapping. Both ``_populate_contract_from_plan_safe`` and the -# ``start_phase=implement`` safety net translate a structural NACK into an -# outcome the empty-contract HITL prose dispatcher (#3046) can key off, so -# centralising the table here keeps the two catch sites from drifting if a -# third reason is added to :class:`ForestValidationError` (forest-shape vs. -# file-overlap-ordering today). Unknown reasons fall back to -# ``FOREST_VIOLATION`` — that's the conservative choice because the operator -# prose for forest violations names the slice DAG generally rather than the -# specific defect, so a new reason without a dedicated outcome still routes -# to actionable (if generic) HITL prose. -_FOREST_REASON_TO_OUTCOME: dict[str, PopulateOutcome] = { - "slice_overlap_violation": PopulateOutcome.SLICE_OVERLAP_VIOLATION, - "forest_violation": PopulateOutcome.FOREST_VIOLATION, -} - - -def _forest_error_to_outcome(err: ForestValidationError) -> PopulateOutcome: - """Map a :class:`ForestValidationError` to the matching populate outcome.""" - return _FOREST_REASON_TO_OUTCOME.get(err.reason, PopulateOutcome.FOREST_VIOLATION) - - -# Recovery options offered by the dedicated plan-preflight HITL emitted -# from the ``start_phase=implement`` safety net (#3100). Plain "Retry -# phase" would respawn into the same metadata-less state; each option -# maps to a concrete operator action that actually changes state. -_PLAN_PREFLIGHT_HITL_OPTIONS = [ - "Fix the plan draft's pr: block and restart implement", - "Restart plan phase", - "Abort pipeline", -] - - -def _plan_preflight_hitl_question( - *, - missing_fields: list[str], - plan_draft_rel: str, -) -> str: - """Build the HITL question for an implement-start pre-flight rejection (#3100). - - Names the missing plan-draft fields inline and maps each recovery - option to its concrete operator action, mirroring - :func:`_empty_contract_hitl_question`'s shape so operators see the - same actionable-decision pattern at both implement-start gates. - """ - fields = ", ".join(missing_fields) - return ( - f"Pipeline blocked at start_phase_implement_plan_preflight: the " - f"plan draft ({plan_draft_rel}) is missing required field(s) " - f"{fields}. The context-PR opener reads contract.pr metadata " - f"from the plan's top-level ``pr:`` block; without it the " - f"work-branch context PR can never open — both runner-side " - f"openers soft-fail with missing_pr_metadata on every slice, " - f"and no advance_phase call runs on the implement-start path " - f"to enforce the #2777 hard-require (#3100). How to proceed?\n" - f"- 'Fix the plan draft's pr: block and restart implement' — add " - f"a top-level ``pr:`` block (title, description, test_plan, " - f"manual_steps) to the draft's ``# yaml-tasks`` fence on the " - f"work branch, then restart_phase implement.\n" - f"- 'Restart plan phase' — restart_phase plan to regenerate the " - f"draft from scratch.\n" - f"- 'Abort pipeline' — cancel_task." - ) - - -def _enforce_implement_start_plan_preflight( - pipeline_id: str, - pipeline: Pipeline, - store: StateStore, - worktree_repo_path: Path, - plan_draft_rel: str, -) -> bool: - """Enforce the #2777 plan pre-flight at the implement-start boundary (#3100). - - The natural plan→implement path runs - :func:`egg_contracts.plan_parser.validate_plan_preflight` at the - ``advance_phase`` REST/MCP site (``routes/phases.py``) and rejects - with a typed 422 when the plan draft lacks the ``pr:`` metadata the - context-PR opener needs. ``start_phase=implement`` submits never - traverse ``advance_phase``, so before #3100 a draft without a - ``pr:`` block sailed straight into the implement phase: every - runner-side opener backstop soft-failed with - ``missing_pr_metadata`` at WARNING level, the slice stack ran with - no context PR, and the operator discovered the gap only by noticing - the PR was absent (observed on pipeline-da68d70c and - pipeline-2d9cc50d, Khan/webapp). - - Runs AFTER the empty-contract gate at the call site, so the - established empty-contract HITL routing (#2627) is unchanged — this - gate fires only when the populate succeeded but the draft lacks the - PR metadata. - - Scope: - - * Remote pipelines only (``pipeline.repo`` or ``pipeline.base_branch`` - set). Local-mode pipelines never open a context PR (the opener's - own local-mode skip in - :func:`_open_context_pr_at_implement_start`), so requiring ``pr:`` - metadata there would fail test pipelines over a PR that would - never exist. - * Infra failures log a WARNING and return False — the gate must - not add a new hard-fail mode for transient errors, and the - populate path's own outcomes already cover an unreadable draft. - Only the two named infra-class exceptions are caught: an - :class:`ImportError` from the ``plan_parser`` import (validator - unavailable on this host) and an :class:`OSError` from the draft - ``read_text`` (file vanished, permission flake). Any other - exception out of :func:`validate_plan_preflight` propagates to - the outer ``_run_pipeline`` Exception handler, mirroring the - ``advance_phase`` site's behaviour — the goal is to never - swallow a real parser bug under a generic "infra" umbrella. - - Returns True when the pipeline was marked FAILED (the caller must - return without spawning implement-phase agents), False when the - pre-flight passed or was legitimately skipped. - """ - if not (pipeline.repo or pipeline.base_branch): - return False - - try: - from egg_contracts.plan_parser import ( - PlanPreflightError, - validate_plan_preflight, - ) - except ImportError as imp_err: - logger.warning( - "Implement-start plan pre-flight: plan_parser import failed " - "(continuing without the gate) (#3100)", - pipeline_id=pipeline_id, - error=str(imp_err), - ) - return False - - try: - plan_text = (worktree_repo_path / plan_draft_rel).read_text() - except OSError as read_err: - logger.warning( - "Implement-start plan pre-flight: failed to read plan draft " - "(continuing without the gate) (#3100)", - pipeline_id=pipeline_id, - error=str(read_err), - ) - return False - - try: - validate_plan_preflight(plan_text) - return False - except PlanPreflightError as preflight_err: - error_msg = ( - "start_phase=implement plan pre-flight failed — plan draft is " - f"missing required field(s) " - f"{', '.join(preflight_err.missing_fields)}: refusing to run " - "the implement phase with no openable context PR (#2777 " - "pre-flight, #3100 implement-start enforcement)" - ) - logger.error( - "OVERSEER_ALERT start_phase_implement_plan_preflight_failed", - pipeline_id=pipeline_id, - missing_fields=preflight_err.missing_fields, - ) - with get_pipeline_state_lock(pipeline_id): - disk_pipeline = store.load_pipeline(pipeline_id) - disk_pipeline.status = PipelineStatus.FAILED - disk_pipeline.error = error_msg - store.save_pipeline(disk_pipeline) - _persist_hitl_decision( - pipeline_id, - disk_pipeline, - store, - question=_plan_preflight_hitl_question( - missing_fields=preflight_err.missing_fields, - plan_draft_rel=plan_draft_rel, - ), - options=list(_PLAN_PREFLIGHT_HITL_OPTIONS), - phase=disk_pipeline.current_phase, - ) - report_pipeline_status( - disk_pipeline, - event_type="pipeline.failed", - message=f"Pipeline failed: {error_msg[:100]}", - ) - _emit_pipeline_event(disk_pipeline, "pipeline.failed") - return True - - -def _empty_contract_hitl_reason( - err: PlanDraftMissingOnLocalError - | PlanDraftMissingOnLocalAndOriginError - | PopulateProducedEmptyContractError, -) -> str: - """Return the ``reason`` field for the empty-contract HITL. - - Dispatches the operator-visible HITL ``reason`` from one of the - three plan-complete fail-loud exceptions: - - * :class:`PlanDraftMissingOnLocalError` → ``plan_draft_missing_on_local`` - * :class:`PlanDraftMissingOnLocalAndOriginError` → - ``plan_draft_missing_on_local_and_origin`` - * :class:`PopulateProducedEmptyContractError` — delegates to - :func:`_populate_outcome_to_hitl_reason` so the outcome → reason - mapping is shared with the ``start_phase=implement`` safety net - (#2627 review). - - Extracted so the plan-complete call site's HITL-reason dispatch - is unit-testable without standing up the full ``_run_pipeline`` - integration setup (#2627 review). - """ - if isinstance(err, PlanDraftMissingOnLocalError): - return "plan_draft_missing_on_local" - if isinstance(err, PlanDraftMissingOnLocalAndOriginError): - return "plan_draft_missing_on_local_and_origin" - return _populate_outcome_to_hitl_reason(err.outcome) - - -def _empty_contract_failure_metadata( - err: PlanDraftMissingOnLocalError - | PlanDraftMissingOnLocalAndOriginError - | PopulateProducedEmptyContractError, -) -> tuple[str, str]: - """Return ``(teardown_reason, log_event)`` for the plan-complete - fail-loud handler in :func:`_run_pipeline`. - - Dispatches on the three #2627 fail-loud exception classes: - - * :class:`PlanDraftMissingOnLocalError` — draft missing from the local - worktree but present on origin (the #2337 silent-failure). - * :class:`PlanDraftMissingOnLocalAndOriginError` — draft missing from - both refs (the #2627 silent-failure). - * :class:`PopulateProducedEmptyContractError` — draft existed but the - populator yielded an empty/broken contract (the orthogonal "draft - existed but populate yielded nothing" failure mode #2627 also - called out). - - Extracted so the dispatch is unit-testable without standing up the - full ``_run_pipeline`` integration setup — a typo that swapped the - branches would otherwise pass the existing populator-helper - tests. Re #2627 review. - - ``log_event`` uses the ``"OVERSEER_ALERT <discriminator>"`` event-name - convention so the plan-complete fail-loud path is visible to the same - log filters operators use for the slice-gate and start_phase - safety-net (#2627 review). The matching pre-raise OVERSEER_ALERTs - (emitted by :func:`_populate_contract_from_plan_safe` for the two - ``PlanDraftMissing*`` cases, and by ``_run_pipeline``'s plan-complete - synthesis for :class:`PopulateProducedEmptyContractError`) use the - same event names so the pre-raise log and the FAILED-cleanup log - share a single discriminator on every branch. - """ - if isinstance(err, PlanDraftMissingOnLocalError): - return ( - "plan draft missing on local", - "OVERSEER_ALERT plan_draft_missing_on_local_but_present_on_origin", - ) - if isinstance(err, PlanDraftMissingOnLocalAndOriginError): - return ( - "plan draft missing on local and origin", - "OVERSEER_ALERT plan_draft_missing_on_local_and_origin", - ) - return ( - f"populate produced {err.outcome.value} outcome", - "OVERSEER_ALERT plan_populate_produced_empty_contract", - ) - - -def _origin_has_plan_draft(repo_path: Path, branch: str, draft_rel: str) -> bool: - """Return True if ``origin/{branch}:{draft_rel}`` resolves locally. - - Uses ``git cat-file -e`` against the local refs to origin (the - immediately preceding ``_sync_worktree_with_remote`` call has already - fetched), so this is a cheap on-disk check, not a network round-trip. - - A False return collapses two cases: origin really doesn't have the - draft, or the ``cat-file`` probe itself failed (transient git error, - timeout, etc.). The natural plan-completion call site treats False - as "definitively missing on origin" and, when local is also missing, - raises :class:`PlanDraftMissingOnLocalAndOriginError` so the pipeline - is marked FAILED rather than advancing to implement with an empty - contract (#2627). This is a deliberate fail-loud choice: a transient - probe failure combined with a missing local draft will fail the - pipeline rather than silently advance. Operators can re-run the - pipeline; silently shipping an empty contract has no recovery path. - """ - try: - result = subprocess.run( - [ - "git", - "-c", - "core.hooksPath=/dev/null", - "-c", - f"safe.directory={repo_path}", - "-C", - str(repo_path), - "cat-file", - "-e", - f"origin/{branch}:{draft_rel}", - ], - capture_output=True, - text=True, - timeout=10, - check=False, - ) - return result.returncode == 0 - except Exception: - return False - - -def _auto_populate_contract_at_implement_start( - worktree_repo_path: Path, - pipeline_id: str, - pipeline_mode: str, - issue_number: int | None, - current_phase: PipelinePhase, - pipeline_branch: str, - *, - gateway: Any, - gateway_mode: str, - base_branch: str | None, -) -> int: - """Attempt to auto-populate an empty contract at implement start (#2915). - - When a pipeline enters the implement phase with zero slices in the - contract, this helper tries to populate it from the plan draft. On - success, commits and pushes the populated contract; on failure, logs - and returns 0 (still empty). - - Returns the number of slices in the contract after the attempt. - - NOTE: restored in slice-4 v4 of #2908 — the slice-4 base merge - (commit 06c5a6cb0) accidentally dropped this function when bringing - slice-1/2/3 work into the coder branch. The orphan import in - ``orchestrator/tests/test_auto_populate_contract.py`` broke - ``pytest --collect-only`` and blocked ``make test`` from running - any tests at all (per tester v3 NACK blocker #1). The function - body matches ``origin/main`` verbatim; the call site at - ``_run_pipeline`` is unchanged. - """ - logger.info( - "Attempting to auto-populate empty contract at implement start (#2915)", - pipeline_id=pipeline_id, - issue_number=issue_number, - ) - try: - _populate_result = _populate_contract_from_plan( - worktree_repo_path, - pipeline_id, - pipeline_mode, - issue_number, - current_phase=current_phase, - ) - except ForestValidationError as _forest_err: - logger.warning( - "Auto-populate contract failed: slice-DAG validation error", - pipeline_id=pipeline_id, - reason=_forest_err.reason, - errors=_forest_err.errors, - ) - return 0 - except Exception as _populate_err: # noqa: BLE001 - logger.warning( - "Auto-populate contract failed at implement start", - pipeline_id=pipeline_id, - error=str(_populate_err), - exc_info=True, - ) - return 0 - - if _populate_result.outcome != PopulateOutcome.POPULATED or _populate_result.slice_count == 0: - logger.warning( - "Auto-populate contract returned empty or failed", - pipeline_id=pipeline_id, - outcome=_populate_result.outcome.value, - slice_count=_populate_result.slice_count, - ) - return 0 - - # Commit the populated contract - try: - _committed = _commit_statefiles_to_worktree( - worktree_repo_path, - "Auto-populate contract at implement start (#2915)", - _pipeline_identifier(issue_number, pipeline_id), - pipeline_id=pipeline_id, - ) - if not _committed: - logger.warning( - "Auto-populate: commit returned False (nothing to commit)", - pipeline_id=pipeline_id, - ) - return 0 - except Exception as _commit_err: # noqa: BLE001 - logger.warning( - "Auto-populate: commit failed", - pipeline_id=pipeline_id, - error=str(_commit_err), - ) - return 0 - - # Push the populated contract. Failure is non-fatal — the contract is - # already committed locally — but mirror the canonical pattern from - # agent_salvage._push_recovery (try/except for transport, then check - # push_result.ok for gateway-reported rejections like non_fast_forward - # / auth_failed / gateway_unreachable). Thread gateway_mode and - # base_branch so private-mode pipelines route correctly and non-FF - # reconcile uses --onto and doesn't replay base-branch commits. - push_succeeded = False - try: - push_result = gateway.push_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(worktree_repo_path), - branch=pipeline_branch, - mode=gateway_mode, - base_branch=base_branch, - ) - except Exception as _push_err: # noqa: BLE001 - logger.warning( - "Auto-populate: push transport failure (non-fatal, contract committed locally)", - pipeline_id=pipeline_id, - error=str(_push_err), - ) - else: - if not push_result.ok: - logger.warning( - "Auto-populate: push rejected by gateway (non-fatal, contract committed locally)", - pipeline_id=pipeline_id, - category=push_result.category, - detail=push_result.detail, - ) - else: - push_succeeded = True - - logger.info( - "Auto-populate contract succeeded" - if push_succeeded - else "Auto-populate contract succeeded locally only (push did not land)", - pipeline_id=pipeline_id, - slice_count=_populate_result.slice_count, - push_succeeded=push_succeeded, - ) - return _populate_result.slice_count - - -def _populate_contract_from_plan_safe( - repo_path: Path, - pipeline_id: str, - pipeline_mode: str = "issue", - issue_number: int | None = None, - *, - source: Literal[ - "plan_complete", - "advance_phase_force", - "hitl_plan_gate_approval", - ] = "advance_phase_force", - branch: str | None = None, - current_phase: PipelinePhase | None = None, -) -> PopulateResult: - """Run :func:`_populate_contract_from_plan` without propagating failures. - - Shared call path for the three code sites that run the populate step - when a pipeline leaves the ``plan`` phase: ``_run_pipeline``'s - post-complete block (``source="plan_complete"``), ``advance_phase`` - (used by the MCP ``advance_phase`` tool, especially with - ``force=true`` — ``source="advance_phase_force"``), and the HITL - plan-gate approval path in :func:`start_pipeline` - (``source="hitl_plan_gate_approval"`` — operator approved the - plan_gate while the pipeline was AWAITING_HUMAN, recovery - re-spawns ``_run_pipeline``). Blocking the phase transition on a - populate failure would defeat the purpose of the advance hammer - or recovery path — see #1941 — so all non-natural call sites - keep the swallow-everything behaviour. - - The natural plan-completion call site (``source="plan_complete"``) - additionally raises: - - * :class:`PlanDraftMissingOnLocalError` when the draft is missing - from local but present on origin — the silent-failure mode - behind #2337. - * :class:`PlanDraftMissingOnLocalAndOriginError` when the draft is missing from - BOTH local and origin — the silent-failure mode behind #2627 - (orchestrator-side delete with no consolidated re-write). - - Caller is expected to mark the pipeline FAILED so the operator can - intervene rather than advancing to implement with an empty - contract. - - Returns a :class:`PopulateResult` so non-raising failure modes are - still inspectable: callers that need to fail-fast on - ``EMPTY_RESULT`` / ``PARSE_FAILED`` (#2627 follow-up) can branch on - the outcome. ``ForestValidationError`` raised by the inner is - caught and translated to ``PopulateResult(FOREST_VIOLATION, 0, 0)``; - any other unexpected exception translates to - ``PopulateResult(UNEXPECTED_EXCEPTION, 0, 0)``. - """ - if source == "plan_complete" and branch is not None: - draft_rel = _get_draft_path("plan", issue_number=issue_number, pipeline_id=pipeline_id) - if draft_rel is not None: - local_path = repo_path / draft_rel - on_local = local_path.exists() - on_origin = _origin_has_plan_draft(repo_path, branch, draft_rel) - if not on_local and on_origin: - logger.error( - "OVERSEER_ALERT plan_draft_missing_on_local_but_present_on_origin", - pipeline_id=pipeline_id, - branch=branch, - draft_rel=draft_rel, - note=( - "_sync_worktree_with_remote returned without bringing " - "agents' plan-phase commits into the local worktree; " - "blocking phase advance to avoid silent demotion to " - "monolithic implement (#2337)" - ), - ) - raise PlanDraftMissingOnLocalError( - f"plan draft {draft_rel} missing on local but present on " - f"origin/{branch} — refusing to advance plan phase" - ) - if not on_local and not on_origin: - logger.error( - "OVERSEER_ALERT plan_draft_missing_on_local_and_origin", - pipeline_id=pipeline_id, - branch=branch, - draft_rel=draft_rel, - note=( - f"plan draft is missing from both the local worktree " - f"and origin/{branch}; advancing would produce an " - f"empty contract and strand implement-phase agents " - f"with nothing to do (#2627)" - ), - ) - raise PlanDraftMissingOnLocalAndOriginError( - f"plan draft {draft_rel} missing on local and " - f"origin/{branch} — refusing to advance plan phase" - ) - - try: - return _populate_contract_from_plan( - repo_path, - pipeline_id, - pipeline_mode, - issue_number, - current_phase=current_phase, - ) - except ForestValidationError as forest_err: - # Slice-DAG structural rejection is the expected #2137 / #3046 - # NACK path — log structurally so the discriminator shows up in - # operator audit, but don't propagate to the wrapper's - # caller (the populator already stashed the structured - # errors on contract.plan_review_feedback so the plan - # reviewer prompt can NACK the architect). The exception's - # ``reason`` selects the matching outcome so operators see an - # accurate discriminator (forest shape vs file-overlap order). - logger.warning( - "contract_phases_ingest_failed", - pipeline_id=pipeline_id, - reason=forest_err.reason, - source="safe_wrapper", - errors=forest_err.errors, - ) - return PopulateResult(_forest_error_to_outcome(forest_err)) - except Exception as pop_err: - logger.warning( - "contract_phases_ingest_failed", - pipeline_id=pipeline_id, - reason="unexpected_exception", - source="safe_wrapper", - error=str(pop_err), - exc_info=True, - ) - return PopulateResult(PopulateOutcome.UNEXPECTED_EXCEPTION) - - -def _merge_preserved_slice_runtime( - new_slices: "list[ContractSlice]", # noqa: UP037 - old_slices: "list[ContractSlice]", # noqa: UP037 -) -> None: - """Carry runtime slice/task state from ``old_slices`` onto ``new_slices`` in place. - - ``_populate_contract_from_plan`` re-parses the plan markdown into a - fresh set of slices on every call — and its safety-net caller fires - on *every* ``start_phase=implement`` restart (deliberately outside - the ``contract_synced`` guard). The plan is the source of truth for - slice/task STRUCTURE (names, descriptions, dependencies, acceptance - criteria); it always parses back as ``PENDING`` with the runtime - bookkeeping fields unset. Blindly assigning ``contract.slices = - <freshly parsed>`` therefore wipes every slice the slice loop had - already advanced — resetting COMPLETE slices to PENDING and dropping - the ``parent_branch_at_creation`` / ``integration_base_sha`` a real - run stamped — so a restarted pipeline re-runs slice-1 forever and can - never reach slice-2 (#2908). - - Mirroring the PR-metadata preservation a few lines down in the - caller, this merges by slice id (and by task id within a slice): the - plan supplies STRUCTURE while RUNTIME state survives a re-populate. - Unmatched ids (a re-plan that adds or removes slices/tasks) simply - keep the plan's fresh ``PENDING`` defaults. - - Task-level runtime fields covered (each is durably written by a - runtime path that the plan parser cannot reconstruct): - - - ``status``, ``commit``, ``checkpoint_id``, ``review_cycles``, - ``escalated``, ``gaps`` — slice-loop / reviewer / tester - bookkeeping. - - ``role`` + ``delegation_attempts`` — paired SYSTEM-owned - impasse-delegation state. ``impasse_routing.py`` flips - ``task.role`` to the suggested alternative and bumps - ``delegation_attempts`` in the same ``apply_mutation`` cycle - under ``Role.SYSTEM`` (only SYSTEM owns these two fields); the - slice-loop dispatcher then routes the task to the new role. - Preserving the counter without the role would re-spawn the - original producer on restart and trip ``DELEGATION_LIMIT`` on - the next impasse, escalating to HITL even though no delegation - visibly happened — so both fields must survive together. - - ``notes`` — APPLIER writes Won't-Do drain failure reasons here - (``pipelines.py`` Won't-Do path) and agents write implementation - narrative via ``mcp__task__update_notes`` / ``egg-contract - update-notes``; the plan parser always emits ``""``. - - ``jira_action_status`` — APPLIER advances ``pending`` → - ``in_flight`` → ``applied``/``failed`` (#1557 risk_analyst R7); - idempotency depends on ``applied`` surviving re-populate so the - next apply skips it instead of re-creating the Jira issue. - - ``jira_key`` — APPLIER writes the freshly-allocated key back after - a ``create`` action so re-runs skip the create; plan parser emits - ``None`` on ``create`` actions, so re-populate would otherwise - strand the applier into creating duplicate tickets. - """ - old_by_id = {s.id: s for s in (old_slices or [])} - for new_slice in new_slices: - old_slice = old_by_id.get(new_slice.id) - if old_slice is None: - continue - # Slice-level runtime state stamped by ``_run_one_slice_inner`` - # and the bootstrap reconciler — never re-derivable from the plan. - new_slice.status = old_slice.status - new_slice.parent_branch_at_creation = old_slice.parent_branch_at_creation - new_slice.integration_base_sha = old_slice.integration_base_sha - new_slice.commit = old_slice.commit - new_slice.review_cycles = old_slice.review_cycles - # Defensive copy so post-merge mutations of the discarded ``old`` - # contract don't alias-leak into the live ``new`` contract. - new_slice.review_feedback = list(old_slice.review_feedback) - new_slice.escalated = old_slice.escalated - new_slice.escalation_reason = old_slice.escalation_reason - # Task-level runtime state: match by task id so a re-plan that - # adds/removes tasks still preserves completion of the survivors. - old_tasks_by_id = {t.id: t for t in old_slice.tasks} - for new_task in new_slice.tasks: - old_task = old_tasks_by_id.get(new_task.id) - if old_task is None: - continue - new_task.status = old_task.status - new_task.commit = old_task.commit - new_task.checkpoint_id = old_task.checkpoint_id - new_task.review_cycles = old_task.review_cycles - new_task.escalated = old_task.escalated - # Paired SYSTEM-owned impasse-delegation state — preserving - # the counter without the role would silently undo the - # delegation on restart (see docstring). - new_task.role = old_task.role - new_task.delegation_attempts = old_task.delegation_attempts - new_task.gaps = list(old_task.gaps) - # Runtime narrative + applier idempotency anchors. The - # plan parser cannot reconstruct any of these — see the - # docstring for the per-field invariants. - new_task.notes = old_task.notes - new_task.jira_action_status = old_task.jira_action_status - new_task.jira_key = old_task.jira_key - - -def _populate_contract_from_plan( - repo_path: Path, - pipeline_id: str, - pipeline_mode: str = "issue", - issue_number: int | None = None, - *, - current_phase: PipelinePhase | None = None, -) -> PopulateResult: - """Read the plan draft and populate the contract with tasks. - - Extracts task structure from markdown headers in the plan draft - and writes tasks + acceptance criteria to the contract. - - Returns a :class:`PopulateResult` whose ``outcome`` discriminates - success from each silent-failure mode (#2627 follow-up). Callers - that need to fail-fast on an empty contract — natural plan-complete - and the ``start_phase=implement`` safety net — branch on ``outcome`` - to surface a dedicated HITL instead of advancing into an implement - phase with nothing to do. ``ForestValidationError`` continues to - raise so HTTP callers keep their structured-422 contract; the - wrapper translates that raise into - ``PopulateResult(FOREST_VIOLATION, 0, 0)``. - - When ``current_phase`` is provided, the contract's - ``current_phase`` is advanced to that value **only if it would move - the phase forward** (REFINE → PLAN → IMPLEMENT → PR). Backward - transitions are silently ignored so a respawn of the safety-net - populator (e.g. when a ``start_phase=implement`` pipeline progresses - to PR and re-enters ``_run_pipeline``) cannot demote the contract. - The advance also appends a ``create_transition_entry`` audit log - entry so operators inspecting the audit trail see the transition. - - This parameter is needed because the natural plan-completion path - advances ``pipeline.current_phase`` (orchestrator-side) but leaves - ``contract.current_phase`` for the reviewer agent / gateway phase - API to advance via ``apply_mutation``. When ``start_phase=implement`` - no plan reviewer runs, so the populator nudges the contract itself - (#2427 sub-bug). - """ - try: - from egg_contracts.loader import load_contract, save_contract - except ImportError: - logger.warning( - "contract_phases_ingest_failed", - pipeline_id=pipeline_id, - reason="egg_contracts_unavailable", - ) - return PopulateResult(PopulateOutcome.EGG_CONTRACTS_UNAVAILABLE) - - # Resolve draft path - draft_rel = _get_draft_path("plan", issue_number=issue_number, pipeline_id=pipeline_id) - if not draft_rel: - logger.warning( - "contract_phases_ingest_failed", - pipeline_id=pipeline_id, - reason="no_draft_path", - ) - return PopulateResult(PopulateOutcome.NO_DRAFT_PATH) - - plan_path = repo_path / draft_rel - if not plan_path.exists(): - logger.warning( - "contract_phases_ingest_failed", - pipeline_id=pipeline_id, - reason="plan_draft_missing", - path=str(plan_path), - ) - return PopulateResult(PopulateOutcome.DRAFT_MISSING) - - try: - contract = load_contract(pipeline_id, repo_path) - except Exception as load_err: - logger.warning( - "contract_phases_ingest_failed", - pipeline_id=pipeline_id, - reason="contract_load_failed", - error=str(load_err), - ) - return PopulateResult(PopulateOutcome.CONTRACT_LOAD_FAILED) - - try: - from egg_contracts.plan_parser import parse_plan - - plan_text = plan_path.read_text() - result = parse_plan(plan_text) - - if not result.success: - logger.warning( - "contract_phases_ingest_failed", - pipeline_id=pipeline_id, - reason="parse_failed", - error=result.error, - ) - return PopulateResult(PopulateOutcome.PARSE_FAILED) - - for warning in result.warnings: - logger.warning( - "Plan parse warning", - pipeline_id=pipeline_id, - warning_message=warning.message, - warning_context=warning.context, - ) - - contract_slices = result.to_contract_slices() - changed = False - - if contract_slices: - # Forest validation (#2137 TASK-2-2): the slice DAG must be - # a forest (every slice has ≤1 DAG parent). Multi-parent - # slices break the stacked-PR invariant and are rejected - # at ingestion so the plan reviewer NACKs the planner. - # - # ``parse_plan`` was already imported unconditionally above, - # so we don't guard ``validate_forest`` import — if the - # parser module is unavailable the populator has already - # failed; silently defaulting ``forest_errors = []`` would - # let a broken-import multi-parent contract slip past the - # gate (reviewer_code_holistic v2 finding #5). - from egg_contracts.plan_parser import ( - validate_forest, - validate_slice_file_overlap, - ) - - forest_errors = validate_forest(contract_slices) - - if forest_errors: - # Stash the structured errors onto the contract's - # ``plan_review_feedback`` so the plan reviewer's - # prompt picks them up and NACKs the planner with the - # error verbatim. The slices are NOT written to the - # contract — leaving ``contract.slices`` empty makes - # downstream phases visibly broken so the violation - # cannot silently leak through. - logger.warning( - "contract_phases_ingest_failed", - pipeline_id=pipeline_id, - reason="forest_violation", - errors=forest_errors, - ) - feedback_lines = [ - "Plan ingestion REJECTED: the slice DAG is not a forest.", - "", - "Each slice must have at most one DAG parent. The " - "implement phase ships every slice as a stacked PR with " - "exactly one base branch — multi-parent slices break " - "this invariant. Re-emit the plan with " - "``serialized_chain_order`` populated on the downstream " - "slice (see issue #2137 plan TASK-2-3 for the rule).", - "", - "Structured errors:", - ] - feedback_lines.extend(f"- {e}" for e in forest_errors) - contract.plan_review_feedback = "\n".join(feedback_lines) - save_contract(contract, repo_path) - # Raise a structured ForestValidationError so any - # caller running this in an HTTP context (e.g. a - # plan-ingestion API endpoint) can surface a 422 with - # the inlined errors. Internal callers - # (``_populate_contract_from_plan_safe`` and the - # pipeline run-loop) catch and log instead — the - # ``plan_review_feedback`` stash above is the durable - # signal the reviewer prompt picks up either way. - raise ForestValidationError("slice DAG is not a forest", errors=forest_errors) - - # File-overlap ordering validation (#3046). The forest is - # valid (≤1 parent per slice), but the implement phase cuts - # each slice's integration branch off its dependency parent - # (roots off ``work``) — so two slices that touch the same - # file MUST be ordered along a dependency chain, or their - # branches fork independently off the shared base and their - # edits collide at integration (the guaranteed modify/delete - # conflict observed on #3023). Reject overlapping-but-unordered - # slices here, with the SAME NACK-the-architect handling as a - # forest violation: stash the structured errors on - # ``plan_review_feedback`` and leave ``contract.slices`` empty - # so the defect cannot silently leak into the implement phase. - overlap_errors = validate_slice_file_overlap(contract_slices) - if overlap_errors: - logger.warning( - "contract_phases_ingest_failed", - pipeline_id=pipeline_id, - reason="slice_overlap_violation", - errors=overlap_errors, - ) - feedback_lines = [ - "Plan ingestion REJECTED: slices touch overlapping files " - "without a dependency ordering.", - "", - "The implement phase cuts each slice's integration branch " - "off its dependency parent (root slices off the ``work`` " - "branch) and ships it as a stacked PR. Two slices that " - "touch the same file must be ordered along a single " - "dependency chain so the later slice's branch is forked " - "from a base that already contains the earlier slice's " - "commits — otherwise both branches fork independently off " - "the shared base and their edits collide at integration " - "(a guaranteed modify/delete conflict). The forest " - "constraint means the fix is always to serialise the " - "overlapping cluster into ONE linear ``dependencies`` " - "chain (you cannot depend on two parents) — or merge the " - "slices into one.", - "", - "Structured errors:", - ] - feedback_lines.extend(f"- {e}" for e in overlap_errors) - contract.plan_review_feedback = "\n".join(feedback_lines) - save_contract(contract, repo_path) - raise ForestValidationError( - "slices share files without a dependency ordering", - errors=overlap_errors, - reason="slice_overlap_violation", - ) - # Preserve runtime slice/task progress across re-populates so - # the safety-net populator (which fires on every - # ``start_phase=implement`` restart) cannot reset COMPLETE - # slices to PENDING and strand the pipeline on slice-1 (#2908). - _merge_preserved_slice_runtime(contract_slices, contract.slices) - contract.slices = contract_slices - changed = True - - # Populate PR metadata from plan if available - if result.pr_title: - from egg_contracts.models import PRMetadata - - # Preserve orchestrator-populated runtime fields on - # ``PRMetadata`` across re-populates. The planner-emitted - # title/description/test_plan/manual_steps flow in fresh - # from the parsed plan; the fields below are populated by - # orchestrator code paths (the up-front context-PR opener - # in ``_open_context_pr_at_implement_start``, the - # conditional-ACK gate at ``complete_phase``) and would - # otherwise be silently dropped when this safety-net - # populator re-runs (e.g. on a ``start_phase=implement`` - # re-entry where ``deferred_actions`` was already populated - # during implement-phase close). - # - # ``deferred_actions`` is the merge-blocking *Pre-merge - # Obligations* handoff written by ``decisions.py`` after a - # conditional-ACK gate resolves; losing it here erases the - # reviewer's only durable handoff for git-mv / migration / - # cross-repo flips. See test - # ``test_populate_contract_from_plan_preserves_deferred_actions``. - preserved_pr_number = contract.pr.context_pr_number if contract.pr is not None else None - preserved_deferred_actions = ( - list(contract.pr.deferred_actions) if contract.pr is not None else [] - ) - contract.pr = PRMetadata( - title=result.pr_title, - description=result.pr_description or "", - test_plan=result.pr_test_plan or "", - manual_steps=result.pr_manual_steps or "", - context_pr_number=preserved_pr_number, - deferred_actions=preserved_deferred_actions, - ) - changed = True - - if current_phase is not None and contract.current_phase != current_phase: - # Forward-only: never demote. Without this guard a respawn - # of _run_pipeline (e.g. when a start_phase=implement pipeline - # progresses past the implement boundary and re-enters the - # safety-net call site) would silently roll - # contract.current_phase back from IMPLEMENT to whatever the - # call site hardcoded. The PR phase was removed in #2777 - # (cq-4); IMPLEMENT is now terminal. - _phase_order = ( - PipelinePhase.REFINE, - PipelinePhase.PLAN, - PipelinePhase.IMPLEMENT, - ) - if ( - contract.current_phase in _phase_order - and current_phase in _phase_order - and _phase_order.index(current_phase) > _phase_order.index(contract.current_phase) - ): - from egg_contracts.audit import create_transition_entry - from egg_contracts.models import AuditRole - - old_phase = contract.current_phase - contract.audit_log.append( - create_transition_entry( - actor="orchestrator", - role=AuditRole.SYSTEM, - from_phase=old_phase.value, - to_phase=current_phase.value, - reason=( - "populator advanced contract.current_phase " - "(no apply_mutation caller for this pipeline; #2427)" - ), - ) - ) - contract.current_phase = current_phase - changed = True - - if changed: - save_contract(contract, repo_path) - slice_count = len(contract.slices) - task_count = sum(len(s.tasks) for s in contract.slices) - logger.info( - "contract_phases_populated", - pipeline_id=pipeline_id, - phase_count=slice_count, - task_count=task_count, - has_pr_metadata=contract.pr is not None, - ) - return PopulateResult( - PopulateOutcome.POPULATED, - slice_count=slice_count, - task_count=task_count, - ) - else: - # Parse succeeded but yielded neither phases nor PR metadata — - # this is the #1931 failure mode (empty contract with no error). - # Emit a discriminator so the gap is visible in audit logs. - logger.warning( - "contract_phases_ingest_failed", - pipeline_id=pipeline_id, - reason="empty_result", - warning_count=len(result.warnings), - ) - return PopulateResult(PopulateOutcome.EMPTY_RESULT) - - except ForestValidationError: - # Re-raise so callers with HTTP context (or the safe wrapper) - # can surface the structured errors. The populator already - # stashed feedback on contract.plan_review_feedback before - # raising. - raise - except Exception as e: - logger.warning( - "contract_phases_ingest_failed", - pipeline_id=pipeline_id, - reason="unexpected_exception", - source="parse_save", - error=str(e), - exc_info=True, - ) - return PopulateResult(PopulateOutcome.UNEXPECTED_EXCEPTION) - - -def _sync_pipeline_decisions_to_contract( - repo_path: Path, - worktree_repo_path: Path, - pipeline_id: str, -) -> None: - """Sync resolved non-phase-gate pipeline decisions to the contract. - - Converts HITLDecision objects from pipeline state into contract Decision - objects so that implement-phase agents can see what was decided during - refine/plan phases. - - Only syncs decisions with decision_type != "phase_gate" (substantive - choices, not process-control gates). Skips decisions already present - in the contract (matched by question text) to avoid duplicates on - re-runs after HITL revision cycles. - - Args: - repo_path: Orchestrator's main repo path — root for the state - store that owns pipeline records. - worktree_repo_path: Pipeline's per-run worktree path — root for - the contract under ``<worktree>/.egg-state/contracts/``. - """ - try: - from egg_contracts.loader import load_contract, save_contract - from egg_contracts.models import Decision, DecisionOption, DecisionType - except ImportError: - logger.warning("egg_contracts not available, skipping decision sync") - return - - # Load pipeline from the orchestrator's state store, NOT the per-run - # worktree. Pipeline records live under ``repo_path``'s persistent - # state-store worktree; the per-run worktree has none. Conflating - # the two silently no-op'd this helper for every issue-mode pipeline - # since #950 (#2345). - store = get_state_store(repo_path) - try: - pipeline = store.load_pipeline(pipeline_id) - except Exception as exc: - logger.warning( - "decision_sync_pipeline_load_failed", - pipeline_id=pipeline_id, - state_store_repo_path=str(repo_path), - error=str(exc), - ) - return - - # Filter to resolved, non-phase-gate decisions - substantive_decisions = [ - d - for d in pipeline.decisions - if d.decision_type != "phase_gate" and d.status == DecisionStatus.RESOLVED - ] - - if not substantive_decisions: - logger.debug("No substantive decisions to sync", pipeline_id=pipeline_id) - return - - try: - contract = load_contract(pipeline_id, worktree_repo_path) - except Exception: - logger.warning( - "Contract not found, skipping decision sync", - pipeline_id=pipeline_id, - ) - return - - # Build set of existing contract decision questions for deduplication - existing_questions = {d.question for d in contract.decisions} - - # Determine next decision ID (continue numbering after existing ones) - max_existing_id = 0 - for d in contract.decisions: - # Extract numeric suffix from "decision-N" - try: - num = int(d.id.split("-")[1]) - max_existing_id = max(max_existing_id, num) - except IndexError, ValueError: - pass - - synced_count = 0 - for pipeline_decision in substantive_decisions: - if pipeline_decision.question in existing_questions: - continue - - max_existing_id += 1 - decision_id = f"decision-{max_existing_id}" - - # Convert pipeline options (list[str]) to contract DecisionOption objects - contract_options = [ - DecisionOption(id=f"opt-{i + 1}", label=opt) - for i, opt in enumerate(pipeline_decision.options) - ] - - contract_decision = Decision( - id=decision_id, - question=pipeline_decision.question, - type=DecisionType.HITL, - options=contract_options, - resolved=True, - resolution=pipeline_decision.resolution, - resolved_by="human", - resolved_at=pipeline_decision.resolved_at, - ) - contract.decisions.append(contract_decision) - existing_questions.add(pipeline_decision.question) - synced_count += 1 - - if synced_count > 0: - save_contract(contract, worktree_repo_path) - logger.info( - "Synced pipeline decisions to contract", - pipeline_id=pipeline_id, - synced_count=synced_count, - total_contract_decisions=len(contract.decisions), - ) - - -# Decision-ledger backstop options (#3390). Bare labels matched -# case-insensitively on the resolution, mirroring the phase_gate's -# keyword handling. -_LEDGER_BACKSTOP_RERUN_OPTION = "Re-run phase to register decisions" -_LEDGER_BACKSTOP_PROCEED_OPTION = "Proceed without a decision ledger" - -# Explicit-none attestation confirmation option (#3462). Paired with -# ``_LEDGER_BACKSTOP_RERUN_OPTION`` on the confirmation decision; only a -# resolution that IS a confirmation (the bare keyword or the full label) -# proceeds — any other text is treated as a re-run directive, mirroring -# the phase_gate's "bare approve advances, notes request changes" posture. -_LEDGER_ATTESTATION_CONFIRM_OPTION = "Confirm — no open decisions this phase" - - -def _ledger_attestation_question(role: str, rationale: str, phase_value: str) -> str: - """Compose the explicit-none confirmation question (#3462). - - A producer's claim that a phase raises no operator decisions is itself - a judgment call about what *is* a judgment call — exactly the class of - decision the HITL contract assigns to the operator. It therefore - surfaces as its own confirmable decision, not a sentence embedded in - the phase_gate question. - """ - return ( - f"The {role} attests the {phase_value} phase deliberately raises " - f"no operator decisions (#3462):\n\n" - f"> {rationale}\n\n" - f"Confirm to proceed to the phase gate, or choose " - f"“{_LEDGER_BACKSTOP_RERUN_OPTION}” to send the phase back so its " - f"agents register the decisions as first-class contract entries " - f"(cq-N). Any free-text reply is treated as a re-run directive and " - f"forwarded to the agents." - ) - - -def _unwrap_choice_resolution(resolution: str) -> str: - """Unwrap the ``{"action":"select","selected":<label>}`` envelope. - - The SDLC HITL CLI resolves a ``choice`` decision with that structured - envelope (mirrors ``routes.decisions._normalize_choice_resolution``); - a bare string / non-JSON resolution passes through unchanged. - """ - try: - payload = json.loads(resolution) - if isinstance(payload, dict) and payload.get("action") == "select": - selected = payload.get("selected") - if isinstance(selected, str): - return selected - except ValueError, TypeError: - pass - return resolution - - -def _ledger_attestation_confirmed(resolution: str) -> bool: - """Return True when ``resolution`` confirms the explicit-none attestation. - - Conservative on purpose (#3462): only the bare keyword ``confirm`` or - the full confirm-option label counts. Anything else — the re-run - option, or free text naming decisions the operator expected — kicks - the phase back, with the text riding along as the directive. - """ - normalized = _unwrap_choice_resolution(resolution).strip().lower() - return normalized in ("confirm", _LEDGER_ATTESTATION_CONFIRM_OPTION.lower()) - - -def _ledger_attestation_rerun_directive(phase_value: str, rationale: str, resolution: str) -> str: - """Compose the re-run directive for a rejected explicit-none attestation (#3462). - - The operator declined to confirm that the phase raises no operator - decisions, so the phase re-runs with an instruction to register each - decision — including ones the producer believes prior context already - resolves. Any free-text resolution (i.e. not the bare re-run option) is - an operator note and rides along verbatim so the agents see the specific - concern. - """ - directive = ( - f"The operator declined to confirm the {phase_value} phase's " - f"no-decisions attestation (#3462). The phase claimed: " - f"“{rationale}”. Register each operator-grade decision via " - f"`egg-contract add-decision` — including decisions you believe " - f"prior context already resolves: register those with your " - f"recommended answer as the first option and cite the resolving " - f"context in its description. Belief about resolution is a " - f"recommended disposition, not a reason to skip registration." - ) - if resolution.strip().lower() != _LEDGER_BACKSTOP_RERUN_OPTION.lower(): - directive += f"\n\nOperator note: {resolution.strip()}" - return directive - - -def _handle_explicit_none_attestation_gate( - *, - pipeline, - pipeline_id: str, - repo_path, - current_phase: PipelinePhase, - ledger_note: str, - explicit_none: tuple[str, str], - store, - spawner, -): - """Surface an explicit-none attestation as a confirmable HITL decision (#3462). - - A producer's claim that a refine/plan phase raises no operator decisions - bypasses the entire register → bridge → resolve chain, and the claim is - itself a judgment call the HITL contract assigns to the operator. Rather - than folding it into the phase_gate question as prose, surface it as its - own confirmable ``choice`` decision: confirming records the operator's - endorsement on the ledger note; rejecting re-runs the phase so producers - register the decisions as first-class ``cq-N`` entries. - - Returns ``(rerun_requested, ledger_note, pipeline)``: - - - ``rerun_requested`` — True when the operator rejected the attestation - and the phase has already been re-run here; the caller must ``continue`` - its poll loop. False when the attestation was confirmed (or fail-open on - a cancelled/non-RESOLVED terminal state); the caller proceeds to the - phase gate. - - ``ledger_note`` — the note to thread into the phase_gate question, - annotated with the confirmation outcome. - - ``pipeline`` — the (possibly reloaded) pipeline the caller must rebind, - since queuing the confirmation decision reloads and mutates state. - """ - attest_role, attest_rationale = explicit_none - attest_question = _ledger_attestation_question( - attest_role, attest_rationale, current_phase.value - ) - # A converge-loop round (or a resume) re-enters this gate with the same - # attestation — do not re-ask a question the operator already answered, - # and reuse a pending one instead of queueing a duplicate (mirrors the - # phase_gate's #1152 guard). - prior_confirm = next( - ( - d - for d in reversed(pipeline.decisions) - if d.decision_type == "choice" - and d.phase == current_phase - and d.question == attest_question - and d.status == DecisionStatus.RESOLVED - and _ledger_attestation_confirmed(str(d.resolution or "")) - ), - None, - ) - if prior_confirm is not None: - return False, ledger_note + " Operator confirmed the attestation.", pipeline - - dq = get_decision_queue(pipeline_id, repo_path) - pending_attest = next( - ( - d - for d in reversed(pipeline.decisions) - if d.decision_type == "choice" - and d.phase == current_phase - and d.question == attest_question - and d.status == DecisionStatus.PENDING - ), - None, - ) - if pending_attest is not None: - attest_decision = pending_attest - newly_created = False - else: - attest_decision = dq.queue_decision( - question=attest_question, - context=ledger_note, - options=[ - _LEDGER_ATTESTATION_CONFIRM_OPTION, - _LEDGER_BACKSTOP_RERUN_OPTION, - ], - decision_type="choice", - phase=current_phase, - ) - newly_created = True - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - pipeline.status = PipelineStatus.AWAITING_HUMAN - phase_execution = pipeline.get_phase_execution(current_phase) - phase_execution.status = PipelineStatus.AWAITING_HUMAN - store.save_pipeline(pipeline) - # Only announce a freshly-created decision. Reusing a pending decision - # across polls must not re-emit ``decision.created`` — a duplicate event - # for a decision the operator is already looking at (#3462 review). - if newly_created: - report_pipeline_status( - pipeline, - event_type="decision.created", - message=( - f"{current_phase.value} phase attests no operator " - f"decisions — awaiting operator confirmation (#3462)" - ), - ) - _emit_pipeline_event(pipeline, "decision.created") - - attest_resolved = dq.wait_for_decision(attest_decision.id) - attest_resolution = _unwrap_choice_resolution( - str(getattr(attest_resolved, "resolution", None) or "") - ).strip() - resolved_ok = attest_resolved.status == DecisionStatus.RESOLVED - confirmed = resolved_ok and _ledger_attestation_confirmed(attest_resolution) - - if resolved_ok and not confirmed: - # Rejected — re-run the phase so producers register the decisions as - # first-class cq-N entries. - rerun_directive = _ledger_attestation_rerun_directive( - current_phase.value, attest_rationale, attest_resolution - ) - logger.info( - "Explicit-none attestation rejected: re-running phase (#3462)", - pipeline_id=pipeline_id, - phase=current_phase.value, - ) - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - pipeline.status = PipelineStatus.RUNNING - phase_execution = pipeline.get_phase_execution(current_phase) - phase_execution.status = PipelineStatus.RUNNING - phase_execution.completed_at = None - phase_execution.hitl_review_cycles += 1 - _alert_threshold = pipeline.config.max_hitl_review_cycles - if phase_execution.hitl_review_cycles >= _alert_threshold: - _broadcast_hitl_nonconvergence_alert( - pipeline_id, - pipeline, - current_phase, - phase_execution.hitl_review_cycles, - _alert_threshold, - ) - _perform_hitl_phase_rerun( - store=store, - spawner=spawner, - pipeline=pipeline, - phase_execution=phase_execution, - pipeline_id=pipeline_id, - current_phase=current_phase, - feedback_text=rerun_directive, - event_message=( - f"Re-running {current_phase.value}: no-decisions attestation rejected (#3462)" - ), - ) - return True, ledger_note, pipeline - - if confirmed: - return False, ledger_note + " Operator confirmed the attestation.", pipeline - # Fail open to the phase gate on a non-RESOLVED terminal state (cancel): - # the gate itself still blocks for approval, mirroring the missing-ledger - # backstop's posture. Record the outcome accurately — do not claim a - # confirmation the operator never gave (#3462 review). - return ( - False, - ledger_note + " Attestation confirmation was cancelled; deferring to the phase gate.", - pipeline, - ) - - -def _find_explicit_none_attestation( - pipeline_id: str, - phase_value: str, -) -> tuple[str, str] | None: - """Find a producer's explicit-none decision-ledger attestation (#3390). - - Scans the phase's ``CONSENSUS_PROPOSE`` messages (newest first) for a - proposal whose attestation carries a non-empty - ``no_decisions_rationale`` — the durable record that a producer - *deliberately* registered no decisions this phase (propose-time - validation guarantees the field was well-formed when accepted). - Returns ``(role, rationale)`` or ``None``; message-store outages - degrade to ``None`` (the caller fails closed into the backstop HITL, - which the operator can resolve either way — never a silent pass). - """ - try: - from message_store import MessageType, get_message_store - - messages = get_message_store().get_messages(pipeline_id, limit=500) - except Exception as exc: # noqa: BLE001 - logger.warning( - "Decision-ledger attestation scan failed (treating as not found)", - pipeline_id=pipeline_id, - phase=phase_value, - error=str(exc), - ) - return None - - for message in reversed(messages): - if message.message_type != MessageType.CONSENSUS_PROPOSE: - continue - if message.phase is not None and message.phase != phase_value: - continue - payload = (message.metadata or {}).get("payload") - if not isinstance(payload, dict): - continue - attestation = payload.get("attestation") - if not isinstance(attestation, dict): - continue - rationale = attestation.get("no_decisions_rationale") - if isinstance(rationale, str) and rationale.strip(): - return message.from_role, rationale.strip() - return None - - -def _collect_decision_ledger_status( - worktree_repo_path: Path, - pipeline_id: str, - pipeline_identifier: int | str, - phase: PipelinePhase, -) -> tuple[str, bool, tuple[str, str] | None]: - """Summarize the phase's decision ledger for the gate surface (#3390). - - Returns ``(note, missing, explicit_none)``: - - - ``note`` — an operator-visible one-liner appended to the phase_gate - question so "N registered" vs "explicitly none" vs "MISSING" is - readable at the gate without a ``get_contract`` round-trip. - - ``missing`` — True only when the phase registered zero decisions - AND no producer attested an explicit empty ledger. With propose-time - enforcement in place this means the gate was reached on a path that - bypassed consensus (force-advance, resume) or the producer's claim - was lost — the caller surfaces a dedicated backstop HITL rather - than silently advancing. - - ``explicit_none`` — the ``(role, rationale)`` of a producer's - explicit-none attestation when that is what stands in for a ledger - (zero registered decisions), else ``None``. The caller surfaces it - as its own confirmable decision (#3462) rather than trusting the - self-attestation. Mutually exclusive with ``missing``. - """ - phase_value = phase.value - registered_ids: list[str] = [] - try: - from egg_contracts.loader import load_contract - - contract = load_contract(pipeline_identifier, worktree_repo_path) - except Exception as exc: # noqa: BLE001 - logger.warning( - "Decision-ledger status: contract not loadable", - pipeline_id=pipeline_id, - phase=phase_value, - error=str(exc), - ) - contract = None - - if contract is not None: - for d in contract.decisions: - d_type = getattr(d.type, "value", d.type) - if d_type != "hitl": - continue - d_phase = getattr(d.phase, "value", d.phase) if d.phase is not None else None - if d_phase is None or d_phase == phase_value: - registered_ids.append(d.id) - - if registered_ids: - resolved = 0 - for d in contract.decisions: - if d.id in registered_ids and d.resolved: - resolved += 1 - return ( - f"Decision ledger: {len(registered_ids)} decision(s) registered this " - f"phase ({', '.join(registered_ids)}), {resolved} resolved.", - False, - None, - ) - - explicit_none = _find_explicit_none_attestation(pipeline_id, phase_value) - if explicit_none is not None: - role, rationale = explicit_none - return ( - f"Decision ledger: explicitly none — {role} attested: {rationale}", - False, - explicit_none, - ) - - return ( - "⚠️ Decision ledger MISSING: this phase registered no HITL decisions " - "and no producer attested an explicit empty ledger (#3390). " - "“0 decisions” here cannot be distinguished from “failed " - "to register”.", - True, - None, - ) - - -def _queue_and_await_contract_decisions( - dq: Any, - worktree_repo_path: Path, - pipeline_id: str, - pipeline_identifier: int | str, - phase: PipelinePhase, -) -> int: - """Promote unresolved contract decisions/feedback into the orchestrator queue. - - Returns the number of contract decisions/feedback this call surfaced and - the operator *resolved* this round — the converge-before-advance signal - (#3392). Decisions that were surfaced but came back non-RESOLVED (e.g. the - operator cancelled them) are **not** counted: the contract ``cq-N`` stays - open, and counting it would re-run the phase, re-surface the still-open - question (carry-forward only adopts *resolved* questions), and loop with no - termination now that the force-advance backstop is gone. A non-zero count - means the operator just answered something, so the caller re-runs the - phase to fold the resolutions into the documents; a zero count means the - round resolved nothing new and the caller may advance. - - - Agents register architectural questions via ``egg-contract add-decision`` - and ``add-feedback``. Those writes only touch ``.egg-state/contracts/ - {identifier}.json`` — the orchestrator's decision queue never sees them, - so approving the phase_gate silently drops the questions and the next - phase's agents have to guess (issue #1889). - - This helper bridges contract-scoped questions for the current phase into - the orchestrator queue after phase_gate approval, so HTTP/MCP callers - (e.g. the ``/sdlc`` skill's Phase 4 handler) surface them as individual - ``choice`` / ``feedback`` decisions. Resolutions are written back to - the contract so implement-phase agents see the human's answers. - - All pending decisions (plus the feedback entry, if any) are queued up - front before any ``wait_for_decision`` call, so ``get_status`` surfaces - them as a single batch. Callers can then prompt for up to 4 at a time - and submit answers in parallel, collapsing what was previously N prompts - and N polling cycles into ~⌈N/4⌉ prompts and one cycle (issue #1956). - - Once the batch is queued, a single ``decision.created`` event is - published to the EventBus so event-driven watchers (the ``wait-status`` - monitor long-polling ``/status/wait``) wake immediately. - ``DecisionQueue.queue_decision`` itself emits no event, so without this - the bridged decisions are created silently and the operator only - discovers them via a manual ``get_status`` (issue #2770). - """ - try: - from egg_contracts.loader import load_contract, save_contract - except ImportError: - logger.warning( - "egg_contracts not available, skipping contract decision bridge", - pipeline_id=pipeline_id, - ) - return 0 - - try: - contract = load_contract(pipeline_identifier, worktree_repo_path) - except Exception as e: - logger.debug( - "Contract not loadable, skipping contract decision bridge", - pipeline_id=pipeline_id, - error=str(e), - ) - return 0 - - phase_value = phase.value - pending_decisions = [ - d - for d in contract.decisions - if not d.resolved - and getattr(d.type, "value", d.type) == "hitl" - and (d.phase is None or getattr(d.phase, "value", d.phase) == phase_value) - ] - fb = contract.feedback - pending_feedback = None - if fb is not None and not fb.submitted: - fb_phase_val = getattr(fb.phase, "value", fb.phase) if fb.phase is not None else None - if fb_phase_val is None or fb_phase_val == phase_value: - pending_feedback = fb - - if not pending_decisions and pending_feedback is None: - return 0 - - logger.info( - "Bridging contract decisions/feedback into orchestrator queue", - pipeline_id=pipeline_id, - phase=phase_value, - decision_count=len(pending_decisions), - has_feedback=pending_feedback is not None, - ) - - def _save_contract_update(mutator: Callable[[Any], bool]) -> None: - try: - latest = load_contract(pipeline_identifier, worktree_repo_path) - except Exception as e: - logger.warning( - "Could not reload contract to persist bridged resolution", - pipeline_id=pipeline_id, - error=str(e), - ) - return - if not mutator(latest): - return - try: - save_contract(latest, worktree_repo_path) - except Exception as e: - logger.warning( - "Failed to save contract after bridged resolution", - pipeline_id=pipeline_id, - error=str(e), - ) - - # Pass 1: queue every pending decision + feedback up front. - queued_decisions: list[tuple[str, Any]] = [] - for contract_decision in pending_decisions: - options_labels = [opt.label for opt in contract_decision.options] - queued = dq.queue_decision( - question=contract_decision.question, - context=( - f"Open contract question {contract_decision.id}, " - f"registered by an agent during the {phase_value} phase." - ), - options=options_labels, - decision_type="choice", - phase=phase, - ) - queued_decisions.append((contract_decision.id, queued)) - - queued_feedback: HITLDecision | None = None - if pending_feedback is not None: - questions_payload = [ - {"id": q.id, "question": q.question, "answer": ""} for q in pending_feedback.questions - ] - queued_feedback = dq.queue_decision( - question=f"Open feedback request {pending_feedback.id}", - context=( - f"Open contract feedback {pending_feedback.id}, " - f"registered by an agent during the {phase_value} phase." - ), - options=[], - decision_type="feedback", - questions=questions_payload, - phase=phase, - ) - - # Surface the freshly-queued batch to event-driven watchers before - # blocking on resolution. ``DecisionQueue.queue_decision`` emits no - # EventBus event, so without this the bridged decisions are created - # silently — the operator's ``wait-status`` monitor never wakes and - # only finds them via a manual ``get_status`` (#2770). The phase_gate - # decision emits ``decision.created`` the same way. - if _emit_event is not None: - _emit_event( - EventType.DECISION_CREATED, - pipeline_id, - data={"phase": phase_value}, - ) - - # Pass 2: wait for each to resolve and persist back to the contract. - # Count only decisions whose queue resolution was RESOLVED — a - # CANCELLED / non-resolved outcome leaves the contract ``cq-N`` open and - # must NOT count toward the convergence signal, or the caller would re-run - # the phase, re-surface the still-open question (carry-forward only adopts - # *resolved* questions), and loop without the operator ever being able to - # break out (#3392 review). - resolved_count = 0 - for contract_id, queued in queued_decisions: - resolved = dq.wait_for_decision(queued.id) - if resolved.status != DecisionStatus.RESOLVED: - continue - resolved_count += 1 - resolution_str = (resolved.resolution or "").strip() - - def _apply(latest: Any, _cd_id: str = contract_id, _res: str = resolution_str) -> bool: - for d in latest.decisions: - if d.id == _cd_id: - d.resolved = True - d.resolution = _res - d.resolved_by = "human" - d.resolved_at = datetime.now(UTC) - return True - return False - - _save_contract_update(_apply) - - feedback_resolved = False - if queued_feedback is not None and pending_feedback is not None: - resolved = dq.wait_for_decision(queued_feedback.id) - if resolved.status == DecisionStatus.RESOLVED: - feedback_resolved = True - answers: dict[str, str] = {} - try: - payload = json.loads(resolved.resolution or "") - if isinstance(payload, dict): - raw_answers = payload.get("answers") - if isinstance(raw_answers, dict): - answers = {str(k): str(v) for k, v in raw_answers.items()} - except json.JSONDecodeError, TypeError: - pass - - fb_id = pending_feedback.id - - def _apply_fb( - latest: Any, _fb_id: str = fb_id, _answers: dict[str, str] = answers - ) -> bool: - if latest.feedback is None or latest.feedback.id != _fb_id: - return False - for q in latest.feedback.questions: - if q.id in _answers: - q.answer = _answers[q.id] - # Always mark submitted after resolution — even if - # individual answers didn't parse, the human responded - # and shouldn't be asked again. - latest.feedback.submitted = True - latest.feedback.submitted_by = "human" - latest.feedback.submitted_at = datetime.now(UTC) - return True - - _save_contract_update(_apply_fb) - - # Convergence signal (#3392): the number of decisions + feedback this - # round the operator actually *resolved* (not merely surfaced). Non-zero ⇒ - # the operator answered something ⇒ caller re-runs the phase to fold the - # resolutions in. A surfaced-but-cancelled decision is deliberately - # excluded: counting it would re-run the phase, re-surface the still-open - # question, and loop indefinitely now that the force-advance backstop is - # gone. - return resolved_count + (1 if feedback_resolved else 0) - - -def _await_unresolved_gap_gate( - store: Any, - pipeline_id: str, - repo_path: Path, - worktree_repo_path: Path, - pipeline_identifier: int | str, - phase: PipelinePhase, - hitl_gates: bool = True, -) -> bool: - """Block phase finalize while the contract carries unresolved TaskGaps. - - A tester→coder :class:`TaskGap` left ``resolved == False`` ships into - the committed contract snapshot and fails ``test_models_gaps.py`` red - in CI on the already-open PR (#3298 class 4). The implement phase is - **not** in ``_HITL_GATE_PHASES``, so the autonomous run loop would - otherwise mark it complete and finalize with the gap open and no - human in the loop. This surfaces a blocking ``phase_gate`` HITL - decision listing the open gaps and waits — mirroring the - unresolved-HITL guard in ``complete_phase`` (#1788). See #3300. - - The operator resolves the gap (set the gap's ``resolved=true`` via - the contract-mutate path, e.g. by re-running/kicking the coder) and - approves, or picks the override option to ship with the gap open. The - contract is re-read after each approval so a stale ``approve`` cannot - advance with a gap still open. Returns ``True`` when a gate was - surfaced and the contract may have changed (resolved or overridden), - ``False`` when the contract was already clean (the common path) or - the escalation could only be logged (autonomous run, below). - - **Autonomous runs.** ``wait_for_decision`` polls indefinitely and - both options require a human, so a fully-autonomous pipeline - (``hitl_gates is False``) has no path forward — blocking here would - convert a red-but-progressing PR into an indefinite stall (the - health monitor would eventually tear it down). When ``hitl_gates is - False`` we therefore *surface* the escalation (event + warning) but - do **not** block: the reactive ``test_models_gaps.py`` CI check - remains the backstop, exactly as it was before this gate existed. - - A best-effort scan: contract load failures fail open (log + return) - so a transient read error can never strand the pipeline. - """ - try: - from egg_contracts.loader import load_contract - except ImportError: - logger.warning( - "egg_contracts not available, skipping unresolved-gap gate", - pipeline_id=pipeline_id, - ) - return False - - def _load_open_gaps() -> list[tuple[str, Any]] | None: - try: - contract = load_contract(pipeline_identifier, worktree_repo_path) - except Exception as e: # noqa: BLE001 - logger.warning( - "Could not load contract for unresolved-gap gate (skipping)", - pipeline_id=pipeline_id, - error=str(e), - ) - return None - return contract.unresolved_gaps() - - open_gaps = _load_open_gaps() - if not open_gaps: - return False - - if not hitl_gates: - # No human in the loop — do not block forever. Both options need a - # human, so blocking would convert a red-but-progressing PR into an - # indefinite stall. Surface the escalation (so observers still see - # it) + log loudly, and let the reactive CI backstop catch the open - # gap on the PR, exactly as before this gate existed. - report_pipeline_status( - store.load_pipeline(pipeline_id), - event_type="phase.gap_gate", - message=f"{phase.value} phase has unresolved coverage gaps", - ) - logger.warning( - "Unresolved-gap gate: open gaps on an autonomous pipeline " - "(hitl_gates=False); surfacing but not blocking", - pipeline_id=pipeline_id, - phase=phase.value, - open_gap_ids=[f"{t}/{g.id}" for t, g in open_gaps], - ) - return False - - def _set_status(status: PipelineStatus) -> Pipeline: - # Mirror the phase_gate block: drive both pipeline and the phase - # box so the DAG visualization renders the gate on the right - # phase, and the operator's wait-status monitor wakes. - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - pipeline.status = status - phase_execution = pipeline.get_phase_execution(phase) - if phase_execution is not None: - phase_execution.status = status - store.save_pipeline(pipeline) - return pipeline - - dq = get_decision_queue(pipeline_id, repo_path) - gated = False - - while open_gaps: - gated = True - gap_lines = "\n".join( - f"- `{task_id}` / `{gap.id}` ({gap.from_role}→{gap.to_role}): {gap.description}" - for task_id, gap in open_gaps - ) - question = ( - f"The {phase.value} phase has {len(open_gaps)} unresolved coverage " - f"gap{'s' if len(open_gaps) != 1 else ''}. Resolve " - f"{'them' if len(open_gaps) != 1 else 'it'} (mark the gap resolved " - "via the contract) and approve, or choose 'override' to finalize " - "with the gap open." - ) - context = ( - "These tester→coder coverage gaps are still open on the contract. " - "Finalizing with an open gap ships it into the committed contract " - "and fails CI (test_models_gaps.py) red on the PR.\n\n" - f"{gap_lines}" - ) - decision = dq.queue_decision( - question=question, - context=context, - options=["approve", "override"], - decision_type="phase_gate", - phase=phase, - ) - - # Mark AWAITING_HUMAN + surface to event watchers, mirroring the - # phase_gate block so the operator's wait-status monitor wakes. - pipeline = _set_status(PipelineStatus.AWAITING_HUMAN) - report_pipeline_status( - pipeline, - event_type="phase.gap_gate", - message=f"{phase.value} phase has unresolved coverage gaps", - ) - if _emit_event is not None: - _emit_event( - EventType.DECISION_CREATED, - pipeline_id, - data={"phase": phase.value}, - ) - - resolved = dq.wait_for_decision(decision.id) - # Restore RUNNING now the gate cleared (re-set to AWAITING_HUMAN - # above on the next loop if gaps remain). - _set_status(PipelineStatus.RUNNING) - - if resolved.status != DecisionStatus.RESOLVED: - # Cancelled / abandoned — don't spin; let the loop proceed so - # a cancel can tear the pipeline down. - logger.warning( - "Unresolved-gap gate ended without resolution; proceeding", - pipeline_id=pipeline_id, - phase=phase.value, - decision_status=getattr(resolved.status, "value", resolved.status), - ) - return gated - - resolution = (resolved.resolution or "").strip().lower() - if "override" in resolution: - logger.warning( - "Unresolved-gap gate overridden — finalizing with open gaps", - pipeline_id=pipeline_id, - phase=phase.value, - open_gap_ids=[f"{t}/{g.id}" for t, g in open_gaps], - ) - # Record the override on the frozen phase artifacts for audit - # parity with the complete_phase endpoint's ``force`` path - # (otherwise the load-bearing run-loop override left only a - # transient log line). Values must be strings — - # PhaseExecution.artifacts is dict[str, str]. - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - phase_execution = pipeline.get_phase_execution(phase) - if phase_execution is not None: - merged = dict(phase_execution.artifacts) - merged["force_completed_gaps"] = json.dumps( - [f"{t}/{g.id}" for t, g in open_gaps] - ) - phase_execution.artifacts = merged - store.save_pipeline(pipeline) - return gated - - # Approval path: re-read the contract. If the operator actually - # marked the gaps resolved, the gate clears; otherwise re-surface. - reloaded = _load_open_gaps() - if reloaded is None: - # Load failed — fail open rather than strand the pipeline. - return gated - open_gaps = reloaded - if open_gaps: - logger.info( - "Unresolved-gap gate approved but gaps still open; re-surfacing", - pipeline_id=pipeline_id, - phase=phase.value, - remaining=len(open_gaps), - ) - - return gated - - -# --------------------------------------------------------------------------- -# Jira-epic SDLC scheduling helpers (issue #1557 — task-1-4 / task-2-7) -# --------------------------------------------------------------------------- - - -def _next_phases_for_epic( - pipeline: Pipeline, - current_phase: PipelinePhase, - default_next_phases: list[PipelinePhase], -) -> list[PipelinePhase]: - """Reroute auto-advance through ``APPLY`` for Jira-epic pipelines. - - Issue #1557: when ``pipeline.is_epic`` is true the orchestrator - inserts the new ``APPLY`` phase between ``PLAN`` and ``IMPLEMENT`` - so the ``APPLIER`` role can drive Jira mutations (epic-Description - write, child create / link / Won't-Do) on HITL approval. Non-epic - pipelines see ``default_next_phases`` returned unchanged so the - pre-#1557 scheduling is preserved bit-for-bit. - - The orchestrator-side scheduler is the authoritative gate per the - architecture's "VALID_TRANSITIONS lists APPLY but the scheduler - decides whether to actually pick it" design (see the comment on - :data:`gateway.phase_transition.VALID_TRANSITIONS`). Returns a - single-element list so the call site's ``next_phases[0]`` indexing - works without change. - """ - if not getattr(pipeline, "is_epic", False): - return default_next_phases - if current_phase == PipelinePhase.PLAN: - return [PipelinePhase.APPLY] - if current_phase == PipelinePhase.APPLY: - return [PipelinePhase.IMPLEMENT] - return default_next_phases - - -def _drain_wontdo_batch_after_apply( - pipeline: Pipeline, - worktree_repo_path: Path, -) -> None: - """Run the orchestrator-only Won't-Do drain after ``APPLY`` consensus. - - Trigger chain (issue #1557 task-2-7): the HITL operator approves - the plan-gate → ``_persist_phase_gate_resolution`` flips state → - the scheduler routes through ``APPLY`` → the applier writes a - handoff JSON at ``.egg-state/agent-outputs/<pipeline>-wontdo.json`` - listing every obsolete child key it could not transition itself - (decision-15: agent-facing routes deny Jira transitions) → the - APPLIER's CONSENSUS_PROPOSE → REVIEWER_CONTRACT ACK confirms → - this hook fires from the auto-advance block, iterates the handoff, - and POSTs to ``/api/v1/jira/ticket/transition`` with the launcher- - secret bearer token. - - Runs **out of band** from ``_persist_phase_gate_resolution`` so a - slow Jira API does not extend the HITL approve POST's latency SLA - (task-2-7 acceptance). Fail-open: a missing handoff file means - "no Won't-Dos to drain" and returns silently; a per-transition - failure surfaces as a logger warning but does not block the - pipeline from advancing to ``IMPLEMENT``. - - Naming note (reviewer_code v1 non-blocking): the handoff file - this function READS is the applier's *output* - (``<pipeline.id>-wontdo.json``), distinct from the applier's - *input* handoff (``<pipeline.id>-apply-handoff.json``) written - by :func:`_write_apply_phase_handoff` just before APPLY spawns. - - Per-Task lifecycle (reviewer_contract v1 finding #3 / task-2-7): - the drain registers an ``on_entry_result`` callback with - ``run_wontdo_drain``. After each transition attempt, the callback - loads the contract via ``egg_contracts.loader.load_contract``, - locates the corresponding Task (by ``task_id`` when the applier - included one in the handoff entry, otherwise by ``jira_key`` - match), and writes ``Task.jira_action_status = 'applied'`` / - ``'failed'`` plus the failure reason into ``Task.notes``. The - write is best-effort: contract-load / save failures surface as a - logger warning so a brittle contract state never breaks the - drain — the operator can re-run later with the same handoff JSON - (the gateway's idempotency cache absorbs the duplicate transition - calls within the 5-minute window). - """ - handoff_path = ( - Path(worktree_repo_path) / ".egg-state" / "agent-outputs" / f"{pipeline.id}-wontdo.json" - ) - if not handoff_path.exists(): - logger.debug( - "Won't-Do drain skipped — no handoff file produced by applier", - pipeline_id=pipeline.id, - handoff_path=str(handoff_path), - ) - return - - # Per-entry contract writeback callback (reviewer_contract v1 #3). - # Each invocation looks up the task by ``task_id`` (when the - # applier set it on the handoff entry) or by ``jira_key`` match - # otherwise, flips ``jira_action_status`` to ``'applied'`` / - # ``'failed'`` and records the failure reason in ``Task.notes``. - def _on_entry_result(entry: Any, ok: bool, reason: str) -> None: - try: - try: - from egg_contracts.loader import load_contract, save_contract - except ImportError: # pragma: no cover - defensive - logger.warning( - "Won't-Do drain: egg_contracts loader unavailable; " - "skipping per-Task lifecycle writeback", - pipeline_id=pipeline.id, - ) - return - try: - contract = load_contract(pipeline.id, worktree_repo_path) - except Exception as load_err: # noqa: BLE001 - logger.warning( - "Won't-Do drain: contract load failed; skipping per-Task lifecycle writeback", - pipeline_id=pipeline.id, - error=str(load_err), - ) - return - target_task = None - entry_task_id = getattr(entry, "task_id", None) - entry_key = getattr(entry, "jira_key", None) - for sl in getattr(contract, "slices", []) or []: - for tsk in getattr(sl, "tasks", []) or []: - if entry_task_id and tsk.id == entry_task_id: - target_task = tsk - break - if ( - not entry_task_id - and entry_key - and getattr(tsk, "jira_key", None) == entry_key - ): - target_task = tsk - break - if target_task is not None: - break - if target_task is None: - # No matching task — applier-written handoff may have - # entries for keys outside the contract's task list - # (e.g. consolidate-into "obsolete-only" rows). Log - # at DEBUG since this is expected for split / consolidate - # patterns. - logger.debug( - "Won't-Do drain: no contract task matches handoff entry; " - "skipping lifecycle writeback for this row", - pipeline_id=pipeline.id, - entry_task_id=entry_task_id, - entry_key=entry_key, - ) - return - target_task.jira_action_status = "applied" if ok else "failed" - if not ok: - existing_notes = target_task.notes or "" - failure_note = f"wontdo drain failed: {reason}" - target_task.notes = existing_notes + ("\n" if existing_notes else "") + failure_note - try: - save_contract(contract, worktree_repo_path) - except Exception as save_err: # noqa: BLE001 - logger.warning( - "Won't-Do drain: contract save failed after lifecycle writeback", - pipeline_id=pipeline.id, - error=str(save_err), - ) - except Exception as cb_err: # noqa: BLE001 - defensive - logger.warning( - "Won't-Do drain: per-Task callback raised (continuing)", - pipeline_id=pipeline.id, - error=str(cb_err), - ) - - # Contract-state idempotency gate. The drain consults this predicate - # before posting each transition so a benign re-run (orchestrator - # restart, manual re-drain, re-entry of the apply phase) does not - # double-POST transitions whose outcomes the gateway's 5-minute - # idempotency cache has long since forgotten — and does not flip an - # ``'applied'`` Task back to ``'failed'`` when Jira returns 400 for - # an already-transitioned ticket. - def _entry_already_applied(entry: Any) -> bool: - try: - try: - from egg_contracts.loader import load_contract - except ImportError: # pragma: no cover - defensive - logger.warning( - "Won't-Do drain idempotency gate disarmed: egg_contracts.loader not importable", - pipeline_id=pipeline.id, - ) - return False - try: - contract = load_contract(pipeline.id, worktree_repo_path) - except Exception as load_err: # noqa: BLE001 - defensive - # Contract unreadable / corrupted: idempotency gate is - # disarmed for this drain run. The drain re-POSTs every - # entry, Jira returns 400 for already-transitioned ones, - # and ``_on_entry_result`` flips ``'applied'`` → - # ``'failed'`` — surface this loudly so the operator can - # repair the contract before the next re-run. - logger.warning( - "Won't-Do drain idempotency gate disarmed: load_contract failed", - pipeline_id=pipeline.id, - error=str(load_err), - ) - return False - entry_task_id = getattr(entry, "task_id", None) - entry_key = getattr(entry, "jira_key", None) - for sl in getattr(contract, "slices", []) or []: - for tsk in getattr(sl, "tasks", []) or []: - matches_task = bool(entry_task_id and tsk.id == entry_task_id) - matches_key = bool( - not entry_task_id - and entry_key - and getattr(tsk, "jira_key", None) == entry_key - ) - if matches_task or matches_key: - return getattr(tsk, "jira_action_status", None) == "applied" - return False - except Exception as predicate_err: # noqa: BLE001 - defensive - logger.warning( - "Won't-Do drain idempotency gate raised; treating entry as not-yet-applied", - pipeline_id=pipeline.id, - error=str(predicate_err), - ) - return False - - try: - # Reviewer_code v1 non-blocking note: mirror the dual-import - # pattern used elsewhere in this module (e.g. ``from - # jira_epic import resolve_epic_mode``) so the helper still - # resolves when ``orchestrator/`` is imported as a package - # rather than treated as ``sys.path`` root. - try: - from wontdo_drain import run_wontdo_drain - except ImportError: # pragma: no cover — packaged-import fallback - from orchestrator.wontdo_drain import run_wontdo_drain # type: ignore[no-redef] - - result = run_wontdo_drain( - handoff_path=handoff_path, - on_entry_result=_on_entry_result, - is_already_applied=_entry_already_applied, - ) - except Exception as exc: # noqa: BLE001 — defensive: drain must not crash auto-advance - logger.warning( - "Won't-Do drain failed after APPLY phase (continuing)", - pipeline_id=pipeline.id, - error=str(exc), - ) - return - logger.info( - "Won't-Do drain complete after APPLY phase", - pipeline_id=pipeline.id, - succeeded=len(result.succeeded), - failed=len(result.failed), - skipped=len(result.skipped), - ) - - -def _write_apply_phase_handoff( - pipeline: Pipeline, - worktree_repo_path: Path, - approved_phase: str, -) -> None: - """Write the applier handoff JSON before the ``APPLY`` phase spawns. - - The applier prompt consumes a one-line JSON identifying which - artifact was just approved so it can branch between refine-apply - (writing the analysis to the epic Description) and plan-apply - (walking ``Task.jira_action`` + driving the Jira CLI per task). - - The handoff lands at - ``.egg-state/agent-outputs/<pipeline-id>-apply-handoff.json`` - inside the per-pipeline worktree so the applier (running in a - sandbox container with the same worktree mounted) reads from a - deterministic path. Fail-open: I/O errors surface as a logger - warning but never abort phase advancement. - """ - handoff_dir = Path(worktree_repo_path) / ".egg-state" / "agent-outputs" - try: - handoff_dir.mkdir(parents=True, exist_ok=True) - except OSError as exc: - logger.warning( - "Failed to create agent-outputs dir for applier handoff (continuing)", - pipeline_id=pipeline.id, - error=str(exc), - ) - return - contract_path = Path(worktree_repo_path) / ".egg-state" / "contracts" / f"{pipeline.id}.json" - draft_path = ( - Path(worktree_repo_path) - / ".egg-state" - / "brc-history" - / f"{pipeline.id}-{approved_phase}.md" - ) - payload = { - "approved_phase": approved_phase, - "contract_path": str(contract_path), - "draft_path": str(draft_path), - } - handoff_path = handoff_dir / f"{pipeline.id}-apply-handoff.json" - try: - handoff_path.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8") - except OSError as exc: - logger.warning( - "Failed to write applier handoff JSON (continuing)", - pipeline_id=pipeline.id, - handoff_path=str(handoff_path), - error=str(exc), - ) - return - logger.info( - "Applier handoff JSON written for APPLY phase", - pipeline_id=pipeline.id, - approved_phase=approved_phase, - handoff_path=str(handoff_path), - ) - - -def _persist_phase_gate_resolution( - repo_path: Path, - pipeline_id: str, - decision: HITLDecision, - phase: str, - issue_number: int | None = None, -) -> None: - """Persist a phase-gate resolution to the contract and draft. - - After a human approves a phase gate, the resolution context needs to be - visible to agents in the next phase. This function: - - 1. Adds the resolution as a HITL decision in the contract so next-phase - agents see it when they load the contract. - 2. Appends a ``## HITL Resolution`` section to the phase draft file so - agents reading the draft also see the human's decisions. - - See: #1295 - """ - # Extract structured context from JSON resolution, or use raw string - resolution_context: str = "" - raw = (decision.resolution or "").strip() - if raw: - try: - payload = json.loads(raw) - if isinstance(payload, dict): - resolution_context = payload.get("context", "") or payload.get("feedback", "") - if not resolution_context: - logger.debug( - "Phase gate approved without context, nothing to persist", - pipeline_id=pipeline_id, - phase=phase, - ) - return - else: - resolution_context = raw - except json.JSONDecodeError, TypeError: - resolution_context = raw - - if not resolution_context: - logger.debug( - "Phase gate resolution has no context to persist", - pipeline_id=pipeline_id, - phase=phase, - ) - return - - # --- 1. Sync to contract --- - try: - from egg_contracts.loader import load_contract, save_contract - from egg_contracts.models import Decision, DecisionOption, DecisionType - - contract = load_contract(pipeline_id, repo_path) - - existing_questions = {d.question for d in contract.decisions} - question_text = f"[Phase gate: {phase}] {decision.question}" - - if question_text not in existing_questions: - # Determine next decision ID - max_existing_id = 0 - for d in contract.decisions: - try: - num = int(d.id.split("-")[1]) - max_existing_id = max(max_existing_id, num) - except IndexError, ValueError: - pass - - contract_options = [ - DecisionOption(id=f"opt-{i + 1}", label=opt) - for i, opt in enumerate(decision.options) - ] - - contract_decision = Decision( - id=f"decision-{max_existing_id + 1}", - question=question_text, - type=DecisionType.HITL, - options=contract_options, - resolved=True, - resolution=resolution_context, - resolved_by="human", - resolved_at=decision.resolved_at, - ) - contract.decisions.append(contract_decision) - save_contract(contract, repo_path) - logger.info( - "Persisted phase gate resolution to contract", - pipeline_id=pipeline_id, - phase=phase, - ) - except ImportError: - logger.warning("egg_contracts not available, skipping phase gate contract sync") - except Exception: - logger.warning( - "Failed to persist phase gate resolution to contract (continuing)", - pipeline_id=pipeline_id, - phase=phase, - exc_info=True, - ) - - # --- 2. Append to draft --- - try: - draft_rel = _get_draft_path(phase, issue_number, pipeline_id) - if draft_rel: - draft_path = repo_path / draft_rel - if draft_path.exists(): - existing = draft_path.read_text(encoding="utf-8") - if "## HITL Resolution" not in existing: - section = ( - f"\n\n## HITL Resolution\n\n" - f"The following was approved by a human reviewer at the " - f"{phase} phase gate:\n\n{resolution_context}\n" - ) - draft_path.write_text(existing + section, encoding="utf-8") - logger.info( - "Appended HITL resolution to draft", - pipeline_id=pipeline_id, - phase=phase, - draft=draft_rel, - ) - except Exception: - logger.warning( - "Failed to append phase gate resolution to draft (continuing)", - pipeline_id=pipeline_id, - phase=phase, - exc_info=True, - ) - - -def _spawn_pipeline_run_thread( - pipeline_id: str, - repo_path: Path, - run_epoch: datetime, -) -> threading.Thread: - """Spawn a fresh ``_run_pipeline`` driver thread. - - Callers (all use the ``pipeline-{id}-{epoch}`` naming scheme): - - - ``advance_phase`` (manual phase advance via REST) - - ``restart_phase`` (manual phase restart via REST) - - the auto-advance block in ``_run_pipeline`` (#2165) - - The other ``_run_pipeline`` thread spawn sites — ``start_pipeline``'s - initial-spawn and AWAITING_HUMAN-recovery paths, plus the spurious-PNFE - respawn inside ``_run_pipeline`` — use different naming or take extra - kwargs (e.g. ``_respawn_attempt``) and are deliberately left inline. - - Without a fresh thread per phase, a mid-execution exception in the new - phase's first iteration takes down the whole pipeline (#2165). - """ - thread = threading.Thread( - target=_run_pipeline, - args=(pipeline_id, repo_path), - daemon=True, - name=f"pipeline-{pipeline_id}-{int(run_epoch.timestamp())}", - ) - thread.start() - return thread - - -def has_live_pipeline_driver(pipeline_id: str) -> bool: - """Return True if a live ``_run_pipeline`` driver thread owns this pipeline. - - Driver threads are named ``pipeline-{id}`` (``start_pipeline``'s initial - and AWAITING_HUMAN-recovery spawns), ``pipeline-{id}-{epoch}`` - (``_spawn_pipeline_run_thread``), or ``pipeline-{id}-respawn-...`` (the - spurious-PNFE recovery). Every variant is either exactly ``pipeline-{id}`` - or carries a ``pipeline-{id}-`` prefix, so the literal-hyphen boundary - keeps a pipeline whose id is a prefix of another (``issue-3`` vs - ``issue-32``) from matching. - - After an orchestrator restart the process holds no driver threads, which - is precisely the orphaned-parked condition behind #3233: a pipeline left - AWAITING_HUMAN with a pending decision has no thread polling - ``wait_for_decision``, so a later resolution is recorded with no consumer - and the pipeline hangs silently. - """ - exact = f"pipeline-{pipeline_id}" - prefix = exact + "-" - for t in threading.enumerate(): - if not t.is_alive(): - continue - if t.name == exact or t.name.startswith(prefix): - return True - return False - - -def relaunch_driverless_running_pipelines(store) -> int: - """Relaunch drivers for RUNNING pipelines orphaned by a restart (#3469). - - Called once per repo store at orchestrator startup, after - ``startup_reconciliation.reconcile_stale_containers`` has settled each - pipeline's status. A pipeline still RUNNING at that point was mid-flight - when the previous orchestrator process died: its consensus state is fully - reconciled at boot, but its ``_run_pipeline`` driver thread — and the BRC - event loop the driver owns — died with the old process, and no other code - path revives it. ``restart_agent`` delegates the respawn to the (dead) - event loop and returns success, while ``start_pipeline`` rejects - status=RUNNING with a 409, so without this sweep the pipeline is - permanently driverless and never spawns another pod (#3469). - - Relaunching reuses the proven resume path (the same one - ``restart_agent``'s inactive-pipeline branch relies on, #3244): - ``_run_pipeline`` re-enters ``pipeline.current_phase``, re-syncs the - worktree with the remote, and restarts the event loop, which respawns - one-shot agent Jobs within one poll. The persisted ``run_epoch`` is - deliberately NOT bumped: the old process is gone so no stale thread can - contend for the epoch, and the relaunched thread derives its own epoch - from persisted state exactly as the original did. - - AWAITING_HUMAN pipelines are out of scope — their drivers are revived on - decision resolution by ``maybe_revive_orphaned_awaiting_human_driver`` - (#3233). - - The sweep iterates ``store.get_active_pipelines()`` rather than the full - ``list_pipelines()`` so terminal/historical records (COMPLETE, FAILED, - CANCELLED) are skipped without a redundant load — reconciliation already - walked every pipeline immediately before this, and re-scanning the whole - store would double the boot-time git reads on repos with many historical - pipelines. - - Returns the number of drivers relaunched. Failures are isolated at two - layers: a record that fails to load with ``StateStoreError`` (corruption) - is skipped inside ``get_active_pipelines()``, and a per-pipeline failure - during the relaunch itself (driver probe or thread spawn) is logged and - skipped so one bad pipeline cannot strand the rest. The one case that is - *not* isolated: a load failure other than ``StateStoreError`` propagates out - of ``get_active_pipelines()`` and the outer ``except`` aborts the sweep - (returns 0) — an accepted trade for using the canonical active-pipeline - accessor, matching how ``get_active_pipelines()`` behaves for its other - callers. - """ - try: - active_pipelines = store.get_active_pipelines() - except Exception as e: # noqa: BLE001 - startup sweep must not raise - logger.warning( - "Driver relaunch sweep skipped: could not list active pipelines", - error=str(e), - ) - return 0 - - relaunched = 0 - for pipeline in active_pipelines: - try: - if pipeline.status != PipelineStatus.RUNNING: - continue - if has_live_pipeline_driver(pipeline.id): - continue - run_epoch = pipeline.run_epoch or pipeline.created_at - _spawn_pipeline_run_thread(pipeline.id, store.repo_path, run_epoch) - relaunched += 1 - logger.warning( - "Relaunched _run_pipeline driver for RUNNING pipeline with no " - "live driver thread (orchestrator restart recovery, #3469)", - pipeline_id=pipeline.id, - phase=pipeline.current_phase.value, - run_epoch=run_epoch.isoformat(), - ) - except Exception as e: # noqa: BLE001 - per-pipeline isolation - logger.warning( - "Failed to relaunch driver for RUNNING pipeline (continuing sweep)", - pipeline_id=getattr(pipeline, "id", "unknown"), - error=str(e), - ) - return relaunched - - -def _broadcast_orphaned_driver_alert(pipeline_id: str, pipeline: Pipeline) -> None: - """Surface an orphaned-driver revival as an overseer alert (#3233). - - A resolved decision on a driver-less pipeline used to return ``success`` - and hang invisibly. Emit an OVERSEER_ALERT alongside the WARNING log so - the recovery is visible on the bus, not just in orchestrator logs. - Best-effort: a broadcast failure never blocks the revival itself. - """ - try: - from message_store import Message, MessageType - - store_fn = _get_message_store() - if store_fn is None: - return - msg_store = store_fn() - phase = pipeline.current_phase.value if pipeline.current_phase else None - msg_store.add_message( - Message( - pipeline_id=pipeline_id, - from_role="orchestrator", - to_role="all", - message_type=MessageType.OVERSEER_ALERT, - subject="orphaned_driver_revived: orchestrator [medium]", - body=( - "A HITL decision was resolved on a pipeline whose " - "_run_pipeline driver thread did not survive an " - "orchestrator restart. The driver is being re-launched so " - "the resolution is acted on (no manual start_pipeline " - "needed). See #3233." - ), - metadata={"reason": "restart_orphaned_awaiting_human"}, - phase=phase, - ) - ) - except Exception as alert_err: # noqa: BLE001 - logger.warning( - "Failed to broadcast orphaned-driver revival alert (non-fatal)", - pipeline_id=pipeline_id, - error=str(alert_err), - ) - - -def maybe_revive_orphaned_awaiting_human_driver(pipeline_id: str, repo_path: Path) -> bool: - """Re-launch the driver for an AWAITING_HUMAN pipeline orphaned by a restart. - - Called from the decision-resolve path (#3233). When the orchestrator - restarts while a pipeline is parked AWAITING_HUMAN at a phase gate, the - in-memory ``_run_pipeline`` driver blocked on ``wait_for_decision`` is - gone and startup reconciliation deliberately leaves the still-pending - decision as-is (``startup_reconciliation.py``). Resolving the decision - then flips it to RESOLVED with no consumer and the pipeline hangs - silently — the operator sees ``success`` and nothing happens. - - This detects that no live driver owns the pipeline and, once the queue - has no remaining pending decisions, routes through ``start_pipeline``'s - proven AWAITING_HUMAN recovery branch (advance-or-rerun + driver respawn) - so the resolution self-heals without a manual ``start_pipeline``. - - No-ops (returns ``False``) when a live driver is already polling — the - normal in-process path consumes the resolution — or when the pipeline - isn't in the orphaned-parked state. Must be called from a Flask request - context (it reuses the lifecycle-secret-guarded ``start_pipeline`` route), - which the resolve-decision handler satisfies. - """ - store = get_state_store(repo_path) - try: - pipeline = store.load_pipeline(pipeline_id) - except Exception: - return False - - if pipeline.status != PipelineStatus.AWAITING_HUMAN: - return False - # A multi-decision batch (e.g. the contract-decision bridge) is only - # ready to resume once every decision is resolved; leave it parked while - # siblings are still pending. - if pipeline.get_pending_decisions(): - return False - if has_live_pipeline_driver(pipeline_id): - return False - - logger.warning( - "Decision resolved on AWAITING_HUMAN pipeline with no live driver " - "thread (orphaned by an orchestrator restart); reviving via " - "start_pipeline recovery so the resolution is acted on (#3233)", - pipeline_id=pipeline_id, - current_phase=pipeline.current_phase.value if pipeline.current_phase else None, - ) - _broadcast_orphaned_driver_alert(pipeline_id, pipeline) - - try: - _resp, status_code = start_pipeline(pipeline_id) - except Exception as revive_err: # noqa: BLE001 - logger.warning( - "Orphaned-driver revival raised (decision is still resolved; an " - "operator can recover manually via start_pipeline) (#3233)", - pipeline_id=pipeline_id, - error=str(revive_err), - ) - return False - - if status_code != 200: - logger.warning( - "Orphaned-driver revival did not start the pipeline (#3233)", - pipeline_id=pipeline_id, - status_code=status_code, - ) - return False - - logger.info( - "Orphaned AWAITING_HUMAN pipeline revived after decision resolution (#3233)", - pipeline_id=pipeline_id, - ) - return True - - -# Tunables for the spurious-PipelineNotFoundError recovery path in -# ``_run_pipeline``. The verify retry covers the empty-file race window -# during a ``git commit`` truncate-and-rewrite on the state worktree -# (typical: <100ms); 3 × 200ms gives ~600ms of total slack. The respawn -# cap bounds how aggressively a persistent transient can leak threads, -# overseer containers, and state-branch commits before we fail the -# pipeline outright. See #2155. -_PNFE_VERIFY_ATTEMPTS = 3 -_PNFE_VERIFY_INTERVAL = 0.2 # seconds between verify retries -_PNFE_RESPAWN_MAX_ATTEMPTS = 5 # cap on respawn cascade -_PNFE_RESPAWN_BACKOFF_CAP = 30 # seconds, exponential backoff ceiling - - -def _run_pipeline( - pipeline_id: str, - repo_path: Path, - _respawn_attempt: int = 0, -) -> None: - """Run a pipeline by spawning containers for each phase. - - This runs in a background thread. For each phase it: - 1. Spawns agent containers via concurrent BRC execution - (_run_concurrent_phase) for all phases. - 2. For reviewed phases (refine, implement, plan): reviewers participate - in the BRC consensus protocol alongside workers, then the phase - loops back with feedback if revision is needed. - 3. Advances to the next phase once approved. - - Args: - pipeline_id: Pipeline ID - repo_path: Path to repository - _respawn_attempt: Internal — counts how many times this thread - has been respawned by the spurious-PNFE recovery path. - Bounded by ``_PNFE_RESPAWN_MAX_ATTEMPTS`` to prevent a - persistent transient from cascading into an unbounded - thread/overseer/commit storm. - """ - from routes.phases import PHASE_TRANSITIONS - - # Track which run of the pipeline this thread owns. If the pipeline - # is deleted and recreated with the same ID while we're still running, - # the new run creates its own worktrees under the same path. Without - # this guard, our finally block would delete the *new* run's worktrees. - run_epoch: datetime | None = None - overseer_container_id: str | None = None - phase_overseer_active: bool = False - overseer_lock = threading.Lock() - health_monitor_instance = None - health_monitor_timer: threading.Event | None = None - poll_thread: threading.Thread | None = None - - try: - store = get_state_store(repo_path) - spawner = _get_spawner() - pipeline = store.load_pipeline(pipeline_id) - run_epoch = pipeline.run_epoch or pipeline.created_at - pipeline_mode = "issue" if pipeline.issue_number is not None else "prompt" - transitions = PHASE_TRANSITIONS - - def _make_overseer_teardown_hook( - *, - reason: str, - container_id: str | None, - phase: PipelinePhase, - ) -> Callable[[], None]: - """Build a pre_event_hook that tears down the per-phase overseer. - - ``container_id`` and ``phase`` are snapshotted as function - parameters (frozen per-call), so the returned closure binds - the loop-iteration values that were current when the - post-phase cleanup branch fired — late binding would race a - subsequent loop iteration. ``reason`` differs between the - doubly-failed and hard-reset-recovered call sites and is - forwarded to :func:`_teardown_phase_overseer`. - - #2797 follow-up: collapses the two duplicated closure - definitions at the two post-phase hard-reset emission sites - into one shared factory. The closure remains inside - ``_run_pipeline`` because the ``phase_overseer_active`` - bool is a local nonlocal of this function. - """ - - def _hook() -> None: - nonlocal phase_overseer_active - with overseer_lock: - if container_id and phase_overseer_active: - phase_overseer_active = False - _teardown_phase_overseer( - spawner, - container_id, - pipeline_id, - phase_label=str(phase), - reason=reason, - ) - - return _hook - - # Map pipeline to gateway session mode. - gateway_mode, detected_visibility = _compute_gateway_mode(pipeline) - if not pipeline.network_mode and pipeline.repo: - if detected_visibility is not None: - logger.info( - "Auto-detected network mode from repo visibility", - repo=pipeline.repo, - visibility=detected_visibility, - gateway_mode=gateway_mode, - ) - else: - logger.warning( - "Could not detect repo visibility, defaulting to public mode", - repo=pipeline.repo, - ) - - # Parse host repo map for volume mounts. When the orchestrator - # runs inside Docker, EGG_REPO_PATH is the *container* path but - # volume mounts need *host* paths (since the Docker socket - # operates on the host daemon). EGG_HOST_REPO_MAP provides a - # JSON mapping of repo_name -> host_path, auto-generated from - # repositories.yaml by the egg launcher. - host_repo_map_raw = os.environ.get("EGG_HOST_REPO_MAP", "{}") - try: - host_repo_map: dict[str, str] = json.loads(host_repo_map_raw) - except json.JSONDecodeError as exc: - logger.error( - "Failed to parse EGG_HOST_REPO_MAP — no repos will be mounted in sandbox containers", - raw_value=host_repo_map_raw, - ) - raise ValueError( - f"EGG_HOST_REPO_MAP contains invalid JSON: {host_repo_map_raw!r}" - ) from exc - - # Create a pipeline-level worktree via the gateway. This worktree - # is used by the orchestrator for reading/writing contracts, drafts, - # and state files. Individual agents get their own per-agent - # worktrees at spawn time (created in container_spawner.py) so - # concurrent agents cannot stomp on each other's uncommitted work. - # See #1481 for the per-agent worktree isolation design. - # - # We use the pipeline_id as the worktree container_id for the - # orchestrator-side worktree. Agent worktrees use - # "{pipeline_id}-{role}" as their container_id. - worktree_id = pipeline_id - repo_volumes: dict[str, str] = {} - worktree_repo_path = repo_path # default; overridden when worktrees exist - host_uid = int(os.environ.get("HOST_UID", 1000)) - host_gid = int(os.environ.get("HOST_GID", 1000)) - pipeline_repos = [pipeline.repo] if pipeline.repo else [] - - if host_repo_map: - try: - # Request repos in owner/repo format if available, else bare names - wt_repos = pipeline_repos if pipeline_repos else list(host_repo_map.keys()) - # When the pipeline specifies a base_branch, pass it through - # so the worktree is branched from that ref instead of the - # repo's default branch. Otherwise let the gateway resolve - # the remote default branch per-repo (see #860). - # Retry worktree creation on transient gateway errors - # (e.g., 500s from concurrent pipeline starts contending - # on per-repo locks). See #1386. - wt_max_attempts = 3 - wt_backoff = 2.0 - wt_result = None - for wt_attempt in range(1, wt_max_attempts + 1): - try: - wt_result = spawner.gateway.create_worktrees( - container_id=worktree_id, - repos=wt_repos, - uid=host_uid, - gid=host_gid, - base_branch=pipeline.base_branch, - ) - break # Success — exit retry loop - except GatewayError as gw_err: - is_transient = gw_err.status_code is None or gw_err.status_code >= 500 - if not is_transient or wt_attempt == wt_max_attempts: - # Surface gw_err.details so per-repo failures - # captured by the gateway aren't dropped. See - # #2186. - logger.error( - "Worktree creation failed permanently", - pipeline_id=pipeline_id, - attempts=wt_attempt, - status_code=gw_err.status_code, - error_message=gw_err.message, - details=gw_err.details, - ) - detail_suffix = ( - f" (details: {gw_err.details})" if gw_err.details else "" - ) - raise RuntimeError( - f"Failed to create worktrees for pipeline {pipeline_id} " - f"after {wt_max_attempts} attempts: " - f"{gw_err.message}{detail_suffix}" - ) from gw_err - logger.warning( - "Worktree creation failed, retrying", - pipeline_id=pipeline_id, - attempt=wt_attempt, - max_attempts=wt_max_attempts, - error=str(gw_err), - details=gw_err.details, - ) - time.sleep(wt_backoff) - wt_backoff *= 2 - - if wt_result and wt_result.success and wt_result.worktrees: - # Gateway returns worktrees keyed by the full ``owner/repo`` - # slug (#3393 slice-3, operator ruling #6). The on-disk - # worktree directory (and the container mount target) is - # still the bare repo name at /home/egg/repos/<name>, so - # the path reconstruction below strips the owner prefix - # from each key. - repo_volumes = wt_result.worktrees - - # Derive the orchestrator-accessible worktree path. - # Reviewer containers write verdict/draft/check files into - # the worktree, so the orchestrator must read from there. - # Match against pipeline.repo (full owner/repo slug, which - # is now the map key) explicitly to avoid picking the wrong - # repo in multi-repo pipelines. - matched = False - if pipeline.repo and pipeline.repo in wt_result.worktrees: - repo_short = pipeline.repo.split("/")[-1] - candidate = WORKTREE_BASE_DIR / worktree_id / repo_short - if candidate.exists(): - worktree_repo_path = candidate - matched = True - if not matched: - # Fallback: take the first existing worktree path. - # Keys are ``owner/repo``; the on-disk dir is the bare - # leaf, so strip the owner prefix before joining. - for owner_repo in wt_result.worktrees: - candidate = WORKTREE_BASE_DIR / worktree_id / owner_repo.split("/")[-1] - if candidate.exists(): - worktree_repo_path = candidate - break - - logger.info( - "Worktrees created for pipeline", - pipeline_id=pipeline_id, - worktrees=list(repo_volumes.keys()), - ) - else: - raise RuntimeError( - f"Worktree creation returned no worktrees for pipeline {pipeline_id}: " - f"errors={wt_result.errors}" - ) - - if wt_result.errors: - for err in wt_result.errors: - logger.warning("Worktree error", pipeline_id=pipeline_id, error=err) - - except RuntimeError: - raise # Re-raise our own RuntimeError - except Exception as wt_err: - raise RuntimeError( - f"Failed to create worktrees for pipeline {pipeline_id}: {wt_err}" - ) from wt_err - - if not repo_volumes: - raise RuntimeError( - f"No repo volumes available for pipeline {pipeline_id} — " - f"worktree creation is required" - ) - - # Sync worktree with remote before starting pipeline phases. After an - # orchestrator restart, the local worktree branch may be behind origin: - # commits pushed by agents in previous phases (contracts, drafts, - # statefiles) exist on the remote but not in the local checkout. - # Fetching and resetting ensures downstream code (contract loading, - # draft reading) sees the full pipeline state from prior phases. - if worktree_repo_path != repo_path: - # Determine whether the most recent prior phase completed - # successfully — this controls whether local-ahead commits are - # pushed (success) or discarded (failure). - prior_phase_succeeded = True - current_phase = pipeline.current_phase - phase_order = [ - PipelinePhase.REFINE, - PipelinePhase.PLAN, - PipelinePhase.IMPLEMENT, - ] - current_idx = phase_order.index(current_phase) if current_phase in phase_order else 0 - if current_idx > 0: - prior_phase = phase_order[current_idx - 1] - prior_exec = pipeline.phases.get(prior_phase.value) - if prior_exec and prior_exec.status in ( - PipelineStatus.FAILED, - PipelineStatus.CANCELLED, - ): - prior_phase_succeeded = False - - # #2979: sync the worktree, pausing for a manual reconcile if - # it diverges and the rebase autoresolve can't reconcile it. - # The helper blocks (AWAITING_HUMAN) on a reconcile HITL and - # resumes the phase start once the operator acks — nothing is - # discarded and the pipeline is never failed for a recoverable - # divergence. - phase_start_sync_outcome, phase_start_sync_aborted = ( - _sync_worktree_reconciling_divergence( - spawner, - pipeline_id, - store, - repo_path, - worktree_repo_path=worktree_repo_path, - phase=current_phase, - gateway_mode=gateway_mode, - base_branch=pipeline.base_branch, - pipeline_branch=pipeline.branch, - prior_phase_succeeded=prior_phase_succeeded, - ) - ) - if phase_start_sync_aborted: - # Operator aborted the manual reconcile (or the pause - # budget was exhausted). Fail the pipeline; the local - # commits remain pinned under the backup ref for offline - # recovery — nothing was discarded. - _fail_pipeline_after_divergence_abort( - pipeline_id, - store, - phase=current_phase, - backup_ref=phase_start_sync_outcome.backup_ref, - local_only_commit_shas=phase_start_sync_outcome.local_only_commit_shas, - ) - return - - # When resuming a stale pipeline branch (cancelled run from - # days/weeks ago), rebase origin/<branch> onto origin/<base> - # before any orchestrator/agent commits land — otherwise the - # final PR carries 70+ stale-from-main commits as ancestors - # (#2098). No-op for fresh pipelines and for branches already - # caught up with base. - if pipeline.branch and pipeline.base_branch: - try: - _rebase_pipeline_branch_onto_base( - spawner, - pipeline_id, - worktree_repo_path, - pipeline_branch=pipeline.branch, - base_branch=pipeline.base_branch, - gateway_mode=gateway_mode, - ) - except StalePipelineBranchError as stale_err: - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - pipeline.status = PipelineStatus.FAILED - pipeline.error = str(stale_err) - store.save_pipeline(pipeline) - return - - # Remove legacy unprefixed draft files (analysis.md, plan.md) - # that may have been left by earlier pipelines on this branch. - # Uses git rm so deletions are committed directly. See #1559. - cleanup_committed = _cleanup_stale_generic_drafts(worktree_repo_path) - if cleanup_committed and pipeline.branch: - try: - spawner.gateway.push_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(worktree_repo_path), - branch=pipeline.branch, - mode=gateway_mode, - base_branch=pipeline.base_branch, - ) - except Exception: - logger.warning( - "Failed to push stale draft cleanup (continuing)", - pipeline_id=pipeline_id, - ) - - # Resolve the certs named volume for gateway CA trust. - # The docker-compose stack creates ${COMPOSE_PROJECT_NAME:-egg}-certs. - certs_volume_raw = os.environ.get( - "EGG_CERTS_VOLUME", - os.environ.get("COMPOSE_PROJECT_NAME", "egg") + "-certs", - ) - # Validate volume name: Docker allows [a-zA-Z0-9][a-zA-Z0-9_.-]* - # We use a permissive check that rejects obvious shell metacharacters. - if not re.match(r"^[a-zA-Z0-9][a-zA-Z0-9_.-]*$", certs_volume_raw): - logger.warning( - "Invalid certs volume name, using default", - raw_name=certs_volume_raw, - ) - certs_volume = "egg-certs" - else: - certs_volume = certs_volume_raw - - # Capture source_branch before _read_source_branch_artifacts clears - # it on success — the contract-pull path below (#2035) runs inside - # the contract_synced block and otherwise wouldn't see the value. - source_branch_for_contract_pull = pipeline.source_branch - - # Read artifacts from source branch if specified and inline values - # were not provided. This populates pipeline.plan and - # pipeline.analysis so the contract creation block below can use them. - if pipeline.source_branch and not ( - pipeline.plan is not None and pipeline.analysis is not None - ): - # source_branch is cleared inside _read_source_branch_artifacts - # when artifacts are actually found. - try: - _read_source_branch_artifacts( - repo_path=worktree_repo_path, - source_branch=pipeline.source_branch, - issue_number=pipeline.issue_number, - pipeline_id=pipeline_id, - store=store, - pipeline=pipeline, - source_artifact_prefix=pipeline.source_artifact_prefix, - spawner=spawner, - gateway_mode=gateway_mode, - ) - except Exception: - logger.warning( - "Failed to read artifacts from source branch", - source_branch=pipeline.source_branch, - pipeline_id=pipeline_id, - exc_info=True, - ) - - # Write source-branch artifacts to disk so the safety-net - # _populate_contract_from_plan() call below can find them. - # The inline-plan path writes drafts inside the contract_synced - # block, but that block is skipped on pipeline restarts - # (contract already synced). Writing here ensures the draft - # files exist regardless of contract_synced state. - if pipeline.plan is not None or pipeline.analysis is not None: - drafts_dir = worktree_repo_path / ".egg-state" / "drafts" - drafts_dir.mkdir(parents=True, exist_ok=True) - - if pipeline.plan is not None: - plan_rel = _get_draft_path( - "plan", - issue_number=pipeline.issue_number, - pipeline_id=pipeline_id, - ) - if plan_rel: - plan_path = worktree_repo_path / plan_rel - plan_path.write_text(pipeline.plan, encoding="utf-8") - logger.info( - "Wrote source-branch plan draft to worktree", - pipeline_id=pipeline_id, - path=plan_rel, - ) - - if pipeline.analysis is not None: - analysis_rel = _get_draft_path( - "refine", - issue_number=pipeline.issue_number, - pipeline_id=pipeline_id, - ) - if analysis_rel: - analysis_path = worktree_repo_path / analysis_rel - analysis_path.write_text(pipeline.analysis, encoding="utf-8") - logger.info( - "Wrote source-branch analysis draft to worktree", - pipeline_id=pipeline_id, - path=analysis_rel, - ) - - # Create companion contract in the worktree (deferred from pipeline - # creation so it doesn't pollute the main repo working directory). - if not pipeline.contract_synced: - try: - from egg_contracts.loader import compose_task_description, create_contract - - # Every entry path (GitHub issue, JIRA, free-text) anchors - # the task the same way (#3163): identity first, then the - # operator's submit description. Before #3163 issue - # pipelines deliberately got ``None`` here (#3042 "agents - # fetch the live body"), which left the #3123 binding - # prompt section empty for the most common pipeline type. - issue_url = ( - f"https://github.com/{pipeline.repo}/issues/{pipeline.issue_number}" - if pipeline.issue_number is not None - else None - ) - task_description = compose_task_description( - description=pipeline.prompt, - issue_number=pipeline.issue_number, - issue_url=issue_url, - jira_ticket=pipeline.jira_ticket, - ) - - # When source_branch is set, try to carry over the contract - # (with any resolved HITL decisions) from there instead of - # overwriting with a fresh zero-state contract (#2035). - pulled_contract = False - if source_branch_for_contract_pull: - try: - pulled_contract = _pull_contract_from_source_branch( - repo_path=worktree_repo_path, - source_branch=source_branch_for_contract_pull, - issue_number=pipeline.issue_number, - pipeline_id=pipeline.id, - spawner=spawner, - gateway_mode=gateway_mode, - task_description=task_description, - ) - except Exception: - logger.warning( - "Unexpected error pulling contract from source branch — falling back to fresh contract", - pipeline_id=pipeline_id, - source_branch=source_branch_for_contract_pull, - exc_info=True, - ) - pulled_contract = False - - if not pulled_contract: - if pipeline.issue_number is not None: - create_contract( - issue_number=pipeline.issue_number, - title=f"Issue #{pipeline.issue_number}", - url=issue_url or "", - pipeline_id=pipeline.id, - repo_root=worktree_repo_path, - task_description=task_description, - ) - else: - # ``pipeline.issue_number is None`` covers both - # free-text submits and JIRA-driven pipelines - # (``pipeline.jira_ticket`` set). The event-pump - # never delivers the orchestrator-built spawn - # prompt to the agent, so the contract (read via - # ``egg-contract show`` + the #3123 prompt - # section) is the reliable channel for the - # complete task; the ``title`` arg is only used - # for the ``IssueInfo`` label and is dropped - # without an ``issue_number``, so it is not a - # substitute (#3033). - create_contract( - pipeline_id=pipeline.id, - title=(pipeline.prompt or "")[:100], - task_description=task_description, - repo_root=worktree_repo_path, - ) - - # Write pre-generated drafts for short-flow pipelines so the - # existing plan parser can populate the contract with tasks. - if pipeline.analysis or pipeline.plan: - drafts_dir = worktree_repo_path / ".egg-state" / "drafts" - drafts_dir.mkdir(parents=True, exist_ok=True) - - if pipeline.analysis: - analysis_rel = _get_draft_path( - "refine", - issue_number=pipeline.issue_number, - pipeline_id=pipeline_id, - ) - if analysis_rel: - (worktree_repo_path / analysis_rel).write_text( - pipeline.analysis, encoding="utf-8" - ) - logger.info( - "Wrote pre-generated analysis draft", - pipeline_id=pipeline_id, - path=analysis_rel, - ) - - if pipeline.plan: - plan_rel = _get_draft_path( - "plan", - issue_number=pipeline.issue_number, - pipeline_id=pipeline_id, - ) - if plan_rel: - (worktree_repo_path / plan_rel).write_text( - pipeline.plan, encoding="utf-8" - ) - logger.info( - "Wrote pre-generated plan draft", - pipeline_id=pipeline_id, - path=plan_rel, - ) - - # Populate the contract from the plan's yaml-tasks appendix - _inline_plan_populate_result = _populate_contract_from_plan( - worktree_repo_path, - pipeline_id, - pipeline_mode, - pipeline.issue_number, - ) - # #2627 follow-up: warn-and-continue on non-POPULATED. - # This is the initial-contract creation path (a - # pre-generated plan handed to ``start_pipeline``); - # failing here would block legitimate pipelines that - # recover via the natural plan-phase populator a few - # blocks later. We only attach the structured - # outcome as audit signal. - if _inline_plan_populate_result.outcome != PopulateOutcome.POPULATED: - logger.warning( - "Pre-generated plan populate produced non-POPULATED outcome", - pipeline_id=pipeline_id, - outcome=_inline_plan_populate_result.outcome.value, - ) - - # Commit all .egg-state/ files so they're on the feature branch - issue_ref = ( - f"issue #{pipeline.issue_number}" - if pipeline.issue_number is not None - else f"pipeline {pipeline_id}" - ) - try: - _commit_statefiles_to_worktree( - worktree_repo_path, - f"Initialize SDLC contract for {issue_ref}", - pipeline_identifier=_pipeline_identifier( - pipeline.issue_number, pipeline_id - ), - pipeline_id=pipeline_id, - ) - except Exception as git_err: - # Catch broadly so TimeoutExpired/OSError also produce - # an explicit FAILED state rather than silently - # propagating to the outer handler (#2219). - logger.error( - "Failed to commit initial statefiles — aborting pipeline", - pipeline_id=pipeline_id, - error=str(git_err), - ) - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - pipeline.status = PipelineStatus.FAILED - pipeline.contract_synced = False - pipeline.error = f"Failed to commit initial statefiles: {git_err}" - store.save_pipeline(pipeline) - return - - # Push contract statefiles to remote so agents see them. - # This MUST succeed before agents start — otherwise agents' - # diffs will include .egg-state/ files they can't push (#1431). - push_succeeded = False - # For prompt-driven pipelines, pipeline.branch is None at this - # point — the branch name is only persisted later when the - # agent container is spawned (line ~6279). Derive it here so - # the push actually happens. The worktree was already created - # on this branch by the gateway. - push_branch = pipeline.branch or f"egg/{pipeline_id}/work" - if not pipeline.branch: - pipeline.branch = push_branch - with get_pipeline_state_lock(pipeline_id): - p = store.load_pipeline(pipeline_id) - if not p.branch: - p.branch = push_branch - store.save_pipeline(p) - logger.info( - "Recorded generated branch on pipeline (pre-push)", - pipeline_id=pipeline_id, - branch=push_branch, - ) - if worktree_repo_path != repo_path: - push_err_msg = "" - # push_worktree_branch reconciles non-fast-forward - # rejections internally (fetch+rebase+retry), so a - # single call is sufficient — no outer retry needed. - try: - push_result = spawner.gateway.push_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(worktree_repo_path), - branch=push_branch, - mode=gateway_mode, - base_branch=pipeline.base_branch, - ) - push_succeeded = bool(push_result) - if not push_succeeded: - push_err_msg = push_result.describe() - except Exception as push_err: - push_succeeded = False - push_err_msg = str(push_err) - - if not push_succeeded: - logger.error( - "Contract init push failed after retry — aborting pipeline", - pipeline_id=pipeline_id, - error=push_err_msg, - ) - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - pipeline.status = PipelineStatus.FAILED - pipeline.contract_synced = False - pipeline.error = ( - f"Failed to push contract init to remote: {push_err_msg}" - ) - store.save_pipeline(pipeline) - return - else: - logger.warning( - "Skipped contract init push — worktree path equals repo path", - pipeline_id=pipeline_id, - worktree_repo_path=str(worktree_repo_path), - repo_path=str(repo_path), - ) - - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - pipeline.contract_synced = push_succeeded - store.save_pipeline(pipeline, commit=False) - logger.info( - "Pipeline contract created in worktree", - pipeline_id=pipeline_id, - mode=pipeline_mode, - ) - except Exception as contract_err: - logger.error( - "Failed to create contract in worktree", - pipeline_id=pipeline_id, - error=str(contract_err), - ) - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - pipeline.status = PipelineStatus.FAILED - pipeline.error = f"Failed to create contract: {contract_err}" - store.save_pipeline(pipeline) - return - - # Safety net: when start_phase=implement, the plan phase is - # skipped so the plan-completion hook at the end of the phase loop - # never fires. The inline-plan path above calls - # _populate_contract_from_plan inside the contract_synced block, - # but that block is skipped on pipeline restarts (contract already - # synced) and when _read_source_branch_artifacts writes the draft - # file to the worktree without going through the inline-plan - # branch. This catch-all ensures the contract has phases and - # tasks before agents spawn when the plan phase was skipped. - # When start_phase=plan, the plan phase runs normally and the - # plan-completion hook populates the contract, so no safety net - # is needed. - if pipeline.config.start_phase == "implement": - plan_draft_rel = _get_draft_path( - "plan", - issue_number=pipeline.issue_number, - pipeline_id=pipeline.id, - ) - if plan_draft_rel and (worktree_repo_path / plan_draft_rel).exists(): - # Advance contract.current_phase alongside slice/PR - # ingestion. In the natural flow contract.current_phase - # is mutated by the plan reviewer agent (or the gateway - # phase API) via apply_mutation; with start_phase=implement - # no such reviewer ever runs, so the contract would stay - # at REFINE forever (#2427 sub-bug). We pass - # pipeline.current_phase rather than a hardcoded literal - # so the right value follows automatically if start_phase - # ever supports values other than 'implement'. The - # populator enforces forward-only advancement, so a - # respawn during the PR phase cannot demote the contract. - # Note: the *outer* guard above remains hardcoded to - # ``"implement"``; widening it to other start_phase values - # is a two-line change (this guard plus the matching - # ``initial_phase`` mapping in start_pipeline). - # Catch ``ForestValidationError`` here so a malformed - # plan landing at the safety-net path lands on the - # dedicated empty-contract HITL — the same recovery - # surface the natural plan-complete path uses via - # :func:`_populate_contract_from_plan_safe`'s - # forest-violation translation. Without this catch the - # safety net (which calls the inner directly so the - # ``PlanDraftMissing*`` raises don't fire here) would - # propagate the exception to the outer pipeline - # ``except`` and the operator would see a generic - # ``status: failed`` instead of the actionable - # repopulate/restart-plan/abort decision (#2627 review). - try: - _safety_net_populate_result = _populate_contract_from_plan( - worktree_repo_path, - pipeline_id, - pipeline_mode, - pipeline.issue_number, - current_phase=pipeline.current_phase, - ) - except ForestValidationError as forest_err: - # #3046 — overlap violations map to their own outcome so - # the empty-contract HITL prose matches the discriminator. - logger.warning( - "contract_phases_ingest_failed", - pipeline_id=pipeline_id, - reason=forest_err.reason, - source="safety_net", - errors=forest_err.errors, - ) - _safety_net_populate_result = PopulateResult( - _forest_error_to_outcome(forest_err) - ) - # #2627 follow-up: fail-fast whenever the safety-net populate - # did not produce a contract with tasks. Without this guard - # the implement phase spawns into the same empty-contract - # state that #2627 surfaced — the slice-gate at - # implement-phase entry would eventually catch it, but at - # that point the pipeline has already advanced and the - # operator sees the empty-contract divergence after the - # loop is running. Catching it here is earlier and cheaper. - # - # Routes through :func:`_populate_result_is_empty_contract` - # so the two empty-contract call sites (this safety net - # and the natural plan-complete handler below) can't drift - # out of agreement. See that helper's docstring for the - # full discriminator rules. - if _populate_result_is_empty_contract(_safety_net_populate_result): - # Reason dispatch shared with the plan-complete handler - # via :func:`_populate_outcome_to_hitl_reason` so the - # POPULATED → "populated_but_empty_slices" translation - # (and any future special-cased outcome) can't drift - # between the two call sites (#2627 review follow-up). - _safety_net_reason = _populate_outcome_to_hitl_reason( - _safety_net_populate_result.outcome - ) - if _safety_net_populate_result.outcome == PopulateOutcome.POPULATED: - _safety_net_error = ( - "start_phase=implement safety-net populate " - "completed but produced 0 slices/tasks — refusing " - "to spawn implement-phase agents on an empty " - "contract (#2627)" - ) - else: - _safety_net_error = ( - f"start_phase=implement safety-net populate produced " - f"{_safety_net_populate_result.outcome.value} outcome — " - f"refusing to spawn implement-phase agents on an " - f"empty contract (#2627)" - ) - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - pipeline.status = PipelineStatus.FAILED - pipeline.error = _safety_net_error - store.save_pipeline(pipeline) - # Emit the dedicated empty-contract HITL inline so the - # operator sees an actionable decision instead of a - # generic ``status: failed`` with no recovery path - # other than ``restart_phase implement`` (which would - # respawn into the same empty-contract state). - _emit_empty_contract_hitl( - pipeline_id, - pipeline, - store, - reason=_safety_net_reason, - draft_slice_count=None, - gate="start_phase_implement_safety_net", - phase=pipeline.current_phase, - ) - logger.error( - "OVERSEER_ALERT start_phase_implement_safety_net_empty_contract", - pipeline_id=pipeline_id, - outcome=_safety_net_populate_result.outcome.value, - slice_count=_safety_net_populate_result.slice_count, - reason=_safety_net_reason, - ) - report_pipeline_status( - pipeline, - event_type="pipeline.failed", - message=f"Pipeline failed: {_safety_net_error[:100]}", - ) - _emit_pipeline_event(pipeline, "pipeline.failed") - return - - # #3100: the natural plan→implement path enforces the - # #2777 plan pre-flight (``validate_plan_preflight``) at - # the advance_phase site; implement-start submits skip - # that site entirely, so a plan draft without a ``pr:`` - # block previously entered the implement phase and every - # context-PR opener backstop soft-failed with - # ``missing_pr_metadata`` forever. Enforce the same - # validator here — after the empty-contract gate so the - # #2627 HITL routing above is unchanged. - if _enforce_implement_start_plan_preflight( - pipeline_id, - pipeline, - store, - worktree_repo_path, - plan_draft_rel, - ): - return - - # Operator directives + prior iteration history are persisted on - # ``PhaseExecution`` and accumulate across HITL kickbacks (#2795). - # They are read directly off the phase below each loop iteration — - # no separate "read once and clear" stash is needed. - - # Initialize the Tier 1 health monitor so deterministic tripwires - # (heartbeat timeout, container exit, repeated errors, message rate, - # progress stall) fire during pipeline execution. The monitor - # subscribes to EventBus events reactively, but check_heartbeats() - # and check_progress() need periodic polling. - try: - from events import get_event_bus - from health_monitor import init_health_monitor - - health_monitor_instance = init_health_monitor( - get_event_bus(), pipeline_id, pipeline.config - ) - # Sync the phase-aware threshold with the current pipeline phase - health_monitor_instance.set_current_phase(pipeline.current_phase.value) - - # Wake stuck producers directly when check_brc_progress fires - # so the deterministic detector actually drives remediation - # instead of relying on the overseer agent's discretion (#2079). - # The closure reads the monitor's current phase at fire time so - # the message records the phase the producer is actually in. - def _on_health_escalation(escalation: dict[str, Any]) -> None: - phase = health_monitor_instance.get_current_phase() - _send_brc_confirmation_nudge(escalation, pipeline_id, phase) - - health_monitor_instance.on_escalation(_on_health_escalation) - - # Start a background polling thread for time-based tripwires - health_monitor_timer = threading.Event() - - # SHAs we've already raised a branch-divergence alert for - # (#2224 PR 3). Per-pipeline dedupe so we fire once per - # offending commit, not once per 30s tick. - divergence_alerted_shas: set[str] = set() - - def _health_monitor_poll(monitor, stop_event: threading.Event, interval: float = 30.0): - while not stop_event.is_set(): - try: - # Tier 1 no longer sends nudges directly — it raises - # alerts and fires escalation callbacks internally. - # The overseer (Tier 2) decides whether to nudge. - monitor.check_tripwires() - except Exception as poll_err: - logger.debug( - "Health monitor poll error", - pipeline_id=pipeline_id, - error=str(poll_err), - ) - - # Branch-divergence detector (#2224 PR 3). Helper - # re-loads pipeline state each tick so a - # base_branch / branch update mid-pipeline is - # picked up. Dedupe set is mutated in place. - _branch_divergence_tick( - pipeline_id=pipeline_id, - worktree_repo_path=worktree_repo_path, - store=store, - alerted_shas=divergence_alerted_shas, - ) - - # NOTE (#2270 slice-5): the standing-pod overseer respawn loop - # was removed here. The overseer is no longer a respawned - # standing pod — orchestrator-side detection (slice-4 - # ``health_checks.detection_plane``) runs in-process and the - # only agent spawned is the on-demand adjudicator. Any - # surviving restart need is served by the general - # agent-restart machinery (``restart_agent``), not a bespoke - # overseer respawn. This also means a multi-hour zero-agent - # HITL park spawns nothing from this loop (§3). - - stop_event.wait(interval) - - poll_thread = threading.Thread( - target=_health_monitor_poll, - args=(health_monitor_instance, health_monitor_timer), - daemon=True, - name=f"health-monitor-{pipeline_id[:8]}", - ) - poll_thread.start() - logger.info( - "Health monitor initialized", - pipeline_id=pipeline_id, - ) - except Exception as hm_err: - # Non-fatal: pipeline can run without Tier 1 monitoring - logger.warning( - "Failed to initialize health monitor (continuing without Tier 1 monitoring)", - pipeline_id=pipeline_id, - error=str(hm_err), - ) - - while True: - try: - pipeline = store.load_pipeline(pipeline_id) - except Exception: - # Pipeline was deleted — exit quietly - logger.info( - "Pipeline no longer exists, exiting thread", - pipeline_id=pipeline_id, - ) - return - - # Detect recreation/restart: another run now owns this pipeline ID - _current_epoch = pipeline.run_epoch or pipeline.created_at - if _current_epoch != run_epoch: - logger.info( - "Pipeline was recreated, exiting old thread", - pipeline_id=pipeline_id, - ) - return - - if pipeline.status in (PipelineStatus.FAILED, PipelineStatus.CANCELLED): - logger.info( - "Pipeline stopped", pipeline_id=pipeline_id, status=pipeline.status.value - ) - break - - current_phase = pipeline.current_phase - - # Start the current phase - phase_execution = pipeline.get_phase_execution(current_phase) - if phase_execution.status == PipelineStatus.PENDING: - # Record branch tip SHA for completion signal verification. - # This allows the completion handler to detect if a commit - # was pushed to a different branch than expected. - # NOTE: Intentional TOCTOU — the SHA is captured before - # acquiring the state lock, so a push between rev-parse and - # lock acquisition could make it stale. Acceptable because - # phase_start_sha is only used for advisory "no new commits" - # logging, not for correctness decisions. - phase_start_sha: str | None = None - try: - _sha_result = subprocess.run( - ["git", "rev-parse", f"origin/{pipeline.branch}"], - capture_output=True, - text=True, - cwd=str(worktree_repo_path), - timeout=10, - check=False, - ) - if _sha_result.returncode == 0: - phase_start_sha = _sha_result.stdout.strip() - except Exception: - pass # Non-fatal — verification is best-effort - - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - phase_execution = pipeline.get_phase_execution(current_phase) - phase_execution.status = PipelineStatus.RUNNING - phase_execution.started_at = datetime.now(UTC) - phase_execution.phase_start_sha = phase_start_sha - pipeline.status = PipelineStatus.RUNNING - store.save_pipeline(pipeline) - - # Report phase start to collaborator - report_pipeline_status( - pipeline, - event_type="phase.started", - message=f"Phase {current_phase.value} started", - ) - _emit_pipeline_event(pipeline, "phase.started") - - # #2777 (cq-4, TASK-1-2) — implement-phase entry - # backstop. Calls the new - # ``_open_context_pr_at_implement_start`` opener for - # the runner-driven paths that bypass - # ``advance_phase`` REST (inline ``_run_pipeline`` - # auto-advance and the HITL-approval recovery in - # ``start_pipeline`` both leave - # ``phase_execution.status`` as PENDING and spawn the - # runner directly; the backstop catches both per - # #2593). The opener is idempotent so re-firing here - # after a successful advance_phase call is a one- - # round-trip ``gh pr list`` no-op. - # - # reviewer_code_holistic blocker 1 fix: v1 deleted - # this site under the (incorrect) "single canonical - # site" plan AC; the four soft-fail call sites are in - # fact the only context-PR opener calls on the - # runner-driven paths, so the deletion silently - # stranded slice stacks on ``egg/<id>/work``. - # Restored under the new idempotent opener. - if current_phase == PipelinePhase.IMPLEMENT: - try: - _open_context_pr_at_implement_start(pipeline_id, repo_path=repo_path) - except ContextPrCreationError as ctx_err: - logger.warning( - "Context PR opener: implement-entry backstop " - "failed (continuing — hard-require enforced at " - "advance_phase and the implement-start plan " - "pre-flight gate) (#2777, #3100)", - pipeline_id=pipeline_id, - reason=ctx_err.reason, - error=str(ctx_err), - ) - except Exception as backstop_err: # noqa: BLE001 - logger.warning( - "Context PR opener: implement-entry backstop " - "outer wrapper raised (continuing) (#2777)", - pipeline_id=pipeline_id, - error=str(backstop_err), - ) - - # Spawn overseer container for this phase's health monitoring. - # The overseer is phase-scoped: spawned at phase start and torn - # down at phase completion/advance/failure. Each phase gets a - # fresh overseer instance with no accumulated state. - # - # #2270 slice-5: gate overseer presence on "agents actually - # running". During a zero-agent HITL park the pipeline has no phase - # agents in flight, so spawning an overseer there is pure churn - # (§3). The respawn loop that used to keep it alive across such - # parks was removed; this gate stops the phase-start spawn from - # doing the same thing. The agent count is the deterministic phase - # roster the concurrent executor itself consults — the cohort this - # phase is about to run. - _phase_agent_count = _count_phase_agents(pipeline, current_phase) - if pipeline.config.overseer_enabled and _overseer_should_be_present( - running_agent_count=_phase_agent_count, - pipeline_status=pipeline.status, - ): - try: - overseer_result = _spawn_overseer_agent( - spawner=spawner, - pipeline_id=pipeline_id, - issue_number=pipeline.issue_number, - gateway_mode=gateway_mode, - pipeline_repos=pipeline_repos if pipeline_repos else None, - max_turns=pipeline.config.overseer_max_turns, - decision_model=pipeline.config.overseer_decision_maker_model, - ) - with overseer_lock: - overseer_container_id = overseer_result.container_info.container_id - phase_overseer_active = True - logger.info( - "Overseer container spawned for phase", - pipeline_id=pipeline_id, - phase=current_phase.value, - container_id=overseer_container_id[:12], - ) - except (ContainerSpawnError, KubernetesSpawnError) as e: - # Non-fatal: pipeline can run without overseer monitoring - logger.warning( - "Failed to spawn overseer container (continuing without monitoring)", - pipeline_id=pipeline_id, - phase=current_phase.value, - error=str(e), - ) - - # Common sandbox environment for all containers in this phase. - # GATEWAY_URL, RUNTIME_UID/GID, proxy vars, DNS lockdown, and - # extra_hosts are now handled by the shared build_sandbox_config() - # inside spawn_agent_container(). Only pipeline-specific vars go here. - if gateway_mode == "private": - orchestrator_ip = ORCHESTRATOR_ISOLATED_IP - else: - orchestrator_ip = ORCHESTRATOR_EXTERNAL_IP - orchestrator_url = f"http://{orchestrator_ip}:{ORCHESTRATOR_PORT}" - sandbox_env: dict[str, str] = { - "EGG_PIPELINE_ID": pipeline_id, - "EGG_PIPELINE_PHASE": current_phase.value, - "EGG_PIPELINE_MODE": pipeline_mode, - "EGG_ORCHESTRATOR_URL": orchestrator_url, - "EGG_ORCHESTRATOR_MODE": "distributed", - } - # ``EGG_BRANCH`` is intentionally NOT set here. The spawner - # is the single source of truth for the agent's assigned - # branch (#2428): ``KubernetesSpawner.spawn_agent_job`` - # derives ``EGG_BRANCH`` from its ``branch`` parameter, - # which the slice scheduler populates with the slice - # integration branch via - # ``ConcurrentPhaseExecutor.get_worktree_branch``. Stuffing - # ``pipeline.branch`` into ``sandbox_env`` here used to be - # threaded through ``extra_env``, where the spawner's - # override loop runs after the default-from-``branch`` - # assignment — deterministic precedence, not a race — so - # the pipeline-level value silently won and slice agents - # were downgraded to the pipeline tip, breaking every - # slice-coder push. The branch persistence below is the - # only side-effect the run loop still needs. - if not pipeline.branch: - generated_branch = f"egg/{pipeline_id}/work" - # Persist the generated branch so the PR phase can use it - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - if not pipeline.branch: - pipeline.branch = generated_branch - store.save_pipeline(pipeline) - logger.info( - "Recorded generated branch on pipeline", - pipeline_id=pipeline_id, - branch=generated_branch, - ) - if pipeline.prompt: - sandbox_env["EGG_PIPELINE_PROMPT"] = pipeline.prompt - - if pipeline.repo: - repos = [pipeline.repo] - sandbox_env["EGG_REPO"] = pipeline.repo - else: - repos = [] - - # Jira ticket advisory env vars (issue #1556). These give sandbox - # agents a stable handle for the ticket the pipeline is working - # against (``jira ticket get "$EGG_JIRA_TICKET"``) without - # hard-coding the key. They are ADVISORY — the gateway's project - # allowlist is the only hard boundary, and we never export - # Atlassian credentials (JIRA_BASE_URL / JIRA_USERNAME / - # JIRA_API_TOKEN) to the sandbox. An empty string is exported - # when no ticket is configured so agent wrappers can rely on - # variable presence. - jira_ticket_value = getattr(pipeline, "jira_ticket", None) or "" - sandbox_env["EGG_JIRA_TICKET"] = jira_ticket_value - if jira_ticket_value and "-" in jira_ticket_value: - sandbox_env["EGG_JIRA_PROJECT"] = jira_ticket_value.split("-", 1)[0] - else: - sandbox_env["EGG_JIRA_PROJECT"] = "" - - # Jira-epic SDLC support (issue #1557). Export ``EGG_IS_EPIC`` - # (bool-string) and ``EGG_EPIC_MODE`` (one of - # 'epic-fresh', 'epic-reassess', 'ticket', 'github_issue') - # so the refiner / task-planner / applier prompts can select - # the right mode block. Mapping is derived via - # ``prompt_loader.derive_pipeline_mode`` so the orchestrator - # and any auxiliary callers agree on the canonical rule. - # - # Note: ``EGG_PIPELINE_MODE`` is already taken (PipelineMode: - # 'issue' — set above at L19349). - # ``EGG_EPIC_MODE`` is the orthogonal Jira-epic dimension. - try: - from prompt_loader import derive_pipeline_mode - except ImportError: # pragma: no cover - defensive - derive_pipeline_mode = None # type: ignore[assignment] - _is_epic_flag = bool(getattr(pipeline, "is_epic", False)) - _pipeline_mode_attr = getattr(pipeline, "pipeline_mode", None) - sandbox_env["EGG_IS_EPIC"] = "true" if _is_epic_flag else "false" - if derive_pipeline_mode is not None: - sandbox_env["EGG_EPIC_MODE"] = derive_pipeline_mode( - is_epic=_is_epic_flag, - pipeline_mode=_pipeline_mode_attr, - jira_ticket=jira_ticket_value or None, - ) - else: - sandbox_env["EGG_EPIC_MODE"] = "github_issue" if not jira_ticket_value else "ticket" - - # Issue #1557 reviewer_code v1 finding #4: run the reassess - # sweep before the planner / applier spawn on reassess-mode - # epic pipelines so the task-planner prompt's ``[mode: epic- - # reassess]`` branch and the applier's in-flight refusal - # have the children classification on disk. The sweep - # writes two JSON files under ``.egg-state/agent-outputs/``; - # we export both paths into the sandbox env so the prompts - # read them by env var rather than re-querying the gateway. - # Fail-open: a sweep failure logs a warning but never aborts - # the phase — the planner falls back to fresh-mode treatment - # of the children (which is safe because every action carries - # an explicit ``jira_action`` and the applier's in-flight - # refusal hinges on the sweep file's presence). - if ( - _is_epic_flag - and _pipeline_mode_attr == "reassess" - and current_phase.value in ("plan", "apply") - and jira_ticket_value - ): - try: - from jira_reassess import ( - run_reassess_sweep, - serialise_sweep_to_disk, - ) - except ImportError: # pragma: no cover - defensive - run_reassess_sweep = None # type: ignore[assignment] - serialise_sweep_to_disk = None # type: ignore[assignment] - if run_reassess_sweep is not None and serialise_sweep_to_disk is not None: - try: - sweep_result = run_reassess_sweep( - epic_key=jira_ticket_value, - state_store=store, - ) - agent_outputs_dir = ( - Path(worktree_repo_path) / ".egg-state" / "agent-outputs" - ) - sweep_path, done_path = serialise_sweep_to_disk( - result=sweep_result, - agent_outputs_dir=agent_outputs_dir, - pipeline_id=pipeline_id, - ) - sandbox_env["EGG_REASSESS_SWEEP_PATH"] = str(sweep_path) - sandbox_env["EGG_DONE_CHILDREN_PATH"] = str(done_path) - logger.info( - "Reassess sweep complete", - pipeline_id=pipeline_id, - epic_key=jira_ticket_value, - child_count=len(sweep_result.children), - done_count=len(sweep_result.done), - warnings=sweep_result.warnings, - ) - except Exception as sweep_err: # noqa: BLE001 — fail-open - logger.warning( - "Reassess sweep failed (continuing without sweep handoff)", - pipeline_id=pipeline_id, - epic_key=jira_ticket_value, - error=str(sweep_err), - ) - - phase_failed = False - tester_gap_summary: str | None = None - - # --- Inner review cycle --- - # NOTE: the legacy PR phase (and its auto-PR / slice-DAG-skip - # branches) was deleted in #2777 (cq-4 / TASK-2-2). The context - # PR now opens up-front via ``_open_context_pr_at_implement_start`` - # at the plan→implement boundary, slice PRs stack on it, and - # IMPLEMENT is the terminal phase — no per-phase auto-PR creation - # logic is reachable here for ``current_phase.value == "pr"``. - if True: - while True: - # Reset tester gaps each cycle so stale findings don't accumulate - tester_gap_summary = None - - # Reload to get latest review_cycles count - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - phase_execution = pipeline.get_phase_execution(current_phase) - review_cycle = phase_execution.review_cycles - - # Reset status to RUNNING at cycle start so that a - # previous cycle's FAILED status doesn't persist and - # cause _derive_subphase_status() to misreport (see - # issue #1178). - phase_execution.status = PipelineStatus.RUNNING - pipeline.status = PipelineStatus.RUNNING - - # Record when actual agent work begins (excludes sandbox setup - # and HITL waiting time from the phase duration). - phase_execution.work_started_at = datetime.now(UTC) - - # Capture HEAD commit for delta reviews: reviewers in - # subsequent cycles can diff against this to see only - # the changes made since the last review. - cycle_commit_sha: str | None = None - try: - _git_result = subprocess.run( - ["git", "rev-parse", "HEAD"], - capture_output=True, - text=True, - cwd=str(worktree_repo_path), - timeout=10, - ) - if _git_result.returncode == 0: - cycle_commit_sha = _git_result.stdout.strip() - except Exception: - pass # Non-fatal — delta review is best-effort - - phase_execution.cycle_timings.append( - CycleTiming( - cycle=review_cycle, - started_at=phase_execution.work_started_at, - commit_sha=cycle_commit_sha, - ) - ) - store.save_pipeline(pipeline) - - # 1. Spawn workers — always use concurrent BRC execution. - logger.info( - "Spawning concurrent phase execution", - pipeline_id=pipeline_id, - phase=current_phase, - review_cycle=review_cycle, - mode=gateway_mode, - ) - - # Read structured operator directives + prior iteration - # history off the phase so iteration N+1 prompts can render - # them with precedence prose (#2795). These lists accumulate - # across kickbacks and are never cleared, so no read-and- - # clear stash is needed. - _phase_operator_directives: list[OperatorDirective] = [] - _phase_iteration_history: list[IterationSummary] = [] - try: - with get_pipeline_state_lock(pipeline_id): - _fb_pipeline = store.load_pipeline(pipeline_id) - _fb_phase = _fb_pipeline.get_phase_execution(current_phase) - _phase_operator_directives = list(_fb_phase.operator_directives) - _phase_iteration_history = list(_fb_phase.iteration_history) - except Exception as e: - logger.debug("Failed to read operator directives for phase", error=str(e)) - - # #2137: route the implement phase through the slice - # DAG iterator when the contract has more than one - # slice. Single-slice and no-slice contracts continue - # to use the legacy monolithic path so existing - # pipelines are unaffected. - _use_slice_loop = False - _slice_gate_failure: SliceGateMonolithicBlock | None = None - if current_phase.value == "implement": - try: - from egg_contracts.loader import ( - load_contract as _load_contract_for_slice_check, - ) - - _check_contract = _load_contract_for_slice_check( - pipeline_id, worktree_repo_path - ) - _slice_count = len(getattr(_check_contract, "slices", []) or []) - # #2777 cq-10 — route through ``_is_slice_dag_mode`` - # so the "what counts as slice-DAG" definition has - # a single source of truth. Local ``_slice_count`` - # is still used by the defensive recheck below for - # the structured log when the populator dropped - # slices (#2337). - _use_slice_loop = _is_slice_dag_mode(_check_contract) - - # #2915: Auto-populate contract if empty at implement start - # This fills the gap where start_phase=implement doesn't trigger - # the plan-completion populate path, leaving agents with nothing to do. - if _slice_count == 0: - _slice_count = _auto_populate_contract_at_implement_start( - worktree_repo_path, - pipeline_id, - pipeline_mode, - pipeline.issue_number, - pipeline.current_phase, - pipeline.branch, - gateway=spawner.gateway, - gateway_mode=gateway_mode, - base_branch=pipeline.base_branch, - ) - if _slice_count > 0: - # Reload contract after successful populate - _check_contract = _load_contract_for_slice_check( - pipeline_id, worktree_repo_path - ) - _use_slice_loop = _is_slice_dag_mode(_check_contract) - - # #2337 defensive recheck: if the contract has no - # slices but the on-disk plan draft parses to N>1 - # slices, the populator silently failed earlier. - # Refuse to demote to monolithic. - if _slice_count == 0: - _slice_gate_failure = _slice_gate_block_monolithic_demotion( - worktree_repo_path, - pipeline_id, - pipeline.issue_number, - ) - except Exception as _slice_check_err: # noqa: BLE001 - logger.debug( - "Slice-loop gate: contract load failed, falling back to monolithic", - pipeline_id=pipeline_id, - error=str(_slice_check_err), - ) - - if _slice_gate_failure is not None: - _slice_gate_msg = _slice_gate_failure.message - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - phase_execution = pipeline.get_phase_execution(current_phase) - if phase_execution.cycle_timings: - phase_execution.cycle_timings[-1].completed_at = datetime.now(UTC) - phase_execution.status = PipelineStatus.FAILED - phase_execution.error = _slice_gate_msg - phase_execution.completed_at = datetime.now(UTC) - pipeline.status = PipelineStatus.FAILED - pipeline.error = _slice_gate_msg - store.save_pipeline(pipeline) - # #2627 follow-up: emit a dedicated HITL naming the - # empty-contract root cause inline. The generic - # post-failure Retry/Accept/Abort decision respawns - # implement into the same empty-contract state; this - # HITL's options map to repopulate / restart-plan / - # abort so the operator has a recovery path that - # actually changes state. - _emit_empty_contract_hitl( - pipeline_id, - pipeline, - store, - reason="slice_gate_blocked_monolithic_demotion", - draft_slice_count=_slice_gate_failure.draft_slice_count, - gate="slice_gate", - phase=current_phase, - ) - logger.error( - "OVERSEER_ALERT slice_gate_blocked_monolithic_demotion", - pipeline_id=pipeline_id, - error=_slice_gate_msg, - draft_slice_count=_slice_gate_failure.draft_slice_count, - ) - phase_failed = True - break - - try: - if _use_slice_loop: - exit_code, container_logs = _run_implement_phase_slices( - pipeline_id=pipeline_id, - pipeline=pipeline, - spawner=spawner, - repo_volumes=repo_volumes, - gateway_mode=gateway_mode, - repos=repos, - sandbox_env=sandbox_env, - store=store, - certs_volume=certs_volume, - worktree_repo_path=worktree_repo_path, - run_epoch=run_epoch, - ) - else: - # Pre-#2137 monolithic-implement fallback. The - # impasse-retry wrapper deliberately wraps only - # the slice-loop call site (#2529): impasse - # delegation rewires a *task* between producer - # roles, which only makes sense per-slice. - # Pipelines that don't use the slice loop are - # legacy / single-PR-shape, so an impasse here - # surfaces as a normal slice failure and the - # operator handles it via the existing - # phase-failure HITL path. - exit_code, container_logs = _run_concurrent_phase( - pipeline_id=pipeline_id, - pipeline=pipeline, - phase=current_phase, - spawner=spawner, - repo_volumes=repo_volumes, - gateway_mode=gateway_mode, - repos=repos, - sandbox_env=sandbox_env, - store=store, - certs_volume=certs_volume, - worktree_repo_path=worktree_repo_path, - operator_directives=_phase_operator_directives, - iteration_history=_phase_iteration_history, - run_epoch=run_epoch, - ) - except (ContainerSpawnError, KubernetesSpawnError) as e: - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - phase_execution = pipeline.get_phase_execution(current_phase) - if phase_execution.cycle_timings: - phase_execution.cycle_timings[-1].completed_at = datetime.now(UTC) - phase_execution.status = PipelineStatus.FAILED - phase_execution.error = str(e) - phase_execution.completed_at = datetime.now(UTC) - pipeline.status = PipelineStatus.FAILED - pipeline.error = str(e) - store.save_pipeline(pipeline) - logger.error( - "Failed to spawn concurrent containers", - pipeline_id=pipeline_id, - error=str(e), - ) - phase_failed = True - break - - if exit_code != 0: - # Check if pipeline was restarted while this thread - # was running (e.g. restart_phase bumped run_epoch). - # If so, a new _run_pipeline thread owns this pipeline - # — exit without marking the phase FAILED. See #1638. - _check_pip = store.load_pipeline(pipeline_id) - _check_epoch = _check_pip.run_epoch or _check_pip.created_at - if _check_epoch != run_epoch: - logger.info( - "Pipeline was restarted during phase execution, exiting old thread", - pipeline_id=pipeline_id, - ) - return - - error_msg = f"Container exited with code {exit_code}" - if container_logs: - log_lines = container_logs.strip().splitlines() - tail = "\n".join(log_lines[-10:]) - error_msg += f"\n--- container logs (last 10 lines) ---\n{tail}" - - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - phase_execution = pipeline.get_phase_execution(current_phase) - if phase_execution.cycle_timings: - phase_execution.cycle_timings[-1].completed_at = datetime.now(UTC) - phase_execution.status = PipelineStatus.FAILED - phase_execution.error = error_msg - phase_execution.completed_at = datetime.now(UTC) - pipeline.status = PipelineStatus.FAILED - pipeline.error = error_msg - store.save_pipeline(pipeline) - logger.error( - "Phase failed", - pipeline_id=pipeline_id, - phase=current_phase, - exit_code=exit_code, - container_logs=container_logs[-2000:] if container_logs else "", - ) - phase_failed = True - break - - # 2. Read tester gap findings (concurrent phases include a tester). - # Only read when the phase succeeded — a failed phase may - # have left stale output from a previous cycle on disk. - if not phase_failed: - tester_gap_summary = _read_tester_gaps( - worktree_repo_path, - identifier=_pipeline_identifier(pipeline.issue_number, pipeline_id), - ) - if tester_gap_summary: - logger.info( - "Tester found gaps", - pipeline_id=pipeline_id, - phase=current_phase, - ) - - # Reviewers are handled within the BRC consensus protocol - # (see issue #1178) — advance to next phase. - break - - # If the phase failed, emit the failure event so the SSE stream - # terminates, then break out of the outer loop. - if phase_failed: - # Stop the phase-scoped overseer on failure. - # Hold the lock to prevent the poll thread from seeing the - # container as EXITED and respawning it. - with overseer_lock: - if overseer_container_id and phase_overseer_active: - phase_overseer_active = False - _teardown_phase_overseer( - spawner, - overseer_container_id, - pipeline_id, - phase_label=str(current_phase), - reason="phase failed", - ) - - # report_pipeline_status is a stub (no-op) unless status_reporter - # is installed. The actual SSE emission is _emit_pipeline_event - # below. Kept for consistency with the except block at the - # bottom of this function. - report_pipeline_status( - pipeline, - event_type="pipeline.failed", - message=f"Pipeline failed: {(pipeline.error or 'unknown')[:100]}", - ) - _emit_pipeline_event(pipeline, "pipeline.failed") - - # Best-effort: push worktree branch to remote so work is backed up - if pipeline.branch and worktree_repo_path != repo_path: - try: - spawner.gateway.push_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(worktree_repo_path), - branch=pipeline.branch, - mode=gateway_mode, - base_branch=pipeline.base_branch, - ) - except Exception as push_err: - logger.warning( - "Best-effort push on failure failed", - pipeline_id=pipeline_id, - error=str(push_err), - ) - - break - - # Phase succeeded — mark complete and advance - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - phase_execution = pipeline.get_phase_execution(current_phase) - phase_execution.status = PipelineStatus.COMPLETE - phase_execution.completed_at = datetime.now(UTC) - - store.save_pipeline(pipeline) # Persist phase completion before HITL gate - - # Report phase completion to collaborator - report_pipeline_status( - pipeline, - event_type="phase.completed", - message=f"Phase {current_phase.value} completed", - ) - _emit_pipeline_event(pipeline, "phase.completed") - - # Commit any uncommitted ``.egg-state/`` writes the agents - # made during the phase BEFORE the worktree sync runs. - # ``register_open_question`` / ``request_feedback`` mutate the - # contract live in the shared pipeline worktree (see - # ``orchestrator/contract_store.py``); those writes are - # uncommitted on disk. The ``git reset --hard`` step inside - # ``_sync_worktree_with_remote`` discards them, leaving the - # bridge below with an empty ``contract.decisions`` and - # silently dropping the operator-bound questions (#2488). - # Committing first lets the sync's rebase reconcile them - # against agent-pushed drafts cleanly. - try: - _commit_statefiles_to_worktree( - worktree_repo_path, - f"Persist agent statefile writes before {current_phase.value} sync", - pipeline_identifier=_pipeline_identifier(pipeline.issue_number, pipeline_id), - pipeline_id=pipeline_id, - ) - except Exception as git_err: - logger.warning( - "Failed to commit pre-sync agent statefiles (continuing)", - pipeline_id=pipeline_id, - phase=current_phase.value, - error=str(git_err), - ) - - # Sync worktree with remote before post-phase modifications - # so that agent-pushed commits (including plan drafts) are - # incorporated. This must run BEFORE _populate_contract_from_plan - # and _sync_pipeline_decisions_to_contract so the autoresolve - # rebase inside _sync_worktree_with_remote lands the remote - # state before the populate step reads ``.egg-state/`` — - # otherwise populate would read a stale local view and either - # produce an empty contract or overwrite agent-pushed drafts - # that only exist on origin. (Before #2979 the helper also - # issued ``git reset --hard`` on a doubly-failed divergence, - # which would have reverted local on-disk modifications; that - # destructive path is gone, so the modern rationale is purely - # about the autoresolve rebase, not a hard reset.) - post_phase_sync_outcome: WorktreeSyncOutcome | None = None - post_phase_sync_aborted = False - if pipeline.branch and worktree_repo_path != repo_path: - # Best-effort for transient failures: a sync failure must - # not strand the auto-advance. Without this guard, a - # gateway HTTP error or git subprocess failure inside the - # helper propagates to the outer Exception handler and (if - # marking FAILED also fails) leaves the pipeline wedged with - # phase COMPLETE but no successor (#2219). - # - # #2979: on an unreconciled divergence the helper pauses - # (AWAITING_HUMAN) on a reconcile HITL and blocks until the - # operator acks, then re-runs the sync — nothing is - # discarded and the pipeline is NOT failed for a recoverable - # post-consensus sync. Only an operator abort (or an - # exhausted reconcile budget) returns aborted=True. - try: - post_phase_sync_outcome, post_phase_sync_aborted = ( - _sync_worktree_reconciling_divergence( - spawner, - pipeline_id, - store, - repo_path, - worktree_repo_path=worktree_repo_path, - phase=current_phase, - gateway_mode=gateway_mode, - base_branch=pipeline.base_branch, - pipeline_branch=pipeline.branch, - ) - ) - except Exception as sync_err: - logger.warning( - "Failed to sync worktree with remote after phase (continuing)", - pipeline_id=pipeline_id, - phase=current_phase.value, - error=str(sync_err), - ) - - # #2979: operator aborted the manual reconcile (or the pause - # budget was exhausted). Fail the pipeline; nothing was - # discarded — the local commits remain pinned under the backup - # ref for offline recovery. ``pre_event_hook`` tears down the - # per-phase overseer under its own lock before the public - # ``pipeline.failed`` event, matching the prior ordering. - if post_phase_sync_aborted and post_phase_sync_outcome is not None: - _fail_pipeline_after_divergence_abort( - pipeline_id, - store, - phase=current_phase, - backup_ref=post_phase_sync_outcome.backup_ref, - local_only_commit_shas=post_phase_sync_outcome.local_only_commit_shas, - pre_event_hook=_make_overseer_teardown_hook( - reason="worktree divergence reconcile aborted", - container_id=overseer_container_id, - phase=current_phase, - ), - ) - break - - # After plan phase: populate contract with task structure. - # NOTE: worktree_repo_path is used for both draft reads and - # contract load/save inside _populate_contract_from_plan. - # The contract was created at worktree_repo_path above, so - # both operations must use the same path. - # Called on every successful plan completion (including after - # HITL revision) so the contract reflects the latest approved - # plan, not a previously rejected draft. - # - # Routed through _populate_contract_from_plan_safe so a raised - # exception here cannot skip the HITL gate below (#1890). The - # same helper is invoked from advance_phase so force-advances - # out of plan see the same populate step (#1941). - # - # ``source="plan_complete"`` makes the wrapper raise: - # * PlanDraftMissingOnLocalError — draft missing on local - # but present on origin (#2337 silent demotion). - # * PlanDraftMissingOnLocalAndOriginError — draft missing on BOTH local - # and origin (#2627 silent advance to empty contract). - # We catch either below and mark the pipeline FAILED so the - # operator can intervene rather than implement silently - # shipping slice-1 alone (#2337) or strand 8 agents on an - # empty contract (#2627). - if current_phase.value == "plan": - try: - _plan_complete_populate_result = _populate_contract_from_plan_safe( - worktree_repo_path, - pipeline_id, - pipeline_mode, - pipeline.issue_number, - source="plan_complete", - branch=pipeline.branch, - ) - # #2627 follow-up: populate-succeeded-but-empty is the - # orthogonal failure mode flagged in the issue. The - # draft existed (so neither PlanDraftMissing variant - # fired) but the populator did not produce a contract - # with tasks the implement-phase agents can act on. - # Synthesize a raise so the same FAILED-cleanup - # handler below runs. - # - # Routes through - # :func:`_populate_result_is_empty_contract` so the two - # empty-contract call sites (this handler and the - # ``start_phase=implement`` safety net) can't drift out - # of agreement. This widens the original - # ``EMPTY_RESULT`` / ``PARSE_FAILED`` check to cover - # every non-success outcome plus the POPULATED-with-no- - # slices case (#2627 review). - if _populate_result_is_empty_contract(_plan_complete_populate_result): - # Pre-raise OVERSEER_ALERT mirroring the two - # ``PlanDraftMissing*`` wrapper-side emits at - # :func:`_populate_contract_from_plan_safe` so the - # discriminator the FAILED-cleanup logger uses - # (``OVERSEER_ALERT plan_populate_produced_empty_contract``) - # is also emitted before the raise. Without this - # the third fail-loud branch had no pre-raise log - # while the two draft-missing branches did, - # asymmetric audit (#2627 review). - logger.error( - "OVERSEER_ALERT plan_populate_produced_empty_contract", - pipeline_id=pipeline_id, - branch=pipeline.branch, - outcome=_plan_complete_populate_result.outcome.value, - slice_count=_plan_complete_populate_result.slice_count, - note=( - "plan populate did not produce a contract with " - "tasks the implement-phase agents can act on; " - "blocking phase advance (#2627)" - ), - ) - raise PopulateProducedEmptyContractError( - _plan_complete_populate_result.outcome, - slice_count=_plan_complete_populate_result.slice_count, - ) - except ( - PlanDraftMissingOnLocalError, - PlanDraftMissingOnLocalAndOriginError, - PopulateProducedEmptyContractError, - ) as missing_err: - # Mirror the slice-gate failure handler at the - # implement-phase entry: mark FAILED in state, - # then run the same cleanup sequence as the - # ``if phase_failed:`` block above (teardown phase - # overseer, report pipeline status, best-effort push - # for backup) so both load-bearing failure paths - # have a uniform cleanup story. Re #2337 / #2627 - # reviews. - teardown_reason, log_event = _empty_contract_failure_metadata(missing_err) - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - phase_execution = pipeline.get_phase_execution(current_phase) - phase_execution.status = PipelineStatus.FAILED - phase_execution.error = str(missing_err) - phase_execution.completed_at = datetime.now(UTC) - pipeline.status = PipelineStatus.FAILED - pipeline.error = str(missing_err) - store.save_pipeline(pipeline) - # #2627 follow-up: emit the dedicated empty-contract - # HITL so the operator sees an actionable decision - # (repopulate / restart-plan / abort) inline with the - # FAILED status, instead of having to dig through - # pipeline.error and the generic consensus-timeout - # decision. - _hitl_reason = _empty_contract_hitl_reason(missing_err) - _emit_empty_contract_hitl( - pipeline_id, - pipeline, - store, - reason=_hitl_reason, - draft_slice_count=None, - gate="plan_complete", - phase=current_phase, - ) - logger.error( - log_event, - pipeline_id=pipeline_id, - error=str(missing_err), - ) - # Stop the phase-scoped overseer on failure. - # Hold the lock to prevent the poll thread from seeing - # the container as EXITED and respawning it. - with overseer_lock: - if overseer_container_id and phase_overseer_active: - phase_overseer_active = False - _teardown_phase_overseer( - spawner, - overseer_container_id, - pipeline_id, - phase_label=str(current_phase), - reason=teardown_reason, - ) - report_pipeline_status( - pipeline, - event_type="pipeline.failed", - message=f"Pipeline failed: {(pipeline.error or 'unknown')[:100]}", - ) - _emit_pipeline_event(pipeline, "pipeline.failed") - # Best-effort: push worktree branch to remote so work - # is backed up before the pipeline exits. - if pipeline.branch and worktree_repo_path != repo_path: - try: - spawner.gateway.push_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(worktree_repo_path), - branch=pipeline.branch, - mode=gateway_mode, - base_branch=pipeline.base_branch, - ) - except Exception as push_err: - logger.warning( - "Best-effort push on failure failed", - pipeline_id=pipeline_id, - error=str(push_err), - ) - break - - # After refine and plan phases: sync substantive HITL decisions - # (non-phase-gate) to the contract so implement-phase agents - # can see what was decided. Called for both refine and plan - # phases — refine decisions inform the plan, plan decisions - # inform the implementation. - if current_phase.value in _HITL_GATE_PHASES: - try: - _sync_pipeline_decisions_to_contract( - repo_path, - worktree_repo_path, - pipeline_id, - ) - except Exception as sync_err: - logger.warning( - "Failed to sync pipeline decisions to contract (continuing)", - pipeline_id=pipeline_id, - phase=current_phase.value, - error=str(sync_err), - ) - - # Write BRC consensus history for this phase before committing - # statefiles so the history file is included in the commit. - try: - _write_brc_history( - worktree_repo_path, - pipeline_id, - current_phase.value, - _brc_history_identifier(pipeline), - # Per-slice implement-phase files are owned by each - # slice's integration branch; committing them onto - # ``work`` here would conflict with the slice - # branches' add of the same paths and break slice - # PR merges (#2755). The parameter is a no-op for - # non-implement phases. - write_per_slice=False, - ) - except Exception as brc_err: - logger.debug( - "Failed to write BRC history (continuing)", - pipeline_id=pipeline_id, - phase=current_phase, - error=str(brc_err), - ) - - # Commit any .egg-state/ files produced during this phase - # (drafts, reviews, check results, contract updates). Mirrors - # the GHA workflow's `git add .egg-state/` at phase boundaries. - try: - _commit_statefiles_to_worktree( - worktree_repo_path, - f"Persist statefiles after {current_phase.value} phase", - pipeline_identifier=_pipeline_identifier(pipeline.issue_number, pipeline_id), - pipeline_id=pipeline_id, - ) - except Exception as git_err: - # Catch broadly: the helper does ``subprocess.run(check=True, - # timeout=30)`` which can raise ``TimeoutExpired`` (not a - # CalledProcessError) and ``glob.glob`` which can raise - # ``OSError``. A narrow ``except`` here let either escape - # to the outer handler and stranded the pipeline (#2219). - logger.warning( - "Failed to commit statefiles after phase (continuing)", - pipeline_id=pipeline_id, - phase=current_phase, - error=str(git_err), - ) - - # Push statefiles to remote so the next phase's agents - # don't have unpushed .egg-state/ files in their diff. - if pipeline.branch and worktree_repo_path != repo_path: - try: - spawner.gateway.push_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(worktree_repo_path), - branch=pipeline.branch, - mode=gateway_mode, - base_branch=pipeline.base_branch, - ) - except Exception as push_err: - logger.warning( - "Failed to push statefiles after phase (continuing)", - pipeline_id=pipeline_id, - phase=current_phase, - error=str(push_err), - ) - - # --- Unresolved-gap gate (#3300) --- - # Block finalize while the contract carries an unresolved - # tester→coder TaskGap. Runs after the worktree sync above so - # the contract reflects the agents' final writes, and BEFORE - # the phase_gate / advance / finalize below so the gap can't - # ship into the committed contract (which would fail - # test_models_gaps.py red in CI on the already-open PR — - # #3298 class 4). Scoped to IMPLEMENT, where gaps are written; - # no-ops on a clean contract. On a fully-autonomous pipeline - # (hitl_gates=False) the gate surfaces the escalation but does - # not block — both options need a human, so blocking would - # stall the pipeline indefinitely; the reactive CI check stays - # the backstop there. - if current_phase == PipelinePhase.IMPLEMENT: - try: - gap_gated = _await_unresolved_gap_gate( - store, - pipeline_id, - repo_path, - worktree_repo_path, - _pipeline_identifier(pipeline.issue_number, pipeline_id), - current_phase, - pipeline.config.hitl_gates, - ) - pipeline = store.load_pipeline(pipeline_id) - # The gate ran after the statefile commit+push above, so - # when it changed the contract (operator resolved a gap, - # or the override audit landed) the resolution is still - # uncommitted in the worktree. Re-commit + push so the - # work branch tree CI sees reflects the post-gate - # contract, not the open-gap snapshot pushed earlier. - if gap_gated: - gate_committed = False - try: - gate_committed = _commit_statefiles_to_worktree( - worktree_repo_path, - f"Persist contract after {current_phase.value} gap gate", - pipeline_identifier=_pipeline_identifier( - pipeline.issue_number, pipeline_id - ), - pipeline_id=pipeline_id, - ) - except Exception as git_err: - logger.warning( - "Failed to commit statefiles after gap gate (continuing)", - pipeline_id=pipeline_id, - phase=current_phase.value, - error=str(git_err), - ) - # Skip the follow-up push when nothing was committed - # (e.g. the override path leaves the contract - # unchanged) — it would be a no-op fast-forward - # (#2548). - if gate_committed and pipeline.branch and worktree_repo_path != repo_path: - try: - spawner.gateway.push_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(worktree_repo_path), - branch=pipeline.branch, - mode=gateway_mode, - base_branch=pipeline.base_branch, - ) - except Exception as push_err: - logger.warning( - "Failed to push statefiles after gap gate (continuing)", - pipeline_id=pipeline_id, - phase=current_phase.value, - error=str(push_err), - ) - except Exception as gap_gate_err: # noqa: BLE001 - # Never let a gate bug strand the pipeline — the - # reactive test_models_gaps.py CI check remains the - # backstop if this fails open. - logger.warning( - "Unresolved-gap gate raised (continuing)", - pipeline_id=pipeline_id, - phase=current_phase.value, - error=str(gap_gate_err), - ) - - # --- HITL gate: pause for human approval --- - # Refine/plan are gated by the converge-before-advance loop - # (#3392): it resolves decisions with a human each round, which is - # what lets us drop the force-advance backstop — a human is present - # to resolve and approve. - # - # But a fully-autonomous pipeline (``hitl_gates is False``) has no - # human to resolve or approve, and ``wait_for_decision`` polls - # indefinitely — so unconditionally gating here would convert that - # explicitly-chosen, first-class config into an indefinite hang - # with no operator-facing signal that the flag was ignored. Mirror - # the unresolved-gap gate's autonomous escape (#3300): when - # ``hitl_gates is False`` we *surface* the gate (event + loud - # warning) but do not block, advancing autonomously instead. - # ``hitl_gates`` therefore still governs refine/plan, but only by - # toggling between the human-gated converge loop and an autonomous - # advance — never an indefinite stall. - if current_phase.value in _HITL_GATE_PHASES and not pipeline.config.hitl_gates: - report_pipeline_status( - pipeline, - event_type="phase.gate_skipped", - message=( - f"{current_phase.value} phase gate skipped " - f"(hitl_gates=False) — advancing autonomously" - ), - ) - logger.warning( - "HITL gate: refine/plan gate on an autonomous pipeline " - "(hitl_gates=False); surfacing but not blocking — advancing " - "without human approval (the converge-before-advance loop " - "requires a human, so it cannot run unattended)", - pipeline_id=pipeline_id, - phase=current_phase.value, - ) - # Decision-ledger visibility on the autonomous path (#3390): - # no human is present to resolve a backstop HITL, so mirror - # the gate-skip posture — surface a missing ledger loudly - # (event + warning) but never block. - try: - _ledger_note, _ledger_missing, _ledger_explicit_none = ( - _collect_decision_ledger_status( - worktree_repo_path, - pipeline_id, - _pipeline_identifier(pipeline.issue_number, pipeline_id), - current_phase, - ) - ) - if _ledger_missing: - logger.warning( - "Decision ledger missing at autonomous gate skip (#3390)", - pipeline_id=pipeline_id, - phase=current_phase.value, - ) - report_pipeline_status( - pipeline, - event_type="phase.decision_ledger_missing", - message=( - f"{current_phase.value} phase advanced autonomously " - f"with no decision ledger — {_ledger_note}" - ), - ) - elif _ledger_explicit_none is not None: - # No human is present to confirm the attestation - # (#3462) — mirror the gate-skip posture: surface - # loudly, never block. - report_pipeline_status( - pipeline, - event_type="phase.decision_ledger_explicit_none", - message=( - f"{current_phase.value} phase advanced autonomously " - f"on an unconfirmed no-decisions attestation — " - f"{_ledger_note}" - ), - ) - except Exception as ledger_err: # noqa: BLE001 - logger.warning( - "Decision-ledger check raised on autonomous path (continuing)", - pipeline_id=pipeline_id, - phase=current_phase.value, - error=str(ledger_err), - ) - elif current_phase.value in _HITL_GATE_PHASES: - # --- Decision-ledger backstop (#3390) --- - # Propose-time validation guarantees every refine/plan - # producer attested its ledger, so reaching this gate with - # zero registered decisions AND no explicit-none attestation - # means a path bypassed consensus (force-advance, resume) or - # the claim was lost. Never silently advance past that: - # surface a dedicated HITL whose default remedy is a phase - # re-run (the converge loop's standard corrective), with an - # explicit operator override to proceed. - _ledger_note = "" - _ledger_missing = False - _ledger_explicit_none: tuple[str, str] | None = None - try: - _ledger_note, _ledger_missing, _ledger_explicit_none = ( - _collect_decision_ledger_status( - worktree_repo_path, - pipeline_id, - _pipeline_identifier(pipeline.issue_number, pipeline_id), - current_phase, - ) - ) - except Exception as ledger_err: # noqa: BLE001 - # Never let a helper bug strand the pipeline — the - # propose-time hard gate remains the primary enforcement. - logger.warning( - "Decision-ledger status check raised (continuing)", - pipeline_id=pipeline_id, - phase=current_phase.value, - error=str(ledger_err), - ) - - if _ledger_missing: - dq = get_decision_queue(pipeline_id, repo_path) - _backstop = dq.queue_decision( - question=( - f"The {current_phase.value} phase reached its gate " - f"without a decision ledger (#3390). {_ledger_note}\n\n" - f"Re-running the phase lets its agents register the " - f"decisions the drafts should have surfaced (or attest " - f"an explicit empty ledger); proceeding accepts the " - f"unverified ledger and presents the normal phase gate." - ), - context=_ledger_note, - options=[ - _LEDGER_BACKSTOP_RERUN_OPTION, - _LEDGER_BACKSTOP_PROCEED_OPTION, - ], - decision_type="choice", - phase=current_phase, - ) - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - pipeline.status = PipelineStatus.AWAITING_HUMAN - phase_execution = pipeline.get_phase_execution(current_phase) - phase_execution.status = PipelineStatus.AWAITING_HUMAN - store.save_pipeline(pipeline) - report_pipeline_status( - pipeline, - event_type="decision.created", - message=( - f"Decision ledger missing for {current_phase.value} " - f"phase — awaiting operator direction" - ), - ) - _emit_pipeline_event(pipeline, "decision.created") - - _backstop_resolved = dq.wait_for_decision(_backstop.id) - _backstop_resolution = str( - getattr(_backstop_resolved, "resolution", None) or "" - ).strip() - _proceed = ( - _backstop_resolved.status != DecisionStatus.RESOLVED - or "proceed" in _backstop_resolution.lower() - ) - if not _proceed: - # Default remedy: re-run the phase so producers can - # register (or explicitly attest) the ledger. Any - # free-text resolution rides along as the directive. - _rerun_directive = ( - f"The {current_phase.value} phase reached its gate " - f"without a decision ledger: no HITL decisions were " - f"registered and no producer attested an explicit " - f"empty ledger (#3390). Review your draft for " - f"operator-grade choices; register each via " - f"`egg-contract add-decision` and cite its cq-N in " - f"the draft, or attest `no_decisions_rationale` when " - f"proposing if the phase genuinely raises none." - ) - if _backstop_resolution.lower() != (_LEDGER_BACKSTOP_RERUN_OPTION.lower()): - _rerun_directive += f"\n\nOperator note: {_backstop_resolution}" - logger.info( - "Decision-ledger backstop: re-running phase (#3390)", - pipeline_id=pipeline_id, - phase=current_phase.value, - ) - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - pipeline.status = PipelineStatus.RUNNING - phase_execution = pipeline.get_phase_execution(current_phase) - phase_execution.status = PipelineStatus.RUNNING - phase_execution.completed_at = None - phase_execution.hitl_review_cycles += 1 - _alert_threshold = pipeline.config.max_hitl_review_cycles - if phase_execution.hitl_review_cycles >= _alert_threshold: - _broadcast_hitl_nonconvergence_alert( - pipeline_id, - pipeline, - current_phase, - phase_execution.hitl_review_cycles, - _alert_threshold, - ) - _perform_hitl_phase_rerun( - store=store, - spawner=spawner, - pipeline=pipeline, - phase_execution=phase_execution, - pipeline_id=pipeline_id, - current_phase=current_phase, - feedback_text=_rerun_directive, - event_message=( - f"Re-running {current_phase.value}: decision " - f"ledger missing (#3390)" - ), - ) - continue # Re-enter outer loop → re-run phase - logger.warning( - "Decision-ledger backstop: operator chose to proceed " - "without a ledger (#3390)", - pipeline_id=pipeline_id, - phase=current_phase.value, - resolution=_backstop_resolution[:200], - ) - elif _ledger_explicit_none is not None: - # --- Explicit-none attestation confirmation (#3462) --- - # The producer's claim that this phase raises no operator - # decisions bypasses the entire register → bridge → - # resolve chain, and is itself a judgment call the HITL - # contract assigns to the operator. Surface it as its own - # confirmable decision (see the helper): confirming records - # the operator's endorsement on the ledger note; rejecting - # re-runs the phase to register cq-N entries. - _rerun_requested, _ledger_note, pipeline = ( - _handle_explicit_none_attestation_gate( - pipeline=pipeline, - pipeline_id=pipeline_id, - repo_path=repo_path, - current_phase=current_phase, - ledger_note=_ledger_note, - explicit_none=_ledger_explicit_none, - store=store, - spawner=spawner, - ) - ) - if _rerun_requested: - continue # Re-enter outer loop → re-run phase - - # Check for an existing pending phase_gate decision for this - # phase. A prior agent-exit event may - # have already created one — creating a duplicate confuses the - # human reviewer. See #1152. - existing_pending_gate = any( - d.decision_type == "phase_gate" - and d.phase == current_phase - and d.status == DecisionStatus.PENDING - for d in pipeline.decisions - ) - - if existing_pending_gate: - logger.info( - "HITL gate: reusing existing pending phase_gate decision", - pipeline_id=pipeline_id, - phase=current_phase.value, - ) - # Find the existing decision to wait on - dq = get_decision_queue(pipeline_id, repo_path) - decision = next( - d - for d in reversed(pipeline.decisions) - if d.decision_type == "phase_gate" - and d.phase == current_phase - and d.status == DecisionStatus.PENDING - ) - else: - draft_content = _read_phase_draft( - worktree_repo_path, - current_phase.value, - issue_number=pipeline.issue_number, - pipeline_id=pipeline_id, - branch=pipeline.branch, - ) - phase_label = ( - "analysis" if current_phase.value == "refine" else current_phase.value - ) - - # Warn if draft is missing — the agent may not have written - # it to the expected path. See #1016. - if draft_content is None: - logger.warning( - "HITL gate: draft not found on work branch", - pipeline_id=pipeline_id, - phase=current_phase.value, - worktree_path=str(worktree_repo_path), - ) - draft_content = ( - f"**Warning**: No {phase_label} draft was found on the " - f"work branch. The agent may not have written the output " - f"to the expected path." - ) - - question = ( - f"The {current_phase.value} phase has completed. " - f"Please review the {phase_label} and approve to continue, " - f"or provide feedback to request changes." - ) - # Auditability (#3390): make "N registered" vs "explicitly - # none" vs "MISSING (operator overrode)" readable at the - # gate without a get_contract round-trip. - if _ledger_note: - question += f"\n\n{_ledger_note}" - - # Lead the gate comment with the simplifier's human-focused - # companion (simplified, jargon-free) when present, and link - # the full agent draft for depth. Falls back to the full - # draft inline when no companion exists (older pipelines, - # or the companion failed to land). - human_content = _read_human_phase_draft( - worktree_repo_path, - current_phase.value, - issue_number=pipeline.issue_number, - pipeline_id=pipeline_id, - branch=pipeline.branch, - ) - gate_context = draft_content - if human_content: - full_draft_link = "" - draft_rel = _get_draft_path( - current_phase.value, - issue_number=pipeline.issue_number, - pipeline_id=pipeline_id, - ) - if pipeline.repo and pipeline.branch and draft_rel: - blob = f"https://github.com/{pipeline.repo}/blob/{pipeline.branch}" - full_draft_link = ( - f"\n\n[View the full detailed {phase_label} draft]" - f"({blob}/{draft_rel})" - ) - gate_context = f"{human_content}{full_draft_link}" - - # Detect whether the gate content changed compared to the - # previous phase_gate decision for this phase (if any). - # - # NB: this compares ``gate_context``, which leads with the - # simplifier's human-focused summary when a companion - # exists. That summary is intentionally high-level and - # lossy, so a re-refinement that materially changes the - # detailed agent draft *without* altering the summary will - # report ``content_changed=False``. The flag only feeds the - # overseer's no-op-rerun health heuristic - # (``overseer/monitor.py`` ``_check_rerun_anomaly``) — it - # never gates re-prompting — so a missed change here is at - # worst a suppressed advisory alert, not a correctness - # issue. We compare the gate content (not the full draft) - # deliberately so the heuristic tracks what the operator - # actually sees at the gate. - _content_changed: bool | None = None - _prev_gate = next( - ( - d - for d in reversed(pipeline.decisions) - if d.decision_type == "phase_gate" - and d.phase == current_phase - and d.status == DecisionStatus.RESOLVED - ), - None, - ) - if _prev_gate is not None: - _content_changed = gate_context != _prev_gate.context - - dq = get_decision_queue(pipeline_id, repo_path) - decision = dq.queue_decision( - question=question, - context=gate_context, - options=["approve", "request changes"], - decision_type="phase_gate", - phase=current_phase, - content_changed=_content_changed, - ) - - # Reload pipeline to pick up the decision persisted by queue_decision(), - # otherwise the stale local object overwrites it with an empty decisions list. - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - pipeline.status = PipelineStatus.AWAITING_HUMAN - # Also mark the phase as awaiting human so the DAG visualization - # shows the HITL gate on the correct phase box. - phase_execution = pipeline.get_phase_execution(current_phase) - phase_execution.status = PipelineStatus.AWAITING_HUMAN - store.save_pipeline(pipeline) - - # Report HITL gate to collaborator - report_pipeline_status( - pipeline, - event_type="decision.created", - message=f"Awaiting human approval for {current_phase.value} phase", - ) - _emit_pipeline_event(pipeline, "decision.created") - - dq.wait_for_decision(decision.id) - - # Check resolution — did the human approve or request changes? - resolved_decision = dq.get_decision(decision.id) - resolution = (resolved_decision.resolution or "").strip() - - # JSON-first resolution parsing: try structured payload before - # falling back to keyword matching for legacy bare-string resolutions. - _is_approved = False - _needs_revision = False - _revision_feedback: str | None = None - - try: - payload = json.loads(resolution) - if isinstance(payload, dict) and "action" in payload: - action = payload["action"] - feedback_text = payload.get("feedback", "") - - if action == "approve": - _is_approved = True - elif action == "select": - # Selection from a choice menu — treat as approval - _is_approved = True - elif action == "submit_feedback": - # Feedback submission — treat as approval (info collected) - _is_approved = True - elif action in ("request_changes", "change_approach"): - if feedback_text: - # R-1: Extract readable feedback, not raw JSON - _needs_revision = True - _revision_feedback = feedback_text - else: - # JSON request_changes without feedback — same as bare label - _needs_revision = True - _revision_feedback = None - else: - # Unknown action — fall through to legacy matching - raise json.JSONDecodeError("unknown action", resolution, 0) - else: - # Valid JSON but no action field — fall through to legacy - raise json.JSONDecodeError("no action field", resolution, 0) - except json.JSONDecodeError, TypeError, AttributeError: - # Legacy bare-string resolution — existing keyword matching - if resolution.lower() in _APPROVE_KEYWORDS: - _is_approved = True - elif resolution.lower() in _BARE_OPTION_LABELS: - # Bare "request changes" without feedback - _needs_revision = True - _revision_feedback = None - elif resolution: - # Free-text feedback - _needs_revision = True - _revision_feedback = resolution - - # Holds the operator's resolution from the "bare request → - # asked for specifics → approve-with-context" follow-up path, - # if that path is taken. When set, it (not the original - # ``resolution``) carries any context attached to the final - # gate approval, so the convergence re-run below must thread it - # rather than the stale original resolution (#3392 review). - followup_resolution: str | None = None - - if _needs_revision and _revision_feedback is None: - # Bare request without actionable feedback — ask for specifics. - # This handles both legacy "request changes" and JSON - # {"action":"request_changes"} without feedback text. - logger.info( - "HITL gate: bare option label without feedback, requesting specifics", - pipeline_id=pipeline_id, - phase=current_phase, - resolution=resolution, - ) - # Extract a human-friendly label from the resolution for the - # follow-up prompt (avoid displaying raw JSON to the user). - try: - _parsed = json.loads(resolution) - display_resolution = ( - _parsed.get("action", resolution).replace("_", " ") - if isinstance(_parsed, dict) - else resolution - ) - except json.JSONDecodeError, TypeError, AttributeError: - display_resolution = resolution - followup = dq.queue_decision( - question=( - f'You selected "{display_resolution}" but didn\'t provide specific feedback. ' - f"Please describe what changes you'd like to see in the {phase_label}, " - f"or approve to continue." - ), - context=draft_content, - options=["approve"], - decision_type="phase_gate", - phase=current_phase, - ) - dq.wait_for_decision(followup.id) - resolved_followup = dq.get_decision(followup.id) - followup_resolution = (resolved_followup.resolution or "").strip() - - # Parse follow-up resolution (also JSON-first) - try: - fp = json.loads(followup_resolution) - if isinstance(fp, dict) and "action" in fp: - fa = fp["action"] - if fa == "approve": - _is_approved = True - _needs_revision = False - elif fa in ("request_changes", "change_approach"): - ft = fp.get("feedback", "") - if ft: - _revision_feedback = ft - else: - _is_approved = True - _needs_revision = False - else: - raise json.JSONDecodeError("unknown", followup_resolution, 0) - else: - raise json.JSONDecodeError("no action", followup_resolution, 0) - except json.JSONDecodeError, TypeError, AttributeError: - if ( - followup_resolution.lower() in _APPROVE_KEYWORDS - or followup_resolution.lower() in _BARE_OPTION_LABELS - ): - logger.info( - "HITL follow-up: no actionable feedback, treating as approval", - pipeline_id=pipeline_id, - phase=current_phase, - ) - _is_approved = True - _needs_revision = False - elif followup_resolution: - _revision_feedback = followup_resolution - - if _needs_revision and _revision_feedback: - # Human provided feedback — re-run the phase with corrections - logger.info( - "HITL gate: changes requested, re-running phase", - pipeline_id=pipeline_id, - phase=current_phase, - feedback_preview=_revision_feedback[:200], - ) - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - pipeline.status = PipelineStatus.RUNNING - phase_execution = pipeline.get_phase_execution(current_phase) - phase_execution.status = PipelineStatus.RUNNING - phase_execution.completed_at = None # Reset — phase is re-running - phase_execution.hitl_review_cycles += 1 - - # No force-advance (#3392). The converge-before-advance - # loop is human-gated every round, so an unbounded loop - # cannot burn compute silently and we must never advance - # with the operator's feedback unaddressed. After the - # configured number of rounds, emit a non-fatal overseer - # alert for visibility, then always re-run. The - # ``max_hitl_review_cycles`` config is now this alert - # threshold, not a force-advance budget. - _alert_threshold = pipeline.config.max_hitl_review_cycles - if phase_execution.hitl_review_cycles >= _alert_threshold: - _broadcast_hitl_nonconvergence_alert( - pipeline_id, - pipeline, - current_phase, - phase_execution.hitl_review_cycles, - _alert_threshold, - ) - # #2795: the directive + frozen iteration summary - # accumulate across kickbacks so iteration N+1's prompts - # render them with explicit precedence prose. - _perform_hitl_phase_rerun( - store=store, - spawner=spawner, - pipeline=pipeline, - phase_execution=phase_execution, - pipeline_id=pipeline_id, - current_phase=current_phase, - feedback_text=_revision_feedback, - event_message=f"Human requested changes to {current_phase.value}", - ) - continue # Re-enter outer loop → re-run phase with feedback - - # Before advancing, surface any contract-scoped decisions / - # feedback the phase's agents registered via ``egg-contract``. - # Without this bridge, approving the phase_gate silently - # discards them (#1889). Wrapped in try/except so a bug - # here can never strand the pipeline. - _decisions_resolved_this_round = 0 - try: - _decisions_resolved_this_round = _queue_and_await_contract_decisions( - dq, - worktree_repo_path, - pipeline_id, - _pipeline_identifier(pipeline.issue_number, pipeline_id), - current_phase, - ) - except Exception as bridge_err: - logger.warning( - "Contract decision bridge failed (continuing)", - pipeline_id=pipeline_id, - phase=current_phase.value, - error=str(bridge_err), - ) - - # Converge-before-advance (#3392): if the operator just - # resolved one or more decisions, re-run the phase so the - # documents reflect those resolutions and any decision the - # resolutions induce is surfaced in the next round. Re-asks of - # already-answered questions are suppressed by carry-forward - # (find_resolved_question), so the open-decision set shrinks - # toward a fixpoint; we advance only on a round that resolved - # nothing new. The phase gate is re-presented after the re-run. - if _decisions_resolved_this_round and current_phase.value in _HITL_GATE_PHASES: - # Preserve any operator context attached to the approve so - # the re-run's agents see it (the bridge already persisted - # the decision answers themselves; this carries the gate - # prose that would otherwise be dropped on a re-run round). - # When the operator went through the "bare request → asked - # for specifics → approve-with-context" follow-up path, the - # context lives in ``followup_resolution`` (the final - # answer), not the stale original ``resolution`` — prefer - # it so that context is not silently dropped (#3392 review). - _context_source = ( - followup_resolution if followup_resolution is not None else resolution - ) - _approve_context = "" - try: - _ap = json.loads(_context_source) - if isinstance(_ap, dict): - _approve_context = ( - _ap.get("context") or _ap.get("feedback") or "" - ).strip() - except json.JSONDecodeError, TypeError, AttributeError: - _approve_context = "" - - _rerun_feedback = ( - f"The operator resolved {_decisions_resolved_this_round} HITL " - f"decision(s) for the {current_phase.value} phase. Update the " - f"{current_phase.value} document(s) to reflect the resolved " - f"decisions (read them from the contract's `decisions`), and " - f"register any new decisions the resolutions induce." - ) - if _approve_context: - _rerun_feedback += f"\n\nOperator note at the gate: {_approve_context}" - - logger.info( - "HITL gate: decisions resolved, re-running phase to fold them in", - pipeline_id=pipeline_id, - phase=current_phase.value, - resolved_count=_decisions_resolved_this_round, - ) - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - pipeline.status = PipelineStatus.RUNNING - phase_execution = pipeline.get_phase_execution(current_phase) - phase_execution.status = PipelineStatus.RUNNING - phase_execution.completed_at = None - phase_execution.hitl_review_cycles += 1 - _alert_threshold = pipeline.config.max_hitl_review_cycles - if phase_execution.hitl_review_cycles >= _alert_threshold: - _broadcast_hitl_nonconvergence_alert( - pipeline_id, - pipeline, - current_phase, - phase_execution.hitl_review_cycles, - _alert_threshold, - ) - _perform_hitl_phase_rerun( - store=store, - spawner=spawner, - pipeline=pipeline, - phase_execution=phase_execution, - pipeline_id=pipeline_id, - current_phase=current_phase, - feedback_text=_rerun_feedback, - event_message=( - f"Folding {_decisions_resolved_this_round} resolved " - f"decision(s) into {current_phase.value}" - ), - ) - continue # Re-enter outer loop → re-run phase, re-surface gate - - # Approved — resume and advance - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - pipeline.status = PipelineStatus.RUNNING - # Restore phase status to COMPLETE now that the HITL gate is cleared - phase_execution = pipeline.get_phase_execution(current_phase) - phase_execution.status = PipelineStatus.COMPLETE - if phase_execution.completed_at is None: - phase_execution.completed_at = datetime.now(UTC) - store.save_pipeline(pipeline) - - # Persist phase gate resolution to contract and draft so - # next-phase agents can see the human's decisions. #1295 - _persist_phase_gate_resolution( - worktree_repo_path, - pipeline_id, - resolved_decision, - current_phase.value, - pipeline.issue_number, - ) - - # Commit and push updated statefiles (contract + draft with resolution) - try: - _commit_statefiles_to_worktree( - worktree_repo_path, - f"Persist HITL resolution after {current_phase.value} phase gate", - pipeline_identifier=_pipeline_identifier( - pipeline.issue_number, pipeline_id - ), - pipeline_id=pipeline_id, - ) - except Exception as git_err: - # Catch broadly: see #2219. The helper raises - # ``TimeoutExpired`` and ``OSError`` paths that a - # ``CalledProcessError``-only handler did not catch. - logger.warning( - "Failed to commit statefiles after phase gate resolution (continuing)", - pipeline_id=pipeline_id, - error=str(git_err), - ) - - if pipeline.branch and worktree_repo_path != repo_path: - try: - spawner.gateway.push_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(worktree_repo_path), - branch=pipeline.branch, - mode=gateway_mode, - base_branch=pipeline.base_branch, - ) - except Exception as push_err: - logger.warning( - "Failed to push statefiles after phase gate resolution (continuing)", - pipeline_id=pipeline_id, - error=str(push_err), - ) - - # ---------------------------------------------------------- - # #2777 (cq-4, TASK-1-2) — inline ``_run_pipeline`` - # auto-advance plan→implement transition. Calls the new - # idempotent ``_open_context_pr_at_implement_start`` - # opener directly; auto-advance does NOT route through - # ``routes/phases.py:advance_phase``, so without this call - # site a natural plan-exit (no operator REST call) would - # never get a context PR opened, leaving the slice stack - # stranded on ``egg/<id>/work`` (the #2593 / #2769 - # symptom). reviewer_code_holistic blocker 1 fix: - # restored after v1's incorrect "single canonical site" - # deletion. The opener's ``gh pr list`` pre-flight makes - # a redundant call from any other transition path a one- - # round-trip no-op. - # ---------------------------------------------------------- - if current_phase.value == "plan": - try: - _open_context_pr_at_implement_start(pipeline_id, repo_path=repo_path) - except ContextPrCreationError as ctx_err: - logger.warning( - "Context PR opener: _run_pipeline auto-advance " - "failed (continuing — hard-require enforced at " - "advance_phase and the implement-start plan " - "pre-flight gate) (#2777, #3100)", - pipeline_id=pipeline_id, - reason=ctx_err.reason, - error=str(ctx_err), - ) - except Exception as autoadvance_err: # noqa: BLE001 - logger.warning( - "Context PR opener: _run_pipeline auto-advance " - "outer wrapper raised (continuing) (#2777)", - pipeline_id=pipeline_id, - error=str(autoadvance_err), - ) - - # Tear down the phase-scoped overseer before advancing. - # Each phase gets a fresh overseer instance — no state carries - # over between phases. - # Hold the lock to prevent the poll thread from seeing the - # container as EXITED and respawning it. - with overseer_lock: - if overseer_container_id and phase_overseer_active: - phase_overseer_active = False - _teardown_phase_overseer( - spawner, - overseer_container_id, - pipeline_id, - phase_label=current_phase.value, - reason="phase ended", - ) - - # Determine next phase. Issue #1557: epic-mode pipelines - # route through the new APPLY phase between PLAN and - # IMPLEMENT so the APPLIER role can drive Jira mutations on - # HITL approval. ``_next_phases_for_epic`` returns - # ``transitions.get(current_phase, [])`` unchanged for - # non-epic pipelines so the pre-#1557 scheduling is - # preserved bit-for-bit. - next_phases = _next_phases_for_epic( - pipeline, - current_phase, - transitions.get(current_phase, []), - ) - - if not next_phases: - # Terminal phase — pipeline complete - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - pipeline.status = PipelineStatus.COMPLETE - store.save_pipeline(pipeline) - - # Report pipeline completion to collaborator - report_pipeline_status( - pipeline, - event_type="pipeline.completed", - message="Pipeline completed successfully", - ) - _emit_pipeline_event(pipeline, "pipeline.completed") - logger.info( - "Pipeline complete", - pipeline_id=pipeline_id, - ) - break - - # TEST_MARKER: auto_advance_block (load-bearing: brackets the - # block for TestAutoAdvanceRespawnsThread; do not remove without - # updating that test class). - # Advance to next phase by respawning a fresh _run_pipeline - # thread, mirroring advance_phase (#2165). Bumping run_epoch - # makes this thread's finally cleanup detect itself as superseded - # and skip worktree teardown; the new thread drives the next - # phase from clean local state. Without this, any exception in - # the new phase's first iteration takes the whole pipeline down. - next_phase = next_phases[0] - - # Issue #1557: when the just-completed phase is PLAN and the - # pipeline is_epic, we are advancing into APPLY. Write the - # applier handoff JSON now (before respawning the driver - # thread) so the APPLIER container can read it on its - # first wakeup. ``approved_phase='plan'`` so the applier - # drives plan-apply (Task.jira_action walk → child create / - # edit / link, Won't-Do handoff for the orchestrator drain). - if ( - getattr(pipeline, "is_epic", False) - and current_phase == PipelinePhase.PLAN - and next_phase == PipelinePhase.APPLY - ): - _write_apply_phase_handoff( - pipeline, - worktree_repo_path, - approved_phase="plan", - ) - - # Issue #1557 task-2-7: when the just-completed phase is - # APPLY (BRC consensus confirmed), drain the Won't-Do - # handoff JSON before advancing to IMPLEMENT. The drain - # runs out-of-band from the HITL approve POST so a slow - # Jira API never extends that handler's latency. - if current_phase == PipelinePhase.APPLY: - _drain_wontdo_batch_after_apply(pipeline, worktree_repo_path) - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - pipeline.current_phase = next_phase - pipeline.run_epoch = datetime.now(UTC) - # ``updated_at`` is unconditionally set by ``StateStore.save_pipeline``. - store.save_pipeline(pipeline) - - # Drop the previous phase's in-memory consensus tracker and - # message-store entries (#2502). The other phase-transition - # paths -- ``advance_phase`` REST handler, HITL-revision - # re-run, and the ``recover_pipeline`` resume path -- all - # call this; the auto-advance path used to skip it, leaving - # a stale plan-phase tracker keyed under the bare - # ``pipeline_id`` for ``_get_concurrent_status`` to find and - # report as ``is_complete: True`` long after the implement - # phase had started. ``_write_brc_history`` runs at the - # bottom of each phase iteration with - # ``write_per_slice=False`` (see #2755), so per-slice - # implement-phase transcripts are on the slice integration - # branches, and the work commit picks up only the - # unattributed sibling plus whatever aggregate the writer - # still emits — refine/plan/pr aggregates, and the - # non-slice-implement aggregate that any implement-phase - # run without slice scope lands on work via the ``not - # buckets`` branch — before we wipe the message store here. - from routes.phases import _clear_concurrent_state - - _clear_concurrent_state(pipeline_id) - - logger.info( - "Phase advanced (auto), respawning driver thread", - pipeline_id=pipeline_id, - from_phase=current_phase.value, - to_phase=next_phase.value, - ) - - _spawn_pipeline_run_thread(pipeline_id, repo_path, pipeline.run_epoch) - return - - except PipelineNotFoundError as pnf_err: - # `PipelineNotFoundError` can be raised either because the pipeline - # was actually deleted or because of a transient state-store read - # (e.g., empty content while a concurrent commit on the state - # worktree races with the read). Re-verify before treating it as - # deletion: if the pipeline is still on disk after retry, the - # original exception was spurious — bump ``run_epoch`` so the - # finally cleanup detects this thread as superseded and skips the - # destructive worktree teardown, then relaunch ``_run_pipeline`` so - # the next phase keeps making progress. See #2155. - pipeline_still_exists = False - _verify_store = None - try: - _verify_store = get_state_store(repo_path) - except Exception as verify_store_err: - # Couldn't even open the state store — treat as transient - # (corrupt-but-present > deletion) so we skip the respawn - # rather than amplifying an infrastructure blip. Note: with - # ``_verify_store=None`` the bump path below short-circuits, - # so worktree preservation depends on whether ``run_epoch`` - # was set before the initial PNFE — this path avoids the - # cascade but does not unconditionally preserve worktrees. - logger.warning( - "Failed to obtain state store after PipelineNotFoundError; " - "treating as transient infrastructure failure and skipping respawn", - pipeline_id=pipeline_id, - error=str(verify_store_err), - ) - pipeline_still_exists = True - - if _verify_store is not None: - for _attempt in range(_PNFE_VERIFY_ATTEMPTS): - time.sleep(_PNFE_VERIFY_INTERVAL) - try: - _verify_store.load_pipeline(pipeline_id) - pipeline_still_exists = True - break - except PipelineNotFoundError: - continue - except StateValidationError: - # Corrupt JSON or schema mismatch means the file - # exists but is unreadable right now — that's not - # deletion. Treat as transient: better to risk a - # wasted respawn than to nuke the worktrees on a - # transient corruption. - pipeline_still_exists = True - break - except StateStoreError as verify_err: - # Other state-store failures (transient git read - # errors, etc.) are also not evidence of deletion. - logger.warning( - "State-store error verifying pipeline existence; " - "treating as transient and preserving worktrees", - pipeline_id=pipeline_id, - error=str(verify_err), - ) - pipeline_still_exists = True - break - - if pipeline_still_exists: - # Cap the respawn cascade so a persistent transient can't - # leak threads, overseer containers, and state-branch - # commits without bound. The recovery code is what runs - # exactly when the system is misbehaving — it must not - # amplify the misbehaviour. - if _respawn_attempt >= _PNFE_RESPAWN_MAX_ATTEMPTS: - logger.error( - "Spurious-PipelineNotFoundError recovery exhausted " - "respawn budget; marking pipeline FAILED so an " - "operator can investigate via restart_phase", - pipeline_id=pipeline_id, - attempts=_respawn_attempt, - exc_info=pnf_err, - ) - if _verify_store is not None: - try: - with get_pipeline_state_lock(pipeline_id): - _failed_pipeline = _verify_store.load_pipeline(pipeline_id) - _failed_pipeline.status = PipelineStatus.FAILED - _failed_pipeline.error = ( - "Transient PipelineNotFoundError recovery " - f"exhausted after {_respawn_attempt} respawns" - ) - _verify_store.save_pipeline(_failed_pipeline) - except Exception as fail_err: - logger.warning( - "Failed to mark pipeline FAILED after exhausting respawn budget", - pipeline_id=pipeline_id, - error=str(fail_err), - ) - else: - # Recoverable transient — log at warning so it doesn't - # trip error-rate dashboards every time it self-heals. - logger.warning( - "Spurious PipelineNotFoundError during execution — " - "pipeline still exists after retry; relaunching driver " - "thread and preserving worktrees", - pipeline_id=pipeline_id, - attempt=_respawn_attempt, - exc_info=pnf_err, - ) - # Bump run_epoch so the finally cleanup observes this - # thread as superseded (mirrors the advance_phase - # pattern) and skips worktree teardown. Capture the - # pre-bump epoch into the local ``run_epoch`` so the - # finally guard works even when the *initial* load - # raised PNFE (in that case run_epoch was never set - # at line 11393). - bump_succeeded = False - if _verify_store is not None: - try: - with get_pipeline_state_lock(pipeline_id): - _bumped = _verify_store.load_pipeline(pipeline_id) - run_epoch = _bumped.run_epoch or _bumped.created_at - _bumped.run_epoch = datetime.now(UTC) - _verify_store.save_pipeline(_bumped) - bump_succeeded = True - except Exception as bump_err: - logger.warning( - "Failed to bump run_epoch during spurious-PNFE " - "recovery; skipping respawn so the existing " - "finally cleanup runs without racing a new thread", - pipeline_id=pipeline_id, - error=str(bump_err), - ) - - if bump_succeeded: - # Exponential backoff between respawn attempts so a - # tight cascade can't fire dozens of respawns per - # second. attempt=0 → 1s, 1 → 2s, 2 → 4s, 3 → 8s, - # 4 → 16s, capped at _PNFE_RESPAWN_BACKOFF_CAP. - _backoff = min(2**_respawn_attempt, _PNFE_RESPAWN_BACKOFF_CAP) - time.sleep(_backoff) - threading.Thread( - target=_run_pipeline, - args=(pipeline_id, repo_path), - kwargs={"_respawn_attempt": _respawn_attempt + 1}, - daemon=True, - name=( - f"pipeline-{pipeline_id}-respawn-" - f"{_respawn_attempt + 1}-{time.monotonic_ns()}" - ), - ).start() - else: - logger.info( - "Pipeline was deleted during execution, exiting", - pipeline_id=pipeline_id, - exc_info=pnf_err, - ) - except Exception as e: - logger.error( - "Pipeline execution failed", pipeline_id=pipeline_id, error=str(e), exc_info=True - ) - persisted_ok = False - try: - store = get_state_store(repo_path) - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - - # Don't corrupt a recreated pipeline's state - _fail_epoch = pipeline.run_epoch or pipeline.created_at - if run_epoch and _fail_epoch != run_epoch: - logger.info( - "Pipeline was recreated, not marking new run as failed", - pipeline_id=pipeline_id, - ) - else: - pipeline.status = PipelineStatus.FAILED - pipeline.error = str(e) - store.save_pipeline(pipeline) - persisted_ok = True - - # Report pipeline failure to collaborator - report_pipeline_status( - pipeline, - event_type="pipeline.failed", - message=f"Pipeline failed: {str(e)[:100]}", - ) - _emit_pipeline_event(pipeline, "pipeline.failed") - except Exception as fail_err: - # If FAILED-marking itself fails (state-store contention, lock - # timeout, etc.), the pipeline stays at ``running`` with no - # error recorded — exactly the silent-wedge symptom in #2219. - # Log so the next occurrence is visible in the orchestrator - # log instead of vanishing. - logger.error( - "Failed to mark pipeline FAILED after exception", - pipeline_id=pipeline_id, - original_error=str(e), - mark_error=str(fail_err), - exc_info=True, - ) - # Surface a synthetic ``pipeline.failed`` to the EventBus even - # though the mark-FAILED block raised. Without this, hosts - # blocked on ``/status/wait`` (whose event allowlist requires - # pipeline.failed/completed/cancelled) wait forever on a dead - # runner — the zombie symptom in #2234. ``persisted`` carries - # whether ``save_pipeline`` actually flushed FAILED to disk - # before the inner block raised: True means disk state matches - # the event, False means consumers should treat the event as - # the only authoritative source. - if _emit_event is not None: - try: - _emit_event( - EventType.PIPELINE_FAILED, - pipeline_id, - data={ - "status": PipelineStatus.FAILED.value, - "persisted": persisted_ok, - "original_error": str(e), - "mark_error": str(fail_err), - }, - ) - except Exception as emit_err: - logger.warning( - "Failed to emit synthetic pipeline.failed event", - pipeline_id=pipeline_id, - error=str(emit_err), - ) - finally: - # Stop health monitor polling and unsubscribe from events - if health_monitor_timer is not None: - health_monitor_timer.set() - if poll_thread is not None: - poll_thread.join(timeout=5) - if health_monitor_instance is not None: - try: - health_monitor_instance.stop() - logger.info("Health monitor stopped", pipeline_id=pipeline_id) - except Exception as hm_stop_err: - logger.debug( - "Failed to stop health monitor", - pipeline_id=pipeline_id, - error=str(hm_stop_err), - ) - - # Clean up progress store for this pipeline - try: - from progress_store import get_progress_store - - progress_store = get_progress_store() - if progress_store is not None: - progress_store.clear(pipeline_id) - except Exception as ps_err: - logger.debug( - "Failed to clear progress store", - pipeline_id=pipeline_id, - error=str(ps_err), - ) - - # Stop overseer container if it was spawned - if overseer_container_id: - try: - _spawner = _get_spawner() - _spawner.stop_agent_job( - overseer_container_id, - cleanup_session=True, - timeout=10, - ) - logger.info( - "Overseer container stopped", - pipeline_id=pipeline_id, - container_id=overseer_container_id[:12], - ) - except Exception as overseer_err: - logger.debug( - "Failed to stop overseer container (may have already exited)", - pipeline_id=pipeline_id, - error=str(overseer_err), - ) - - # Clean up pipeline-level worktrees unless the pipeline has been - # recreated (delete + create with the same ID). In that case the - # new run owns the worktrees and we must not remove them. - try: - _spawner = _get_spawner() - _store = get_state_store(repo_path) - skip_cleanup = False - pipeline_was_restarted = False - try: - current = _store.load_pipeline(pipeline_id) - _cleanup_epoch = current.run_epoch or current.created_at - if run_epoch and _cleanup_epoch != run_epoch: - skip_cleanup = True - pipeline_was_restarted = True - logger.info( - "Pipeline was recreated/restarted, skipping worktree cleanup", - pipeline_id=pipeline_id, - old_epoch=run_epoch.isoformat(), - new_epoch=_cleanup_epoch.isoformat(), - ) - elif current.status == PipelineStatus.FAILED: - skip_cleanup = True - logger.info( - "Pipeline failed, preserving worktrees for retry", - pipeline_id=pipeline_id, - ) - except Exception: - # Pipeline was deleted and not recreated — safe to clean up - pass - - if not skip_cleanup: - try: - _spawner.gateway.delete_worktrees( - container_id=pipeline_id, - force=True, - ) - logger.info("Pipeline worktrees cleaned up", pipeline_id=pipeline_id) - except Exception as pipeline_wt_err: - logger.warning( - "Failed to clean up pipeline worktrees", - pipeline_id=pipeline_id, - error=str(pipeline_wt_err), - ) - - # Also clean up per-agent session worktrees. Each agent - # registers a gateway session under container_id - # "egg-{pipeline_id}-{role}" and session_create creates a - # worktree keyed to that name. The per-agent cleanup path - # calls delete_session_by_container with the Docker container - # hash (not the session container_id), so those worktrees are - # never removed via the normal per-container cleanup. Sweep - # them here as a safety net. delete_worktrees is a no-op for - # container IDs that have no worktree directory. - # - # NOTE: This uses the "egg-{pipeline_id}-{role}" naming for - # session-created worktrees. Per-agent worktrees from #1481 - # use "{pipeline_id}-{role}" (no "egg-" prefix) and are - # cleaned up by cleanup_pipeline() which scans both container - # labels and the filesystem. - for role in AgentRole: - agent_container_id = f"egg-{pipeline_id}-{role.value}" - try: - _spawner.gateway.delete_worktrees( - container_id=agent_container_id, - force=True, - ) - except Exception as agent_wt_err: - logger.warning( - "Failed to clean up agent worktrees", - pipeline_id=pipeline_id, - agent_container_id=agent_container_id, - error=str(agent_wt_err), - ) - - except Exception as wt_err: - logger.warning( - "Failed to clean up worktrees", - pipeline_id=pipeline_id, - error=str(wt_err), - ) - - # Safety-net: clean up any orphaned containers for this pipeline. - # If the pipeline failed during startup or cleanup timed out, Docker - # containers may persist. This is a no-op when no containers exist. - # Skip when the pipeline was restarted (run_epoch changed) so the - # new thread's containers are not killed. See #1386, #1638. - if not pipeline_was_restarted: - try: - # ``gateway_mode`` is the mode this pipeline ran under; - # the auto-salvage hook needs it to push recovery refs - # under the same policy (#2429 review). - removed = _spawner.cleanup_pipeline( - pipeline_id, - force=True, - preserve_worktrees=skip_cleanup, - salvage_mode=gateway_mode, - salvage_base_branch=pipeline.base_branch, - ) - if removed > 0: - logger.info( - "Safety-net cleanup removed orphaned containers", - pipeline_id=pipeline_id, - containers_removed=removed, - ) - except Exception as cleanup_err: - logger.warning( - "Safety-net container cleanup failed", - pipeline_id=pipeline_id, - error=str(cleanup_err), - ) - - -@pipelines_bp.route("/<pipeline_id>/start", methods=["POST"]) -@require_lifecycle_secret -def start_pipeline(pipeline_id: str) -> tuple[Response, int]: - """ - Start pipeline execution. - - Spawns containers for each phase in sequence, advancing through - the phase DAG until completion or failure. Runs in a background thread. - - URL params: - pipeline_id: Pipeline ID - - Response: - { - "success": true, - "message": "Pipeline started", - "data": { - "pipeline_id": "local-a1b2c3d4", - "status": "running" - } - } - """ - repo_path = get_repo_path() - - # Parse force / force_reason from body. ``force=true`` skips the - # live-pod orphan guard before the phase reset (#2420). force_reason - # is recorded in the structured warning log, mirroring the - # complete_phase audit pattern. - body = request.get_json(silent=True) or {} - # Strict boolean — `body.get("force") is True` rather than - # `bool(body.get("force"))` so non-boolean truthy values - # (`"false"`, `[]`, `{}`, `1`) don't silently flip the predicate. - force = body.get("force") is True - force_reason = body.get("force_reason") - if force_reason is not None and not isinstance(force_reason, str): - return make_error_response( - "force_reason must be a string", - status_code=400, - reason="invalid_force_reason", - ) - if isinstance(force_reason, str) and not force_reason.strip(): - force_reason = None - - try: - store, pipeline = _resolve_pipeline(pipeline_id, repo_path) - # Use the store's repo_path so _run_pipeline operates on the correct directory - repo_path = store.repo_path - - # Compute gateway mode for session operations in the recovery path - _gw_mode, _gw_vis = _compute_gateway_mode(pipeline) - - if pipeline.status == PipelineStatus.RUNNING: - return make_error_response( - f"Pipeline {pipeline_id} is already running", - status_code=409, - ) - - if pipeline.status == PipelineStatus.AWAITING_HUMAN: - # No pending decisions — the polling thread died (e.g. restart) - # but the human already resolved everything. Recover based on - # the latest phase_gate decision's resolution. - # - # #2593 review issue 1 — initialised before the lock so the - # post-lock deferred context-PR opener invocation has a - # stable name to read regardless of which branch inside the - # lock executes. - _hitl_open_context_pr_after_lock: bool = False - _hitl_pr_worktree_path: Path | None = None - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - - # Re-validate status after acquiring the lock — another - # concurrent start_pipeline call may have already recovered - # this pipeline. - if pipeline.status != PipelineStatus.AWAITING_HUMAN: - return make_error_response( - f"Pipeline {pipeline_id} status changed to " - f"{pipeline.status.value} (concurrent recovery)", - status_code=409, - ) - - pending = pipeline.get_pending_decisions() - if len(pending) > 0: - return make_error_response( - f"Pipeline {pipeline_id} is awaiting human approval " - f"({len(pending)} pending decision(s))", - status_code=409, - ) - - # Find the latest resolved phase_gate decision - phase_gate_decisions = [ - d - for d in reversed(pipeline.decisions) - if d.decision_type == "phase_gate" and d.status.value == "resolved" - ] - latest_resolution = ( - phase_gate_decisions[0].resolution if phase_gate_decisions else None - ) - - # Determine if approved or request_changes using the shared - # parser (handles approve, select, submit_feedback, - # request_changes, change_approach, and legacy bare strings). - is_approved, revision_feedback = _parse_resolution(latest_resolution) - - if is_approved: - # Mark current phase COMPLETE and advance - phase_execution = pipeline.get_phase_execution(pipeline.current_phase) - phase_execution.status = PipelineStatus.COMPLETE - if phase_execution.completed_at is None: - phase_execution.completed_at = datetime.now(UTC) - - # Persist phase gate resolution so next-phase agents see it. #1295 - # - # The contract and phase draft both live under the - # per-pipeline worktree (``<worktree>/.egg-state/``), - # not the orchestrator's main repo. Resolve the - # worktree explicitly here — the inline path inside - # ``_run_pipeline`` already has ``worktree_repo_path`` - # in scope, but this recovery branch only has the - # main ``repo_path``. Passing ``repo_path`` would - # silently no-op the contract write and draft append - # (#2357, same shape as #2345). - if phase_gate_decisions: - worktree_repo_path = _resolve_pipeline_worktree_path(pipeline, repo_path) - if worktree_repo_path == repo_path: - # No materialised worktree — recovery degrades to - # the pre-fix shape (contract write typically - # no-ops via ContractNotFoundError, draft append - # skipped). The contract write *may* succeed if - # the orchestrator's main repo happens to carry a - # contract for this pipeline, but it would land - # against the wrong tree. Surface this either way - # so operators can correlate missing next-phase - # context with worktree-cleanup races. - logger.warning( - "No materialised worktree found for phase gate " - "persistence; falling back to main repo path. " - "Contract write may silently no-op.", - pipeline_id=pipeline_id, - phase=pipeline.current_phase.value, - ) - _persist_phase_gate_resolution( - worktree_repo_path, - pipeline_id, - phase_gate_decisions[0], - pipeline.current_phase.value, - pipeline.issue_number, - ) - - # Commit statefiles so worktrees created by _run_pipeline - # include the contract/draft changes. - try: - _commit_statefiles_to_worktree( - worktree_repo_path, - f"Persist HITL resolution after {pipeline.current_phase.value} phase gate", - pipeline_identifier=_pipeline_identifier( - pipeline.issue_number, pipeline_id - ), - pipeline_id=pipeline_id, - ) - except Exception as git_err: - # Catch broadly: see #2219. The helper raises - # ``TimeoutExpired`` and ``OSError`` paths that a - # ``CalledProcessError``-only handler did not catch. - logger.warning( - "Failed to commit statefiles after phase gate resolution (continuing)", - pipeline_id=pipeline_id, - error=str(git_err), - ) - - # Push if this repo tracks a remote branch and a - # worktree was materialised. Mirrors the inline - # path's guard at pipelines.py:16044 — pushing from - # the orchestrator's main repo would target the - # wrong working tree. - if pipeline.branch and worktree_repo_path != repo_path: - try: - _spawner = _get_spawner() - _spawner.gateway.push_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(worktree_repo_path), - branch=pipeline.branch, - mode=_gw_mode, - base_branch=pipeline.base_branch, - ) - except Exception as push_err: - logger.warning( - "Failed to push statefiles after phase gate resolution (continuing)", - pipeline_id=pipeline_id, - error=str(push_err), - ) - - from routes.phases import PHASE_TRANSITIONS - - transitions = PHASE_TRANSITIONS - current_phase = pipeline.current_phase - # Issue #1557 — route epic pipelines through APPLY - # between PLAN and IMPLEMENT. Non-epic pipelines - # see the default transition unchanged. - next_phases = _next_phases_for_epic( - pipeline, - current_phase, - transitions.get(current_phase, []), - ) - # #2593 — populate contract from the plan draft when - # the HITL recovery is advancing the pipeline out - # of the plan phase. Without this, contract.pr is - # empty (so the PR phase falls back to placeholder - # title/body and the context PR hook short-circuits - # on "contract has no pr block"), and the slice - # stack ends up rooted on ``/work`` with no PR to - # ``main`` — exactly the symptom reported on the - # in-flight #2474 pipeline. Mirrors the plan-exit - # logic in ``advance_phase`` (routes/phases.py) - # and the auto-advance path in ``_run_pipeline``. - # Best-effort: failures warn and continue so a - # transient infra problem cannot strand the HITL - # recovery. The actual context-PR open is - # deferred until after the lock is released - # (#2593 review issue 1) so the multi-second - # gateway sequence does not extend the - # per-pipeline state lock's hold time. - _next_phase_peek = next_phases[0] if next_phases else None - if ( - current_phase == PipelinePhase.PLAN - and _next_phase_peek == PipelinePhase.IMPLEMENT - ): - _hitl_worktree_path = _resolve_pipeline_worktree_path(pipeline, repo_path) - try: - _pipeline_mode = pipeline.mode.value if pipeline.mode else "issue" - _hitl_populate_result = _populate_contract_from_plan_safe( - _hitl_worktree_path, - pipeline_id, - _pipeline_mode, - pipeline.issue_number, - source="hitl_plan_gate_approval", - ) - # #1941: HITL plan-gate approval is a recovery - # hammer like force-advance — blocking it on a - # populate failure defeats the purpose. We log - # the structured outcome but never raise. - if _hitl_populate_result.outcome != PopulateOutcome.POPULATED: - logger.warning( - "HITL plan-gate approval populate produced non-POPULATED outcome", - pipeline_id=pipeline_id, - outcome=_hitl_populate_result.outcome.value, - ) - try: - _commit_statefiles_to_worktree( - _hitl_worktree_path, - "Populate contract from plan on HITL plan-gate approval", - pipeline_identifier=_pipeline_identifier( - pipeline.issue_number, pipeline_id - ), - pipeline_id=pipeline_id, - ) - except Exception as _hitl_commit_err: # noqa: BLE001 - logger.warning( - "Failed to commit populated contract on HITL plan-gate approval (continuing) (#2593)", - pipeline_id=pipeline_id, - error=str(_hitl_commit_err), - ) - - # #2593 review issue 5 — the earlier - # ``push_worktree_branch`` at line ~20598 - # ran *before* this populate commit, so - # the populated ``contract.pr`` only - # exists locally until the IMPLEMENT - # phase's next phase-boundary sync. Push - # again now so any slice-agent container - # that materialises a fresh worktree from - # origin before that sync still sees - # ``contract.pr``. Mirrors the - # auto-advance flow's pre-context-PR push - # in ``_run_pipeline``. - if pipeline.branch and _hitl_worktree_path != repo_path: - try: - _get_spawner().gateway.push_worktree_branch( - pipeline_id=pipeline_id, - repo_path=str(_hitl_worktree_path), - branch=pipeline.branch, - mode=_gw_mode, - base_branch=pipeline.base_branch, - ) - except Exception as _hitl_push_err: # noqa: BLE001 - logger.warning( - "Failed to push populated contract on HITL plan-gate approval (continuing) (#2593)", - pipeline_id=pipeline_id, - error=str(_hitl_push_err), - ) - except Exception as _hitl_pop_err: # noqa: BLE001 - logger.warning( - "Failed to run plan-exit populate on HITL recovery (continuing) (#2593)", - pipeline_id=pipeline_id, - error=str(_hitl_pop_err), - ) - - # Defer the context-PR open until after the - # per-pipeline state lock is released — see - # ``_open_context_pr_at_implement_start``'s - # idempotency docstring on why this multi- - # second network sequence (one ``gh pr list`` - # + maybe one ``gh pr create``) must not run - # under the lock (#2593 review issue 1). - _hitl_open_context_pr_after_lock = True - _hitl_pr_worktree_path = _hitl_worktree_path - - if not next_phases: - # Terminal phase — pipeline complete. - # Bump run_epoch so any lingering old _run_pipeline - # thread (e.g. stuck in its finally block) detects the - # recreation and exits without double-cleaning up. - pipeline.status = PipelineStatus.COMPLETE - pipeline.run_epoch = datetime.now(UTC) - store.save_pipeline(pipeline) - return make_success_response( - "Pipeline recovered and completed", - data={ - "pipeline_id": pipeline_id, - "status": "complete", - "current_phase": pipeline.current_phase.value, - }, - ) - - # Advance to next phase - next_phase = next_phases[0] - pipeline.current_phase = next_phase - - # Issue #1557: PLAN → APPLY transition on epic - # pipelines (mirrors auto-advance path). Write the - # applier handoff JSON before the next _run_pipeline - # thread is respawned so the APPLIER container's - # first read finds it on disk. - if ( - getattr(pipeline, "is_epic", False) - and current_phase == PipelinePhase.PLAN - and next_phase == PipelinePhase.APPLY - ): - _hitl_apply_worktree = _resolve_pipeline_worktree_path(pipeline, repo_path) - _write_apply_phase_handoff( - pipeline, - _hitl_apply_worktree, - approved_phase="plan", - ) - - # Issue #1557 task-2-7: when the resolved phase was - # APPLY (BRC consensus confirmed via HITL recovery - # path), drain the Won't-Do handoff before advancing. - if current_phase == PipelinePhase.APPLY: - _hitl_drain_worktree = _resolve_pipeline_worktree_path(pipeline, repo_path) - _drain_wontdo_batch_after_apply(pipeline, _hitl_drain_worktree) - - # Update health monitor phase threshold before agents spawn - try: - from health_monitor import get_health_monitor - - _hm_instance = get_health_monitor() - if _hm_instance is not None: - _hm_instance.set_current_phase(next_phase.value) - except ImportError: - pass - - else: - # request_changes/change_approach — reset phase for re-run - phase_execution = pipeline.get_phase_execution(pipeline.current_phase) - # #2795: derive iteration_n monotonically. The - # ``max(len(iteration_history), max(directive_idx) + 1)`` - # form does not depend on ``hitl_review_cycles``, so - # this expression is safe to evaluate either before - # or after ``_clear_concurrent_state`` resets the - # per-phase counter. What *is* order-sensitive is - # the tracker snapshot a few lines below: the BRC - # tracker is in-memory only and gets wiped by - # ``_clear_concurrent_state``, so the snapshot MUST - # happen first. On a crash-recovery resolution the - # snapshot will typically have empty verdict detail, - # but the iteration index + artifacts are still - # useful context for iteration N+1's prompts. - # The ``max(...) + 1`` floor ensures a legacy- - # hitl_feedback migration (which synthesises a - # directive but leaves iteration_history empty) - # doesn't restart the index at 0. - _recovery_iteration_n = max( - len(phase_execution.iteration_history), - max( - (d.iteration_n for d in phase_execution.operator_directives), - default=-1, - ) - + 1, - ) - _recovery_tracker = None - try: - from peer_consensus import ( - get_peer_consensus_tracker as _gpct_recovery, - ) - - _recovery_tracker = _gpct_recovery(pipeline_id) - except Exception as tracker_err: # noqa: BLE001 - logger.debug( - "Tracker lookup failed during recovery snapshot", - pipeline_id=pipeline_id, - error=str(tracker_err), - ) - _recovery_summary = _build_iteration_summary_from_tracker( - _recovery_tracker, - iteration_n=_recovery_iteration_n, - artifacts=phase_execution.artifacts, - ) - - if phase_execution.status in ( - PipelineStatus.COMPLETE, - PipelineStatus.FAILED, - PipelineStatus.RUNNING, - PipelineStatus.AWAITING_HUMAN, - ): - # Refuse to clear containers/agents/artifacts when - # pods labeled to this pipeline are still alive — - # the reset would orphan them (#2420). - guard = _guard_live_pods_or_force(pipeline_id, force, force_reason) - if guard is not None: - return guard - phase_execution.status = PipelineStatus.PENDING - phase_execution.started_at = None - phase_execution.work_started_at = None - phase_execution.completed_at = None - phase_execution.error = None - phase_execution.review_cycles = 0 - phase_execution.hitl_review_cycles = 0 - phase_execution.containers = [] - phase_execution.agents = [] - phase_execution.artifacts = {} - - # Clear stale consensus state so re-run doesn't - # short-circuit (issue #1296). - from routes.phases import _clear_concurrent_state - - _clear_concurrent_state(pipeline_id) - - # #2795: append the operator directive + iteration - # summary so iteration N+1 prompts can render them - # with precedence prose. Both lists accumulate - # across kickbacks (no clear). - if revision_feedback: - phase_execution.operator_directives.append( - OperatorDirective( - iteration_n=_recovery_iteration_n, - feedback_text=revision_feedback, - ) - ) - phase_execution.iteration_history.append(_recovery_summary) - - pipeline.error = None - pipeline.run_epoch = datetime.now(UTC) - pipeline.status = PipelineStatus.RUNNING - store.save_pipeline(pipeline) - - # TEST_MARKER: recover_advance_clear (load-bearing: brackets - # the post-lock clear for TestRecoverPipelineClearsConcurrentState; - # do not remove without updating that test class). - # Drop the previous phase's in-memory consensus tracker on - # cross-phase advance (#2502). The request_changes / - # change_approach branch above already cleared inside the - # lock for same-phase re-runs (#1296); the advance branch - # needs its own post-lock clear so persisted state lands - # before the tracker is wiped, matching the persist-then- - # clear-then-spawn order used by ``advance_phase`` and the - # auto-advance block. - if is_approved: - from routes.phases import _clear_concurrent_state - - _clear_concurrent_state(pipeline_id) - - # #2593 review issue 1 — context-PR open moved out of the - # per-pipeline state lock so the multi-second gateway - # sequence does not hold the lock and block concurrent - # ``advance_phase`` / status reads. - # - # #2777 (cq-4, TASK-1-2) — HITL-recovery context-PR site - # calls the new idempotent - # ``_open_context_pr_at_implement_start`` opener directly. - # HITL recovery in ``start_pipeline`` does NOT route - # through ``advance_phase`` REST (the runner thread is - # spawned inline below), so without this call site an - # operator-resumed pipeline would silently strand its - # slice stack on ``egg/<id>/work``. The opener's - # ``gh pr list`` pre-flight makes a redundant call from a - # later ``advance_phase`` invocation a one-round-trip - # no-op (reviewer_code_holistic blocker 1 fix; v1 deleted - # this site under the incorrect "single canonical site" - # plan AC). - if _hitl_open_context_pr_after_lock and _hitl_pr_worktree_path is not None: - try: - _open_context_pr_at_implement_start(pipeline_id, repo_path=repo_path) - except ContextPrCreationError as ctx_err: - logger.warning( - "Context PR opener: HITL-resume failed " - "(continuing — hard-require enforced at " - "advance_phase and the implement-start plan " - "pre-flight gate) (#2777, #3100)", - pipeline_id=pipeline_id, - reason=ctx_err.reason, - error=str(ctx_err), - ) - except Exception as hitl_err: # noqa: BLE001 - logger.warning( - "Context PR opener: HITL-resume outer wrapper raised (continuing) (#2777)", - pipeline_id=pipeline_id, - error=str(hitl_err), - ) - - # Launch runner thread - thread = threading.Thread( - target=_run_pipeline, - args=(pipeline_id, repo_path), - daemon=True, - name=f"pipeline-{pipeline_id}", - ) - thread.start() - - logger.info( - "Pipeline recovered from AWAITING_HUMAN", - pipeline_id=pipeline_id, - recovery_action="advance" if is_approved else "rerun", - ) - - return make_success_response( - "Pipeline recovered and started", - data={ - "pipeline_id": pipeline_id, - "status": "running", - "current_phase": pipeline.current_phase.value, - }, - ) - - if pipeline.status == PipelineStatus.COMPLETE: - return make_error_response( - f"Pipeline {pipeline_id} is already complete", - status_code=409, - ) - - if pipeline.status == PipelineStatus.CANCELLED: - return make_error_response( - f"Pipeline {pipeline_id} is cancelled", - status_code=409, - ) - - with get_pipeline_state_lock(pipeline_id): - pipeline = store.load_pipeline(pipeline_id) - - if pipeline.status == PipelineStatus.FAILED: - # Reset the failed phase so it can be re-run. - # Also reset phases stuck in RUNNING — a pipeline-level exception - # sets the pipeline to FAILED without updating the phase status. - phase_execution = pipeline.get_phase_execution(pipeline.current_phase) - if phase_execution.status in (PipelineStatus.FAILED, PipelineStatus.RUNNING): - # Refuse to clear containers/agents/artifacts when pods - # labeled to this pipeline are still alive — the reset - # would orphan them (#2420). - guard = _guard_live_pods_or_force(pipeline_id, force, force_reason) - if guard is not None: - return guard - prev_status = phase_execution.status.value - phase_execution.status = PipelineStatus.PENDING - phase_execution.started_at = None - phase_execution.work_started_at = None - phase_execution.completed_at = None - phase_execution.error = None - phase_execution.review_cycles = 0 - phase_execution.hitl_review_cycles = 0 - phase_execution.containers = [] - phase_execution.agents = [] - phase_execution.artifacts = {} - logger.info( - "Resetting phase for restart", - pipeline_id=pipeline_id, - phase=pipeline.current_phase.value, - previous_phase_status=prev_status, - ) - pipeline.error = None - - # Bump run_epoch so the old _run_pipeline thread's finally block - # detects the restart and skips worktree cleanup. - pipeline.run_epoch = datetime.now(UTC) - - # Mark pipeline as running - pipeline.status = PipelineStatus.RUNNING - store.save_pipeline(pipeline) - - # Run the pipeline in a background thread - thread = threading.Thread( - target=_run_pipeline, - args=(pipeline_id, repo_path), - daemon=True, - name=f"pipeline-{pipeline_id}", - ) - thread.start() - - logger.info("Pipeline started", pipeline_id=pipeline_id) - - return make_success_response( - "Pipeline started", - data={ - "pipeline_id": pipeline_id, - "status": "running", - "current_phase": pipeline.current_phase.value, - }, - ) - - except InvalidPipelineIdError: - return make_error_response( - f"Invalid pipeline ID format: {pipeline_id}", - status_code=400, - ) - except PipelineNotFoundError: - return make_error_response( - f"Pipeline {pipeline_id} not found", - status_code=404, - ) - - -@pipelines_bp.route("/<pipeline_id>/visualization", methods=["GET"]) -def get_pipeline_visualization(pipeline_id: str) -> tuple[Response, int]: - """ - Get pipeline DAG visualization. - - URL params: - pipeline_id: Pipeline ID - - Query params: - format: Output format - "full" (default), "compact", "text", "json" - ascii: Use ASCII-only characters (default: false) - - Response: - { - "success": true, - "data": { - "pipeline_id": "issue-123", - "visualization": { - "dag": "...", // Full DAG visualization - "compact": "...", // Single-line status - "progress": "..." // Progress bar - }, - "phases": {...}, // Phase status summary - "status": "running", - "current_phase": "implement" - } - } - """ - # Check if visualization module is available (imported at module level) - if not _DAG_VISUALIZER_AVAILABLE: - return make_error_response( - "Visualization module not available", - status_code=500, - ) - - repo_path = get_repo_path() - output_format = request.args.get("format", "full") - use_ascii = request.args.get("ascii", "false").lower() == "true" - - try: - _store, pipeline = _resolve_pipeline(pipeline_id, repo_path) - - if output_format == "json": - # Return structured JSON report - report = generate_status_report(pipeline, use_ascii=use_ascii) - return make_success_response( - "Visualization generated", - data=report, - ) - - elif output_format == "text": - # Return plain text DAG - dag_text = render_pipeline_dag(pipeline, use_ascii=use_ascii) - return Response( - dag_text, - mimetype="text/plain", - status=200, - ) - - elif output_format == "compact": - # Return compact single-line status - compact = render_compact_status(pipeline, use_ascii=use_ascii) - progress = render_progress_bar(pipeline, use_ascii=use_ascii) - return make_success_response( - "Visualization generated", - data={ - "pipeline_id": pipeline.id, - "compact": compact, - "progress": progress, - "status": pipeline.status.value, - "current_phase": pipeline.current_phase.value, - }, - ) - - else: - # Full format with all visualizations - report = generate_status_report(pipeline, use_ascii=use_ascii) - return make_success_response( - "Visualization generated", - data=report, - ) - - except InvalidPipelineIdError: - return make_error_response( - f"Invalid pipeline ID format: {pipeline_id}", - status_code=400, - ) - except PipelineNotFoundError: - return make_error_response( - f"Pipeline {pipeline_id} not found", - status_code=404, - ) - - -@pipelines_bp.route("/stream", methods=["GET"]) -def stream_all_pipelines() -> Response: - """ - Stream unified events for all pipelines via Server-Sent Events (SSE). - - Provides real-time updates for ALL pipeline state changes in a single - SSE connection. Unlike the per-pipeline stream, terminal events for - individual pipelines do not end the stream. - - Query params: - ascii: Use ASCII-only characters (default: false) - active_only: Only include active pipelines (default: true) - full_dag: Include full DAG visualization (default: false) - - Response: - text/event-stream with the following event types: - - snapshot: Initial state of all active pipelines - - pipeline.*: Pipeline lifecycle events - - phase.*: Phase transition events - - agent.*: Agent lifecycle events - - decision.*: HITL decision events - - done: Stream is ending (timeout) - """ - if not _UNIFIED_SSE_AVAILABLE: - return make_error_response( - "Unified SSE streaming module not available", - status_code=500, - ) - - use_ascii = request.args.get("ascii", "false").lower() == "true" - active_only = request.args.get("active_only", "true").lower() == "true" - full_dag = request.args.get("full_dag", "false").lower() == "true" - - repo_path = get_repo_path() - - return Response( - stream_with_context( - create_unified_sse_stream( - repo_path=repo_path, - use_ascii=use_ascii, - active_only=active_only, - full_dag=full_dag, - ) - ), - mimetype="text/event-stream", - headers={ - "Cache-Control": "no-cache", - "X-Accel-Buffering": "no", - }, - ) - - -@pipelines_bp.route("/<pipeline_id>/stream", methods=["GET"]) -def stream_pipeline(pipeline_id: str) -> Response: - """ - Stream pipeline events via Server-Sent Events (SSE). - - Provides real-time updates for pipeline state changes including - phase transitions, agent lifecycle, and DAG visualization. - - URL params: - pipeline_id: Pipeline ID - - Query params: - ascii: Use ASCII-only characters (default: false) - - Response: - text/event-stream with the following event types: - - snapshot: Initial pipeline state - - pipeline.*: Pipeline lifecycle events - - phase.*: Phase transition events - - agent.*: Agent lifecycle events - - decision.*: HITL decision events - - done: Stream is ending (terminal state or timeout) - - error: An error occurred - - The stream automatically closes when the pipeline reaches a - terminal state (completed, failed, cancelled) or after the - maximum connection time (1 hour). - """ - if not _SSE_AVAILABLE: - return make_error_response( - "SSE streaming module not available", - status_code=500, - ) - - use_ascii = request.args.get("ascii", "false").lower() == "true" - - # Validate pipeline exists before starting stream - repo_path = get_repo_path() - try: - _resolve_pipeline(pipeline_id, repo_path) - except InvalidPipelineIdError: - return make_error_response( - f"Invalid pipeline ID format: {pipeline_id}", - status_code=400, - ) - except PipelineNotFoundError: - return make_error_response( - f"Pipeline {pipeline_id} not found", - status_code=404, - ) - - return Response( - stream_with_context( - create_sse_stream(pipeline_id, repo_path=repo_path, use_ascii=use_ascii) - ), - mimetype="text/event-stream", - headers={ - "Cache-Control": "no-cache", - "X-Accel-Buffering": "no", - }, - ) diff --git a/orchestrator/routes/pipelines/__init__.py b/orchestrator/routes/pipelines/__init__.py new file mode 100644 index 0000000000..f8d332d211 --- /dev/null +++ b/orchestrator/routes/pipelines/__init__.py @@ -0,0 +1,1479 @@ +""" +Pipeline CRUD endpoints for egg-orchestrator. +""" + +import concurrent.futures # noqa: F401 — retained for _pkg.concurrent re-export +import functools # noqa: F401 — retained for _pkg.functools re-export +import json # noqa: F401 — retained for _pkg re-export / patch seam +import os +import re +import subprocess # noqa: F401 — retained for _pkg re-export / patch seam +import sys +import threading # noqa: F401 — retained for _pkg re-export / patch seam +import time # noqa: F401 — retained for _pkg re-export / patch seam +from collections.abc import Callable # noqa: F401 — retained for _pkg re-export / patch seam +from datetime import UTC, datetime # noqa: F401 — retained for _pkg re-export / patch seam +from enum import StrEnum +from pathlib import Path +from typing import ( # noqa: F401 — NamedTuple retained for _pkg base-class re-export (_populate) + Any, + Literal, + NamedTuple, +) +from uuid import uuid4 # noqa: F401 — retained for _pkg re-export / patch seam + +try: + from docker.errors import DockerException +except ImportError: + + class DockerException(Exception): # type: ignore[no-redef] + pass + + +from flask import ( # noqa: F401 — retained for _pkg re-export / patch seam + Blueprint, + Response, + jsonify, + request, + stream_with_context, +) + +# Re-export the slice-3 per-event prompt composer so callers can still +# import it via ``orchestrator.routes.pipelines.compose_event_prompt`` +# (the contract assigns this file in TASK-3-1) even though the body +# lives in a sibling module to keep this file under the orchestrator +# decomposition cap (#2261). The slice-3 plan acceptance is satisfied +# by either import path; tests bind on +# ``orchestrator.routes.event_prompt`` directly. +from ..event_prompt import compose_event_prompt # noqa: F401 + + +# Closed enumeration of ``ContextPrCreationError.reason`` values +# (#2777). Producer and downstream tests (TASK-3-8) bind on these +# strings so a single source of truth avoids the synthetic-key +# divergence reviewer_code_holistic flagged. New reasons MUST be +# added here AND to ``ContextPrCreationReason`` so the type narrows. +class ContextPrCreationReason(StrEnum): + """Closed set of typed reasons for :class:`ContextPrCreationError` (#2777).""" + + UNKNOWN = "unknown" + # Lookup of the pipeline / store / spawner failed before any + # gateway call could be attempted. + PIPELINE_LOAD_FAILED = "pipeline_load_failed" + ROUTES_UNAVAILABLE = "routes_unavailable" + LOADER_UNAVAILABLE = "loader_unavailable" + # Pipeline misconfiguration. ``base_branch`` left unset alongside a + # ``repo`` is NOT a misconfiguration — it is the normal "auto-detect + # the repo's default branch" state (#3031), so the opener resolves it + # rather than raising. The remaining genuine misconfigurations are a + # ``base_branch`` with no ``repo`` to open a PR against + # (``missing_repo``) and a remote pipeline with no work branch + # (``missing_branch``). + MISSING_BRANCH = "missing_branch" + MISSING_REPO = "missing_repo" + # Contract / PR-metadata failures encountered after the pipeline + # passed the misconfiguration check. + CONTRACT_LOAD_FAILED = "contract_load_failed" + MISSING_PR_METADATA = "missing_pr_metadata" + SAVE_FAILED = "save_failed" + # Gateway-layer failures wrapping ``lookup_open_pr`` / + # ``create_pr`` outcomes. + LOOKUP_FAILED = "lookup_failed" + GATEWAY_ERROR = "gateway_error" + GATEWAY_NO_URL = "gateway_no_url" + GATEWAY_BAD_URL = "gateway_bad_url" + + +class ContextPrCreationError(Exception): + """Raised by :func:`_open_context_pr_at_implement_start` when the + hard-required up-front context PR cannot be opened (#2777, cq-4). + + Replaces the soft-fail ``return None`` swallow path that the legacy + ``_maybe_open_base_pr_for_plan_to_implement`` wrapper used before + slice-2 deleted it. Under cq-4 the context PR is hard-required at + the plan→implement boundary; a gateway failure here must surface to + the BRC NACK / 422 surface rather than silently strand the slice + stack on ``/work``. + + Attributes: + reason: Machine-readable reason drawn from + :class:`ContextPrCreationReason`. Tests assert on these + constants so producer and tests share one source of + truth; passing an unknown string is a programming error + caught here. The instance attribute is exposed as the + underlying ``str`` value (matching ``.value`` of the + enum) so existing JSON-serialization callers continue to + work without change. + cause: The original exception, if any, that triggered the + error. Preserved so logs and the BRC NACK body show the + gateway/contract failure rather than only this wrapper's + text. + """ + + def __init__( + self, + message: str, + *, + reason: str | ContextPrCreationReason = ContextPrCreationReason.UNKNOWN, + cause: BaseException | None = None, + ) -> None: + super().__init__(message) + # Coerce-and-validate the reason against the closed + # enumeration. Passing a string that is not a known reason + # would normally raise ``ValueError`` from the ``StrEnum`` + # constructor — but the four ``except ContextPrCreationError`` + # handlers at every call site would not match that + # ``ValueError``, so a typo would surface as a 500 instead of + # the typed 422 the handlers contract on + # (egg-reviewer non-blocking #4). Catch and coerce to + # ``UNKNOWN`` so the typed-exception contract holds, and log + # the bad reason loudly so the typo is still visible in the + # operator's logs and CI grep — silent coercion would hide + # the programming error. + try: + self.reason: str = ContextPrCreationReason(reason).value + except ValueError: + logger.warning( + "ContextPrCreationError received unknown reason; " + "coercing to UNKNOWN (#2777, egg-reviewer non-blocking #4)", + bad_reason=repr(reason), + error_message=message, + ) + self.reason = ContextPrCreationReason.UNKNOWN.value + self.cause: BaseException | None = cause + + +class ForestValidationError(Exception): + """Raised by ``_populate_contract_from_plan`` on slice-DAG structural rejection. + + Added in #2137 (TASK-2-2) for the forest-shape violation (a slice + with >1 DAG parent). Generalised in #3046 to also signal the + file-overlap-ordering violation (two slices touching the same file + with no dependency edge between them — see + ``egg_contracts.validate_slice_file_overlap``). Both are slice-DAG + structural defects surfaced at plan ingestion with identical + handling: the slices are NOT written to the contract, the structured + errors are stashed on ``plan_review_feedback`` so the plan reviewer + NACKs the architect, and the exception is raised so HTTP callers can + return a 422. + + The ``reason`` discriminator (``"forest_violation"`` or + ``"slice_overlap_violation"``) selects the operator-facing prose and + the :class:`PopulateOutcome` the safe wrapper maps to. Any future + Flask route that ingests a plan in-band can catch this and + ``body, status = err.to_response(); return jsonify(body), status`` + to surface the structured rejection. Internal callers + (``_populate_contract_from_plan_safe`` and the pipeline run-loop + helpers) catch it and log a warning — the ``plan_review_feedback`` + stash is the durable NACK signal either way. + """ + + def __init__( + self, message: str, *, errors: list[str], reason: str = "forest_violation" + ) -> None: + super().__init__(message) + self.errors: list[str] = list(errors) + self.reason: str = reason + self.status_code: int = 422 + + def to_response(self) -> tuple[dict[str, object], int]: + """Serialise into a Flask-compatible (body, status) tuple.""" + return ({"error": self.reason, "errors": self.errors}, 422) + + +# Completion bases a caller may declare when it has positive, verified +# evidence a slice finished even though not every task is marked COMPLETE +# on the contract (the crash-recovery / merged-skip paths). See +# :class:`SliceCompletionInvariantError`. +_VERIFIED_SLICE_COMPLETION_BASES = frozenset({"merged", "consensus_complete"}) + + +# Add shared directory to path for egg_logging +_shared_path = Path(__file__).parent.parent.parent.parent / "shared" +if _shared_path.exists() and str(_shared_path) not in sys.path: + sys.path.insert(0, str(_shared_path)) + +# Add config directory to path for repo_config module +_config_path = Path(__file__).parent.parent.parent.parent / "config" +if _config_path.exists() and str(_config_path) not in sys.path: + sys.path.insert(0, str(_config_path)) + +try: + from egg_logging import get_logger +except ImportError: + import logging + + def get_logger(name: str, **kwargs) -> logging.Logger: # type: ignore[misc] + return logging.getLogger(name) + + +try: + from repo_config import get_repo_checks +except ImportError: + + def get_repo_checks(repo: str) -> list[dict[str, str]]: # type: ignore[misc] + return [] + + +# Import orchestrator modules - try relative import first +try: + from .. import agent_salvage + from ..container_spawner import ContainerSpawnError, SpawnFailureError, get_container_spawner + from ..decision_queue import get_decision_queue + from ..docker_client import ContainerNotFoundError, ContainerOperationError, DockerClientError + from ..gateway_client import ( + GatewayError, + _rebase_with_agent_output_autoresolve, # noqa: F401 + ) + from ..kubernetes_client import ( + LABEL_AGENT_ROLE, + LABEL_PIPELINE_ID, + LABEL_SLICE_ID, + JobOperationError, + KubernetesClientError, + PodNotFoundError, + ) + from ..kubernetes_spawner import KubernetesSpawnError, get_kubernetes_spawner + from ..models import ( + LIVE_POD_STATUSES, + AgentExecutionStatus, + AgentExitInfo, + AgentRole, + ContainerInfo, + ContainerStatus, # noqa: F401 — retained for _pkg re-export / patch seam + CycleTiming, + DecisionStatus, + HITLDecision, + IterationSummary, + OperatorDirective, + PhaseExecution, + Pipeline, + PipelineMode, + PipelinePhase, + PipelineStatus, + RepoSpec, + ) + from ..slice_id_validation import extract_slice_id + from ..state_store import ( + InvalidPipelineIdError, + PipelineNotFoundError, + StateStore, # noqa: F401 — retained for _pkg re-export / patch seam + StateStoreError, + StateValidationError, + get_pipeline_state_lock, + get_state_store, + ) +except ImportError: + import agent_salvage # type: ignore[no-redef] # noqa: F401 — retained for _pkg re-export / patch seam + from container_spawner import ( # type: ignore + ContainerSpawnError, # noqa: F401 — retained for _pkg re-export / patch seam + SpawnFailureError, # noqa: F401 — retained for _pkg re-export / patch seam + get_container_spawner, # noqa: F401 — retained for _pkg re-export / patch seam + ) + from decision_queue import ( + get_decision_queue, # type: ignore # noqa: F401 — retained for _pkg re-export / patch seam + ) + from docker_client import ( # type: ignore + ContainerNotFoundError, # noqa: F401 — retained for _pkg re-export / patch seam + ContainerOperationError, # noqa: F401 — retained for _pkg re-export / patch seam + DockerClientError, # noqa: F401 — retained for _pkg re-export / patch seam + ) + from gateway_client import ( # type: ignore + GatewayError, # noqa: F401 — retained for _pkg re-export / patch seam + _rebase_with_agent_output_autoresolve, # noqa: F401 + ) + from kubernetes_client import ( # type: ignore + LABEL_AGENT_ROLE, # noqa: F401 — retained for _pkg re-export / patch seam + LABEL_PIPELINE_ID, # noqa: F401 — retained for _pkg re-export / patch seam + LABEL_SLICE_ID, # noqa: F401 — retained for _pkg re-export / patch seam + JobOperationError, # noqa: F401 — retained for _pkg re-export / patch seam + KubernetesClientError, # noqa: F401 — retained for _pkg re-export / patch seam + PodNotFoundError, # noqa: F401 — retained for _pkg re-export / patch seam + ) + from kubernetes_spawner import ( # type: ignore + KubernetesSpawnError, # noqa: F401 — retained for _pkg re-export / patch seam + get_kubernetes_spawner, # noqa: F401 — retained for _pkg re-export / patch seam + ) + from models import ( # type: ignore + LIVE_POD_STATUSES, + AgentExecutionStatus, # noqa: F401 — retained for _pkg re-export / patch seam + AgentExitInfo, # noqa: F401 — retained for _pkg re-export / patch seam + AgentRole, # noqa: F401 — retained for _pkg re-export / patch seam + ContainerInfo, # noqa: F401 — retained for _pkg re-export / patch seam + ContainerStatus, # noqa: F401 — retained for _pkg re-export / patch seam + CycleTiming, # noqa: F401 — retained for _pkg re-export / patch seam + DecisionStatus, # noqa: F401 — retained for _pkg re-export / patch seam + HITLDecision, # noqa: F401 — retained for _pkg re-export / patch seam + IterationSummary, # noqa: F401 — retained for _pkg re-export / patch seam + OperatorDirective, # noqa: F401 — retained for _pkg re-export / patch seam + PhaseExecution, # noqa: F401 — retained for _pkg re-export / patch seam + Pipeline, # noqa: F401 — retained for _pkg re-export / patch seam + PipelineMode, # noqa: F401 — retained for _pkg re-export / patch seam + PipelinePhase, # noqa: F401 — retained for _pkg re-export / patch seam + PipelineStatus, # noqa: F401 — retained for _pkg re-export / patch seam + RepoSpec, # noqa: F401 — retained for _pkg re-export / patch seam + ) + from slice_id_validation import ( + extract_slice_id, # type: ignore # noqa: F401 — retained for _pkg re-export / patch seam + ) + from state_store import ( # type: ignore + InvalidPipelineIdError, # noqa: F401 — retained for _pkg re-export / patch seam + PipelineNotFoundError, # noqa: F401 — retained for _pkg re-export / patch seam + StateStore, # noqa: F401 — retained for _pkg re-export / patch seam + StateStoreError, # noqa: F401 — retained for _pkg re-export / patch seam + StateValidationError, # noqa: F401 — retained for _pkg re-export / patch seam + get_pipeline_state_lock, # noqa: F401 — retained for _pkg re-export / patch seam + get_state_store, # noqa: F401 — retained for _pkg re-export / patch seam + ) + +from egg_contracts.orchestrator import ( # noqa: F401 — retained for _pkg re-export / patch seam + load_agent_output, + save_agent_output, +) +from egg_git.default_branch import ( + get_default_branch, # noqa: F401 — retained for _pkg re-export / patch seam +) +from lifecycle_auth import require_lifecycle_secret + +logger = get_logger("orchestrator.pipelines") + + +# ----------------------------------------------------------------- +# egg_inflight_host_waits metric (issue #1932 TASK-1-3). +# +# Gauge counting in-flight ``/status/wait`` route calls. Paired with +# ``egg_inflight_long_polls`` from ``routes/messages.py`` — both draw +# against the same Waitress thread pool so operators alert on the +# sum when it approaches ``EGG_ORCH_WAITRESS_THREADS``. The +# lame-duck daemon thread that keeps running after the route returns +# (up to ``wait`` seconds of ``message_store.get_messages``) is +# deliberately NOT counted against this gauge — the metric represents +# in-flight *route* calls, not in-flight store waits. +# +# Best-effort registration so missing-metrics-backend deployments +# degrade gracefully (matches the pattern at routes/messages.py:80-85). +# ----------------------------------------------------------------- +try: + from metrics import get_metrics_registry as _get_metrics_registry_for_host_wait + + _inflight_host_waits = _get_metrics_registry_for_host_wait().gauge( + "egg_inflight_host_waits", + labels={"endpoint": "pipelines.status_wait"}, + ) +except Exception: # pragma: no cover - metrics best-effort + _inflight_host_waits = None + + +# ----------------------------------------------------------------- +# Cursor protocol for /status/wait (issue #1932 TASK-1-2). +# +# Opaque compound cursor "msg:<redis_stream_id>|evt:<sequence>": +# * ``msg:<id>`` is the message-store tip ID from the prior call. +# Either half may be empty when the corresponding source has not +# emitted yet (e.g. ``msg:|evt:5`` = "no message seen, EventBus +# tip at seq 5"). +# * ``evt:<sequence>`` is the EventBus per-bus monotonic sequence +# (see ``Event.sequence`` added in TASK-1-1). The sequence is +# signed purely so malformed inputs with leading ``-`` are +# accepted by the regex and handled gracefully by the parser. +# +# The regex is intentionally permissive — unknown halves degrade to +# ``None`` which the route maps to "snap to tip" (``from_tip`` on +# the message bus, ``current_sequence`` on the EventBus) so +# first-call semantics are race-free. +# ----------------------------------------------------------------- +_STATUS_WAIT_CURSOR_RE = re.compile(r"^msg:([^|]*)\|evt:(-?\d*)$") + +# Slice-or-phase id shape used when reading the parent edge from the +# contract for the restart route's ``base_branch`` derivation (#2439). +# ``Slice.id`` permits either ``slice-<N>`` (canonical) or ``phase-<N>`` +# (legacy, pre-#2137) and the contract migration shim only normalises +# the typical case where the input has a top-level ``phases`` key. A +# directly-loaded ``slices`` field with legacy ids is rare but allowed +# by the model — accept either shape here so the gate doesn't false- +# reject a legitimate restart on a long-lived contract. +_SLICE_OR_PHASE_ID_PATTERN = re.compile(r"^(?:slice|phase)-[0-9]+$") + +# Event allowlist for ``/status/wait`` (issue #1932 locked in +# refine HITL decision 2). The route returns early when an event +# matching any of these types is published. ``DECISION_RESOLVED`` +# is intentionally excluded — it is the post-``provide_input`` +# event and would cause the host to self-wake on an action it +# initiated. Agent-lifecycle events are excluded because the host +# does not drive on them. See +# docs/reference/agent-wait-patterns.md §7. +_STATUS_WAIT_EVENT_TYPES = frozenset( + { + "phase.started", + "phase.completed", + "decision.created", + "pipeline.completed", + "pipeline.failed", + "pipeline.cancelled", + } +) + +# Message-type allowlist for ``/status/wait`` (same HITL decision). +# Wired to ``message_store.get_messages(wait_for_types=...)`` so a +# message of a non-matching type does NOT unblock the waiter. +_STATUS_WAIT_MESSAGE_TYPES = ( + "OVERSEER_ALERT", + "CONSENSUS_CONFIRMED", + "CONSENSUS_NACK", + "CONSENSUS_RE_REVIEW", +) + + +# --------------------------------------------------------------------------- +# Overseer authority plane (#2270 slice-6, §4) — the orchestrator-side seams the +# CorrectiveExecutor dispatches to. The overseer ADVISES (returns a verdict); the +# control plane EXECUTES exactly three bounded actions. Agents — including the +# overseer — cannot reach these directly: the gateway file patterns deny agents +# from contract writes (the "403"), and the executor only runs control-plane-side. +# The seams are invoked by CorrectiveExecutor with keyword arguments. See +# orchestrator/overseer/corrective.py and gateway/agent_restrictions.py. +# --------------------------------------------------------------------------- + + +# Base directory where the gateway creates per-pipeline worktrees. +# Must match the gateway's WORKTREE_BASE_DIR and docker-compose volume mounts. +WORKTREE_BASE_DIR = Path("/home/egg/.egg-worktrees") + +# Sentinel header used in tester gap summaries. Checked in prompt-building +# functions to adapt language when tester findings are present. +TESTER_FINDINGS_HEADER = "### tester findings" + + +# Network constants for sandbox container URLs +try: + from egg_config import ( + ORCHESTRATOR_EXTERNAL_IP, + ORCHESTRATOR_ISOLATED_IP, + ORCHESTRATOR_PORT, + ) +except ImportError: + ORCHESTRATOR_ISOLATED_IP = "172.32.0.3" + ORCHESTRATOR_EXTERNAL_IP = "172.33.0.3" + ORCHESTRATOR_PORT = 9849 + +try: + from egg_config.validators import validate_checks +except ImportError: + + def validate_checks(checks: list) -> list[dict[str, str]]: # type: ignore[misc] + if not isinstance(checks, list): + return [] + return [ + {"name": str(c["name"]), "command": str(c["command"])} + for c in checks + if isinstance(c, dict) and "name" in c and "command" in c + ] + + +pipelines_bp = Blueprint("pipelines", __name__, url_prefix="/api/v1/pipelines") + + +# Runtime detection: use Kubernetes spawner when EGG_RUNTIME=kubernetes +_RUNTIME = os.environ.get("EGG_RUNTIME", "docker") + + +# Live-pod status filter (#2420). Hoisted to ``models.LIVE_POD_STATUSES`` +# in #2650 so ``startup_reconciliation`` and this module can't drift; +# this alias preserves the historical underscore-prefixed name used by +# existing tests and prose references. +_LIVE_POD_STATUSES = LIVE_POD_STATUSES + + +from routes import get_repo_path # noqa: E402,F401 — shared helper, retained for _pkg re-export + +try: + from gateway_client import get_gateway_client +except ImportError: + from orchestrator.gateway_client import ( + get_gateway_client, # type: ignore # noqa: F401 — retained for _pkg re-export / patch seam + ) + +# Import status reporter for real-time updates +try: + from status_reporter import get_status_reporter, report_pipeline_status +except ImportError: + # Fallback if status_reporter not available + def get_status_reporter(): # type: ignore[misc] + return None + + def report_pipeline_status(pipeline, event_type=None, message=None): # type: ignore[misc] + pass + + +# Import event bus for SSE streaming. +# report_pipeline_status dispatches to StatusReporter handlers, but the +# SSE stream subscribes to the EventBus — a separate system. We need to +# emit events to both so SSE clients see live updates. +try: + from events import EventType + from events import emit_event as _emit_event +except ImportError: + _emit_event = None # type: ignore[assignment] + +# Map report_pipeline_status event_type strings to EventType enum values +_EVENT_TYPE_MAP: dict[str, EventType] = {} +if _emit_event is not None: + _EVENT_TYPE_MAP = { + "phase.started": EventType.PHASE_STARTED, + "phase.completed": EventType.PHASE_COMPLETED, + "phase.revision_requested": EventType.PHASE_STARTED, # re-entering phase + "pipeline.completed": EventType.PIPELINE_COMPLETED, + "pipeline.failed": EventType.PIPELINE_FAILED, + "pipeline.cancelled": EventType.PIPELINE_CANCELLED, + "decision.created": EventType.DECISION_CREATED, + } + + +# Import visualization modules for DAG endpoint +try: + from dag_visualizer import ( + generate_status_report, # noqa: F401 — retained for _pkg re-export / patch seam + render_compact_status, # noqa: F401 — retained for _pkg re-export / patch seam + render_pipeline_dag, # noqa: F401 — retained for _pkg re-export / patch seam + render_progress_bar, # noqa: F401 — retained for _pkg re-export / patch seam + ) + + _DAG_VISUALIZER_AVAILABLE = True +except ImportError: + _DAG_VISUALIZER_AVAILABLE = False + +# Import SSE streaming support +try: + from sse import create_sse_stream # noqa: F401 — retained for _pkg re-export / patch seam + + _SSE_AVAILABLE = True +except ImportError: + _SSE_AVAILABLE = False + +# Import unified SSE streaming support +try: + from unified_sse import ( + create_unified_sse_stream, # noqa: F401 — retained for _pkg re-export / patch seam + ) + + _UNIFIED_SSE_AVAILABLE = True +except ImportError: + _UNIFIED_SSE_AVAILABLE = False + + +def make_error_response( + message: str, + status_code: int = 400, + details: dict[str, Any] | None = None, + reason: str | None = None, +) -> tuple[Response, int]: + """Create an error response. + + ``reason`` is a stable, machine-readable enum-like code that disambiguates + responses sharing the same HTTP status (especially 409, where distinct + gates would otherwise collapse into one signal). Callers should switch on + ``reason`` rather than parsing ``message``. See #1939. + """ + response: dict[str, Any] = {"success": False, "message": message} + if reason is not None: + response["reason"] = reason + if details: + response["details"] = details + return jsonify(response), status_code + + +def make_success_response( + message: str, + data: dict[str, Any] | None = None, +) -> tuple[Response, int]: + """Create a success response.""" + response: dict[str, Any] = {"success": True, "message": message} + if data: + response["data"] = data + return jsonify(response), 200 + + +@pipelines_bp.route("", methods=["GET"]) +def list_pipelines() -> tuple[Response, int]: + return _list_pipelines_body() + + +@pipelines_bp.route("/<pipeline_id>", methods=["GET"]) +def get_pipeline(pipeline_id: str) -> tuple[Response, int]: + return _get_pipeline_body(pipeline_id) + + +@pipelines_bp.route("", methods=["POST"]) +@require_lifecycle_secret +def create_pipeline() -> tuple[Response, int]: + return _create_pipeline_body() + + +@pipelines_bp.route("/<pipeline_id>", methods=["PATCH"]) +@require_lifecycle_secret +def update_pipeline(pipeline_id: str) -> tuple[Response, int]: + return _update_pipeline_body(pipeline_id) + + +# Config keys the live config-update route accepts. Deliberately a tight +# allowlist (#3174): most of PipelineConfig is consumed at submit time or +# mid-phase in ways a partial update could corrupt. Two families qualify: +# +# * ``agent_models`` is re-resolved from a fresh store load before every +# spawn (the run loop reloads the pipeline at the top of each cycle, and +# the restart_agent / restart_phase paths load fresh state), so mutating +# it on a live pipeline is honored by construction. +# * ``consensus_timeout_minutes*`` is re-resolved from a fresh store load +# by the phase poll loop right before the consensus wall fires (#3490), +# so a widened window takes effect on a running slice without a restart. +# +# Widen only after verifying the same fresh-reload guarantee holds for the +# new key. +_CONSENSUS_TIMEOUT_CONFIG_KEYS = ( + "consensus_timeout_minutes", + "consensus_timeout_minutes_refine", + "consensus_timeout_minutes_plan", + "consensus_timeout_minutes_implement", +) +_MUTABLE_CONFIG_KEYS = frozenset({"agent_models", *_CONSENSUS_TIMEOUT_CONFIG_KEYS}) + + +@pipelines_bp.route("/<pipeline_id>/config", methods=["PATCH"]) +@require_lifecycle_secret +def update_pipeline_config(pipeline_id: str) -> tuple[Response, int]: + return _update_pipeline_config_body(pipeline_id) + + +@pipelines_bp.route("/<pipeline_id>", methods=["DELETE"]) +@require_lifecycle_secret +def delete_pipeline(pipeline_id: str) -> tuple[Response, int]: + return _delete_pipeline_body(pipeline_id) + + +@pipelines_bp.route("/<pipeline_id>/agents/<agent_role>/restart", methods=["POST"]) +@require_lifecycle_secret +def restart_agent(pipeline_id: str, agent_role: str) -> tuple[Response, int]: + return _restart_agent_body(pipeline_id, agent_role) + + +@pipelines_bp.route("/<pipeline_id>/phases/<phase>/restart", methods=["POST"]) +@require_lifecycle_secret +def restart_phase(pipeline_id: str, phase: str) -> tuple[Response, int]: + return _restart_phase_body(pipeline_id, phase) + + +@pipelines_bp.route("/<pipeline_id>/local-commits", methods=["GET"]) +def list_pipeline_local_commits(pipeline_id: str) -> tuple[Response, int]: + return _list_pipeline_local_commits_body(pipeline_id) + + +@pipelines_bp.route("/<pipeline_id>/salvage", methods=["POST"]) +@require_lifecycle_secret +def salvage_pipeline_local_commits(pipeline_id: str) -> tuple[Response, int]: + return _salvage_pipeline_local_commits_body(pipeline_id) + + +@pipelines_bp.route("/<pipeline_id>/status", methods=["GET"]) +def get_pipeline_status(pipeline_id: str) -> tuple[Response, int]: + return _get_pipeline_status_body(pipeline_id) + + +# ----------------------------------------------------------------- +# GET /api/v1/pipelines/<pipeline_id>/status/wait (issue #1932) +# +# Event-driven host-side wait primitive. Blocks up to ``wait`` +# seconds until one of the allowlisted EventBus events or message +# types fires, then returns a small envelope the MCP handler +# enriches with a full status snapshot. See +# docs/reference/agent-wait-patterns.md §7 for the end-to-end +# protocol. +# ----------------------------------------------------------------- +@pipelines_bp.route("/<pipeline_id>/status/wait", methods=["GET"]) +def wait_pipeline_status(pipeline_id: str) -> tuple[Response, int]: + return _wait_pipeline_status_body(pipeline_id) + # Daemon thread is deliberately left running — it exits on + # its own when ``message_store.get_messages`` returns or the + # timeout elapses (plan risk R14, accepted). + + +# Human-focused companion drafts (mandatory, produced by the simplifier). +# Resolved through the same artifact-spec registry as the agent drafts so +# the path knowledge lives in exactly one place. Kept separate from +# ``_get_draft_path`` (which is pinned byte-for-byte by a consistency test +# and switches on the real phase value) rather than overloading its phase +# argument with a synthetic ``refine-human`` key. + + +# Subset of BRC_HISTORY_TYPES that the orchestrator's CONSENSUS_* signal +# handlers tag with ``metadata['slice_id']`` for slice-aware implement +# pipelines (#2548). The implement-phase BRC writer treats a missing +# ``slice_id`` on these as a contract violation (drop with WARNING), +# while the remaining BRC_HISTORY_TYPES (HEARTBEAT, STATUS, HANDOFF, +# AGENT_FAILED, NUDGE, OVERSEER_ALERT) come from emitters that do not +# uniformly carry slice scope — those are routed to the unattributed +# sibling file rather than dropped, so the audit trail stays complete. + + +# --- #3393 slice-5: cross-repo merge-sequencing HITL holds ------------------- +# Stable discriminator prefix on the cross-repo-hold Decision question so +# (a) the poll can idempotently detect an already-registered hold for a +# gate across reconciler ticks / orchestrator restarts, and (b) a future +# dispatch handler in ``routes/decisions.py`` can route on the literal +# substring without a separate context field on the contract Decision. +_CROSS_REPO_HOLD_MARKER_PREFIX = "[#3393 cross-repo-hold" + + +_CROSS_REPO_HOLD_REASON_TEXT = { + "closed_unmerged": ( + "the upstream cross-repo PR was CLOSED without merging, so the " + "automated merge-state hold cannot auto-ready this slice's PR" + ), + "timeout": ( + "the upstream cross-repo PR did not merge within the poll bound, so " + "the automated merge-state hold timed out rather than leaving this " + "slice's PR draft indefinitely" + ), + "beyond_merge_state": ( + "the plan declared this cross-repo dependency a beyond-merge-state " + "condition (release/publish, version-pin, or cannot-continue block), " + "which is released by human decision, never automated detection" + ), +} + + +# The two operator-selectable options on a cross-repo hold Decision. The +# RELEASE option readies the PR; the KEEP option leaves it draft for manual +# handling. Kept as constants so the registration (options list) and the +# resolution reader agree on one shape. +_CROSS_REPO_HOLD_RELEASE_OPTION_ID = "opt-release" +_CROSS_REPO_HOLD_RELEASE_OPTION_LABEL = "Release the hold and mark the PR ready" +_CROSS_REPO_HOLD_KEEP_OPTION_ID = "opt-keep" +_CROSS_REPO_HOLD_KEEP_OPTION_LABEL = "Keep the PR held for manual handling" + + +# Shared PR description guidance injected into planner prompts. +# Kept as a constant so both _build_phase_prompt and _build_agent_prompt +# stay in sync when the guidance evolves. +_PR_DESCRIPTION_GUIDANCE = [ + "**PR description quality**: The `pr.description` field becomes the PR body " + "that reviewers read first. Write 2-3 paragraphs following this structure:", + "1. **Context** — what problem exists and why it matters", + "2. **Changes** — what this PR does, with specifics (e.g. numbered list of " + "key changes with bold headers)", + "3. **Impact** — what behavior changes for users or other components", + "", + "Do NOT write a one-liner — reviewers need enough detail to understand " + "the problem, the approach, and why it was chosen without reading every file.", +] + +_PR_DESCRIPTION_YAML_EXAMPLE = [ + " Explain the problem or need this PR addresses and why it matters.", + "", + " Describe the key changes, ideally as a numbered or bulleted list", + " with bold headers so reviewers can scan quickly. For each change,", + " explain what it does and why.", + "", + " Summarize the impact — what changes for users, callers, or other", + " components as a result.", +] + +# YAML safety guidance for planner prompts. Plain (unquoted) scalars break +# when they contain ``: `` sequences — e.g. "Add `sequence: int = 0` field" +# parses as a nested mapping and raises ScannerError. Block scalars (``|-``) +# take the whole indented block literally, so backticks, colons, quotes, and +# other punctuation are safe. See issue #1974. +_YAML_TASKS_SAFETY_GUIDANCE = [ + "**YAML safety**: Use block scalars (`|-`) for every prose field — " + "`name`, `goal`, `description`, `acceptance`. Plain unquoted scalars " + "break when the text contains `` `code: type` ``, colons in URLs, or " + "other `: ` sequences, because PyYAML reads them as nested mappings " + "and the parser drops back to markdown fallback (silently losing the " + "`pr:` block). Follow the example above literally — do not inline these " + "values on the same line as the key.", +] + +# Permissive subagent-exploration guidance for producer prompts (#2814). +# Producers may delegate deep grep/Read exploration to the Claude Code +# `general-purpose` subagent so large tool-result payloads don't accumulate +# in the producer's main context window. Mitigates the failure surface of +# #2804 (Agent SDK 1MB JSON buffer overflow). Reused across all seven +# producer prompts so the wording stays uniform. +# +# `general-purpose` is the only subagent the Agent SDK ships out of the +# box; we deliberately do not name `Explore` here because the sandbox +# runtime does not register an `Explore` AgentDefinition (no `agents=` +# on ClaudeAgentOptions and no filesystem `.claude/agents/Explore.md`), +# so the example would burn a turn on an unknown-subagent retry. +_EXPLORATION_SUBAGENT_HEADER = "## Subagent use for exploration" +_EXPLORATION_SUBAGENT_GUIDANCE = [ + f"{_EXPLORATION_SUBAGENT_HEADER}\n", + "You **may** use the Agent tool (`subagent_type: general-purpose`) " + "when exploration would otherwise dominate your context window. Use " + "your judgment — a one-off grep or short read doesn't need a " + "subagent; deep investigation of a large file or many call sites " + "usually does. The producer's main context stays lean for synthesis; " + "the subagent returns a focused summary.\n", + "Example signals where subagent use often pays off:", + "- More than ~3 grep/read calls on the same target file or directory.", + "- Walking a primitive's call sites — delegate; ask for `file:line` " + "citations + a few lines of context.", + "- Reading large files (> ~500 lines) — get a subagent summary first; " + "only `Read` the main file yourself if the summary identifies specific " + "line ranges you need to author at.\n", + "Subagent summaries are part of your authoritative work. Verify " + "critical claims (e.g. `file:line` citations) before committing them " + "to your output.", + "", +] + + +# --------------------------------------------------------------------------- +# Multi-agent execution helpers +# --------------------------------------------------------------------------- + + +# Role descriptions for agent roster — maps role names to (short description, +# what artifacts they produce). +_ROLE_DESCRIPTIONS: dict[str, tuple[str, str]] = { + "coder": ( + "Implements code changes", + "commits with source files, tests may be included", + ), + "tester": ( + "Writes comprehensive regression tests AND adversarially probes the " + "coder's implementation for bugs and edge cases (dual role: also " + "reviews coder)", + "test files (including failing tests that demonstrate bugs), check " + "results, gap reports back to the coder", + ), + "documenter": ( + "Documents the current state of the code", + "doc files, README updates, inline documentation", + ), + "refiner": ( + "Refines implementation based on review feedback", + "updated source files addressing review concerns", + ), + "architect": ( + "Designs architecture and component structure", + "architecture analysis, component breakdown", + ), + "task_planner": ( + "Breaks work into implementation tasks", + "task list with acceptance criteria", + ), + "risk_analyst": ( + "Assesses technical risks", + "risk assessment with mitigations", + ), + "reviewer_code": ( + "Reviews code quality, correctness, and security", + "ACK/NACK with file-level feedback", + ), + "reviewer_code_holistic": ( + "Holistic single-pass review for cross-module coherence " + "(use-case end-to-end, doc↔code symmetry, synthetic-key audit, " + "silent-fallback hunt)", + "ACK/NACK with cross-module findings", + ), + "reviewer_contract": ( + "Verifies implementation matches contract/requirements", + "ACK/NACK with task-level verification", + ), + "reviewer_refine": ( + "Reviews refinement changes", + "ACK/NACK on refined implementation", + ), + "first_principles_reviewer": ( + "Adversarially reviews the seed and the refiner's direction from " + "first principles; surfaces significant redirects to the operator as " + "HITL decisions and never NACKs the refiner", + "an ACK on the refiner plus any HITL redirect decisions", + ), + "reviewer_agent_design": ( + "Reviews agent design and architecture decisions", + "ACK/NACK on design choices", + ), + "reviewer_plan": ( + "Reviews plan phase outputs", + "ACK/NACK on architecture, tasks, and risk assessment", + ), + "simplifier": ( + "Distills the producer's draft into a jargon-free, human-focused " + "companion summary (depends on the producer's pushed draft)", + "a simplified `*-human.md` companion to the analysis/plan", + ), +} + + +# Bound on how many times the worktree-divergence reconcile will pause +# for the operator before giving up and failing the pipeline (#2979). +# A small budget guards against an operator repeatedly choosing +# "Reconciled — resume" without actually reconciling the worktree, which +# would otherwise re-pause forever. +_MAX_DIVERGENCE_RECONCILE_PAUSES = 3 + +_DIVERGENCE_RECONCILE_RESUME = "Reconciled — resume" +_DIVERGENCE_RECONCILE_ABORT = "Abort pipeline" +_DIVERGENCE_RECONCILE_HITL_OPTIONS = [ + _DIVERGENCE_RECONCILE_RESUME, + _DIVERGENCE_RECONCILE_ABORT, +] + +# Stable string discriminator set on persisted reconcile HITLs (#2979). The +# non-blocking ``populate_contract`` route uses this to dedupe: when an +# operator re-POSTs against an already-paused pipeline (e.g. an automated +# retry or a refresh through ``/sdlc`` before resolving the prior HITL), the +# route surfaces the existing pending decision rather than appending a fresh +# one. +_DIVERGENCE_RECONCILE_HITL_CONTEXT = "divergence_reconcile_unacked" + +# Stable string discriminator set on the consensus-timeout / incomplete- +# consensus HITL the orchestrator opens when a phase times out without +# converging (the "Consensus timed out; consensus incomplete …" +# Retry/Accept/Abort decision). The convergence-success path uses this to +# auto-withdraw the decision once the phase reaches genuine consensus, so an +# operator is never left disposing of a decision the system already obsoleted +# (#3315 facet c — happens when a superseded thread opens the decision and a +# restarted thread then converges). +_CONSENSUS_TIMEOUT_HITL_CONTEXT = "consensus_timeout_incomplete" + + +# Pipeline-branch divergence alert (#2224 PR 3; #2270 §2 calibration). +# +# Watches ``origin/<pipeline_branch>`` for the contamination shape from +# #2222: the branch has absorbed already-merged main commits (a bad +# rebase / merge re-introduces commits that already live in +# ``origin/<base>``). The original detector keyed on a ``(#NNNN)`` +# subject regex, which both *false-positives* (an agent legitimately +# references a PR number in a commit subject) and *false-negatives* (a +# reabsorbed commit whose subject was rewritten). The #2270 calibration +# replaces that brittle heuristic with a git-history signal: an +# ahead-commit is contamination when its **patch-id matches a commit +# already in ``origin/<base>``** (it is a reabsorbed merged-main commit), +# or — at branch granularity — the branch is neither an ancestor of base +# nor patch-id-equivalent to it. The scan window is capped +# (``_BRANCH_DIVERGENCE_SCAN_CAP``) so a long-lived branch / deep base +# history cannot make the tick unbounded. +# +# Detection latency: the polling thread checks every 30 s, but the +# orchestrator's local ``origin/<pipeline_branch>`` only refreshes +# when it fetches — which happens at pipeline start, phase +# boundaries, and a few resume / signal paths (the polling thread +# itself does not fetch). Contamination introduced mid-phase is +# therefore detected at the next phase boundary's fetch, not within +# 30 s. This is **phase-boundary granularity, not real time** — +# strictly better than detecting at PR open, but defense-in-depth +# only; PR 1 (#2282) remains the primary gate. +BRANCH_DIVERGENCE_THRESHOLD = 20 +# Cap on how many commits we patch-id on each side of the comparison. The +# contamination we care about is recent (a bad rebase during this pipeline), +# so bounding the window keeps the per-tick git work flat regardless of how +# far the branch / base have grown. +_BRANCH_DIVERGENCE_SCAN_CAP = 200 + + +# Phases that pause for human approval before advancing (HITL gates) +_HITL_GATE_PHASES = {"refine", "plan"} + +# Keywords that indicate human approval at HITL gates +_APPROVE_KEYWORDS = {"approved", "approve", "lgtm", "yes", ""} + +# Bare option labels that indicate "request changes" without actionable feedback +_BARE_OPTION_LABELS = {"request changes", "request_changes"} + + +# Minimum characters of non-heading content required for a synthesized plan +# draft to be written. This prevents writing near-empty drafts that contain +# only section headings (e.g. when agents produced no meaningful output). +# A short but valid single-section output like "No architectural risks +# identified." is ~40 chars, so 50 provides a small buffer while still +# catching truly empty drafts. +_MIN_PLAN_DRAFT_CONTENT_LENGTH = 50 + + +# Recovery options offered by the dedicated empty-contract HITL emitted +# from the slice-gate, start_phase=implement safety net, and plan-complete +# paths. Plain "Retry phase" would respawn into the same empty-contract +# state (#2627 incident); these options map each choice to a concrete +# operator action that actually changes state. +_EMPTY_CONTRACT_HITL_OPTIONS = [ + "Repopulate contract from plan draft and retry", + "Restart plan phase", + "Abort pipeline", +] + + +# Per-reason divergence prose used by :func:`_empty_contract_hitl_question` +# when no parsed-slice count is available. The generic fallback wording +# ("draft is missing, unparseable, or yielded no tasks") was written when +# only ``EMPTY_RESULT`` / ``PARSE_FAILED`` / ``DRAFT_MISSING`` / ``NO_DRAFT_PATH`` +# routed through this HITL. The widened +# :func:`_populate_result_is_empty_contract` check now also routes +# ``FOREST_VIOLATION`` / ``CONTRACT_LOAD_FAILED`` / +# ``EGG_CONTRACTS_UNAVAILABLE`` / ``UNEXPECTED_EXCEPTION`` plus the +# orthogonal ``populated_but_empty_slices`` case through here, where +# the operator would otherwise read a contradictory message: the +# prose says "draft missing/unparseable/yielded no tasks" while +# ``reason=forest_violation`` says the draft parsed fine but the slice +# DAG was rejected (#2627 review). Reasons NOT in this dict +# (``empty_result``, ``parse_failed``, ``draft_missing``, +# ``no_draft_path``, ``plan_draft_missing_on_local``, +# ``plan_draft_missing_on_local_and_origin``) fall through to the +# generic line, which describes them accurately. +_DIVERGENCE_LINE_BY_REASON: dict[str, str] = { + "forest_violation": ( + "contract.slices is empty because the plan slice DAG was rejected as not a forest" + ), + "slice_overlap_violation": ( + "contract.slices is empty because the plan slice DAG was rejected: two or more " + "slices touch overlapping files with no dependency ordering between them (#3046)" + ), + "contract_load_failed": ( + "contract.slices is empty because the parsed contract on disk failed to deserialize" + ), + "egg_contracts_unavailable": ( + "contract.slices is empty because the egg-contracts library could " + "not be imported during populate" + ), + "unexpected_exception": ( + "contract.slices is empty because the populator raised an unexpected exception" + ), + "populated_but_empty_slices": ( + "contract.slices is empty because the populator ran but produced 0 slices/tasks" + ), +} + + +# NOTE: _FOREST_REASON_TO_OUTCOME (the ForestValidationError.reason -> +# PopulateOutcome table) moved to _populate.py alongside the PopulateOutcome +# enum it references at definition time (#3312 slice-4); it re-exports through +# the barrel, so _pkg._FOREST_REASON_TO_OUTCOME still resolves. + + +# Recovery options offered by the dedicated plan-preflight HITL emitted +# from the ``start_phase=implement`` safety net (#3100). Plain "Retry +# phase" would respawn into the same metadata-less state; each option +# maps to a concrete operator action that actually changes state. +_PLAN_PREFLIGHT_HITL_OPTIONS = [ + "Fix the plan draft's pr: block and restart implement", + "Restart plan phase", + "Abort pipeline", +] + + +# Decision-ledger backstop options (#3390). Bare labels matched +# case-insensitively on the resolution, mirroring the phase_gate's +# keyword handling. +_LEDGER_BACKSTOP_RERUN_OPTION = "Re-run phase to register decisions" +_LEDGER_BACKSTOP_PROCEED_OPTION = "Proceed without a decision ledger" + +# Explicit-none attestation confirmation option (#3462). Paired with +# ``_LEDGER_BACKSTOP_RERUN_OPTION`` on the confirmation decision; only a +# resolution that IS a confirmation (the bare keyword or the full label) +# proceeds — any other text is treated as a re-run directive, mirroring +# the phase_gate's "bare approve advances, notes request changes" posture. +_LEDGER_ATTESTATION_CONFIRM_OPTION = "Confirm — no open decisions this phase" + + +# --------------------------------------------------------------------------- +# Jira-epic SDLC scheduling helpers (issue #1557 — task-1-4 / task-2-7) +# --------------------------------------------------------------------------- + + +# Tunables for the spurious-PipelineNotFoundError recovery path in +# ``_run_pipeline``. The verify retry covers the empty-file race window +# during a ``git commit`` truncate-and-rewrite on the state worktree +# (typical: <100ms); 3 × 200ms gives ~600ms of total slack. The respawn +# cap bounds how aggressively a persistent transient can leak threads, +# overseer containers, and state-branch commits before we fail the +# pipeline outright. See #2155. +_PNFE_VERIFY_ATTEMPTS = 3 +_PNFE_VERIFY_INTERVAL = 0.2 # seconds between verify retries +_PNFE_RESPAWN_MAX_ATTEMPTS = 5 # cap on respawn cascade +_PNFE_RESPAWN_BACKOFF_CAP = 30 # seconds, exponential backoff ceiling + + +@pipelines_bp.route("/<pipeline_id>/start", methods=["POST"]) +@require_lifecycle_secret +def start_pipeline(pipeline_id: str) -> tuple[Response, int]: + return _start_pipeline_body(pipeline_id) + + +@pipelines_bp.route("/<pipeline_id>/visualization", methods=["GET"]) +def get_pipeline_visualization(pipeline_id: str) -> tuple[Response, int]: + return _get_pipeline_visualization_body(pipeline_id) + + +@pipelines_bp.route("/stream", methods=["GET"]) +def stream_all_pipelines() -> Response: + return _stream_all_pipelines_body() + + +@pipelines_bp.route("/<pipeline_id>/stream", methods=["GET"]) +def stream_pipeline(pipeline_id: str) -> Response: + return _stream_pipeline_body(pipeline_id) + + +# Review-criteria builders live in _criteria.py (#3312 slice-4); re-exported +# here so `from routes.pipelines import X` and patch("routes.pipelines.X") +# keep resolving through the barrel. +# context_pr helpers live in _context_pr.py (#3312 slice-4); re-exported here. +# brc_history helpers live in _brc_history.py (#3312 slice-4); re-exported here. +# alerts helpers live in _alerts.py (#3312 slice-4); re-exported here. +from ._alerts import ( # noqa: E402,F401 + _branch_divergence_tick, + _check_branch_divergence_for_alert, + _check_brc_progress_gate, + _emit_divergence_reconcile_hitl, + _emit_empty_contract_hitl, + _emit_producer_death_alert, + _fail_pipeline_after_divergence_abort, + _handle_brc_consensus_timeout, + _latest_active_role_heartbeat, + _publish_branch_divergence_alert, + _publish_consensus_timeout_alert, + _sync_worktree_reconciling_divergence, + _unresolved_contract_hitl_ids, + detect_branch_divergence, +) +from ._brc_history import ( # noqa: E402,F401 + BRC_HISTORY_TYPES, + CONSENSUS_BRC_TYPES, + _commit_slice_brc_history_to_integration_branch, + _get_message_store, + _persist_phase_brc_history, + _render_brc_history_markdown, + _rewrite_brc_history_for_pr, + _write_brc_history, + _write_brc_history_file, +) +from ._context_pr import ( # noqa: E402,F401 + _build_brc_history_link_line, + _build_pre_merge_obligations_section, + _collect_pre_merge_obligations, + _compose_context_pr_body, + _maybe_open_secondary_context_prs, + _open_context_pr_at_implement_start, + _open_secondary_context_prs, + _persist_context_pr_number, + _refresh_context_pr_body, + _repos_with_slices, +) +from ._criteria import ( # noqa: E402,F401 + _get_agent_design_criteria, + _get_code_review_criteria, + _get_code_review_holistic_criteria, + _get_concurrency_review_criteria, + _get_contract_review_criteria, + _get_first_principles_review_criteria, + _get_plan_review_criteria, + _get_refine_review_criteria, + _get_review_criteria_for_type, + _get_reviewer_scope_preamble, + _get_security_review_criteria, + _human_companion_review_criteria, + _read_shared_criteria, +) +from ._decisions import ( # noqa: E402,F401 + _cancel_consensus_timeout_decisions, + _divergence_reconcile_hitl_question, + _divergence_reconcile_is_abort, + _find_pending_divergence_reconcile_decision, + _format_nack_summary, + _incomplete_consensus_decision_text, + _persist_hitl_decision, + _withdraw_arms_exhausted_decisions, +) + +# drafts helpers live in _drafts.py (#3312 slice-4); re-exported here. +from ._drafts import ( # noqa: E402,F401 + _HUMAN_SPEC_BY_PHASE, + _cleanup_stale_generic_drafts, + _draft_filename, + _get_draft_path, + _get_generic_draft_path, + _get_human_draft_path, + _git_show_draft, + _pull_contract_from_source_branch, + _read_human_phase_draft, + _read_phase_draft, + _read_source_branch_artifacts, + _verdict_path_for_type, +) +from ._drivers import ( # noqa: E402,F401 + _broadcast_orphaned_driver_alert, + _spawn_pipeline_run_thread, + has_live_pipeline_driver, + maybe_revive_orphaned_awaiting_human_driver, + relaunch_driverless_running_pipelines, +) +from ._first_principles import ( # noqa: E402,F401 + _restart_refine_phase, + apply_first_principles_redirect, +) +from ._hitl_rerun import ( # noqa: E402,F401 + _apply_inline_hitl_kickback_to_phase, + _broadcast_hitl_nonconvergence_alert, + _build_iteration_summary_from_tracker, + _build_phase_iteration_context, + _perform_hitl_phase_rerun, +) +from ._ledger import ( # noqa: E402,F401 + _await_unresolved_gap_gate, + _collect_decision_ledger_status, + _drain_wontdo_batch_after_apply, + _find_explicit_none_attestation, + _handle_explicit_none_attestation_gate, + _ledger_attestation_confirmed, + _ledger_attestation_question, + _ledger_attestation_rerun_directive, + _next_phases_for_epic, + _persist_phase_gate_resolution, + _queue_and_await_contract_decisions, + _sync_pipeline_decisions_to_contract, + _unwrap_choice_resolution, + _write_apply_phase_handoff, +) +from ._lifecycle_helpers import ( # noqa: E402,F401 + _assert_repo_set_uniform, + _cleanup_remote_branches, + _clear_pipeline_runtime_state, + _compute_gateway_mode, + _mark_pipeline_records_terminated, + _normalize_submission_repos, +) +from ._overseer import ( # noqa: E402,F401 + _build_overseer_corrective_executor, + _consume_adjudicator_verdict, + _corrective_nudge_agent, + _corrective_open_operator_hitl, + _corrective_respawn_cohort, + _count_phase_agents, + _escalate_finding_to_adjudicator, + _execute_overseer_verdicts, + _overseer_should_be_present, + _run_overseer_detection_plane, + _send_brc_confirmation_nudge, + _spawn_overseer_agent, + _teardown_phase_overseer, +) +from ._pod_liveness import ( # noqa: E402,F401 + _count_live_pods_for_pipeline, + _get_spawner, + _guard_live_pods_or_force, + _live_event_agents, + _slice_agents_alive, +) +from ._populate import ( # noqa: E402,F401 + _FOREST_REASON_TO_OUTCOME, + PlanDraftMissingOnLocalAndOriginError, + PlanDraftMissingOnLocalError, + PopulateOutcome, + PopulateProducedEmptyContractError, + PopulateResult, + SliceGateMonolithicBlock, + _auto_populate_contract_at_implement_start, + _empty_contract_failure_metadata, + _empty_contract_hitl_question, + _empty_contract_hitl_reason, + _enforce_implement_start_plan_preflight, + _forest_error_to_outcome, + _merge_preserved_slice_runtime, + _origin_has_plan_draft, + _plan_preflight_hitl_question, + _populate_contract_from_plan, + _populate_contract_from_plan_safe, + _populate_outcome_to_hitl_reason, + _populate_result_is_empty_contract, + _slice_gate_block_monolithic_demotion, + _synthesize_plan_draft, +) +from ._prompt_agent import ( # noqa: E402,F401 + _build_agent_prompt, + _build_file_boundary_section, +) +from ._prompt_phase import ( # noqa: E402,F401 + _build_brc_preamble, + _build_phase_prompt, + _contract_enforcer_role_names, +) +from ._prompt_review import ( # noqa: E402,F401 + _build_impasse_escape_hatch_section, + _build_review_prompt, + _build_role_context, + _build_role_restrictions_section, + _extract_plan_overview, + _render_contract_tasks, + _summarize_issue, +) +from ._prompt_reviewer import ( # noqa: E402,F401 + _build_agent_roster, + _build_producer_orientation, + _build_reviewer_preparation, + _re_review_priming_block, +) +from ._resolve import ( # noqa: E402,F401 + _brc_history_identifier, + _collect_all_pipelines, + _emit_pipeline_event, + _ensure_pipeline_work_ref, + _pipeline_identifier, + _resolve_pipeline, + _slice_namespace_root, +) + +# reviews helpers live in _reviews.py (#3312 slice-4); re-exported here. +from ._reviews import ( # noqa: E402,F401 + _aggregate_review_verdicts, + _read_review_verdict, + _read_tester_gaps, +) +from ._routes_crud import ( # noqa: E402,F401 + _create_pipeline_body, + _delete_pipeline_body, + _update_pipeline_body, + _update_pipeline_config_body, +) +from ._routes_lifecycle import ( # noqa: E402,F401 + _list_pipeline_local_commits_body, + _salvage_pipeline_local_commits_body, + _start_pipeline_body, +) +from ._routes_read import ( # noqa: E402,F401 + _get_pipeline_body, + _list_pipelines_body, +) +from ._routes_restart import ( # noqa: E402,F401 + _restart_agent_body, + _restart_phase_body, +) +from ._routes_status import ( # noqa: E402,F401 + _get_pipeline_status_body, + _get_pipeline_visualization_body, + _wait_pipeline_status_body, +) +from ._routes_stream import ( # noqa: E402,F401 + _stream_all_pipelines_body, + _stream_pipeline_body, +) +from ._run_concurrent import _run_concurrent_phase # noqa: E402,F401 +from ._run_concurrent_retry import ( # noqa: E402,F401 + _run_concurrent_phase_with_impasse_retry, +) +from ._run_concurrent_support import ( # noqa: E402,F401 + _latest_proposal_ts_impl, + _record_container_exit_impl, + _record_spawned_agents_impl, + _retry_transient_spawn_failures_impl, + _stop_running_containers_impl, + _superseded_by_restart_impl, + _update_agents_complete_impl, +) +from ._run_hitl_gate import ( # noqa: E402,F401 + _run_hitl_gate_converge, +) +from ._run_implement import ( # noqa: E402,F401 + _run_implement_phase_slices, +) +from ._run_implement_support import ( # noqa: E402,F401 + _commit_and_push_slice_statefiles_impl, + _contract_loader_impl, + _persist_slice_status_complete_impl, +) +from ._run_phase import ( # noqa: E402,F401 + _run_phase_execution, +) +from ._run_phase_blocks import ( # noqa: E402,F401 + _run_implement_advance, + _run_pending_phase_init, + _run_plan_advance, +) +from ._run_pipeline import ( # noqa: E402,F401 + _run_pipeline, +) +from ._run_pipeline_setup import ( # noqa: E402,F401 # noqa: E402,F401 # noqa: E402,F401 # noqa: E402,F401 # noqa: E402,F401 + _map_host_repos, + _resolve_worktree_repo, + _start_phase_setup, + _sync_contract_setup, + _sync_source_branch_drafts, +) +from ._run_pipeline_support import ( # noqa: E402,F401 + _health_monitor_poll_impl, + _on_health_escalation_impl, +) +from ._run_support import ( # noqa: E402,F401 + _clear_stale_impasses_for_producers, + _parse_resolution, + _pipeline_superseded_by_restart, + _spawn_and_wait, +) +from ._salvage import ( # noqa: E402,F401 + _filter_salvage_worktrees, + _serialize_commit_report, + _serialize_salvage_result, +) +from ._slice_completion import ( # noqa: E402,F401 + SliceCompletionInvariantError, + _slice_produced_commits, + _validate_slice_completion_basis, +) +from ._slice_state import ( # noqa: E402,F401 + _check_slice_evidence_reachability, + _classify_non_complete_slice, + _cross_repo_hold_marker, + _cross_repo_hold_resolution, + _escalate_blocked_slice_to_hitl, + _escalate_corrupt_slice_to_hitl, + _escalate_layer_c_hitl, + _is_slice_dag_mode, + _lookup_peer_consensus_tracker_or_none, + _register_cross_repo_hold, + _resolve_pipeline_worktree_path, + _resolve_slice_base_branch, + _resolve_slice_gate_repo, + _resolve_slice_worktree_path, + _slice_has_pending_decision, +) +from ._stacked_pr import ( # noqa: E402,F401 + _start_stacked_pr_reconciler, +) + +# statefiles helpers live in _statefiles.py (#3312 slice-4); re-exported here. +from ._statefiles import ( # noqa: E402,F401 + _commit_statefiles_to_worktree, + _detect_default_branch, + _ensure_statefiles_on_branch, + _fetch_pr_state, + _resolve_origin_ref, + persist_contract_statefiles, +) +from ._status_view import ( # noqa: E402,F401 + _build_slice_diff_summary, + _consensus_block, + _get_concurrent_status, + _get_pr_info, +) +from ._status_wait import ( # noqa: E402,F401 + _build_minimal_status_envelope, + _build_status_wait_cursor, + _message_store_tip_id, + _parse_status_wait_cursor, + _track_host_wait_end, + _track_host_wait_start, +) + +# worktree_sync helpers live in _worktree_sync.py (#3312 slice-4); re-exported here. +from ._worktree_sync import ( # noqa: E402,F401 + StalePipelineBranchError, + WorktreeSyncOutcome, + _build_sync_recovery_backup_ref, + _collect_local_only_commits, + _create_sync_recovery_backup_ref, + _read_tree_head, + _rebase_pipeline_branch_onto_base, + _refresh_pipeline_branch_against_current_base, + _restore_missing_state_files_from_head, + _sync_worktree_with_remote, +) diff --git a/orchestrator/routes/pipelines/_alerts.py b/orchestrator/routes/pipelines/_alerts.py new file mode 100644 index 0000000000..2bb70e20ce --- /dev/null +++ b/orchestrator/routes/pipelines/_alerts.py @@ -0,0 +1,1277 @@ +"""alerts helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim; patched/barrel-resident globals reached via _pkg so +patch("routes.pipelines.<name>") keeps intercepting. +""" + +from __future__ import annotations + +import subprocess +import time +from collections.abc import Callable +from datetime import UTC, datetime +from pathlib import Path +from typing import TYPE_CHECKING, Any, Literal + +import routes.pipelines as _pkg # noqa: E402,F401 + +if TYPE_CHECKING: + try: + from ..container_spawner import ContainerSpawner # noqa: F401 + except ImportError: # pragma: no cover + from container_spawner import ContainerSpawner # type: ignore # noqa: F401 + +from events import EventType +from models import Pipeline, PipelinePhase, PipelineStatus +from state_store import StateStore + +from ._worktree_sync import WorktreeSyncOutcome + + +def _emit_divergence_reconcile_hitl( + pipeline_id: str, + store, # noqa: ANN001 — StateStore (avoid import cycle) + *, + phase: PipelinePhase | None, + backup_ref: str | None, + local_only_commit_shas: tuple[str, ...] | list[str], + rebase_category: str | None = None, + rebase_detail: str | None = None, +): + """Pin pipeline+phase to AWAITING_HUMAN and persist the reconcile HITL (#2979). + + Used by the non-blocking ``populate_contract`` route, which cannot + block on the operator the way the in-loop phase-boundary callers do. + Sets the pipeline + phase to ``AWAITING_HUMAN`` (NOT ``FAILED`` — the + divergence is recoverable and nothing was discarded) and persists the + reconcile HITL under the same lock so a reader never observes + ``AWAITING_HUMAN`` without the pending decision, then broadcasts a + ``decision.created`` event. + + Returns the persisted decision (or None on persistence failure). The + operator reconciles the worktree, resolves this decision, and re-runs + ``populate_contract`` against the now-reconciled worktree. + """ + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + if phase is not None: + phase_execution = pipeline.get_phase_execution(phase) + if phase_execution is not None: + phase_execution.status = PipelineStatus.AWAITING_HUMAN + pipeline.status = PipelineStatus.AWAITING_HUMAN + store.save_pipeline(pipeline) + decision = _pkg._persist_hitl_decision( + pipeline_id, + pipeline, + store, + question=_pkg._divergence_reconcile_hitl_question( + pipeline_id=pipeline_id, + phase=phase, + backup_ref=backup_ref, + local_only_commit_shas=tuple(local_only_commit_shas), + rebase_category=rebase_category, + rebase_detail=rebase_detail, + ), + options=list(_pkg._DIVERGENCE_RECONCILE_HITL_OPTIONS), + phase=phase, + context=_pkg._DIVERGENCE_RECONCILE_HITL_CONTEXT, + ) + _pkg.report_pipeline_status( + pipeline, + event_type="decision.created", + message=( + f"Awaiting manual worktree reconcile for " + f"{phase.value if phase else 'current phase'} phase" + ), + ) + _pkg._emit_pipeline_event(pipeline, "decision.created") + return decision + + +def _fail_pipeline_after_divergence_abort( + pipeline_id: str, + store, # noqa: ANN001 — StateStore (avoid import cycle) + *, + phase: PipelinePhase | None, + backup_ref: str | None, + local_only_commit_shas: tuple[str, ...] | list[str], + budget_exhausted: bool = False, + pre_event_hook: Callable[[], None] | None = None, +) -> None: + """Pin pipeline+phase to FAILED after an aborted divergence reconcile (#2979). + + Reached when the operator resolved the reconcile HITL with + ``Abort pipeline`` (or the reconcile pause budget was exhausted). No + HITL is emitted here — the reconcile decision was already surfaced and + resolved. Mirrors the FAILED-write + ``pipeline.failed`` broadcast of + the old destructive-recovery helper, minus the discard: the committed + work is still on HEAD and pinned under ``backup_ref`` for offline + recovery. + + ``pre_event_hook`` runs after the FAILED-write but before the public + ``pipeline.failed`` broadcast (the post-phase site uses it to tear down + the per-phase overseer container). + """ + phase_label = phase.value if phase is not None else "current phase" + reason = ( + "the reconcile pause budget was exhausted" + if budget_exhausted + else "the operator chose to abort" + ) + error_message = ( + f"Worktree diverged from origin at {phase_label} and could not be " + f"auto-reconciled; {reason} (#2979). Local-only commits are " + f"preserved under {backup_ref or '(backup ref write failed)'} " + f"({len(local_only_commit_shas)} commit(s))." + ) + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + if phase is not None: + phase_execution = pipeline.get_phase_execution(phase) + if phase_execution is not None: + phase_execution.status = PipelineStatus.FAILED + phase_execution.error = error_message + phase_execution.completed_at = datetime.now(UTC) + pipeline.status = PipelineStatus.FAILED + pipeline.error = error_message + store.save_pipeline(pipeline) + if pre_event_hook is not None: + pre_event_hook() + _pkg.report_pipeline_status( + pipeline, + event_type="pipeline.failed", + message=f"Pipeline failed: {error_message[:100]}", + ) + _pkg._emit_pipeline_event(pipeline, "pipeline.failed") + + +def _sync_worktree_reconciling_divergence( + spawner: "ContainerSpawner", # noqa: UP037 + pipeline_id: str, + store, # noqa: ANN001 — StateStore (avoid import cycle) + repo_path: Path, + *, + worktree_repo_path: Path, + phase: PipelinePhase | None, + gateway_mode: Literal["public", "private"] = "public", + base_branch: str | None = None, + pipeline_branch: str | None = None, + prior_phase_succeeded: bool = True, + max_reconcile_pauses: int = _pkg._MAX_DIVERGENCE_RECONCILE_PAUSES, +) -> tuple[WorktreeSyncOutcome, bool]: + """Sync the worktree, pausing for a manual reconcile on divergence (#2979). + + Runs :func:`_sync_worktree_with_remote`. When the helper reports an + unreconciled divergence (``diverged_unreconciled``), the worktree is + left non-destructively at the local HEAD; this function pauses the + pipeline (``AWAITING_HUMAN``) on a reconcile HITL and **blocks** the + ``_run_pipeline`` thread on ``wait_for_decision`` — the same proven + pause primitive the phase-approval gate uses. When the operator + resolves the HITL with "Reconciled — resume", the pipeline returns to + ``RUNNING`` and the sync re-runs; the caller then continues the same + phase's post-processing from where it paused, with no full re-run and + nothing discarded. + + Returns ``(outcome, aborted)``. ``aborted`` is True when the operator + chose "Abort pipeline" or the reconcile-pause budget was exhausted; the + caller should fail the pipeline via + :func:`_fail_pipeline_after_divergence_abort`. When ``aborted`` is + False the worktree is reconciled (or never diverged) and the caller + proceeds normally. + + Only call this from inside the ``_run_pipeline`` loop thread, which is + allowed to block; route handlers that cannot block use + :func:`_emit_divergence_reconcile_hitl` instead. + """ + dq = _pkg.get_decision_queue(pipeline_id, repo_path) + phase_label = phase.value if phase is not None else "current phase" + + outcome = _pkg._sync_worktree_with_remote( + spawner, + pipeline_id, + worktree_repo_path, + prior_phase_succeeded=prior_phase_succeeded, + gateway_mode=gateway_mode, + base_branch=base_branch, + pipeline_branch=pipeline_branch, + ) + + pauses = 0 + while outcome.diverged_unreconciled: + if pauses >= max_reconcile_pauses: + _pkg.logger.error( + "OVERSEER_ALERT worktree_divergence_reconcile_budget_exhausted", + pipeline_id=pipeline_id, + phase=phase_label, + pauses=pauses, + backup_ref=outcome.backup_ref, + ) + return outcome, True + pauses += 1 + + # Persist the reconcile HITL and flip to AWAITING_HUMAN under the + # (reentrant) state lock so a reader never sees AWAITING_HUMAN + # without the pending decision. + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + pipeline.status = PipelineStatus.AWAITING_HUMAN + if phase is not None: + phase_execution = pipeline.get_phase_execution(phase) + if phase_execution is not None: + phase_execution.status = PipelineStatus.AWAITING_HUMAN + store.save_pipeline(pipeline) + decision = _pkg._persist_hitl_decision( + pipeline_id, + pipeline, + store, + question=_pkg._divergence_reconcile_hitl_question( + pipeline_id=pipeline_id, + phase=phase, + backup_ref=outcome.backup_ref, + local_only_commit_shas=outcome.local_only_commit_shas, + rebase_category=outcome.rebase_category, + rebase_detail=outcome.rebase_detail, + ), + options=list(_pkg._DIVERGENCE_RECONCILE_HITL_OPTIONS), + phase=phase, + context=_pkg._DIVERGENCE_RECONCILE_HITL_CONTEXT, + ) + if decision is None: + # Could not persist the HITL — fail closed rather than spin on + # a pause the operator can never see. + _pkg.logger.error( + "worktree_divergence_reconcile_hitl_persist_failed", + pipeline_id=pipeline_id, + phase=phase_label, + ) + return outcome, True + + _pkg.logger.error( + "OVERSEER_ALERT worktree_divergence_reconcile_pause", + pipeline_id=pipeline_id, + phase=phase_label, + backup_ref=outcome.backup_ref, + local_only_commit_count=len(outcome.local_only_commit_shas), + rebase_category=outcome.rebase_category, + rebase_detail=outcome.rebase_detail, + pause_attempt=pauses, + ) + + # Once AWAITING_HUMAN is persisted, an unexpected exception + # between here and the resume-write (e.g. a broadcast IO error, + # a decision-queue runtime error, a transient store failure on + # ``get_decision``) would leave the pipeline pinned to + # AWAITING_HUMAN on disk while ``_run_pipeline``'s outer + # ``try/except`` catches the error and moves on — stranding the + # operator with no waiter ever returning. Guard the + # wait-and-resolve span: on unexpected error revert to RUNNING + # before re-raising so the caller still observes the failure + # but the pipeline is not left in an unrecoverable paused state. + # The abort path (operator chose ``Abort pipeline``) returns + # normally with ``aborted=True`` so the caller can flip to + # FAILED — that's not an exception and skips the revert. + try: + _pkg.report_pipeline_status( + pipeline, + event_type="decision.created", + message=f"Awaiting manual worktree reconcile for {phase_label} phase", + ) + _pkg._emit_pipeline_event(pipeline, "decision.created") + + dq.wait_for_decision(decision.id) + + resolved = dq.get_decision(decision.id) + resolution = (resolved.resolution or "") if resolved is not None else "" + if _pkg._divergence_reconcile_is_abort(resolution): + _pkg.logger.warning( + "worktree_divergence_reconcile_aborted_by_operator", + pipeline_id=pipeline_id, + phase=phase_label, + ) + return outcome, True + + # Operator reconciled the worktree — return to RUNNING and re-run + # the sync. If it still diverges, loop and re-pause (bounded). + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + pipeline.status = PipelineStatus.RUNNING + if phase is not None: + phase_execution = pipeline.get_phase_execution(phase) + if phase_execution is not None: + phase_execution.status = PipelineStatus.RUNNING + store.save_pipeline(pipeline) + except Exception: + # Best-effort revert: load fresh, flip AWAITING_HUMAN→RUNNING + # only if still pinned, then re-raise. Swallow secondary + # errors from the revert itself — losing the revert is bad, + # but masking the original failure with a save error is + # worse. The operator can still recover via the pending + # decision (the decision queue may have replayed it on + # restart) or via the backup ref. + try: + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + if pipeline.status == PipelineStatus.AWAITING_HUMAN: + pipeline.status = PipelineStatus.RUNNING + if phase is not None: + phase_execution = pipeline.get_phase_execution(phase) + if ( + phase_execution is not None + and phase_execution.status == PipelineStatus.AWAITING_HUMAN + ): + phase_execution.status = PipelineStatus.RUNNING + store.save_pipeline(pipeline) + except Exception: + _pkg.logger.warning( + "worktree_divergence_reconcile_revert_failed", + pipeline_id=pipeline_id, + phase=phase_label, + exc_info=True, + ) + raise + _pkg.logger.info( + "worktree_divergence_reconcile_resume", + pipeline_id=pipeline_id, + phase=phase_label, + pause_attempt=pauses, + ) + outcome = _pkg._sync_worktree_with_remote( + spawner, + pipeline_id, + worktree_repo_path, + prior_phase_succeeded=prior_phase_succeeded, + gateway_mode=gateway_mode, + base_branch=base_branch, + pipeline_branch=pipeline_branch, + ) + + return outcome, False + + +def _emit_empty_contract_hitl( + pipeline_id: str, + pipeline: Pipeline, + store: StateStore, + *, + reason: str, + draft_slice_count: int | None, + gate: Literal[ + "slice_gate", + "start_phase_implement_safety_net", + "plan_complete", + ], + phase: PipelinePhase | None = None, +): + """Persist a dedicated HITL naming the empty-contract divergence (#2627). + + Built on top of :func:`_persist_hitl_decision` so it inherits the + "load → mutate → save under lock" persistence semantics that make + the decision survive the FAILED-write the calling block does next. + Best-effort: a persistence failure logs and returns None so the + surrounding FAILED-cleanup is not blocked. + + Returns the persisted decision (or None on persistence failure). + + Plain "Retry phase" against this HITL would respawn the implement + phase into the same empty-contract state, so the option set is + distinct from the generic phase-failure decision: callers are + expected to wire each option to its concrete recovery action + (see :data:`_EMPTY_CONTRACT_HITL_OPTIONS` for the mapping). + """ + # ``_empty_contract_hitl_question`` is defined further down the + # module alongside the other #2627 follow-up helpers; importing + # the symbol here keeps the call-site test isolated from module + # top-level ordering. + return _pkg._persist_hitl_decision( + pipeline_id, + pipeline, + store, + question=_pkg._empty_contract_hitl_question( + pipeline_id=pipeline_id, + reason=reason, + draft_slice_count=draft_slice_count, + gate=gate, + ), + options=list(_pkg._EMPTY_CONTRACT_HITL_OPTIONS), + phase=phase, + ) + + +def _check_brc_progress_gate( + pipeline_id: str, + slice_id: str | None, + active_role_names: list[str], + gate_seconds: float, +) -> tuple[bool, str | None]: + """Return (defer, reason) for the BRC consensus-timeout progress gate (#2243). + + Defers the consensus-timeout ``OVERSEER_ALERT`` (#2264; previously + an auto-``choice`` HITL decision) when *any* of the following has + fired within ``gate_seconds``: + + * The BRC tracker's most recent ``CONSENSUS_PROPOSE`` (producer + proposal) timestamp. + * The most recent ACK/NACK timestamp on the approval matrix. + * The most recent container heartbeat for any role in + ``active_role_names`` (filters out cross-phase pollution in the + shared :class:`HealthMonitor` singleton). + + The gate is the operator-friendly half of the issue-2243 fix: at + :data:`consensus_timeout_minutes` we previously opened a `choice` + decision unconditionally, even when producers were minutes from + their first commit. With the gate, the polling loop keeps polling + while signals are alive; the alert is only published once the bus + and containers have both gone quiet for ``gate_seconds``. + + ``gate_seconds <= 0`` disables the gate (returns ``(False, None)``). + Failures in any signal source are logged at WARNING and treated as + "no signal from that source" — never as a gate defer, since a + crashed signal collector must not silently keep us off the alert + surface. + + Heartbeat-cadence contract: the coder-mid-merge-conflict path + (no ``CONSENSUS_PROPOSE`` yet, only container heartbeats — the + original incident's ``decision-17`` flavour, pre-#2264) relies on + container heartbeats firing at least every ``gate_seconds``. Sandbox + heartbeats (see ``shared/egg_agent`` heartbeat scheduler and + ``orchestrator/health_monitor.py``) cadence today is well under + 300s, but a long uninterruptible subprocess (e.g. ``git rebase`` + blocked on a merge driver) could starve them; once that happens + the gate falls open and the pre-fix behaviour returns. Tracked as + a follow-up under #2243. + + TODO(#2243 step 2): same-role cross-phase pollution. The role-name + filter handles different-role ghosts (refiner heartbeat lingering + during a coder phase) but not same-role ghosts: ``coder`` reappears + across implement / implement-fix / fix-on-PR phases and + ``HealthMonitor._last_heartbeat['coder']`` is only popped on + ``clear_agent_state``. A phase boundary clear (or stamping the + heartbeat key with the phase) would close it; per-phase timeouts + in step 2 of the issue plan will likely subsume it. + """ + if gate_seconds <= 0: + return False, None + + # Two clocks, deliberately. ``now_dt`` is used for tracker + # timestamps (datetime in UTC). ``now_wallclock`` is the float + # epoch ``time.time()`` returns, matching the wall-clock values + # ``HealthMonitor._last_heartbeat`` is populated with. Despite the + # earlier name ``now_mono``, these are NOT monotonic — an NTP step + # on the orchestrator host can make ``(now - latest_hb)`` negative + # or skip the gate window. Acceptable today; revisit alongside the + # per-phase-timeout follow-up. + now_dt = datetime.now(UTC) + now_wallclock = time.time() + + # 1. BRC bus signals (proposal + ACK/NACK timestamps). + try: + try: + from peer_consensus import get_peer_consensus_tracker + except ImportError: + from ..peer_consensus import ( + get_peer_consensus_tracker, # type: ignore[no-redef] + ) + tracker = get_peer_consensus_tracker(pipeline_id, slice_id) + if tracker is not None: + ts = tracker.get_latest_progress_timestamp() + if ts is not None and (now_dt - ts).total_seconds() < gate_seconds: + age = (now_dt - ts).total_seconds() + return True, f"BRC bus active {age:.0f}s ago" + except Exception as e: + _pkg.logger.warning( + "BRC progress-gate tracker check failed", + pipeline_id=pipeline_id, + error=str(e), + ) + + # 2. Container heartbeats. Filter by active roles so a stale + # heartbeat from a prior phase in the singleton HealthMonitor + # doesn't keep us out of the HITL surface forever. An empty + # ``active_role_names`` means the caller has no live containers + # to gate on, so match nothing rather than every stale heartbeat. + if not active_role_names: + return False, None + try: + from health_monitor import get_health_monitor + + hm = get_health_monitor() + if hm is not None: + active_set = set(active_role_names) + latest_hb: float | None = None + with hm._lock: # noqa: SLF001 — read-only snapshot + hb_snapshot = dict(hm._last_heartbeat) # noqa: SLF001 + for agent_id, hb_time in hb_snapshot.items(): + if agent_id not in active_set: + continue + if latest_hb is None or hb_time > latest_hb: + latest_hb = hb_time + if latest_hb is not None and (now_wallclock - latest_hb) < gate_seconds: + age = now_wallclock - latest_hb + return True, f"container heartbeat {age:.0f}s ago" + except Exception as e: + _pkg.logger.warning( + "BRC progress-gate heartbeat check failed", + pipeline_id=pipeline_id, + error=str(e), + ) + + return False, None + + +def _latest_active_role_heartbeat(active_role_names: list[str]) -> datetime | None: + """Return the most recent heartbeat timestamp across ``active_role_names``. + + Mirrors the heartbeat half of :func:`_check_brc_progress_gate` so the + consensus-timeout ``OVERSEER_ALERT`` carries a meaningful + ``latest_heartbeat_at`` value. Filters by active role to avoid + pollution from stale entries in the singleton ``HealthMonitor``. + + Returns ``None`` when no live heartbeat is available (no roles + given, no health monitor, or any failure in the lookup — failures + are logged at WARNING and treated as "no signal", consistent with + the gate). + """ + if not active_role_names: + return None + try: + from health_monitor import get_health_monitor + + hm = get_health_monitor() + if hm is None: + return None + active_set = set(active_role_names) + latest_hb: float | None = None + with hm._lock: # noqa: SLF001 — read-only snapshot + hb_snapshot = dict(hm._last_heartbeat) # noqa: SLF001 + for agent_id, hb_time in hb_snapshot.items(): + if agent_id not in active_set: + continue + if latest_hb is None or hb_time > latest_hb: + latest_hb = hb_time + if latest_hb is None: + return None + return datetime.fromtimestamp(latest_hb, tz=UTC) + except Exception as e: + _pkg.logger.warning( + "Consensus-timeout alert heartbeat lookup failed", + error=str(e), + exc_info=True, + ) + return None + + +def _unresolved_contract_hitl_ids( + pipeline_id: str, + pipeline: Pipeline, + phase_str: str, +) -> list[str]: + """Return ids of unresolved contract HITL (``cq-N``) decisions gating ``phase_str``. + + Feeds the consensus-timeout HITL gate (#3426): while an agent-registered + contract question (``register_open_question`` / impasse escalation) for + the running phase awaits an operator answer, the slice is *operator-gated* + — a reviewer correctly withholding its ACK pending the ruling is not a + convergence failure, so the consensus-timeout clock must not expire the + phase. Scoped to decisions whose ``phase`` matches the running phase; + phase-less decisions are skipped, mirroring + ``_collect_unresolved_phase_decisions`` (we cannot prove they gate this + phase, and an eternally-unanswered legacy entry must not suspend the + timeout forever). + + Contract decisions have no slice tag, so during a sliced implement phase + any unresolved implement-tagged question suspends every slice's timeout. + That errs toward parking rather than failing — acceptable, since the + overseer's "wedged on HITL" alert stays sticky and the operator's answer + releases the gate. + + The gate keys on the *existence* of an operator-facing HITL decision + tagged to this phase, not on causal proof that decision is what a + reviewer is withholding an ACK for — decisions carry no link to the + ACK they block. An unrelated implement-tagged HITL therefore suspends + the timeout too; that is the conservative "park rather than fail" + direction, self-corrected by the clock reset on release (a genuine + stall times out on the fresh window) and by the overseer's other + health checks. + + Fail-open: any failure (missing worktree, unloadable contract) returns + ``[]`` so a broken scan degrades to the pre-#3426 timeout behaviour + rather than suspending the clock indefinitely. Matching the sibling + ``_collect_unresolved_phase_decisions``, the except set is narrowed to + the IO/validation failures a real scan can hit and logged at + ``warning`` (so a broken scan is observable, not a silent no-op), + while programming errors (``AttributeError``/``TypeError``/``NameError``) + are left to propagate so they surface during development. + """ + try: + import contract_store + from egg_contracts import load_contract + from egg_contracts.loader import ( + ContractNotFoundError, + ContractValidationError, + ) + except ImportError: + _pkg.logger.warning( + "Consensus-timeout HITL gate: egg_contracts unavailable, cannot scan", + pipeline_id=pipeline_id, + exc_info=True, + ) + return [] + + try: + worktree = contract_store.resolve_pipeline_worktree(pipeline_id) + if worktree is None: + return [] + identifier = _pkg._pipeline_identifier(getattr(pipeline, "issue_number", None), pipeline_id) + contract = load_contract(identifier, worktree) + except OSError, ValueError, ContractNotFoundError, ContractValidationError: + # OSError: filesystem failures resolving the worktree / reading the + # contract. ValueError: identifier / path-resolution failures from + # ``_pipeline_identifier`` (``load_contract`` wraps pydantic-V2 + # validation errors as ContractValidationError, so a raw ValueError + # here does not come from schema validation). Contract*: missing or + # corrupt contract JSON. All fail open to ``[]``. + _pkg.logger.warning( + "Consensus-timeout HITL gate contract scan failed", + pipeline_id=pipeline_id, + exc_info=True, + ) + return [] + + ids: list[str] = [] + for d in contract.decisions or []: + if d.resolved: + continue + if getattr(d.type, "value", d.type) != "hitl": + continue + if getattr(d.phase, "value", d.phase) != phase_str: + continue + ids.append(d.id) + return ids + + +def _publish_consensus_timeout_alert( + pipeline: Pipeline, + pipeline_id: str, + consensus_timeout: float, + blocking_agents: list[str], + *, + priority: str, + latest_proposal_at: datetime | None, + latest_heartbeat_at: datetime | None, + slice_id: str | None, +) -> None: + """Publish a consensus-timeout ``OVERSEER_ALERT`` (#2264). + + Replaces the old auto-``choice`` HITL decision the orchestrator + used to open at ``consensus_timeout_minutes``. The SDLC skill's + existing ``OVERSEER_ALERT`` flow surfaces this as a non-blocking + notification (Check agent logs / Acknowledge / Cancel pipeline) + rather than gating the pipeline on a binary choice. + + Best-effort: if the message store import or write fails, log at + WARNING and return — the orchestrator log is the always-on + fallback (mirrors the slice-cascade alert path). + """ + timeout_minutes = int(consensus_timeout / 60) + phase_value = ( + pipeline.current_phase.value + if hasattr(pipeline.current_phase, "value") + else str(pipeline.current_phase) + ) + # Subject role slot follows the SDLC skill convention + # ``<anomaly_type>: <agent_role> [<priority>]`` (skills/sdlc/SKILL.md + # §"Overseer Alert Detection") so "Check agent logs" extracts a role + # the host can pass to ``get_container_logs``. Fall back to the phase + # only when no blocking role is reported — the phase still appears in + # ``metadata.phase`` regardless. + subject_role = blocking_agents[0] if blocking_agents else phase_value + subject = f"consensus-timeout: {subject_role} [{priority}]" + blockers_render = ", ".join(blocking_agents) if blocking_agents else "(none reported)" + proposal_render = ( + latest_proposal_at.isoformat() if latest_proposal_at is not None else "no proposals seen" + ) + heartbeat_render = ( + latest_heartbeat_at.isoformat() + if latest_heartbeat_at is not None + else "no recent heartbeat" + ) + body = ( + f"BRC consensus has not converged after {timeout_minutes} minutes " + f"in phase '{phase_value}'.\n" + f"Blocking agents: {blockers_render}\n" + f"Latest proposal: {proposal_render}\n" + f"Latest heartbeat (active roles): {heartbeat_render}\n\n" + "The pipeline continues to poll for convergence (up to ~60 min " + "before still-running containers are force-killed). If you want " + "to intervene, use `cancel_task` to stop the pipeline or " + "`restart_phase` to retry." + ) + metadata: dict[str, Any] = { + "anomaly_type": "consensus-timeout", + "phase": phase_value, + "blocking_agents": list(blocking_agents), + "latest_proposal_at": ( + latest_proposal_at.isoformat() if latest_proposal_at is not None else None + ), + "latest_heartbeat_at": ( + latest_heartbeat_at.isoformat() if latest_heartbeat_at is not None else None + ), + "consensus_timeout_minutes": timeout_minutes, + "priority": priority, + } + if slice_id is not None: + metadata["slice_id"] = slice_id + + try: + try: + from message_store import Message, MessageType + except ImportError: + from ..message_store import ( # type: ignore[no-redef] + Message, + MessageType, + ) + store_fn = _pkg._get_message_store() + if store_fn is None: + _pkg.logger.warning( + "Consensus-timeout alert: message store unavailable", + pipeline_id=pipeline_id, + ) + return + msg_store = store_fn() + msg_store.add_message( + Message( + pipeline_id=pipeline_id, + from_role="orchestrator", + to_role="all", + message_type=MessageType.OVERSEER_ALERT, + subject=subject, + body=body, + metadata=metadata, + phase=phase_value, + ) + ) + except Exception as e: + _pkg.logger.warning( + "Failed to publish consensus-timeout OVERSEER_ALERT", + pipeline_id=pipeline_id, + error=str(e), + exc_info=True, + ) + + +def _emit_producer_death_alert( + *, + pipeline_id: str, + role: str, + phase: str, + slice_id: str | None, + exit_code: int, +) -> None: + """Publish a high-priority ``OVERSEER_ALERT`` for permanent producer death (#2806). + + Fires from ``_run_concurrent_phase`` when a producer's + consensus-wrapper container exits with a non-clean code after + exhausting its retry budget. The pipeline (or slice) is about to + transition to FAILED — the alert is what makes the operator notice + rather than waiting for the consensus-timeout / overseer + ``stuck-phase-transition`` alert to fire 30+ minutes later. + + Best-effort: failures to write to the message store degrade to a + WARNING log, mirroring ``_publish_consensus_timeout_alert``. + """ + phase_value = phase if isinstance(phase, str) else getattr(phase, "value", str(phase)) + # ``is not None`` (not truthy) so subject and metadata agree on edge + # values like ``slice_id == ""``: metadata at 15349 also uses ``is + # not None`` (#2811 round 3 item 1). In practice ``slice_id`` is + # validated to ``slice-<N>`` upstream, so the asymmetry can't fire + # today — keeping the two checks aligned avoids a future footgun. + subject_slice = f" slice={slice_id}" if slice_id is not None else "" + subject = f"producer-permanent-death: {role} exit={exit_code}{subject_slice} [high]" + slice_render = f" (slice {slice_id})" if slice_id is not None else "" + body = ( + f"Producer '{role}'{slice_render} died permanently in phase " + f"'{phase_value}': container exited with code {exit_code} after the " + f"consensus-wrapper exhausted its retry budget.\n\n" + "The slice/pipeline state machine cannot replace a permanently " + "dead producer, so the pipeline is being transitioned to FAILED " + "(Option A, issue #2806). The agent's committed work — if any — " + "is still on the per-role branch; use `restart_phase` to resume " + "from the prior known-good state, or `cancel_task` to abort." + ) + metadata: dict[str, Any] = { + "anomaly_type": "producer-permanent-death", + "phase": phase_value, + "role": role, + "exit_code": exit_code, + "priority": "high", + } + if slice_id is not None: + metadata["slice_id"] = slice_id + + try: + try: + from message_store import Message, MessageType + except ImportError: + from ..message_store import ( # type: ignore[no-redef] + Message, + MessageType, + ) + store_fn = _pkg._get_message_store() + if store_fn is None: + _pkg.logger.warning( + "Producer-death alert: message store unavailable", + pipeline_id=pipeline_id, + role=role, + ) + return + msg_store = store_fn() + msg_store.add_message( + Message( + pipeline_id=pipeline_id, + from_role="orchestrator", + to_role="all", + message_type=MessageType.OVERSEER_ALERT, + subject=subject, + body=body, + metadata=metadata, + phase=phase_value, + ) + ) + except Exception as e: + _pkg.logger.warning( + "Failed to publish producer-permanent-death OVERSEER_ALERT", + pipeline_id=pipeline_id, + role=role, + error=str(e), + exc_info=True, + ) + + +def detect_branch_divergence(snapshot: Any) -> Any | None: + """Calibration detector for the ``branch_divergence`` corpus rows (#2222/#2224). + + Keys on the git-history signal in ``snapshot.git_state`` rather than the + brittle PR-subject regex: the branch is genuinely diverged only when it is + **neither** an ancestor of base **nor** patch-id-equivalent to the merged + commit. A branch that is an ancestor of base, or whose patch-id matches the + merged commit, is NOT diverged — even if its PR-style subject would have + tripped the old regex. Deterministic and cheap → ``requires_adjudication= + False``. + """ + from health_checks.types import Finding, FindingClass, Severity + + git_state = getattr(snapshot, "git_state", {}) or {} + if not isinstance(git_state, dict): + return None + + is_ancestor = bool(git_state.get("is_ancestor_of_base")) + patch_id_matches = bool(git_state.get("patch_id_matches")) + # An ancestor-of-base branch (or a patch-id match against the merged commit) + # is fully accounted for in main — not divergence. + if is_ancestor or patch_id_matches: + return None + + return Finding( + finding_class=FindingClass.BRANCH_DIVERGENCE, + severity=Severity.MEDIUM, + evidence={ + "branch": git_state.get("branch"), + "is_ancestor_of_base": is_ancestor, + "patch_id_matches": patch_id_matches, + "pr_subject_divergence": bool(git_state.get("pr_subject_divergence")), + }, + recommended_action=( + "Pipeline branch is neither an ancestor of base nor patch-id-" + "equivalent to the merged commit — it has genuinely diverged " + "(see #2222 recovery: rebase --onto the correct base)." + ), + requires_adjudication=False, + detector_key="branch_divergence", + ) + + +def _check_branch_divergence_for_alert( + pipeline_id: str, + worktree_repo_path: Path, + pipeline_branch: str, + base_branch: str, + threshold: int = _pkg.BRANCH_DIVERGENCE_THRESHOLD, + scan_cap: int = _pkg._BRANCH_DIVERGENCE_SCAN_CAP, +) -> tuple[int, list[tuple[str, str]]]: + """Return ``(ahead_count, offenders)``. + + ``offenders`` is the list of ahead-commits that are **reabsorbed merged-main + commits** — an ahead-commit whose patch-id matches a commit already present + in ``origin/<base>`` (within the capped scan window) — when the pipeline + branch is more than ``threshold`` commits ahead of base. This replaces the + old ``(#NNNN)`` subject regex with a patch-id signal that neither + false-positives on legitimate PR references nor false-negatives on rewritten + subjects. Returns ``(ahead, [])`` when the branch is not far enough ahead, + nothing reabsorbed matches, or any git invocation fails (best-effort — + observability must never block the pipeline). + """ + if not pipeline_branch or not base_branch or pipeline_branch == base_branch: + return 0, [] + + git_base = [ + "git", + "-c", + "core.hooksPath=/dev/null", + "-c", + f"safe.directory={worktree_repo_path}", + "-C", + str(worktree_repo_path), + ] + + def _run(args: list[str]) -> subprocess.CompletedProcess[str] | None: + try: + return subprocess.run( + [*git_base, *args], + capture_output=True, + text=True, + timeout=15, + check=False, + ) + except (subprocess.TimeoutExpired, OSError) as exc: + _pkg.logger.debug( + "branch-divergence: git command failed", + pipeline_id=pipeline_id, + git_args=args, + error=str(exc), + ) + return None + + def _patch_id_to_sha(rev_range: str) -> dict[str, str]: + """Map ``patch_id -> sha`` for up to ``scan_cap`` commits in ``rev_range``. + + Runs ``git log -p | git patch-id --stable``. Best-effort: any failure + yields an empty map (the caller degrades to "no offenders"). + """ + log_p = _run( + [ + "log", + "-p", + "--no-merges", + f"--max-count={scan_cap}", + rev_range, + ] + ) + if log_p is None or log_p.returncode != 0 or not log_p.stdout: + return {} + try: + pid = subprocess.run( + [*git_base, "patch-id", "--stable"], + input=log_p.stdout, + capture_output=True, + text=True, + timeout=15, + check=False, + ) + except subprocess.TimeoutExpired, OSError: + return {} + if pid.returncode != 0: + return {} + mapping: dict[str, str] = {} + for line in (pid.stdout or "").splitlines(): + parts = line.split() + if len(parts) >= 2: + mapping[parts[0]] = parts[1] + return mapping + + count = _run( + [ + "rev-list", + "--count", + f"origin/{base_branch}..origin/{pipeline_branch}", + ] + ) + if count is None or count.returncode != 0: + return 0, [] + try: + ahead = int((count.stdout or "0").strip() or "0") + except ValueError: + return 0, [] + if ahead <= threshold: + return ahead, [] + + # Patch-ids present in recent base history — the set an ahead-commit must + # collide with to count as a reabsorbed merged-main commit. + base_patch_ids = set(_patch_id_to_sha(f"origin/{base_branch}").keys()) + if not base_patch_ids: + return ahead, [] + ahead_sha_by_patch_id = _patch_id_to_sha(f"origin/{base_branch}..origin/{pipeline_branch}") + contaminated_shas = {sha for pid, sha in ahead_sha_by_patch_id.items() if pid in base_patch_ids} + if not contaminated_shas: + return ahead, [] + + # Re-read subjects (capped, ordered newest-first) for the alert body. + log = _run( + [ + "log", + "--no-merges", + "--pretty=format:%H%x09%s", + f"--max-count={scan_cap}", + f"origin/{base_branch}..origin/{pipeline_branch}", + ] + ) + if log is None or log.returncode != 0: + return ahead, [] + + offenders: list[tuple[str, str]] = [] + for line in (log.stdout or "").splitlines(): + line = line.strip() + if not line: + continue + sha, _, subject = line.partition("\t") + if not sha: + continue + if sha in contaminated_shas: + offenders.append((sha, subject or "(no subject)")) + return ahead, offenders + + +def _publish_branch_divergence_alert( + pipeline: Pipeline, + pipeline_id: str, + *, + pipeline_branch: str, + base_branch: str, + ahead_count: int, + offenders: list[tuple[str, str]], +) -> None: + """Publish an ``OVERSEER_ALERT`` for branch-divergence contamination. + + Best-effort: import or write failures are logged at WARNING and + swallowed — the orchestrator log is the always-on fallback. + """ + phase_value = ( + pipeline.current_phase.value + if hasattr(pipeline.current_phase, "value") + else str(pipeline.current_phase) + ) + subject = f"branch-divergence: {pipeline_branch} contains merged-main commits" + offender_render = "\n".join(f" {sha[:12]} {subj}" for sha, subj in offenders[:10]) + if len(offenders) > 10: + offender_render += f"\n ... and {len(offenders) - 10} more" + body = ( + f"Pipeline branch ``origin/{pipeline_branch}`` is {ahead_count} commits " + f"ahead of ``origin/{base_branch}`` and contains {len(offenders)} " + f"commit(s) whose **patch-id matches a commit already merged into " + f"base** — i.e. reabsorbed merged-main commits. This is the " + f"contamination shape investigated in #2222 (Phase 4 / #2224 " + f"detector; #2270 §2 patch-id calibration).\n\n" + f"Offending commits:\n{offender_render}\n\n" + f"If this is real contamination, the resulting PR will show a " + f"borked diff against current main — see #2222 recovery procedure " + f"(rebase ``--onto`` the right base)." + ) + metadata: dict[str, Any] = { + "anomaly_type": "branch-divergence", + "phase": phase_value, + "pipeline_branch": pipeline_branch, + "base_branch": base_branch, + "ahead_count": ahead_count, + "offending_shas": [sha for sha, _ in offenders], + } + + try: + try: + from message_store import Message, MessageType + except ImportError: + from ..message_store import ( # type: ignore[no-redef] + Message, + MessageType, + ) + store_fn = _pkg._get_message_store() + if store_fn is None: + _pkg.logger.warning( + "Branch-divergence alert: message store unavailable", + pipeline_id=pipeline_id, + ) + return + msg_store = store_fn() + msg_store.add_message( + Message( + pipeline_id=pipeline_id, + from_role="orchestrator", + to_role="all", + message_type=MessageType.OVERSEER_ALERT, + subject=subject, + body=body, + metadata=metadata, + phase=phase_value, + ) + ) + except Exception as e: + _pkg.logger.warning( + "Failed to publish branch-divergence OVERSEER_ALERT", + pipeline_id=pipeline_id, + error=str(e), + exc_info=True, + ) + + +def _branch_divergence_tick( + pipeline_id: str, + worktree_repo_path: Path, + store: StateStore, + alerted_shas: set[str], +) -> None: + """One iteration of the branch-divergence detector. + + Extracted from the ``_health_monitor_poll`` closure so the + dedupe + reset behavior is unit-testable. Mutates ``alerted_shas`` + in place: adds newly-fired SHAs, and clears the set when the + contamination window goes empty so re-introduction (same SHA, + e.g. agent re-runs a bad rebase) re-fires per the issue's + "rather over-alert than miss" stance. + + All errors are logged-and-swallowed — observability must never + block the pipeline. + """ + try: + pipeline = store.load_pipeline(pipeline_id) + branch = pipeline.branch + base = pipeline.base_branch + if not branch or not base: + return + ahead, offenders = _pkg._check_branch_divergence_for_alert( + pipeline_id=pipeline_id, + worktree_repo_path=worktree_repo_path, + pipeline_branch=branch, + base_branch=base, + ) + if not offenders and alerted_shas: + # Note: transient git errors in ``_check_branch_divergence_for_alert`` + # also surface as ``offenders == []`` and therefore flush the dedupe + # set; this is intentional per #2224's "rather over-alert than miss" + # posture — a flaky git tick will re-fire on the next clean tick. + alerted_shas.clear() + new_offenders = [(sha, subj) for sha, subj in offenders if sha not in alerted_shas] + if new_offenders: + _pkg._publish_branch_divergence_alert( + pipeline, + pipeline_id, + pipeline_branch=branch, + base_branch=base, + ahead_count=ahead, + offenders=new_offenders, + ) + alerted_shas.update(sha for sha, _ in new_offenders) + except Exception as div_err: + _pkg.logger.debug( + "Branch-divergence check failed", + pipeline_id=pipeline_id, + error=str(div_err), + ) + + +def _handle_brc_consensus_timeout( + pipeline: Pipeline, + pipeline_id: str, + consensus_timeout: float, + blocking_agents: list[str], + store: StateStore, # noqa: ARG001 — kept for call-site compatibility (#2264) + slice_id: str | None = None, + active_role_names: list[str] | None = None, +) -> None: + # Extracted from _run_concurrent_phase so k3s-style top-level-module + # layouts (and tests) can exercise this path in isolation — issue #1783. + # ``slice_id`` is propagated so per-slice trackers (#2137) are looked + # up under the nested ``{pipeline_id}/{slice_id}`` key. + # + # Issue #2264: the auto-``choice`` HITL decision this used to open + # was the wrong protocol shape — the platform should not gate the + # pipeline on a binary choice when the operator already has the + # levers (`cancel_task`, `restart_phase`, `provide_input`). The + # two former decision paths now publish ``OVERSEER_ALERT`` messages + # so the SDLC skill's existing alert flow surfaces them as + # notifications rather than a blocking decision. + _brc_handled = False + _brc_timeout_result: dict | None = None + _brc_tracker = None + try: + try: + from peer_consensus import get_peer_consensus_tracker + except ImportError: + from ..peer_consensus import ( + get_peer_consensus_tracker, # type: ignore[no-redef] + ) + + _brc_tracker = get_peer_consensus_tracker(pipeline_id, slice_id) + if _brc_tracker is not None: + _brc_timeout_result = _brc_tracker.handle_timeout() + _brc_handled = _brc_tracker.is_timeout_handled() + _pkg.logger.info( + "BRC timeout handler result", + pipeline_id=pipeline_id, + action=(_brc_timeout_result.get("action") if _brc_timeout_result else None), + brc_handled=_brc_handled, + ) + except Exception as e: + _pkg.logger.warning( + "BRC timeout check failed, falling back to OVERSEER_ALERT", + pipeline_id=pipeline_id, + error=str(e), + ) + + latest_proposal_at: datetime | None = None + if _brc_tracker is not None: + try: + latest_proposal_at = _brc_tracker.get_latest_proposal_timestamp() + except Exception as e: + _pkg.logger.warning( + "Consensus-timeout alert proposal lookup failed", + pipeline_id=pipeline_id, + error=str(e), + exc_info=True, + ) + latest_heartbeat_at = _latest_active_role_heartbeat(active_role_names or []) + + if ( + _brc_handled + and _brc_timeout_result is not None + and _brc_timeout_result.get("action") == "escalate" + ): + # Narrow the alert's blocking_agents to the *critical* blockers + # the tracker just escalated on. The caller-supplied + # ``blocking_agents`` is the full unconfirmed-roles set + # (advisory + critical) from ``evaluate()`` — surfacing + # advisory roles on a high-priority alert dilutes the signal. + critical_entries = _brc_timeout_result.get("critical_blockers") or [] + critical_role_names: list[str] = [] + for entry in critical_entries: + for role in (entry.get("reviewer_role"), entry.get("producer_role")): + if role and role not in critical_role_names: + critical_role_names.append(role) + escalate_blocking = critical_role_names or blocking_agents + _publish_consensus_timeout_alert( + pipeline, + pipeline_id, + consensus_timeout, + escalate_blocking, + priority="high", + latest_proposal_at=latest_proposal_at, + latest_heartbeat_at=latest_heartbeat_at, + slice_id=slice_id, + ) + elif not _brc_handled: + if _pkg._emit_event is not None: + _pkg._emit_event( + EventType.CONSENSUS_TIMEOUT, + pipeline_id, + data={ + "timeout_minutes": consensus_timeout / 60, + "blocking_agents": blocking_agents, + }, + ) + _publish_consensus_timeout_alert( + pipeline, + pipeline_id, + consensus_timeout, + blocking_agents, + priority="medium", + latest_proposal_at=latest_proposal_at, + latest_heartbeat_at=latest_heartbeat_at, + slice_id=slice_id, + ) diff --git a/orchestrator/routes/pipelines/_brc_history.py b/orchestrator/routes/pipelines/_brc_history.py new file mode 100644 index 0000000000..3a87eb6d24 --- /dev/null +++ b/orchestrator/routes/pipelines/_brc_history.py @@ -0,0 +1,961 @@ +"""brc history helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim; patched/barrel-resident globals reached via _pkg so +patch("routes.pipelines.<name>") keeps intercepting. +""" + +from __future__ import annotations + +import json +import subprocess +from pathlib import Path +from typing import TYPE_CHECKING, Any, Literal + +import routes.pipelines as _pkg # noqa: E402,F401 + +if TYPE_CHECKING: + try: + from ..container_spawner import ContainerSpawner # noqa: F401 + except ImportError: # pragma: no cover + from container_spawner import ContainerSpawner # type: ignore # noqa: F401 + +import yaml +from models import Pipeline, PipelineStatus +from slice_id_validation import SLICE_ID_PATTERN +from state_store import StateStore + +BRC_HISTORY_TYPES = frozenset( + { + "CONSENSUS_PROPOSE", + "CONSENSUS_ACK", + "CONSENSUS_NACK", + "CONSENSUS_WITHDRAW", + "CONSENSUS_CONFIRMED", + "CONSENSUS_RE_REVIEW", + # In-cycle conditional-ACK obligation resolution (#2338). Captured + # in the BRC history file so the audit trail survives orchestrator + # teardown — closes the gap that resolution was previously only + # an in-memory event. + "CONSENSUS_OBLIGATION_RESOLVED", + "STATUS", + "HANDOFF", + "AGENT_FAILED", + "NUDGE", + "OVERSEER_ALERT", + # HEARTBEAT (issue #1897) — structured per-agent state messages. + "HEARTBEAT", + # QUESTION removed per issue #1897 Phase 7. The enum member + # remains for backward-compat until the tester updates + # test_brc_history / test_checkpoint fixtures; see + # MessageType.QUESTION. + } +) + + +CONSENSUS_BRC_TYPES = frozenset( + { + "CONSENSUS_PROPOSE", + "CONSENSUS_ACK", + "CONSENSUS_NACK", + "CONSENSUS_WITHDRAW", + "CONSENSUS_CONFIRMED", + "CONSENSUS_RE_REVIEW", + "CONSENSUS_OBLIGATION_RESOLVED", + } +) + + +def _get_message_store(): + """Import and return the message store factory function, or None if unavailable.""" + try: + from message_store import get_message_store + except ImportError: + try: + from ..message_store import get_message_store # type: ignore[import-not-found] + except ImportError: + return None + return get_message_store + + +def _render_brc_history_markdown( + brc_messages: list[Any], + pipeline_id: str, + phase: str, + *, + slice_id: str | None = None, +) -> str: + """Render *brc_messages* as a chronological markdown log. + + The output shape mirrors the legacy aggregate file: a heading line, + a generated-timestamp footer, and one ``### [ts] role (TYPE): subject`` + section per message with a fenced YAML metadata block. + + ``Generated:`` is derived from the *latest* message timestamp (not + wall-clock time) so regenerating the file from the same message set + produces byte-identical output. This keeps the PR-phase safety-net + rewrite (:func:`_rewrite_brc_history_for_pr`) idempotent: when no new + BRC messages arrived between phase completion and PR creation, the + rewritten file matches the previous commit and the follow-up commit is + skipped by :func:`_commit_statefiles_to_worktree`. See #1714. + """ + message_timestamps = [m.timestamp for m in brc_messages if m.timestamp is not None] + if message_timestamps: + generated_str = max(message_timestamps).strftime("%Y-%m-%dT%H:%M:%SZ") + else: + generated_str = "unknown" + # The "unattributed" bucket is not a slice — it holds cross-cutting + # non-CONSENSUS messages that lack canonical slice scope (HEARTBEAT, + # OVERSEER_ALERT, AGENT_FAILED, …) routed to a sibling file so the + # audit trail stays complete. Rendering it as "Slice: unattributed" + # would mislead a reviewer who lands on the file via a link line — + # special-case the heading and metadata block instead. + is_unattributed = slice_id == "unattributed" + lines: list[str] = [] + if is_unattributed: + lines.append(f"# BRC Consensus History — {phase} phase, cross-cutting (unattributed)") + elif slice_id: + lines.append(f"# BRC Consensus History — {phase} phase, {slice_id}") + else: + lines.append(f"# BRC Consensus History — {phase} phase") + lines.append("") + lines.append(f"Generated: {generated_str}") + lines.append(f"Pipeline: {pipeline_id}") + if is_unattributed: + lines.append("Section: cross-cutting (unattributed)") + elif slice_id: + lines.append(f"Slice: {slice_id}") + lines.append("") + + for msg in brc_messages: + ts = msg.timestamp.strftime("%Y-%m-%dT%H:%M:%SZ") if msg.timestamp else "unknown" + # Include to_role for directed messages (not broadcast "all") + if msg.to_role and msg.to_role != "all": + header = ( + f"### [{ts}] {msg.from_role} → {msg.to_role} ({msg.message_type}): {msg.subject}" + ) + else: + header = f"### [{ts}] {msg.from_role} ({msg.message_type}): {msg.subject}" + lines.append(header) + if msg.body: + lines.append("") + lines.append(msg.body) + + # Emit a YAML metadata block with id, phase, and non-empty metadata + meta_block: dict[str, Any] = {} + if msg.id: + meta_block["id"] = msg.id + if msg.phase: + meta_block["phase"] = msg.phase + if msg.metadata: + meta_block["metadata"] = msg.metadata + if meta_block: + lines.append("") + lines.append("````yaml") + lines.append( + yaml.safe_dump(meta_block, sort_keys=False, default_flow_style=False).rstrip() + ) + lines.append("````") + lines.append("") + return "\n".join(lines) + + +def _write_brc_history_file( + worktree_path: Path, + pipeline_id: str, + phase: str, + identifier: int | str, + brc_messages: list[Any], + *, + slice_id: str | None = None, +) -> None: + """Render and persist the markdown + JSON companion files for one bucket. + + ``slice_id``, when provided, switches the on-disk filename from the + aggregate ``{identifier}-{phase}.{md,json}`` shape used by + refine/plan/pr to the per-slice ``{identifier}-{phase}-{slice_id}.{md,json}`` + shape used by implement (#2548 — hard switchover, no aggregate + implement file is produced). + """ + if slice_id: + stem = f"{identifier}-{phase}-{slice_id}" + else: + stem = f"{identifier}-{phase}" + + history_dir = worktree_path / ".egg-state" / "brc-history" + history_dir.mkdir(parents=True, exist_ok=True) + history_file = history_dir / f"{stem}.md" + + # Write the markdown history file + try: + history_file.write_text( + _render_brc_history_markdown( + brc_messages, + pipeline_id, + phase, + slice_id=slice_id, + ) + ) + except Exception as md_err: + _pkg.logger.warning( + "Failed to write BRC history markdown file", + pipeline_id=pipeline_id, + phase=phase, + slice_id=slice_id, + error=str(md_err), + ) + + # Write a JSON companion artifact containing the full message dicts + json_file = history_dir / f"{stem}.json" + try: + json_data = [msg.to_dict() for msg in brc_messages] + json_file.write_text(json.dumps(json_data, indent=2, default=str)) + except Exception as json_err: + _pkg.logger.warning( + "Failed to write BRC history JSON companion file", + pipeline_id=pipeline_id, + phase=phase, + slice_id=slice_id, + error=str(json_err), + ) + + _pkg.logger.info( + "Wrote BRC history file", + pipeline_id=pipeline_id, + phase=phase, + slice_id=slice_id, + path=str(history_file), + message_count=len(brc_messages), + ) + + +def _write_brc_history( + worktree_path: Path, + pipeline_id: str, + phase: str, + identifier: int | str, + *, + write_per_slice: bool = True, +) -> None: + """Write BRC consensus message history for a phase to .egg-state. + + Retrieves BRC-related messages for the given phase from the message store + and writes them as a chronological markdown log to + ``.egg-state/brc-history/{identifier}-{phase}.md``. + + For the ``implement`` phase the writer auto-detects slice-aware vs + aggregate mode (#2548): + + * If at least one BRC message carries a canonical + ``metadata['slice_id']`` (validated against + ``SLICE_ID_PATTERN``), the writer partitions messages per-slice + and writes one file per slice as + ``{identifier}-implement-{slice_id}.{md,json}``. + Per-message attribution rules: + + - ``CONSENSUS_*`` messages without a canonical slice_id are + dropped with a single aggregate WARNING — the orchestrator's + CONSENSUS_* signal handlers tag every implement-phase write + under D4, so a missing slice_id is a contract violation. + - Other ``BRC_HISTORY_TYPES`` (HEARTBEAT, STATUS, HANDOFF, + AGENT_FAILED, NUDGE, OVERSEER_ALERT) come from emitters that + do not uniformly carry slice scope. When they lack a + canonical slice_id they are routed to a sibling + ``{identifier}-implement-unattributed.{md,json}`` file rather + than dropped, so the audit trail stays complete and reviewers + of any per-slice transcript can cross-reference. + + * If **no** BRC message carries a slice_id (non-slice pipelines), + the writer falls back to the aggregate + ``{identifier}-implement.{md,json}`` filename. + + No-ops gracefully when the message store is unavailable or contains no + BRC messages for the pipeline and phase. + + Args: + worktree_path: Path to the worktree repo directory + pipeline_id: The pipeline ID to retrieve messages for + phase: The pipeline phase name (e.g. "implement", "plan") + identifier: The pipeline identifier for file naming + write_per_slice: When False and ``phase == "implement"`` in a + slice-aware pipeline, skip writing the per-slice + ``{identifier}-implement-{slice_id}.{md,json}`` files. The + ``unattributed`` sibling and any non-slice aggregate file + are still written. Per-slice files are owned by their + slice's integration branch (committed by + :func:`_commit_slice_brc_history_to_integration_branch`); + duplicating them onto ``work`` causes add/add merge + conflicts when slice PRs target ``work`` (#2755). Default + ``True`` preserves the historical behavior for the slice + hook itself and for any out-of-tree callers. + """ + _pkg.logger.info( + "_write_brc_history: entering", + pipeline_id=pipeline_id, + phase=phase, + identifier=str(identifier), + ) + + store_fn = _pkg._get_message_store() + if store_fn is None: + _pkg.logger.info( + "_write_brc_history: early return — message store unavailable", + pipeline_id=pipeline_id, + phase=phase, + ) + return + + try: + store = store_fn() + messages = store.get_messages(pipeline_id, limit=10000) + except Exception as e: + _pkg.logger.warning( + "_write_brc_history: early return — failed to retrieve messages", + pipeline_id=pipeline_id, + phase=phase, + error=str(e), + ) + return + + if not messages: + _pkg.logger.info( + "_write_brc_history: early return — no messages in store", + pipeline_id=pipeline_id, + phase=phase, + ) + return + + brc_messages = [m for m in messages if m.message_type in BRC_HISTORY_TYPES and m.phase == phase] + if not brc_messages: + _pkg.logger.info( + "_write_brc_history: early return — no BRC messages for phase", + pipeline_id=pipeline_id, + phase=phase, + total_messages=len(messages), + ) + return + + if phase == "implement": + # Implement-phase BRC messages are partitioned per-slice (#2548) + # for slice-aware pipelines (issue mode with `contract.slices`): + # the orchestrator's CONSENSUS_* signal handlers tag every + # implement-phase consensus message with `metadata['slice_id']`, + # and this writer buckets them into one transcript file per + # slice. Non-slice pipelines have no slice scope on any message, + # so they fall back to the aggregate + # `{identifier}-implement.{md,json}` filename. + # + # ``metadata['slice_id']`` is interpolated into the on-disk + # filename below, so this is a gateway-facing seam in the same + # sense as ``signals.py`` / the restart route / + # ``concurrent_executor`` branch builders: every value MUST be + # validated against the canonical ``SLICE_ID_PATTERN`` before + # use, otherwise an attacker-controlled metadata blob (any role + # can post arbitrary metadata via ``messages.py``) could smuggle + # path separators into the filename and write outside + # ``.egg-state/brc-history/``. See ``slice_id_validation.py`` + # for the invariant. ``SLICE_ID_PATTERN`` is already imported at + # module top (the same try/except sandbox-vs-orchestrator dual + # import that imports ``extract_slice_id``); no local re-import + # is needed. + + buckets: dict[str, list[Any]] = {} + # ``unattributed_consensus`` holds CONSENSUS_* messages that lack + # a canonical slice_id — those are a D4 contract violation and + # are dropped with a single aggregate WARNING. ``unattributed_other`` + # holds non-CONSENSUS BRC types (HEARTBEAT, STATUS, HANDOFF, + # AGENT_FAILED, NUDGE, OVERSEER_ALERT) whose emitters do not + # uniformly carry slice scope; those are written to the + # ``unattributed`` sibling file so the audit trail stays complete. + unattributed_consensus: list[Any] = [] + unattributed_other: list[Any] = [] + for msg in brc_messages: + # ``Message.metadata`` is a Pydantic dict[str, Any] field with a + # default_factory=dict (message_store.Message), so it is always a + # dict at this point — no need to guard with getattr/isinstance. + raw_slice_id = msg.metadata.get("slice_id") + if isinstance(raw_slice_id, str) and SLICE_ID_PATTERN.fullmatch(raw_slice_id): + buckets.setdefault(raw_slice_id, []).append(msg) + continue + if str(getattr(msg, "message_type", "")) in CONSENSUS_BRC_TYPES: + unattributed_consensus.append(msg) + else: + unattributed_other.append(msg) + + if not buckets: + # No slice-attributed messages anywhere — this is a non-slice + # pipeline (an implement-phase run that never spawned slice + # scopes). Fall back to the aggregate + # `{identifier}-implement.{md,json}` filename so we never + # silently drop the entire BRC stream when no per-slice + # bucketing is possible. See #2548 reviewer_code_holistic + # finding #3. + _pkg.logger.info( + "_write_brc_history: no slice-attributed implement-phase " + "messages — writing aggregate file (non-slice pipeline)", + pipeline_id=pipeline_id, + phase=phase, + total_brc_messages=len(brc_messages), + ) + _write_brc_history_file( + worktree_path, + pipeline_id, + phase, + identifier, + brc_messages, + ) + return + + # Slice-aware pipeline: at least one canonical slice_id was + # observed. CONSENSUS_* messages that lack a canonical slice_id + # are a D4 hard-switchover contract violation — drop them with + # a loud aggregate WARNING (count + sample types) so an operator + # notices the asymmetry rather than silently shipping a thinned- + # out transcript. + if unattributed_consensus: + sample_types = sorted( + {str(getattr(m, "message_type", "")) for m in unattributed_consensus[:8]} + ) + _pkg.logger.warning( + "_write_brc_history: dropped implement-phase CONSENSUS_* messages " + "without canonical metadata.slice_id (hard switchover, #2548)", + pipeline_id=pipeline_id, + phase=phase, + dropped_count=len(unattributed_consensus), + sample_message_types=sample_types, + attributed_count=sum(len(v) for v in buckets.values()), + ) + + # Non-CONSENSUS BRC types without a canonical slice_id come from + # emitters that do not uniformly attach slice scope (HealthMonitor + # nudges, overseer respawn alerts, AGENT_FAILED broadcasts, + # CLI-routed HANDOFF/NUDGE messages, etc.). Route them to a + # sibling ``{identifier}-implement-unattributed.{md,json}`` file + # so the audit trail stays complete — reviewers reading any + # per-slice transcript can cross-reference. See #2548 + # reviewer_code blocking finding. + if unattributed_other: + _write_brc_history_file( + worktree_path, + pipeline_id, + phase, + identifier, + unattributed_other, + slice_id="unattributed", + ) + + if not write_per_slice: + # Caller opted out of per-slice writes (#2755). The + # ``unattributed`` sibling has already been written above + # (when ``unattributed_other`` was non-empty); skip the + # per-slice bucket loop so we don't add files that the + # slice branches already own. See the docstring's + # ``write_per_slice`` arg for the merge-conflict rationale. + _pkg.logger.info( + "_write_brc_history: skipping per-slice writes (write_per_slice=False)", + pipeline_id=pipeline_id, + phase=phase, + slice_bucket_count=len(buckets), + ) + return + + # Natural sort by the integer suffix so a 12-slice pipeline iterates + # `slice-1, slice-2, ..., slice-12` rather than the lexicographic + # `slice-1, slice-10, slice-11, slice-12, slice-2`. Every key is + # already SLICE_ID_PATTERN-validated (`^slice-[0-9]+$`) above, so the + # int() parse is total. + for slice_id, slice_msgs in sorted( + buckets.items(), key=lambda kv: int(kv[0].rsplit("-", 1)[1]) + ): + _write_brc_history_file( + worktree_path, + pipeline_id, + phase, + identifier, + slice_msgs, + slice_id=slice_id, + ) + return + + # Refine, plan, and pr phases continue to write the aggregate + # `{identifier}-{phase}.{md,json}` file — only implement is per-slice. + _write_brc_history_file( + worktree_path, + pipeline_id, + phase, + identifier, + brc_messages, + ) + + +def _rewrite_brc_history_for_pr( + worktree_path: Path, + pipeline_id: str, + pipeline_phases: dict, + identifier: int | str, +) -> None: + """Re-write BRC history for all completed phases before PR creation. + + Iterates ``pipeline_phases`` (a mapping of phase name → phase execution + objects with a ``.status`` attribute) and calls :func:`_write_brc_history` + for each phase whose status is ``PipelineStatus.COMPLETE``. + + Errors from individual phase writes are logged at warning level and + do not prevent other phases from being processed. + + After re-writing history files, commits the results via + :func:`_commit_statefiles_to_worktree`. Commit failures are also + logged and swallowed so the PR creation can proceed. + """ + completed_phases = [ + name for name, ex in pipeline_phases.items() if ex.status == PipelineStatus.COMPLETE + ] + _pkg.logger.info( + "_rewrite_brc_history_for_pr: entering", + pipeline_id=pipeline_id, + total_phases=len(pipeline_phases), + completed_phase_count=len(completed_phases), + completed_phases=completed_phases, + ) + for phase_name, phase_exec in pipeline_phases.items(): + if phase_exec.status == PipelineStatus.COMPLETE: + try: + _pkg._write_brc_history( + worktree_path, + pipeline_id, + phase_name, + identifier, + # Per-slice implement-phase files are owned by each + # slice's integration branch (#2548 D2/D5); committing + # them onto ``work`` would re-introduce the add/add + # merge conflict from #2755. Only the aggregate / + # unattributed sibling lands on ``work``. + write_per_slice=False, + ) + except Exception as brc_err: + _pkg.logger.warning( + "Failed to re-write BRC history for PR (continuing)", + pipeline_id=pipeline_id, + phase=phase_name, + error=str(brc_err), + ) + try: + _pkg._commit_statefiles_to_worktree( + worktree_path, + "Persist BRC history files for PR", + pipeline_identifier=identifier, + pipeline_id=pipeline_id, + ) + _pkg.logger.info( + "_rewrite_brc_history_for_pr: commit step completed successfully", + pipeline_id=pipeline_id, + ) + except subprocess.CalledProcessError as git_err: + _pkg.logger.warning( + "Failed to commit BRC history for PR (continuing)", + pipeline_id=pipeline_id, + error=str(git_err), + ) + _pkg.logger.info( + "_rewrite_brc_history_for_pr: exiting", + pipeline_id=pipeline_id, + ) + + +def _persist_phase_brc_history( + pipeline: Pipeline, + store: StateStore, + phase: str, +) -> None: + """Persist BRC history for *phase* and commit it, best-effort. + + Mirrors the per-phase write+commit sequence that ``_run_pipeline`` + runs inline at phase completion, so external phase-transition paths + (the ``complete_phase`` / ``advance_phase`` REST+MCP handlers) do + not silently drop BRC transcripts when ``_clear_concurrent_state`` + wipes the message store. See #1827. + + Note: this commits but does **not** push. Callers must ensure a + push happens downstream — in ``advance_phase`` the spawned + ``_run_pipeline`` thread pushes the branch, carrying this commit + along; in a standalone ``complete_phase`` the caller is expected to + trigger a subsequent advance or push. + """ + worktree_path = _pkg._resolve_pipeline_worktree_path(pipeline, store.repo_path) + try: + _pkg._write_brc_history( + worktree_path, + pipeline.id, + phase, + _pkg._brc_history_identifier(pipeline), + # Per-slice implement-phase files are owned by the slice's + # integration branch (committed by + # :func:`_commit_slice_brc_history_to_integration_branch`); + # the work-branch worktree must not duplicate them, otherwise + # slice PRs targeting ``work`` hit add/add merge conflicts + # (#2755). The parameter is a no-op for non-implement phases. + write_per_slice=False, + ) + except Exception as brc_err: + _pkg.logger.warning( + "Failed to persist BRC history before phase transition (continuing)", + pipeline_id=pipeline.id, + phase=phase, + error=str(brc_err), + ) + return + + try: + _pkg._commit_statefiles_to_worktree( + worktree_path, + f"Persist statefiles after {phase} phase", + pipeline_identifier=_pkg._pipeline_identifier(pipeline.issue_number, pipeline.id), + # Contract files are keyed by pipeline_id, not the issue-number + # prefix; without this the restart-time persist skipped the + # contract entirely (#1829 gap, observed in #3427). + pipeline_id=pipeline.id, + ) + except subprocess.CalledProcessError as git_err: + _pkg.logger.warning( + "Failed to commit BRC history before phase transition (continuing)", + pipeline_id=pipeline.id, + phase=phase, + error=str(git_err), + ) + + +def _commit_slice_brc_history_to_integration_branch( + pipeline, + spawner: "ContainerSpawner", # noqa: UP037 + worktree_repo_path: Path, + slice_id: str, + integration_branch: str, + *, + gateway_mode: Literal["public", "private"] = "public", +) -> bool: + """Commit a slice's per-slice BRC history onto its integration branch (#2548). + + Runs after the slice's implement-phase consensus is reached and + before the slice PR is opened, so reviewers approaching the slice + PR see the full BRC consensus transcript that approved the slice's + code as part of the diff. + + Steps: + + 1. Materialise a per-tick temp directory under + ``WORKTREE_BASE_DIR`` (gateway-allowlisted; see #2684) and + render the per-slice BRC history files into a ``staging/`` + subdirectory via :func:`_write_brc_history`. The writer pulls + messages from the message store; the staging directory is + scoped to this hook tick so concurrent slice hooks do not + cross-write each other (#2755). + 2. Materialise a temporary **detached** git worktree on + ``origin/<integration_branch>`` (the slice's integration branch). + A detached worktree claims no branch ref, so it never collides + with the slice's own agent worktrees — which hold the + integration branch checked out for the duration of the slice + run — nor with a prior tick that crashed mid-flight (#2778). + 3. Copy ONLY this slice's per-slice BRC files + (``<identifier>-implement-<slice_id>.{json,md}``) from the + staging directory to the integration worktree. Other slices' + files (or the unattributed sibling) are deliberately not + copied — each slice PR carries only its own BRC transcript per + D2 / D5 of #2548. + 4. Commit via :func:`_commit_statefiles_to_worktree` + (orchestrator-authored, ``--no-verify``, idempotent: skips when + staged is empty). + 5. Push via :meth:`GatewayClient.push_worktree_branch` (launcher- + auth so we bypass agent-facing push restrictions on + ``.egg-state/brc-history/``). + + Returns ``True`` on success or no-op (files already committed and + push is a fast-forward no-op). Returns ``False`` on any failure; + the caller treats this as best-effort and proceeds with PR + creation. The per-slice BRC files do not exist on the work + worktree under this design (#2755) — the integration branch is + the only on-disk surface that carries them, so a failure here + means the slice PR opens without its consensus transcript. + + Idempotency: every step is convergent — re-running mid-flight + against an already-committed integration branch produces no new + commit (``_commit_statefiles_to_worktree`` skips when nothing is + staged) and a no-op fast-forward push. + + Concurrency: this hook runs from ``_run_one_slice_inner``, which + is itself invoked concurrently across slices in a thread pool. + Each invocation creates its own ``mkdtemp``-rooted staging + directory, so two slices reaching consensus near-simultaneously + do not share any filesystem state (#2755 fix). Each slice copies + only its own per-slice files to its integration worktree + (Step 3), so concurrent writes do not cross-pollinate slice PRs. + """ + pipeline_id = pipeline.id + + if not pipeline.repo: + _pkg.logger.info( + "Per-slice BRC commit: pipeline has no remote repo, skipping (#2548)", + pipeline_id=pipeline_id, + slice_id=slice_id, + ) + return False + + identifier = _pkg._brc_history_identifier(pipeline) + + import shutil + import tempfile + + git_base = [ + "git", + "-c", + "core.hooksPath=/dev/null", + "-c", + f"safe.directory={worktree_repo_path}", + "-C", + str(worktree_repo_path), + ] + + # Root under WORKTREE_BASE_DIR so the temp path falls inside the + # gateway's repo-path allowlist (gateway/git_client.py + # ALLOWED_REPO_PATHS). A ``/tmp`` location is rejected by + # ``validate_repo_path``, which silently failed the BRC-history + # push and left slice PRs without their consensus transcript + # (#2684). Falls back to system temp when the base dir is absent + # (e.g. unit tests) — emit a warning on that branch so a broken + # docker volume mount in production is noisy rather than silently + # recreating the #2684 push-rejection. + if _pkg.WORKTREE_BASE_DIR.exists(): + tmp_dir_base = str(_pkg.WORKTREE_BASE_DIR) + else: + _pkg.logger.warning( + "Per-slice BRC commit: WORKTREE_BASE_DIR missing — falling " + "back to system temp (likely a broken volume mount in " + "production; the push to the integration branch will be " + "rejected by the gateway allowlist) (#2684)", + pipeline_id=pipeline_id, + slice_id=slice_id, + integration_branch=integration_branch, + worktree_base_dir=str(_pkg.WORKTREE_BASE_DIR), + ) + tmp_dir_base = None + tmp_worktree = Path( + tempfile.mkdtemp( + prefix=f"egg-slice-brc-{pipeline_id}-{slice_id}-", + dir=tmp_dir_base, + ) + ) + # Per-tick staging directory so concurrent slice hooks do not + # share the writer's output (#2755). ``_write_brc_history`` + # renders into ``<staging>/.egg-state/brc-history/`` — same + # relative layout it uses against a worktree — so the + # ``Path.relative_to(staging)`` step below preserves the + # canonical on-disk path when copying onto the integration + # worktree. + staging = tmp_worktree / "staging" + wt_path = tmp_worktree / "wt" + + try: + # --- Step 1: render the per-slice BRC files into the staging + # directory. The writer pulls messages from the message store + # and writes all per-slice files for the implement phase; we + # filter to this slice's files below. + try: + _pkg._write_brc_history( + staging, + pipeline_id, + "implement", + identifier, + ) + except Exception as brc_err: # noqa: BLE001 + _pkg.logger.warning( + "Per-slice BRC commit: failed to render BRC history into " + "staging dir, skipping integration-branch commit (#2548)", + pipeline_id=pipeline_id, + slice_id=slice_id, + error=str(brc_err), + ) + return False + + # The per-slice files we will copy onto the integration worktree. + # Both files are produced by ``_write_brc_history`` (markdown and + # JSON companion). Missing files are tolerated — the writer logs + # at warning level but still succeeds on the other format, so we + # copy whichever exists. + history_dir = staging / ".egg-state" / "brc-history" + per_slice_files: list[Path] = [] + for ext in ("md", "json"): + candidate = history_dir / f"{identifier}-implement-{slice_id}.{ext}" + if candidate.is_symlink(): + # Defense-in-depth: a planted symlink could point outside + # ``.egg-state/`` and leak unrelated content onto the slice + # PR. The staging directory is freshly minted under + # ``tempfile.mkdtemp`` per hook tick, so a symlink at this + # path would have to come from the writer itself — the + # check is cheap and protects against any future writer + # change that might honour an attacker-controlled + # metadata blob when synthesising the filename. + _pkg.logger.warning( + "Per-slice BRC commit: skipping symlink in brc-history (#2548)", + pipeline_id=pipeline_id, + slice_id=slice_id, + path=str(candidate), + ) + continue + if candidate.is_file(): + per_slice_files.append(candidate) + + if not per_slice_files: + _pkg.logger.warning( + "Per-slice BRC commit: no per-slice BRC files produced " + "for slice — skipping integration-branch commit (#2548)", + pipeline_id=pipeline_id, + slice_id=slice_id, + identifier=str(identifier), + ) + return False + + # --- Step 2: refresh the local remote-tracking ref for the + # integration branch. The slice's agents pushed directly to + # ``origin/<integration_branch>`` during the run, so the work + # worktree's local tracking ref may lag. Best-effort: a failure + # here usually means the agent-side push has not yet propagated; + # the worktree-add below would then fail and we'd return False. + try: + spawner.gateway.fetch_branch( + pipeline_id, + str(worktree_repo_path), + args=[f"+refs/heads/{integration_branch}:refs/remotes/origin/{integration_branch}"], + mode=gateway_mode, # type: ignore[arg-type] + ) + except Exception as fetch_err: # noqa: BLE001 + _pkg.logger.warning( + "Per-slice BRC commit: fetch of integration branch failed (continuing) (#2548)", + pipeline_id=pipeline_id, + slice_id=slice_id, + integration_branch=integration_branch, + error=str(fetch_err), + ) + + try: + subprocess.run( + [ + *git_base, + "worktree", + "add", + # Detached, not ``-B <integration_branch>``: a branch + # can live in only one linked worktree, and the + # slice's agent worktrees already hold it — ``-B`` + # lost that race with ``fatal: ... already used by + # worktree`` (#2778). See Step 2 in the docstring. + "--detach", + str(wt_path), + f"origin/{integration_branch}", + ], + capture_output=True, + text=True, + check=True, + timeout=60, + ) + except subprocess.CalledProcessError as wt_err: + _pkg.logger.warning( + "Per-slice BRC commit: worktree add failed, skipping (#2548)", + pipeline_id=pipeline_id, + slice_id=slice_id, + integration_branch=integration_branch, + stderr=(wt_err.stderr or "")[:500], + ) + return False + + # --- Step 3: copy ONLY this slice's BRC files onto the integration + # worktree. Each file lands at the same relative path it occupies + # in the staging dir (``.egg-state/brc-history/...``). + for src in per_slice_files: + try: + rel = src.relative_to(staging) + except ValueError: + _pkg.logger.warning( + "Per-slice BRC commit: file outside staging dir, skipping it (#2548)", + pipeline_id=pipeline_id, + slice_id=slice_id, + src=str(src), + ) + continue + dst = wt_path / rel + dst.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(src, dst) + + # --- Step 4: commit (idempotent — skips when staged is empty) --- + try: + _pkg._commit_statefiles_to_worktree( + wt_path, + f"Persist BRC history for {slice_id} (#2548)", + pipeline_identifier=identifier, + pipeline_id=pipeline_id, + ) + except Exception as commit_err: # noqa: BLE001 + _pkg.logger.warning( + "Per-slice BRC commit: commit failed, skipping (#2548)", + pipeline_id=pipeline_id, + slice_id=slice_id, + error=str(commit_err), + ) + return False + + # --- Step 5: push to origin/<integration_branch>. Fast-forward + # no-op when the local tip matches origin (e.g. when the + # commit step was a no-op because everything was already + # committed on a prior tick). + try: + push_result = spawner.gateway.push_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(wt_path), + branch=integration_branch, + mode=gateway_mode, # type: ignore[arg-type] + base_branch=pipeline.base_branch, + ) + except Exception as push_err: # noqa: BLE001 + _pkg.logger.warning( + "Per-slice BRC commit: push raised, skipping (#2548)", + pipeline_id=pipeline_id, + slice_id=slice_id, + error=str(push_err), + ) + return False + if not push_result.ok: + _pkg.logger.warning( + "Per-slice BRC commit: push failed, skipping (#2548)", + pipeline_id=pipeline_id, + slice_id=slice_id, + category=getattr(push_result, "category", None), + detail=getattr(push_result, "detail", None), + ) + return False + + _pkg.logger.info( + "Per-slice BRC commit: pushed BRC history to integration branch (#2548)", + pipeline_id=pipeline_id, + slice_id=slice_id, + integration_branch=integration_branch, + files=[str(p.relative_to(staging)) for p in per_slice_files], + ) + return True + finally: + # Best-effort cleanup of the temp worktree. A failure here is a + # housekeeping problem, not a pipeline-blocker. + try: + subprocess.run( + [*git_base, "worktree", "remove", "--force", str(wt_path)], + capture_output=True, + text=True, + check=False, + timeout=30, + ) + except Exception as cleanup_err: # noqa: BLE001 + _pkg.logger.debug( + "Per-slice BRC commit: worktree remove failed (continuing) (#2548)", + pipeline_id=pipeline_id, + slice_id=slice_id, + error=str(cleanup_err), + ) + try: + shutil.rmtree(tmp_worktree, ignore_errors=True) + except Exception: # noqa: BLE001 + pass diff --git a/orchestrator/routes/pipelines/_context_pr.py b/orchestrator/routes/pipelines/_context_pr.py new file mode 100644 index 0000000000..82a9163687 --- /dev/null +++ b/orchestrator/routes/pipelines/_context_pr.py @@ -0,0 +1,1220 @@ +"""context pr helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim; patched/barrel-resident globals reached via _pkg so +patch("routes.pipelines.<name>") keeps intercepting. +""" + +from __future__ import annotations + +import re +from pathlib import Path +from typing import Any + +import routes.pipelines as _pkg # noqa: E402,F401 +from egg_contracts.markdown import unwrap_soft_breaks +from models import PipelinePhase + +from ._drafts import _get_human_draft_path + + +def _build_pre_merge_obligations_section( + pipeline_id: str, + contract_deferred_actions: list[Any] | None = None, +) -> str: + """Render the "Pre-merge Obligations" section from active conditional ACKs. + + Two sources, in order of preference: + + 1. ``contract_deferred_actions`` — ``DeferredAction`` objects (or legacy + strings) previously persisted to ``contract.pr.deferred_actions`` when + a human approved the conditional-ACK HITL gate (#2004). This is the + durable path: the tracker may have been torn down by the time PR + creation runs, and the contract survives. + 2. The live consensus tracker (#1998). Used when the contract has + no deferred_actions — either because the gate landed before + tracker teardown, or the gate was never required. + + The markdown composition (open vs. resolved sections, banner copy) + is delegated to :mod:`orchestrator.pr_obligations`. Pre-#2777 cq-6 + the slice-DAG terminal slice rendered the same section from this + shared shape; under cq-4 the obligations live solely on the + up-front context PR (``egg/<id>/work → main``) opened by + :func:`_open_context_pr_at_implement_start`, so only this + ``_auto_create_pr`` callsite renders them now. The shared shape + stays so a future caller (re-introducing per-slice obligation + rendering, etc.) has parity. + + Returns an empty string if neither source yields obligations, so + callers can unconditionally append the result to the PR body. + """ + try: + from pr_obligations import render_obligations_section_from_normalized + except ImportError: + from ..pr_obligations import ( # type: ignore[import-not-found,no-redef] + render_obligations_section_from_normalized, + ) + obligations = _collect_pre_merge_obligations(pipeline_id, contract_deferred_actions) + return render_obligations_section_from_normalized(obligations) + + +def _collect_pre_merge_obligations( + pipeline_id: str, + contract_deferred_actions: list[Any] | None, +) -> list[dict[str, str]]: + """Normalize obligations from contract or live tracker into a uniform shape. + + Returns a list of ``{reviewer, condition, resolved_in_diff}`` dicts. The + contract source takes precedence over the live tracker when present. + + .. note:: + + Under #2777 cq-4 obligations live on the up-front context PR + (``egg/<id>/work → main``) opened by + :func:`_open_context_pr_at_implement_start`, not on individual + slice PRs — so the slice-loop no longer calls this helper. The + pipeline-level tracker fallback survives because the + ``_auto_create_pr`` path that still calls this helper uses the + pipeline-level tracker; future re-introducers of per-slice + obligation rendering would need to thread a slice-keyed tracker + (see ``peer_consensus._tracker_key`` ⇒ + ``{pipeline_id}/{slice_id}``) through here. + """ + try: + from pr_obligations import normalize_deferred_actions + except ImportError: + from ..pr_obligations import ( # type: ignore[import-not-found,no-redef] + normalize_deferred_actions, + ) + normalized = normalize_deferred_actions(contract_deferred_actions) + if normalized: + return normalized + + # Tier 2 — live tracker (pre-#2004 path; kept so conditions still + # render if the HITL gate hasn't resolved yet, e.g. under force=true). + try: + from peer_consensus import get_peer_consensus_tracker + except ImportError: + from ..peer_consensus import get_peer_consensus_tracker # type: ignore[import-not-found] + tracker = get_peer_consensus_tracker(pipeline_id) + if tracker is None: + return [] + try: + conditions = tracker.get_pre_merge_conditions() + except Exception as e: # defensive — never block PR creation on this + _pkg.logger.warning( + "Failed to read pre-merge conditions from tracker", + pipeline_id=pipeline_id, + error=str(e), + ) + return [] + + tracker_normalized: list[dict[str, str]] = [] + for c in conditions: + condition = str(c.get("condition", "")).strip() + if not condition: + continue + tracker_normalized.append( + { + "reviewer": str(c.get("reviewer", "") or "").strip(), + "condition": condition, + "resolved_in_diff": str(c.get("resolved_in_diff", "") or "").strip(), + } + ) + return tracker_normalized + + +def _build_brc_history_link_line( + worktree_repo_path: Path, + identifier: int | str | None, + link_base: str | None = None, +) -> str: + """Build a one-line pointer to the committed BRC history transcripts. + + Scans ``.egg-state/brc-history/`` for ``{identifier}-<phase>.md`` files + written by :func:`_write_brc_history` and returns a sentence linking + each phase's transcript, ordered by canonical execution order + (``refine`` → ``plan`` → ``implement`` → ``pr``; unknown names sorted + alphabetically after). + + ``link_base`` (#3115): when set (e.g. + ``https://github.com/<repo>/blob/<branch>``), links are rendered as + branch-qualified absolute URLs instead of the default ``./``-relative + form. GitHub resolves relative links in PR bodies against the repo's + default branch, where ``.egg-state/`` does not exist — so any caller + embedding this line in a PR body must pass ``link_base``. + + Returns an empty string when ``identifier`` is ``None`` or no + transcripts exist on disk. + """ + if identifier is None: + return "" + history_dir = worktree_repo_path / ".egg-state" / "brc-history" + if not history_dir.is_dir(): + return "" + prefix = f"{identifier}-" + phases: list[str] = [] + for path in history_dir.glob(f"{prefix}*.md"): + stem = path.stem + if stem.startswith(prefix): + phases.append(stem[len(prefix) :]) + if not phases: + return "" + + canonical = [p.value for p in PipelinePhase] + rank = {name: i for i, name in enumerate(canonical)} + + # Per-slice implement files (#2548) carry the stem + # ``implement-slice-{N}``; cluster them at the canonical ``implement`` + # rank so the rendered link order is + # ``refine → plan → implement[-slice-N] → implement-unattributed → + # pr`` instead of pushing the per-slice files past pr to the end of + # the list. Within the implement cluster, sort by the integer slice + # index so a 12-slice pipeline renders ``slice-1, slice-2, …, + # slice-12`` rather than the lexicographic ``slice-1, slice-10, + # slice-11, slice-12, slice-2``. The ``implement-unattributed`` + # sibling (cross-cutting non-CONSENSUS BRC types without slice scope, + # see ``_write_brc_history``) sorts after every per-slice file so a + # reviewer reads each slice transcript first, then the cross-cutting + # context. + def _sort_key(name: str) -> tuple[int, int, str]: + if name == "implement": + return (rank["implement"], -1, "") + if name == "implement-unattributed": + return (rank["implement"], 1 << 30, name) + if name.startswith("implement-slice-"): + try: + idx = int(name.rsplit("-", 1)[1]) + except ValueError: + idx = 1 << 30 # malformed → sort last within cluster + return (rank["implement"], idx, name) + return (rank.get(name, len(canonical)), 0, name) + + phases.sort(key=_sort_key) + + prefix_url = f"{link_base.rstrip('/')}/" if link_base else "./" + links = ", ".join( + f"[`{phase}`]({prefix_url}.egg-state/brc-history/{identifier}-{phase}.md)" + for phase in phases + ) + return f"_Per-phase BRC transcripts: {links}._" + + +def _compose_context_pr_body( + *, + contract, + pipeline, + worktree_repo_path: Path, + identifier: int | str, + context_repo: str | None = None, + sibling_context_prs: list[dict[str, Any]] | None = None, +) -> str: + """Compose the context-PR body from contract + pipeline state (#3115). + + Before #3115 the context PR's body was ``contract.pr.description`` + verbatim, which dropped ``test_plan`` / ``manual_steps`` on the + floor (the composer that rendered them died with the PR phase in + #2777 even though the plan preflight still requires both fields) + and linked to none of the pipeline artifacts the orchestrator + deterministically knows about. This helper restores the full shape: + + 1. The planner's ``description`` (narrative, verbatim). + 2. ``## Test Plan`` / ``## Manual Steps`` from the contract fields + (Title Case matches the global PR template and the slice PR's + inline-narrative branch in ``gateway_client.py``). + 3. A generated ``## Pipeline context`` footer: pipeline id, + originating issue, the slice table, and links to the refine + analysis draft, the plan draft, and the per-phase BRC + transcripts committed on the work branch. + + Artifact links are branch-qualified absolute URLs + (``https://github.com/<repo>/blob/<work-branch>/...``) — GitHub + resolves relative links in PR bodies against the default branch, + where ``.egg-state/`` does not exist. Draft links are only emitted + for files that exist in the worktree, so a pipeline that skipped + refine does not link a 404. + + Pure string composition over already-loaded state — no git or + gateway calls — so the opener's failure surface is unchanged. + """ + pr = contract.pr + sections: list[str] = [] + + # Soft-break unwrapping (#3122): the ``pr:`` block fields arrive as + # YAML block scalars hard-wrapped at ~75 chars, and GitHub renders + # every newline in a PR body as a line break — join the wraps back + # into paragraphs, leaving real markdown structure alone. + description = unwrap_soft_breaks(pr.description if pr else None).strip() + if description: + sections.append(description) + + test_plan = unwrap_soft_breaks(pr.test_plan if pr else None).strip() + if test_plan: + sections.append(f"## Test Plan\n\n{test_plan}") + + manual_steps = unwrap_soft_breaks(pr.manual_steps if pr else None).strip() + if manual_steps: + sections.append(f"## Manual Steps\n\n{manual_steps}") + + # Build the footer body first; only emit the ``## Pipeline context`` + # header when *more than* the bare pipeline-id line gets added (a + # single ``- Pipeline: <id>`` line under its own ``##`` header is + # noise — every reviewer can read that off the URL). + body_lines: list[str] = [f"- Pipeline: `{pipeline.id}`"] + has_meaningful_content = False + if pipeline.issue_number: + # Bare ``#N`` autolinks within the same repo, which is where + # the pipeline's originating issue lives. + body_lines.append(f"- Issue: #{pipeline.issue_number}") + has_meaningful_content = True + + # #3393 slice-4 / task-4-2: the repo this context PR lives in. A + # slice PR in this same repo cross-links as a bare ``#N`` autolink; + # a slice PR in a DIFFERENT repo of the pipeline must be qualified + # as ``owner/repo#N`` (a bare ``#N`` would resolve against the wrong + # repo). Defaults to the pipeline primary — the repo the up-front + # opener composes the primary context PR for. For an N=1 pipeline + # every slice resolves to the primary, so every link stays bare and + # the body is byte-identical to the single-repo shape. + this_context_repo = context_repo or getattr(pipeline, "primary_repo", None) or pipeline.repo + try: + from models import resolve_slice_repo # type: ignore[no-redef] + except ImportError: + from ..models import resolve_slice_repo # type: ignore[no-redef] + + slices = list(contract.slices or []) + if slices: + body_lines.append(f"- Slices ({len(slices)}):") + for s in slices: + name = " ".join((s.name or s.id).split()) + # Strip both ``slice-`` and the legacy ``phase-`` prefix — + # ``Slice.id`` still permits the latter (models.py) and + # ``_migrate_phases_to_slices`` only rewrites it on JSON + # load, so a directly-constructed Slice can still carry it. + number = s.id.removeprefix("slice-").removeprefix("phase-") + line = f" {number}. {name} (`{s.id}`)" + # Cross-link the stack (#3122): once the slice's PR is open + # its number is persisted on the contract and the run loop + # re-composes this body, so the entry gains a link. + if getattr(s, "pr_number", None): + s_repo = resolve_slice_repo(s, pipeline) + if s_repo and this_context_repo and s_repo != this_context_repo: + # Cross-repo sibling — repo-qualify so GitHub resolves + # the autolink to the right repo (#3393 slice-4). + line += f" — {s_repo}#{s.pr_number}" + else: + # Same-repo (or repo unknown): bare ``#N`` autolinks + # within the repo this context PR lives in. + line += f" — #{s.pr_number}" + body_lines.append(line) + has_meaningful_content = True + + link_base: str | None = None + if pipeline.repo and pipeline.branch: + link_base = f"https://github.com/{pipeline.repo}/blob/{pipeline.branch}" + + if link_base: + doc_links: list[str] = [] + for phase, label in (("refine", "Refine analysis"), ("plan", "Implementation plan")): + rel_path = _pkg._get_draft_path( + phase, issue_number=pipeline.issue_number, pipeline_id=pipeline.id + ) + if rel_path and (worktree_repo_path / rel_path).is_file(): + doc_links.append(f"[{label}]({link_base}/{rel_path})") + # Human-focused companion (the simplifier's ``*-human.md``), when present. + human_rel = _get_human_draft_path( + phase, issue_number=pipeline.issue_number, pipeline_id=pipeline.id + ) + if human_rel and (worktree_repo_path / human_rel).is_file(): + doc_links.append(f"[{label} (human summary)]({link_base}/{human_rel})") + if doc_links: + body_lines.append(f"- Docs: {', '.join(doc_links)}") + has_meaningful_content = True + brc_line = _build_brc_history_link_line(worktree_repo_path, identifier, link_base=link_base) + if brc_line: + body_lines.append("") + body_lines.append(brc_line) + has_meaningful_content = True + + if has_meaningful_content: + sections.append("\n".join(["## Pipeline context", "", *body_lines])) + + # #3393 slice-4 / task-4-2: cross-reference the pipeline's context + # PRs in OTHER repos. Rendered only for a multi-repo pipeline (the + # opener passes ``sibling_context_prs`` when it coordinates >1 + # repo); an N=1 pipeline passes ``None`` and this section is + # omitted, keeping the body byte-identical to the single-repo shape. + coord_lines: list[str] = [] + for ref in sibling_context_prs or []: + ref_repo = (ref.get("repo") or "").strip() + ref_number = ref.get("number") + if not ref_repo or not isinstance(ref_number, int) or isinstance(ref_number, bool): + continue + if ref_number < 1: + continue + # ``owner/repo#N`` autolinks cross-repo (a bare ``#N`` would + # resolve against the repo this body lives in). + coord_lines.append(f"- {ref_repo}#{ref_number}") + if coord_lines: + sections.append( + "\n".join( + [ + "## Coordinated repos", + "", + "This pipeline coordinates PRs across multiple repos (#3393):", + "", + *coord_lines, + ] + ) + ) + return "\n\n".join(sections) + + +def _persist_context_pr_number( + pipeline_id: str, + pr_number: int, + *, + worktree_repo_path: Path, + identifier: int | str, + pr_url: str | None = None, +) -> None: + """Persist context-PR linkage on both the contract and the pipeline (#2777). + + Single-purpose helper extracted so the new + :func:`_open_context_pr_at_implement_start` opener is not a + non-transactional state mutator. Wraps the contract write under + the existing per-pipeline state lock so concurrent advance_phase / + backstop callers serialise on the same lock instance the rest of + the orchestrator uses, then calls ``save_contract`` to atomically + rewrite ``.egg-state/contracts/...`` on disk. + + The helper is the SOLE writer of ``context_pr_number`` after + slice-2 (#2777, TASK-2-1) deleted the legacy + ``_persist_context_pr_linkage_on_contract``. It is called exactly + once per ``_open_context_pr_at_implement_start`` invocation, + immediately after either the ``gh pr list`` idempotency hit or the + successful ``gh pr create``. The same persistence write fires on + the idempotent path so a resume-from-orphaned-pipeline where the + contract lost ``context_pr_number`` mid-run still recovers (the + unit test in TASK-3-8 asserts this). + + In slice-2 (#2777 TASK-2-2 cross-reviewer NACK fix) the helper was + extended to ALSO write ``pipeline.pr_url`` and ``pipeline.pr_number`` + on the pipeline record. Three downstream consumers depend on these + pipeline-level fields: + + * :func:`_get_pr_info` at the pipeline-status endpoint + (``/api/v1/pipelines/<id>/status``) reports them. + * :meth:`PipelineToolHandler._make_pipeline_summary` (the MCP + ``get_pipeline_status`` tool) reports them. + * ``orchestrator.jira_reassess.pipelines_for_ticket_pr_url`` powers + the #1557 reverse-index in-flight detection that prevents the + Jira reassess sweep from re-mutating issues whose parent egg run + still has an open PR. + + Before this rewire the dedicated writer for the pipeline fields was + the deleted ``_finalize_pr_phase_failed`` (TASK-2-2 of #2777 + deleted it lock-step with the PR phase). Without the explicit + rewrite each of the three consumers above would silently report + ``None``. + + ``pr_url`` is synthesised from ``pipeline.repo`` + ``pr_number`` + when not supplied (the idempotent ``gh pr list`` hit only carries + the number; the create_pr path knows the URL directly from gh's + stdout). The synthesis mirrors GitHub's canonical PR URL shape and + keeps ``_get_pr_info``'s regex parse working unchanged. + + Persistence surface (egg-reviewer non-blocking #3): + + ``save_contract`` is a file-level atomic write — it rewrites + the contract on disk but does NOT commit-and-push it to the + worktree branch. The legacy + ``_persist_context_pr_linkage_on_contract`` (slice-2 deletes + it) wrapped the save in + ``_commit_statefiles_to_worktree`` + ``push_worktree_branch``; + the new opener intentionally does NOT, because the opener + runs at the canonical advance_phase REST site BEFORE + ``_spawn_pipeline_run_thread`` spawns the runner. That makes + the on-disk write durable for the runner's first read, but + the runner's ``_sync_worktree_with_remote`` has hard-reset + paths that can later wipe an uncommitted contract change. + Convergence is by the four runner-side backstops (slice-loop + entry, implement-entry backstop, ``_run_pipeline`` auto- + advance, HITL resume), which call the opener again — its + ``gh pr list`` idempotency hit re-persists ``context_pr_number`` + on disk after a reset. Across the full lifecycle the persisted + value converges; within a single advance_phase call the helper + is best-effort-on-disk-pending-runner-commit, not transactional. + + Raises: + ContextPrCreationError: when the contract cannot be loaded or + saved. Unlike the soft-fail legacy helper this propagates + so the caller surfaces a typed failure rather than leaving + the contract out-of-sync with GitHub. + """ + try: + from egg_contracts.loader import load_contract, save_contract + except ImportError as imp_err: + raise _pkg.ContextPrCreationError( + "egg_contracts.loader unavailable while persisting context_pr_number", + reason="loader_unavailable", + cause=imp_err, + ) from imp_err + + try: + with _pkg.get_pipeline_state_lock(pipeline_id): + contract_local = load_contract(identifier, worktree_repo_path) + if contract_local.pr is None: + # The contract MUST have a PR record by the time we + # reach the plan→implement boundary — populate writes + # it from the plan's ``pr:`` block. Missing PRMetadata + # here is a structural failure, not a persistence + # nuance; surface it loudly. + raise _pkg.ContextPrCreationError( + "contract has no PRMetadata; cannot persist " + "context_pr_number (populate-from-plan must run first)", + reason="missing_pr_metadata", + ) + contract_local.pr.context_pr_number = pr_number + save_contract(contract_local, worktree_repo_path) + + # Pipeline-level mirror (#2777 cross-reviewer NACK fix). + # Load → mutate → save under the same lock so the contract + # write and pipeline write are atomic for downstream + # observers (status endpoint, MCP tool, jira_reassess). + # Pull the state store via the same lazy-import pattern the + # rest of pipelines.py uses; the soft-fail import shape is + # intentional so a stripped-down test harness that mocks + # only the contract loader does not crash here. + # ``get_state_store`` requires the repo path explicitly + # (state_store.py:1356); pass ``worktree_repo_path`` so the + # store resolves under the same root we just wrote the + # contract to. + try: + from state_store import get_state_store # type: ignore[no-redef] + except ImportError: + from ..state_store import get_state_store # type: ignore[no-redef] + store = get_state_store(worktree_repo_path) + try: + reloaded = store.load_pipeline(pipeline_id) + except Exception as pipe_load_err: # noqa: BLE001 + # Don't fail the whole opener because the pipeline + # mirror couldn't be loaded — the contract write + # already succeeded above. Log + continue so the + # context PR opens; the mirror will be re-applied + # on the next idempotent opener tick. + _pkg.logger.warning( + "Context PR opener: could not mirror pipeline.pr_url / " + "pipeline.pr_number (continuing — contract write succeeded)", + pipeline_id=pipeline_id, + pr_number=pr_number, + error=str(pipe_load_err), + ) + return + mirror_url = pr_url + if mirror_url is None: + # Idempotent path (``gh pr list`` hit) only carries the + # number; synthesise the canonical PR URL from + # pipeline.repo + pr_number so all three consumers + # still see a populated ``pr_url`` string. Skip the + # synthesis when ``repo`` is unset (local-mode + # pipelines have no remote PR). + if reloaded.repo: + mirror_url = f"https://github.com/{reloaded.repo}/pull/{pr_number}" + reloaded.pr_number = pr_number + if mirror_url: + reloaded.pr_url = mirror_url + store.save_pipeline(reloaded) + except _pkg.ContextPrCreationError: + raise + except Exception as save_err: # noqa: BLE001 + raise _pkg.ContextPrCreationError( + f"failed to persist context_pr_number={pr_number}: {save_err}", + reason="save_failed", + cause=save_err, + ) from save_err + + +def _refresh_context_pr_body( + pipeline_id: str, + *, + pipeline: Any, + spawner: Any, + worktree_repo_path: Path, + identifier: int | str, + gateway_mode: str = "public", +) -> bool: + """Re-compose and push the context PR's body to GitHub (#3122). + + Called by the run loop after a slice PR opens and its number is + persisted on the contract, so the context PR's slice table gains a + link to each slice PR as the stack materialises + (:func:`_compose_context_pr_body` renders ``— #N`` for every slice + with a recorded ``pr_number``). + + The context PR body is machine-owned: the refresh fully regenerates + it from contract + pipeline state through the same composer the + opener used, clobbering any manual edits. Best-effort by design — + a body refresh is cosmetic, so every failure (contract load, + composition, gateway) logs a warning and returns ``False`` without + raising; no slice outcome may depend on it. + + **Concurrency contract**: the caller must hold + ``get_pipeline_state_lock(pipeline_id)`` for the entire load + + compose + push sequence — without it, two slices completing in the + same wave can interleave so the slice whose refresh lands later + clobbers a body that already included both links. Because no + later slice fires a refresh after the last one, the final slice's + ``— #N`` link would stay missing forever if the race fired on it. + Serializing inside the per-pipeline lock eliminates the race; the + sole production caller (``_run_implement_phase_slices``) already + holds it. + """ + if not pipeline.repo: + return False + + try: + from egg_contracts.loader import load_contract + + contract = load_contract(identifier, worktree_repo_path) + except Exception as load_err: # noqa: BLE001 + # Lazy import + contract load: ImportError, loader validation + # errors, OSError on the contract file read. + _pkg.logger.warning( + "Context PR body refresh: contract load failed (skipping)", + pipeline_id=pipeline_id, + error=str(load_err), + ) + return False + + context_pr_number = ( + contract.pr.context_pr_number if contract.pr else None + ) or pipeline.pr_number + if not context_pr_number: + # No context PR to refresh — reachable on #3100-degraded + # contracts where the opener never persisted linkage. + return False + + try: + body = _compose_context_pr_body( + contract=contract, + pipeline=pipeline, + worktree_repo_path=worktree_repo_path, + identifier=identifier, + ) + except Exception as compose_err: # noqa: BLE001 + # Pure string composition over loaded state; a raise here is a + # programming error, but the cosmetic-refresh contract still + # holds — log and skip rather than fail the slice. + _pkg.logger.warning( + "Context PR body refresh: composition failed (skipping)", + pipeline_id=pipeline_id, + pr_number=context_pr_number, + error=str(compose_err), + ) + return False + + return spawner.gateway.update_pr_body( + pipeline_id, + pipeline.repo, + pr_number=context_pr_number, + body=body, + issue_number=pipeline.issue_number, + # Attribute the action in the gateway audit log; matches + # sibling orchestrator-driven PR mutations (create_slice_pr, + # rebase_onto). + agent_role="orchestrator", + mode=gateway_mode, + ) + + +def _open_context_pr_at_implement_start( + pipeline_id: str, repo_path: Path | None = None +) -> int | None: + """Hard-required, idempotent up-front context PR opener (#2777, cq-4). + + Single up-front context-PR opener for the plan→implement boundary. + Replaces the soft-fail ``_maybe_open_base_pr_for_plan_to_implement`` + wrapper (deleted by slice-2 TASK-2-1 in #2777) that swallowed every + gateway failure with ``return None`` and the four retry-point call + sites it required. Under the new topology the context PR is + ``egg/<id>/work → main`` (rather than a dedicated + ``egg/<id>/context`` branch) and is opened ONCE at the plan→implement + transition; the slice stack cascades onto it. + + Behaviour: + + 1. Look up the pipeline + worktree from ``pipeline_id``. + 2. If the pipeline has neither ``repo`` nor ``base_branch`` set + (local mode), return ``None`` without raising — there is no + remote PR to open. This matches the legacy wrapper's silent-skip + behaviour for local pipelines so the new hard-required contract + does not regress in-house test pipelines. A ``repo`` with no + ``base_branch`` is the normal "auto-detect the default branch" + state (#3031), NOT a misconfiguration: the base is resolved via + :func:`_detect_default_branch` and used for the lookup + create. + A ``base_branch`` with no ``repo`` is a genuine misconfiguration + and raises ``ContextPrCreationError(reason="missing_repo")``. + 3. Otherwise call ``GatewayClient.lookup_open_pr(head, base)`` — the + same control-plane idempotency primitive ``create_slice_pr`` uses + — to find the open PR whose head is the pipeline's work branch and + whose base is the pipeline's base branch. The gateway runs the + narrow ``gh pr list --head --base --state open`` filter server-side + (launcher auth, ``/api/v1/gh/find_open_pr``), so both PR-idempotency + sites share one seam instead of this opener enumerating every open + PR and filtering client-side (#2934). On hit, persist the PR number + via :func:`_persist_context_pr_number` and return it (no + ``gh pr create`` invocation). + 4. On miss, read ``contract.pr.title`` and compose the body via + :func:`_compose_context_pr_body` (#3115) — the planner's + ``description`` plus rendered ``test_plan`` / ``manual_steps`` + and a generated pipeline-context footer (issue, slice table, + analysis/plan draft + BRC transcript links on the work branch). + Call ``GatewayClient.create_pr`` to open the PR, persist the PR + number, and return it. + + Raises: + ContextPrCreationError: on any of (a) pipeline lookup failure, + (b) contract load failure / missing PR metadata, + (c) an unexpected ``lookup_open_pr`` failure (the primitive + itself soft-fails a transient gateway/parse error to ``None``, + so this only fires on a programming error), (d) ``create_pr`` + failure, (e) persistence failure. NO soft-fail + ``return None`` for any of these — + the failure must reach the BRC NACK / 422 surface so the + operator sees the failure rather than silently stranding + the slice stack on ``/work``. The test in TASK-3-8 asserts + no swallow path exists. + + Returns: + Existing or newly-created PR number on the happy path, OR + ``None`` ONLY when the pipeline legitimately has no remote + (local mode). The two outcomes are disambiguated by inspecting + the pipeline's ``repo`` / ``base_branch`` ahead of the call; + the run-loop never needs to branch on ``None`` because + local-mode pipelines never reach the slice loop with remote + operations queued. + + Idempotency contract: + Calling the function twice for the same pipeline is safe — the + second call sees the already-open PR via ``lookup_open_pr`` and + re-persists the number through :func:`_persist_context_pr_number`. + No second ``create_pr`` invocation occurs. Tests in TASK-3-8 + verify this by asserting ``create_pr`` is called zero times on + the idempotent path AND ``_persist_context_pr_number`` IS + called with the existing PR number. + """ + # Step 1: resolve the pipeline + worktree path. ``get_state_store_for_pipeline`` + # handles the multi-repo case so the opener works the same way the + # legacy wrapper did from every call site. + try: + from routes import get_state_store_for_pipeline, resolve_worktree_path + except ImportError as imp_err: + raise _pkg.ContextPrCreationError( + "routes helpers unavailable while resolving pipeline", + reason="routes_unavailable", + cause=imp_err, + ) from imp_err + + try: + store, pipeline = get_state_store_for_pipeline(pipeline_id, repo_path=repo_path) + except Exception as load_err: + raise _pkg.ContextPrCreationError( + f"pipeline {pipeline_id!r} could not be loaded: {load_err}", + reason="pipeline_load_failed", + cause=load_err, + ) from load_err + + # Step 2: local-mode short-circuit + base-branch resolution. + # + # ``repo`` AND ``base_branch`` both empty ⇒ local mode (no remote PR + # to open); return ``None`` without raising. + # + # ``repo`` set but ``base_branch`` empty is the NORMAL state, not a + # misconfiguration: ``Pipeline.base_branch`` defaults to ``None`` + # ("auto-detected from repo's default branch") and the standard + # ``submit_task`` path never populates it, so essentially every + # remote pipeline reaches here with ``base_branch=None``. #2777 cq-4 + # collapsed the final ``work → main`` PR into this up-front opener + # but dropped the default-branch resolution the deleted PR phase did, + # making the opener the only ``base_branch`` consumer that hard- + # raised on ``None`` instead of resolving it — stranding every + # standard pipeline's slice stack on ``/work`` (#3031). Resolve it + # here the way every other consumer does + # (``base_branch or _detect_default_branch``) and thread the resolved + # value through both the idempotency lookup and ``create_pr``. + # + # A ``base_branch`` set with no ``repo`` IS a genuine + # misconfiguration (nothing to open a PR against); surface it as a + # typed error so the operator notices. + repo_set = bool(pipeline.repo) + base_set = bool(pipeline.base_branch) + if not repo_set and not base_set: + _pkg.logger.info( + "Context PR opener: skipping local-mode pipeline (no repo, no base_branch)", + pipeline_id=pipeline_id, + ) + return None + if base_set and not repo_set: + raise _pkg.ContextPrCreationError( + f"pipeline {pipeline_id!r} has a base_branch " + f"({pipeline.base_branch!r}) but no repo; cannot open a context " + "PR with no remote", + reason="missing_repo", + ) + + if not pipeline.branch: + # A remote pipeline without a configured work branch is a + # structural failure; raise so the operator notices instead of + # silently skipping (which would re-introduce the soft-fail + # behaviour cq-4 explicitly removes). + raise _pkg.ContextPrCreationError( + f"pipeline {pipeline_id!r} has no branch set; cannot open context PR", + reason="missing_branch", + ) + + worktree_repo_path = resolve_worktree_path(pipeline_id, store.repo_path) + # Resolve ``base_branch=None`` to the repo's default branch (#3031). + # ``_detect_default_branch`` reads ``origin/HEAD`` from the worktree + # and falls back to ``main``/``master`` then the literal ``"main"``, + # so it never raises and always yields a concrete base ref for the + # lookup + create_pr calls below. + effective_base = pipeline.base_branch or _pkg._detect_default_branch(worktree_repo_path) + identifier = _pkg._pipeline_identifier(pipeline.issue_number, pipeline_id) + gateway_mode, _vis = _pkg._compute_gateway_mode(pipeline) + + # Step 3: idempotency pre-flight. Reuse the same control-plane + # ``lookup_open_pr(head, base)`` primitive the per-slice path + # (``create_slice_pr``) uses, so both PR-idempotency sites share the + # narrow server-side ``gh pr list --head --base`` filter on the + # launcher-auth route rather than this opener enumerating every open + # PR and filtering client-side (#2934). ``lookup_open_pr`` returns a + # clean ``int | None`` (the head/base discrimination and number + # coercion happen server-side + in the primitive), so the client-side + # match loop and the malformed-``number`` guard the old + # ``list_open_prs`` path needed are gone. The primitive soft-fails a + # transient gateway/parse error to ``None`` — matching the slice path, + # and safe because ``gh pr create`` would reject a duplicate + # ``head → base`` PR server-side anyway. The ``try`` is the opener's + # typed-error backstop for an unexpected raise (e.g. a misconfigured + # gateway client), preserving the cq-4 no-raw-exception contract. + spawner = _pkg._get_spawner() + try: + existing_pr_number = spawner.gateway.lookup_open_pr( + pipeline_id=pipeline_id, + repo=pipeline.repo, + head=pipeline.branch, + base=effective_base, + ) + except Exception as lookup_err: + raise _pkg.ContextPrCreationError( + f"gateway lookup_open_pr failed for context-PR idempotency check: {lookup_err}", + reason="lookup_failed", + cause=lookup_err, + ) from lookup_err + + if existing_pr_number is not None: + # Idempotent path. Persist the number even though it MAY + # already be on the contract: the resume-from-orphaned-pipeline + # case (contract lost ``context_pr_number`` mid-run) recovers + # here. The TASK-3-8 unit test asserts the persistence call. + _pkg._persist_context_pr_number( + pipeline_id, + existing_pr_number, + worktree_repo_path=worktree_repo_path, + identifier=identifier, + ) + _pkg.logger.info( + "Context PR opener: idempotent hit on existing PR (no create_pr call)", + pipeline_id=pipeline_id, + pr_number=existing_pr_number, + head=pipeline.branch, + base=effective_base, + ) + _maybe_open_secondary_context_prs( + pipeline_id, + pipeline=pipeline, + primary_pr_number=existing_pr_number, + work_branch=pipeline.branch, + worktree_repo_path=worktree_repo_path, + identifier=identifier, + gateway_mode=gateway_mode, + spawner=spawner, + ) + return existing_pr_number + + # Step 4: open a new context PR. Read title/description from the + # canonical ``contract.pr`` fields (populated from the plan's + # ``pr:`` block by ``_populate_contract_from_plan``). + try: + from egg_contracts.loader import load_contract + except ImportError as imp_err: + raise _pkg.ContextPrCreationError( + "egg_contracts.loader unavailable while reading PR metadata", + reason="loader_unavailable", + cause=imp_err, + ) from imp_err + + try: + contract = load_contract(identifier, worktree_repo_path) + except Exception as load_err: + raise _pkg.ContextPrCreationError( + f"failed to load contract for {identifier!r}: {load_err}", + reason="contract_load_failed", + cause=load_err, + ) from load_err + + if contract.pr is None or not (contract.pr.title or "").strip(): + raise _pkg.ContextPrCreationError( + "contract.pr.title is missing or empty; cannot open context PR", + reason="missing_pr_metadata", + ) + pr_title = contract.pr.title.strip() + # #3115: render the full context-PR body (description + test plan + + # manual steps + generated pipeline-context footer) instead of the + # bare ``contract.pr.description``. + pr_body = _compose_context_pr_body( + contract=contract, + pipeline=pipeline, + worktree_repo_path=worktree_repo_path, + identifier=identifier, + ) + + try: + pr_url = spawner.gateway.create_pr( + pipeline_id=pipeline_id, + repo=pipeline.repo, + title=pr_title, + body=pr_body, + head=pipeline.branch, + base=effective_base, + issue_number=pipeline.issue_number, + mode=gateway_mode, + ) + except Exception as create_err: + raise _pkg.ContextPrCreationError( + f"gateway create_pr failed for context PR: {create_err}", + reason="gateway_error", + cause=create_err, + ) from create_err + + if not pr_url: + raise _pkg.ContextPrCreationError( + "gateway create_pr returned no URL; cannot derive context PR number", + reason="gateway_no_url", + ) + + # Extract the PR number from the URL — gh prints + # ``https://github.com/<owner>/<repo>/pull/<N>`` on stdout. + # Use a trailing-boundary pattern (end-of-string OR a non-digit + # path/query separator) so that a hypothetical + # ``/pull/12345/files`` or ``/pull/12345?diff=split`` URL still + # parses correctly but a digit-suffixed slug like + # ``/pulled-files/12345`` cannot smuggle a wrong number through + # (reviewer_concurrency non-blocking #2 hardening). + match = re.search(r"/pull/(\d+)(?:[/?#]|$)", pr_url) + if not match: + raise _pkg.ContextPrCreationError( + f"could not parse PR number from create_pr URL: {pr_url!r}", + reason="gateway_bad_url", + ) + try: + new_pr_number = int(match.group(1)) + except (TypeError, ValueError) as parse_err: + raise _pkg.ContextPrCreationError( + f"could not coerce PR number from create_pr URL: {pr_url!r}", + reason="gateway_bad_url", + cause=parse_err, + ) from parse_err + + _pkg._persist_context_pr_number( + pipeline_id, + new_pr_number, + worktree_repo_path=worktree_repo_path, + identifier=identifier, + pr_url=pr_url, + ) + + _pkg.logger.info( + "Context PR opener: opened new PR at plan→implement boundary (#2777)", + pipeline_id=pipeline_id, + pr_number=new_pr_number, + head=pipeline.branch, + base=effective_base, + url=pr_url, + ) + _maybe_open_secondary_context_prs( + pipeline_id, + pipeline=pipeline, + primary_pr_number=new_pr_number, + work_branch=pipeline.branch, + worktree_repo_path=worktree_repo_path, + identifier=identifier, + gateway_mode=gateway_mode, + spawner=spawner, + ) + return new_pr_number + + +def _repos_with_slices(contract, pipeline) -> list[str]: + """Repos that own ≥1 slice — the lazy-per-repo participation set (#3393, slice-4). + + A repo *participates* (gets its own ``egg/<id>/work`` branch + context + PR) iff at least one slice resolves to it via + :func:`models.resolve_slice_repo`. The result is ordered by + ``pipeline.repos`` and de-duplicated; a submitted repo that ends up + owning no slices is excluded (operator ruling #1). For an N=1 pipeline + this returns the single repo. This is the invariant the context-PR + opener's per-repo iteration honours (task-4-2). + """ + try: + from models import resolve_slice_repo # type: ignore[no-redef] + except ImportError: + from ..models import resolve_slice_repo # type: ignore[no-redef] + + slices = getattr(contract, "slices", None) or [] + owning = {resolve_slice_repo(s, pipeline) for s in slices} + return [spec.repo for spec in (pipeline.repos or []) if spec.repo in owning] + + +def _maybe_open_secondary_context_prs( + pipeline_id: str, + *, + pipeline: Any, + primary_pr_number: int, + work_branch: str | None, + worktree_repo_path: Path, + identifier: int | str, + gateway_mode: str, + spawner: Any, +) -> None: + """Guarded, never-raising entry to the lazy per-repo context opener (#3393). + + No-op unless the pipeline coordinates more than one repo, so the N=1 + single-repo path in :func:`_open_context_pr_at_implement_start` + performs zero extra work (no contract load, no gateway calls) and is + byte-for-byte unchanged. Requires a resolvable primary repo + work + branch; both are guaranteed set on the multi-repo remote path that + reaches here (the opener already returned for local-mode pipelines). + """ + if len(getattr(pipeline, "repos", None) or []) <= 1: + return + primary_repo = pipeline.primary_repo + if not primary_repo or not work_branch: + return + try: + _open_secondary_context_prs( + pipeline_id, + pipeline=pipeline, + primary_repo=primary_repo, + primary_pr_number=primary_pr_number, + work_branch=work_branch, + worktree_repo_path=worktree_repo_path, + identifier=identifier, + gateway_mode=gateway_mode, + spawner=spawner, + ) + except Exception as sec_err: # noqa: BLE001 + _pkg.logger.warning( + "Lazy per-repo context PRs raised (continuing — primary context PR unaffected) (#3393)", + pipeline_id=pipeline_id, + error=str(sec_err), + ) + + +def _open_secondary_context_prs( + pipeline_id: str, + *, + pipeline: Any, + primary_repo: str, + primary_pr_number: int, + work_branch: str, + worktree_repo_path: Path, + identifier: int | str, + gateway_mode: str, + spawner: Any, +) -> dict[str, int]: + """Open the lazy per-repo context PRs for a multi-repo pipeline (#3393, slice-4 / task-4-2). + + :func:`_open_context_pr_at_implement_start` opens the PRIMARY repo's + context PR (``egg/<id>/work → base``) exactly as it always has. This + helper adds the *other* repos: it iterates the set of repos that own + ≥1 slice (via ``resolve_slice_repo`` over the contract's slices), + drops the primary, and for each remaining repo opens that repo's own + ``egg/<id>/work`` context PR (same branch naming, per repo). A + submitted repo with NO slices is skipped — lazy-per-repo, operator + ruling #1. Every opened context PR (primary + secondaries) then has + its body refreshed to cross-reference the sibling context PRs in the + other repos (``## Coordinated repos``). + + It is only invoked when ``len(pipeline.repos) > 1``; for an N=1 + pipeline the caller never reaches here, so the single-repo path is + byte-for-byte unchanged. + + Prerequisite / current limit (honest scope note): opening a context + PR in a secondary repo requires that repo's ``egg/<id>/work`` branch + to exist on its remote, which in turn needs a secondary-repo worktree + to push it. Threading the full repo set into worktree CREATION was + explicitly deferred by slice-3 (the worktree map is owner/repo-keyed + and list-shaped, but only the primary repo is materialised today), so + until that later wiring lands the secondary ``create_pr`` will + typically fail on a missing head branch. This helper therefore: + + * uses the launcher-auth ``lookup_open_pr`` idempotency primitive + (which works per-repo with no worktree) to ADOPT an already-open + secondary context PR, and + * ATTEMPTS ``create_pr`` otherwise, soft-failing (log, continue) so a + missing secondary branch never strands the pipeline. + + The iteration + cross-referencing structure is therefore complete and + forward-compatible: once secondary-repo worktree/branch creation is + wired, secondary context PRs open with no further change here. + + Every failure is caught and logged; the helper never raises. Returns + the ``{repo: pr_number}`` map of context PRs known after the pass + (always including the primary), for logging / tests. + """ + opened: dict[str, int] = {primary_repo: primary_pr_number} + + try: + from egg_contracts.loader import load_contract + except ImportError: + _pkg.logger.warning( + "Secondary context PRs: egg_contracts.loader unavailable (skipping) (#3393)", + pipeline_id=pipeline_id, + ) + return opened + + try: + contract = load_contract(identifier, worktree_repo_path) + except Exception as load_err: # noqa: BLE001 + _pkg.logger.warning( + "Secondary context PRs: contract load failed (skipping) (#3393)", + pipeline_id=pipeline_id, + error=str(load_err), + ) + return opened + + # Repos owning ≥1 slice (ordered by ``pipeline.repos``), minus the + # primary — the lazy-per-repo participation set (task-4-2). + secondary_repos = [r for r in _repos_with_slices(contract, pipeline) if r != primary_repo] + + if not secondary_repos: + # Multi-repo pipeline whose slices all resolve to the primary + # (e.g. no slice pinned a secondary repo). Nothing lazy to open. + return opened + + base_by_repo = {spec.repo: spec.base_branch for spec in (pipeline.repos or [])} + context_pr_title = ( + contract.pr.title.strip() + if contract.pr and (contract.pr.title or "").strip() + else f"{identifier} context" + ) + + for repo in secondary_repos: + # ``base_branch=None`` ⇒ the repo's default branch. Without a + # secondary worktree we cannot run ``_detect_default_branch`` + # here, so fall back to ``main`` (the create call resolves the + # real default server-side when base is omitted anyway). + base = base_by_repo.get(repo) or "main" + try: + existing = spawner.gateway.lookup_open_pr( + pipeline_id=pipeline_id, + repo=repo, + head=work_branch, + base=base, + ) + if existing is not None: + opened[repo] = existing + _pkg.logger.info( + "Secondary context PR: adopted existing PR (#3393)", + pipeline_id=pipeline_id, + repo=repo, + pr_number=existing, + ) + continue + + body = _compose_context_pr_body( + contract=contract, + pipeline=pipeline, + worktree_repo_path=worktree_repo_path, + identifier=identifier, + context_repo=repo, + sibling_context_prs=[ + {"repo": r, "number": n} for r, n in opened.items() if r != repo + ], + ) + pr_url = spawner.gateway.create_pr( + pipeline_id=pipeline_id, + repo=repo, + title=context_pr_title, + body=body, + head=work_branch, + base=base, + issue_number=pipeline.issue_number, + mode=gateway_mode, # type: ignore[arg-type] + ) + match = re.search(r"/pull/(\d+)(?:[/?#]|$)", pr_url or "") + if match: + opened[repo] = int(match.group(1)) + _pkg.logger.info( + "Secondary context PR: opened new PR (#3393)", + pipeline_id=pipeline_id, + repo=repo, + pr_number=opened[repo], + head=work_branch, + base=base, + ) + else: + _pkg.logger.warning( + "Secondary context PR: create returned no parseable URL (#3393)", + pipeline_id=pipeline_id, + repo=repo, + url=pr_url, + ) + except Exception as sec_err: # noqa: BLE001 + # Best-effort: a missing secondary ``egg/<id>/work`` branch + # (the deferred-worktree limit above) surfaces here as a + # gateway create failure. Log + continue so the primary + # context PR + slice stack are unaffected. + _pkg.logger.warning( + "Secondary context PR deferred (continuing) — secondary-repo " + "work branch likely absent until secondary worktree creation " + "is wired (#3393)", + pipeline_id=pipeline_id, + repo=repo, + error=str(sec_err), + ) + + # Cross-reference pass: refresh every opened context PR body so each + # links the sibling context PRs in the other repos. Best-effort and + # cosmetic — a failed refresh never affects the slice stack. + if len(opened) > 1: + for repo, number in opened.items(): + try: + body = _compose_context_pr_body( + contract=contract, + pipeline=pipeline, + worktree_repo_path=worktree_repo_path, + identifier=identifier, + context_repo=repo, + sibling_context_prs=[ + {"repo": r, "number": n} for r, n in opened.items() if r != repo + ], + ) + spawner.gateway.update_pr_body( + pipeline_id=pipeline_id, + repo=repo, + pr_number=number, + body=body, + issue_number=pipeline.issue_number, + mode=gateway_mode, # type: ignore[arg-type] + ) + except Exception as refresh_err: # noqa: BLE001 + _pkg.logger.warning( + "Coordinated-repos cross-reference refresh failed (continuing) (#3393)", + pipeline_id=pipeline_id, + repo=repo, + error=str(refresh_err), + ) + + return opened diff --git a/orchestrator/routes/pipelines/_criteria.py b/orchestrator/routes/pipelines/_criteria.py new file mode 100644 index 0000000000..603f01390f --- /dev/null +++ b/orchestrator/routes/pipelines/_criteria.py @@ -0,0 +1,961 @@ +"""Review-criteria builders for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pre-split barrel. The only patched module +global these reach is ``_read_shared_criteria`` (and ``logger``); both are +reached through ``import routes.pipelines as _pkg`` so +``patch("routes.pipelines._read_shared_criteria")`` keeps intercepting. +""" + +from __future__ import annotations + +from pathlib import Path # noqa: F401 -- used by _read_shared_criteria + +import routes.pipelines as _pkg # noqa: E402 -- package barrel for patch seams + + +def _read_shared_criteria( + filename: str, + user_override: str | None = None, + repo_path: str | None = None, +) -> str | None: + """Read shared criteria from file, checking user override first. + + Search order: + 1. .egg/<user_override> in the repo (if user_override provided) + 2. shared/prompts/<filename> relative to source tree + 3. /app/prompts/<filename> (Docker container path) + + Returns the file content, or None if no file found (caller uses inline fallback). + """ + # Check user override first + if user_override and repo_path: + override_path = Path(repo_path) / ".egg" / user_override + if override_path.is_file() and override_path.stat().st_size > 0: + return override_path.read_text() + + # Try source tree path (development / tests) + source_path = Path(__file__).parent.parent.parent.parent / "shared" / "prompts" / filename + if source_path.is_file(): + return source_path.read_text() + + # Try Docker container path (production) + docker_path = Path("/app/prompts") / filename + if docker_path.is_file(): + return docker_path.read_text() + + return None + + +def _get_agent_design_criteria() -> str: + """Return agent-mode design review criteria.""" + content = _pkg._read_shared_criteria("agent-design-criteria.md") + if content is not None: + return content + _pkg.logger.warning("Shared agent-design-criteria.md not found, using inline fallback") + return ( + "Flag these **clear** anti-patterns:\n\n" + "1. **Excessive pre-fetching** — Baking large diffs (10KB+) or full file contents " + "into prompts instead of letting the agent fetch what it needs\n" + "2. **Structured output for humans** — Requiring JSON when output goes directly " + "to humans rather than machines\n" + "3. **Post-processing pipelines** — Scripts that parse agent output to take actions " + "the agent could take directly\n" + "4. **Rigid procedures** — Micromanaging step-by-step procedures when objectives " + "would suffice\n" + "5. **Prompt-level security** — Using instructions for constraints that should be " + "sandbox-enforced\n" + "6. **Direct LLM API calls outside sandbox** — Calling the Anthropic API from " + "orchestrator, gateway, or shared code instead of delegating to sandbox containers\n" + "7. **Direct API calls bypassing the Agent SDK** — Using raw HTTP calls to the " + "Anthropic API instead of run_agent() (in-sandbox) or build_agent_command() " + "(orchestrator-spawned containers). Unlike item 6 (scoped to infra code), " + "this applies everywhere including sandbox code.\n" + "8. **Hardcoded model identifiers** — Using full model IDs (date-pinned or " + "version-pinned) instead of short aliases (sonnet, opus, haiku)\n" + ) + + +def _get_code_review_criteria(repo_path: str | None = None) -> str: + """Return code review criteria.""" + content = _pkg._read_shared_criteria( + "code-review-criteria.md", + user_override="review-rules.md", + repo_path=repo_path, + ) + if content is not None: + return content + _pkg.logger.warning("Shared code-review-criteria.md not found, using inline fallback") + return ( + "### Security (highest priority)\n" + "- Injection vulnerabilities (SQL, command, XSS, LDAP, path traversal)\n" + "- Authentication/authorization flaws\n" + "- Credential exposure, hardcoded secrets\n" + "- SSRF, open redirects, unsafe deserialization\n\n" + "### Correctness\n" + "- Logic errors, off-by-one, boundary conditions\n" + "- Race conditions, deadlocks, concurrency bugs\n" + "- Null/undefined handling, missing error paths\n" + "- Resource leaks (connections, file handles, memory)\n" + "- End-to-end feature functionality: verify new features work in their " + "real execution environment\n\n" + "### Robustness\n" + "- Missing input validation at trust boundaries\n" + "- Unhandled exceptions that could crash the system\n" + "- Missing retry logic for transient failures\n" + "- Inadequate timeouts for external calls\n\n" + "### Design\n" + "- Violations of existing codebase patterns\n" + "- Breaking changes to public interfaces\n" + "- Tight coupling that will hinder future changes\n\n" + "### Severity Classification\n\n" + "**Blocking** (request changes):\n" + "- Security vulnerabilities\n" + "- Non-functional features — the feature's core purpose does not work " + "end-to-end\n" + "- Logic errors that produce incorrect results\n" + "- Breaking changes to existing functionality\n" + "- Resource leaks or crashes\n" + "- Pre-existing broken or inconsistent behavior in code the PR " + "modifies\n\n" + "**Non-blocking** (suggestions):\n" + "- Code quality improvements (naming, structure, duplication)\n" + "- Defense-in-depth additions\n" + "- Missing edge case handling that doesn't affect the core feature\n" + "- Documentation gaps\n" + "- Style or convention deviations not caught by linters\n\n" + "**Do not dismiss issues as 'not a regression'**: If a PR modifies " + "code that has existing broken or inconsistent behavior, the issue is " + "blocking even if the PR didn't introduce it. A PR that adds a new " + "code path through already-inconsistent logic makes the inconsistency " + "worse.\n\n" + "**Beware of false analogies**: When comparing new code to existing " + "patterns, verify the analogy holds at the execution-model level. " + "Two features may look structurally similar in config but have " + "completely different execution paths. If the existing pattern works " + "via mechanism A but the new code relies on mechanism B that doesn't " + "exist, the comparison is invalid — classify based on actual " + "functionality, not superficial similarity.\n\n" + "### Skip\n\n" + "- Style issues handled by linters (formatting, import order)\n" + "- Type annotation completeness (type checkers handle this)\n" + "- Auto-generated files (migrations, lock files)\n" + "- `.egg-state/` pipeline artifacts (contracts, drafts, BRC history " + "— managed by the orchestrator)\n" + ) + + +def _get_contract_review_criteria(repo_path: str | None = None) -> str: + """Return contract verification criteria.""" + content = _pkg._read_shared_criteria( + "contract-review-criteria.md", + user_override="contract-rules.md", + repo_path=repo_path, + ) + if content is not None: + return content + _pkg.logger.warning("Shared contract-review-criteria.md not found, using inline fallback") + return ( + "### Task Verification\n" + "For each task in the contract, verify:\n" + "1. The described functionality is present in the code\n" + "2. The acceptance criteria for the task is satisfied\n" + "3. If a commit is linked, verify it relates to the task\n" + "4. Where applicable, tests cover the new functionality\n\n" + "### Phase Consistency\n" + "- All tasks in completed phases are actually implemented\n" + "- Phase status matches task completion state\n" + "- No orphaned code exists that isn't covered by any task\n\n" + "### Acceptance Criteria Verification\n" + "For each acceptance criterion:\n" + "1. Examine the implementation to verify it meets the criterion\n" + "2. Note any gaps in your review\n\n" + "### Contract Integrity\n" + "- No implementation changes violate previously verified criteria\n" + "- New changes don't break existing contract compliance\n" + "- All required files listed in tasks are present\n" + ) + + +def _get_refine_review_criteria() -> str: + """Return review criteria for the dedicated refine reviewer.""" + return ( + "### 1. Problem Understanding\n" + "- Does the analysis correctly identify the core problem or feature request?\n" + "- Is the current behavior (if applicable) accurately described?\n" + "- Are the goals and desired outcomes clear?\n\n" + "### 2. Research Quality\n" + "- Has the agent explored the relevant parts of the codebase?\n" + "- Are existing patterns and conventions identified?\n" + "- Is the technical context accurate and thorough?\n\n" + "### 3. Options Analysis\n" + "- Are the proposed options meaningfully different?\n" + "- Are trade-offs clearly articulated for each option?\n" + "- Is the reasoning logical and well-founded?\n\n" + "### 4. Constraints and Dependencies\n" + "- Are technical constraints identified (performance, compatibility, etc.)?\n" + "- Are dependencies on other code or systems noted?\n" + "- Are potential risks or complications surfaced?\n\n" + "### 5. Open Questions\n" + "- Are open questions specific enough for a human to answer?\n" + "- Do questions address genuine ambiguities?\n" + "- Are questions actionable?\n" + "- **Does each question require a human, or could the planner decide it?** " + "NACK questions that ask about work decomposition / slice-DAG shape / " + "PR packaging — those belong to the plan phase's HITL gate, not the " + "refine gate. NACK questions about implementation strategy " + "(API shape, migration approach, fallback design, detector design) " + "unless the answer is a fact only the operator knows (product intent, " + "scope boundary, external commitment, user-visible behavior). Good " + "refine questions are about *what the problem is* and *what's in/out " + "of scope*; the planner handles *how to build it*.\n\n" + "### 6. Recommendation Quality\n" + "- Is there a clear recommended approach?\n" + "- Is the recommendation justified with specific reasons?\n" + "- Does the recommendation align with the analysis findings?\n\n" + "### 7. HITL Decision Registration & Un-surfaced Decisions (#3390)\n" + "- Run `egg-contract show` and verify a contract decision or feedback " + "item exists for every open question in the analysis, and that each " + "decision-bearing section cites its `cq-N` (the `--format markdown` " + "output of `egg-contract add-decision` embeds it). Open questions as " + "bare prose with no registered `cq-N` ⇒ **NACK** — the producer must " + "register each via `egg-contract add-decision` / " + "`egg-contract add-feedback` and re-propose. (Deterministic " + "propose-time checks already validate the producer's *attested* ids; " + "your job is the judgment half the validators cannot do.)\n" + "- **Un-surfaced decisions — NACK.** Read the draft for choices it " + "quietly *commits to* that should be the operator's call — e.g. " + '"we will drop the legacy filter", a scope narrowing/widening, a ' + "user-visible behavior change, abandoning a stated requirement — " + "with no registered `cq-N` backing the choice. Consensus must not " + "close on a draft that bakes in a human-grade decision outside the " + "HITL channel: the producer either registers the decision (the gate " + "then surfaces it) or rewrites the draft to remove the unilateral " + "commitment.\n" + "- **Calibration — do not over-NACK.** An implementation choice the " + "planner can make from the analysis (API shape, migration approach, " + "fallback design, detector shape) is NOT a human-grade decision; do " + "not force registration of those. The bar is the same as §5: answers " + "only the operator owns (product intent, scope boundaries, external " + "commitments, user-visible behavior).\n" + "- If the ledger is deliberately empty (the producer attested " + "`no_decisions_rationale`), verify the rationale holds: requirements " + "genuinely unambiguous, no assumptions made silently. NACK if you " + "find a hidden operator-grade choice.\n" + "- **Task-named decisions — NACK an explicit-none ledger (#3462).** " + "If the task description names decisions as the operator's to make " + "(or directs that decisions be surfaced as HITL questions), each " + "must have a registered `cq-N` — even when the draft argues prior " + 'context already resolves it. "Already resolved" is a recommended ' + "disposition to register (recommended option citing the resolving " + "context), not a reason to skip; a `no_decisions_rationale` " + "attestation on such a task is a **NACK** regardless of how " + "defensible the rationale reads.\n\n" + + _human_companion_review_criteria( + companion="`*-analysis-human.md`", + parent="the refine analysis", + producer="refiner", + ) + ) + + +def _human_companion_review_criteria(*, companion: str, parent: str, producer: str) -> str: + """Forcing checklist for verifying the simplifier's human companion. + + Shared by the refine and plan reviewer rubrics so the companion is + judged at VERDICT time (#3381), not merely mentioned in the reviewer's + "while waiting" preparation text. The simplifier is a producer-only role + whose companion is gated CRITICAL by this reviewer; the companion is the + only artifact with no automated content check, so this reviewer is the + sole gate on its format. Walk every item before ACKing the **simplifier** + (this section governs the simplifier's proposal, never the {producer}'s). + """ + return ( + f"### Human-Focused Companion ({companion} — the simplifier's " + "proposal)\n" + f"The simplifier produces {companion}, a plain-language companion to " + f"{parent} for a **broad audience — engineers, PMs, and managers**. " + "It is gated CRITICAL by you and has no automated content check, so " + "you are the only gate on its format. **You must walk this checklist " + "and answer every item before you ACK the simplifier** (this is a " + f"separate verdict from your review of the {producer}; NACK the " + "**simplifier**, never the " + f"{producer}, for companion defects):\n" + f"1. **Is it a summary, not a review?** Open {companion} and the full " + f"{parent} side-by-side. NACK if the companion reads as a " + "review/critique of the draft rather than a summary of it — any " + 'verdict/scoring framing ("verdict", "what I verified", "I ' + 'affirm", "sound", ACK/NACK language), any directives aimed at a ' + 'later phase ("the plan should commit to", "don\'t let the plan ' + 'inflate", "guardrails", "anti-pattern to reject"), or any ' + "constraint lists. The companion explains the change to a human; it " + "does not judge it.\n" + "2. **Is it free of implementation minutiae?** NACK if it contains " + "`file:line` references (e.g. `foo.go::Bar` / `L209`), function / " + "struct / field / type names or other code identifiers, or per-field " + "enumerations. It should describe behaviour and user-visible impact, " + "not the code.\n" + "3. **Is it materially lighter than the parent?** NACK if it is a " + "near-copy or as long/dense as the full draft. It must be " + "substantially shorter and more digestible — plain prose and short " + "lists.\n" + "4. **Is it readable by a non-engineer?** NACK if a PM or manager " + "could not follow *what is changing and why it matters*, or if it " + "leaks egg-internal jargon (BRC, consensus, slice-DAG, contract, " + "phase, agent-role terms).\n" + f"5. **Is it faithful?** NACK if it misrepresents {parent}, omits a " + "material point, or introduces new scope/claims.\n" + "A missing or empty companion is a NACK — it is mandatory.\n" + ) + + +def _get_first_principles_review_criteria() -> str: + """Return review criteria for the adversarial first-principles reviewer. + + The escalation instructions interpolate the accept-path's sentinel option + labels from ``routes.decisions`` so the labels the agent writes (here) and + the labels the resolve hook matches stay a single source of truth — they + cannot drift. Lazy import avoids a module-load cycle. + """ + from routes.decisions import ( + FIRST_PRINCIPLES_ADOPT_OPTION, + FIRST_PRINCIPLES_CANCEL_OPTION, + FIRST_PRINCIPLES_PROCEED_OPTION, + ) + + return ( + "You are the **first-principles reviewer**. Your subject is the " + "pipeline's **seed** — the operator's task statement (run " + "`egg-contract show` and read `task_description`, plus the linked " + "issue) — and the **direction** the refiner's analysis is taking. You " + "judge whether the *premise is sound and the direction is " + "appropriate*, NOT the quality of the analysis — that is " + "`reviewer_refine`'s job, so do not duplicate it.\n\n" + "### 1. Interrogate the premise\n" + "- Is the stated problem real, and is solving it worth the work?\n" + "- Is the premise contradicted by what's actually in the codebase — " + "the thing it proposes to build already exists, or the problem is " + "already handled?\n" + "- Will the stated direction actually achieve the stated goal, or does " + "it solve something adjacent?\n\n" + "### 2. Surface significant redirects (where warranted)\n" + "Raise a redirect only when you can name a concrete, evidence-backed " + "alternative — never a vague 'have you considered'. Valid redirects:\n" + "- A **materially simpler path** that achieves the same goal.\n" + "- A **fundamentally different approach** that is better on the " + "merits.\n" + "- A **scope change** — widen it if the seed under-reaches the real " + "goal, narrow it if it over-reaches.\n" + "- **Don't build it** — the work is unnecessary, already solved, or " + "solves a non-problem.\n" + "Back each redirect with evidence: a codebase fact (`file:line`), the " + "seed's own stated goal, or a specific contradiction. Raising more " + "than one is fine — it is acceptable to be relatively noisy — but " + "consolidate related concerns and hold every one to the " + "concrete-and-evidenced bar.\n\n" + "### 3. What NOT to raise (stay in your lane)\n" + "- Analysis-quality issues (research depth, option trade-offs, " + "completeness) — `reviewer_refine` owns those.\n" + "- Work decomposition, slice-DAG shape, PR packaging, or " + "implementation strategy (API shape, migration approach) — those " + "belong to the plan phase and the planner.\n" + "- Taste, stylistic preference, or 'did you consider X' with no " + "concrete better alternative.\n" + "If the premise and direction are sound, say so briefly and ACK — a " + "clean pass is a common and correct outcome. Do not manufacture an " + "objection to look diligent.\n\n" + "### 4. How to act — escalate, never NACK\n" + "- **Never NACK the refiner on first-principles grounds.** A NACK only " + "re-runs the refiner, which cannot change the operator-owned seed; " + "premise and direction are the operator's call, not the refiner's to " + "fix.\n" + "- When you have a redirect, **file one phase-scoped HITL decision** " + "via the `mcp__sdlc__register_open_question` tool so the operator can " + "act on it with one click (the **accept-path**). Pass these args:\n" + ' - `phase`: `"refine"`.\n' + " - `question`: state the concern, then the concrete redirect and " + "why (operator-facing prose).\n" + " - `options`: these EXACT labels, in this order — do NOT paraphrase, " + "the orchestrator matches them verbatim to drive the accept-path: " + f'`["{FIRST_PRINCIPLES_ADOPT_OPTION}", ' + f'"{FIRST_PRINCIPLES_PROCEED_OPTION}", ' + f'"{FIRST_PRINCIPLES_CANCEL_OPTION}"]`.\n' + " - `redirect_seed`: the FULL rewritten seed — the complete " + "`task_description` as it should read if the operator adopts your " + "redirect (not a diff, not just the objection). This rides the same " + "RPC that files the decision, so the orchestrator can read it back " + "directly; do NOT write it to a free-standing file (a reviewer " + "worktree has no path to carry one to the orchestrator).\n" + " On the operator's choice the orchestrator will: **adopt** → rewrite " + "the seed to your `redirect_seed` and re-run the refine phase against " + "it; **proceed** → leave the direction unchanged; **don't build** → " + "cancel the pipeline. If you have only an objection with no concrete " + "alternative direction, you do not have a redirect — do not file the " + "decision (omit `redirect_seed`).\n" + "- Then **ACK the refiner**: your first-principles pass is done and " + "any concerns are filed for the operator. Your ACK does not endorse " + "the direction — it records that you reviewed it; the open decision " + "independently holds the refine→plan gate until the operator resolves " + "it.\n" + ) + + +def _get_plan_review_criteria() -> str: + """Return review criteria for the dedicated plan reviewer.""" + return ( + "### 1. Alignment with Analysis\n" + "- Does the plan implement the recommended approach from the analysis?\n" + "- If the plan deviates from the analysis, is the reason explained?\n" + "- Are all requirements from the analysis addressed?\n\n" + "### 2. Task Breakdown\n" + "- Are tasks discrete, actionable, and properly scoped?\n" + "- Is each task small enough to implement in a single pass?\n" + "- Are task boundaries clear (no overlapping responsibilities)?\n\n" + "### 3. Acceptance Criteria\n" + "- Does each task have clear, testable acceptance criteria?\n" + "- Are criteria specific enough to verify completion?\n" + "- Do criteria cover both happy path and edge cases?\n\n" + "### 4. Dependency Ordering\n" + "- Are task dependencies correctly identified?\n" + "- Is the ordering logical (foundations before features)?\n" + "- Are there opportunities for parallelism that are missed?\n\n" + "### 5. Risk Assessment\n" + "- Are technical risks identified (security, performance, compatibility)?\n" + "- Are mitigation strategies concrete and actionable?\n" + "- Is the rollback plan realistic?\n\n" + "### 6. Test Strategy\n" + "- Is the test strategy appropriate for the scope of changes?\n" + "- Are both unit and integration tests considered?\n" + "- Are test scenarios aligned with acceptance criteria?\n\n" + "### 7. Completeness\n" + "- Does the plan cover all aspects of the original request?\n" + "- Are documentation updates included where needed?\n" + "- Are there any obvious gaps or missing tasks?\n\n" + "### 8. Task Role ↔ Files Alignment (deterministic, see #2527)\n" + "- Task role↔files alignment is enforced **orchestrator-side** at " + "`CONSENSUS_PROPOSE`: a planner proposal whose task `role:` " + "assignments cannot push their `files:` (per " + "`shared/egg_restrictions/patterns.py`, the same blocklist the " + "gateway uses) is rejected with HTTP 400 before the proposal " + "reaches you. By the time you act on a `CONSENSUS_PROPOSE`, " + "structural role↔files alignment is therefore already validated — " + "no manual check is required for this dimension.\n" + "- If you want belt-and-suspenders verification, you can run the " + "validator yourself against the proposed plan: " + '`python3 -c "from egg_contracts.plan_parser import parse_plan_file, ' + "validate_task_role_alignment as v; r = parse_plan_file('<plan-path>'); " + "print('\\n'.join(v(r.to_contract_slices())))\"`. " + "Errors here would predict a push-time `403 " + "restricted_path_modified` — NACK the planner and quote the " + "structured errors verbatim if any surface.\n\n" + "### 9. Primitive-Existence Audit (hard NACK, see #2594)\n" + "Plans are cheap to NACK at this phase and expensive to NACK " + "at implement-phase (8+ pod spawns per slice, ~60–90 min " + "wall clock per implement cycle). For #2474, a single " + "`grep -rn ScriptedProvider sandbox/ k8s/ orchestrator/` " + "returning zero hits would have prevented ~10.7 h of " + "compute. Do that grep **now**.\n\n" + "For every primitive the plan names — class, function, HTTP " + "route, env var, ConfigMap key, test fixture, CLI flag, " + "decorator — produce a small evidence table in your review " + "document. Example shape:\n\n" + "| primitive | kind | grep | result |\n" + "|-----------|------|------|--------|\n" + "| `ScriptedProvider` | class | `grep -rn 'class ScriptedProvider' sandbox/ k8s/ orchestrator/` | 0 hits → NACK |\n" + "| `orchestrator_url` fixture | fixture | `grep -rn 'def orchestrator_url' integration_tests/` | `integration_tests/local_pipeline/conftest.py:NN` — sibling, not parent (see §10) |\n\n" + "Prescribed greps by kind:\n" + "- **class / function**: `grep -rn '<NAME>' <relevant dirs>` " + "finds at least one definition site.\n" + "- **HTTP route**: blueprint registers the path + method the " + "plan uses (search `orchestrator/routes/` and `gateway/`).\n" + "- **env var / ConfigMap key**: a consumer the plan assumes " + "actually reads it.\n" + "- **test fixture**: defined in a conftest **reachable from " + "the test's directory** (parent vs sibling matters — see §10).\n" + "- **CLI flag**: parser registers it.\n\n" + "**NACK rule**: any named primitive whose grep returns zero " + "hits in the directories the plan implies is a hard NACK. " + "Quote the failed command verbatim in your verdict so the " + "planner can re-draft. If the primitive exists but in a " + "different form than the plan assumes (different module, " + "different signature, different scope — e.g. unit-test-only " + "vs deployed-pod), NACK and quote the actual `file:line`.\n\n" + "**Exception — `(NEW — task TASK-X-Y)` annotations.** Plans " + "introduce new primitives by design; the producer prompt " + "tells the planner to mark such primitives " + "`(NEW — task TASK-X-Y)` so the audit doesn't false-NACK " + "the very task that creates them. When you see this " + "annotation: **do not NACK on missing-grep evidence**. " + "Instead verify that the referenced task's acceptance " + "criteria genuinely create the primitive in the form the " + "plan uses (right kind, right module, right scope), and " + "that downstream tasks consuming the primitive depend on " + "the creating task. NACK only if the creating task does " + "not actually produce the primitive or the dependency " + "ordering is wrong.\n\n" + "### 10. Trust-Boundary Audit (hard NACK, see #2594)\n" + "Some primitives exist but are not available in the " + "execution context the plan assumes. The canonical example: " + "`ScriptedProvider` is a unit-test-only fake; deployed agent " + "pods (`sandbox/`) run the real provider, so a k3s " + "integration test cannot inject canned LLM trajectories " + "into a deployed pod without separate infra work. The " + "`integration_tests/` fixture layout encodes a parallel " + "distinction along the **pytest-fixture** axis: the " + "`gateway_url` and `orchestrator_url` fixtures are both " + "defined only in `integration_tests/local_pipeline/conftest.py` " + "and both transitively depend on `local_pipeline_stack`, " + "which `pytest.skip`s when kubectl is unavailable. The " + "parent `integration_tests/conftest.py` exposes `egg_stack` " + "(also kubectl-gated) — `egg_stack.gateway_url` is an " + "attribute on the `EggStack` dataclass, not a standalone " + "fixture. There is no `in-sandbox-agent`-runnable pytest " + "fixture in `integration_tests/` today; the in-sandbox-agent " + "tier reaches the gateway via the `GATEWAY_URL` env at " + "agent runtime, which is a separate surface from pytest " + "fixtures.\n\n" + "For each task that interacts with the orchestrator, " + "gateway, or k3s cluster, identify the **execution context** " + "and confirm the named primitives are available in that " + "context:\n\n" + "- **in-sandbox-agent** — driven by an egg agent pod. " + "Production code the agent writes reaches gateway-mediated " + "routes via the `GATEWAY_URL` env var. No `orchestrator_url`. " + "No lifecycle-secret-gated routes. Cannot inject " + "ScriptedProvider into a pod. **No pytest fixture in " + "`integration_tests/` resolves here today** — every fixture " + "is kubectl-gated and skips in the sandbox.\n" + "- **trusted-CI-runner** — driven by pytest from outside " + "the cluster (CI / dev machine running `make test` against " + "k3s). Sees every pytest fixture in `integration_tests/` " + "(parent and `local_pipeline/`), including `gateway_url`, " + "`orchestrator_url`, lifecycle-secret-gated routes, and " + "`kubectl` pod-log access. Test files live under " + "`integration_tests/` (gateway-only) or " + "`integration_tests/local_pipeline/` (orchestrator-scoped).\n" + "- **human-operator** — manual / `egg-orch` CLI. Not a " + "test-execution context; flag any task that implicitly " + "requires this.\n\n" + "See " + "`docs/architecture/integration-test-trust-boundary.md` " + "for the authoritative tier → fixture / route mapping.\n\n" + "**NACK rule**: if a task's named primitives are not " + "available in its declared (or implied) execution context, " + "NACK and name the specific mismatch. Common forms — NACK " + "each one:\n\n" + '- "task TASK-1-8 writes an in-sandbox-agent pytest test ' + "depending on the `gateway_url` fixture, but that fixture is " + '`trusted-CI-runner`-only and skips when kubectl is absent"\n' + '- "task TASK-2-3 places a test that imports ' + "`orchestrator_url` under `integration_tests/foo/` — pytest " + "resolves fixtures lexically from the nearest conftest " + "upward, so a sibling of `local_pipeline/` cannot see that " + 'fixture and the test fails at collection time"\n' + '- "task TASK-3-1 calls a `@require_lifecycle_secret` route ' + "from an `in-sandbox-agent`-context handler — " + "`EGG_LIFECYCLE_SECRET` is not present in sandbox pods, so " + 'the route returns 403"\n' + '- "task TASK-4-2 references `ScriptedProvider` from ' + "`sandbox/` (or any deployed-pod path) — it is a unit-test " + "double under `shared/tests/`, not a runtime-injectable " + 'provider"\n\n' + "### 11. Slice Sizing (hard NACK, judgment-based — see #2809)\n" + "Slice sizing is owned by the **architect**, not the " + "task_planner. ``reviewer_plan`` is empowered AND required to " + "hard-NACK the architect when a slice is oversized for one " + "BRC cycle. This is a separate rubric key from the slice-DAG " + "shape checks so the NACK is unambiguously routed to the " + "architect for slice re-shaping (re-spawn ``architect`` with " + "the subdivision feedback).\n\n" + "**No fixed tasks-per-slice budget.** Use judgment. NACK when " + "any of the following holds:\n\n" + "- A single slice touches **more than ~3 distinct " + "file-categories** (e.g. orchestrator + gateway + schema + " + "tests + docs all in one slice probably wants subdivision).\n" + "- A single slice combines **deletion-heavy work** with " + "**new-API-introduction work** — these usually want different " + "review attention and ship better as separate slices.\n" + "- A single slice would require the implementing producer to " + "**commit-propose-revise more than 3–4 times** to converge " + "(typical signal: many independent commit clusters with " + "different reviewer surfaces).\n" + "- A single slice contains **independent task groups with no " + "internal dependency** — natural seams for parallel " + "sub-slices.\n\n" + "**NACK format**: name the seam where subdivision is " + "appropriate so the architect's re-propose is actionable. " + "Examples:\n\n" + '- "slice-1 bundles gateway allowlist edits, orchestrator ' + "route handlers, and shared/egg_contracts schema changes — " + "three distinct file-categories with different reviewer " + "surfaces. Subdivide along the gateway / orchestrator " + '/ schema seam."\n' + '- "slice-2 bundles ~600 LOC of removals across "' + "orchestrator/* with ~200 LOC of new gateway-Jira routes — " + "deletion-heavy + new-API in one cycle. Ship the removals " + 'as one slice and the new routes as a downstream slice."\n' + '- "slice-3 contains 9 tasks across 4 independent feature ' + "areas (search, profile, settings, notifications) with no " + "cross-area dependency — subdivide into one slice per " + 'area."\n\n' + "The architect re-proposes with the subdivision applied (the " + "existing BRC re-review loop handles convergence). " + "task_planner re-consumes the revised " + "``architect-slices.yaml`` scaffold on the next BRC cycle. " + "**Refiner / operator can override sizing concerns** if there " + "is a deliberate reason to ship a large slice (e.g. atomic " + "schema migration that cannot be split safely) — in that " + "case the architect should cite the override in the analysis " + "and the reviewer can ACK once the rationale is on the " + "record.\n\n" + "### 12. Slice File-Overlap Ordering (deterministic hard NACK — see #3046)\n" + "Complements §11. When slices are subdivided, any two that touch " + "the **same file** must be **ordered** along one dependency chain — " + "one a transitive ``dependencies`` ancestor of the other — never " + "left as parallel roots or siblings. The implement phase cuts each " + "slice's integration branch off its dependency parent (roots off " + "``work``), so two overlapping slices with no edge between them fork " + "independently off the shared base and their edits to the shared " + "file collide at integration (a guaranteed modify/delete conflict — " + "the #3023 incident, where three slices all touched " + "``consensus_wrapper.py``, one deleting it).\n" + "This is enforced **orchestrator-side at plan ingestion**: an " + "overlapping-but-unordered DAG is rejected before the slices are " + "written to the contract, surfacing as a ``slice_overlap_violation`` " + "discriminator (or a 'Plan ingestion REJECTED: slices touch " + "overlapping files' block on ``plan_review_feedback``). When you see " + "it, NACK the **architect** and quote the structured errors " + "verbatim; instruct it to serialise the overlapping cluster into one " + "linear ``dependencies`` chain — a slice that deletes/retires a file " + "depends on every slice that modifies it — or to merge the slices. " + "Disjoint slices stay parallel so they still run concurrently.\n" + "Belt-and-suspenders self-check: " + '`python3 -c "from egg_contracts.plan_parser import parse_plan_file, ' + "validate_slice_file_overlap as v; r = parse_plan_file('<plan-path>'); " + "print('\\n'.join(v(r.to_contract_slices())))\"`.\n\n" + "### 13. Test Co-location (hard NACK — see #3411)\n" + "Complements §12 on the test dimension. When a slice removes, " + "renames, or rewrites code, the tests exercising that code must be " + "updated, removed, or skip-guarded **in the same slice** — never in " + "a later one. Every cumulative slice tip must be independently " + "green: the per-slice green gate (#3398) executes the repo's " + "checks at the slice tip before opening the PR and blocks while " + "any check is red, so a plan that parks test obsolescence in a " + "later slice guarantees gate blocks and repair-loop churn on " + "slices whose only sin is plan topology (the #3280 stack shipped " + "a 46-failure window across slices 3–4 exactly this way: slice-3 " + "removed ``spawn_overseer_*`` from the spawner, the tests " + "exercising them were only touched in slice-5).\n" + "For each slice whose tasks remove or rename symbols, check: do " + "the test files that statically reference those symbols appear in " + "that slice's task ``files:`` (or a ``dependencies`` ancestor's)? " + "If they appear only in a LATER slice — or nowhere — NACK the " + "**architect** (slice shape is architect-owned, #2809), naming " + "the code files, the referencing test files, and the slice each " + "currently sits in, so the re-propose moves the test updates into " + "the removing slice.\n" + "Belt-and-suspenders self-check (repos shipping the changeset-" + "aware selector; this repo does): `python3 " + "scripts/select_tests/__main__.py --impacted-tests <file>...` " + "prints every test file that transitively imports the named files " + "— the same import graph `make test` narrowing uses. Exit 2 means " + "the closure could not be computed: fall back to grepping the " + "removed symbols in the test trees, and never read empty output " + "on exit 2 as 'no impacted tests'.\n\n" + "### 14. HITL Decision Registration & Un-surfaced Decisions (#3390)\n" + "- Run `egg-contract show` and check the plan-phase decision ledger: " + "every plan-phase open question must be a registered contract " + "decision (`cq-N`), and the plan draft must cite the id where the " + "question is raised. A plan-grade question living only in prose ⇒ " + "**NACK** the producer that owns it (task_planner for the plan " + "draft, architect for slice-shape questions, risk_analyst for " + "risk-acceptance questions).\n" + "- **Un-surfaced decisions — NACK.** A plan that silently commits to " + "a choice only the operator owns — dropping a requirement, changing " + "user-visible behavior, accepting a risk the operator never saw, " + "de-scoping acceptance criteria — without a registered `cq-N` bakes " + "a human-grade decision into the pipeline outside the HITL channel. " + "NACK: the producer registers the decision or removes the " + "unilateral commitment.\n" + "- **Calibration — do not over-NACK.** Design calls the plan phase " + "legitimately owns (task decomposition, API shape, migration " + "approach, slice ordering within the architect's constraints) are " + "NOT operator decisions — do not force registration of those. The " + "bar is answers only the operator owns (product intent, scope " + "boundaries, external commitments, user-visible behavior).\n" + "- A deliberately empty ledger arrives as a producer's " + "`no_decisions_rationale` attestation — verify it holds; NACK if " + "the plan hides an operator-grade choice.\n" + "- **Task-named decisions — NACK an explicit-none ledger (#3462).** " + "If the task description or refine analysis names decisions as the " + "operator's to make (or directs that decisions be surfaced as HITL " + "questions), each must have a registered `cq-N` — even when the " + 'plan argues prior context already resolves it. "Already ' + 'resolved" is a recommended disposition to register (recommended ' + "option citing the resolving context), not a reason to skip; a " + "`no_decisions_rationale` attestation on such a task is a **NACK** " + "regardless of how defensible the rationale reads.\n\n" + + _human_companion_review_criteria( + companion="`*-plan-human.md`", + parent="the implementation plan", + producer="task_planner", + ) + ) + + +def _get_security_review_criteria(repo_path: str | None = None) -> str: + """Return security-lens review criteria (issue #1965). + + The shared file inherits from ``code-review-criteria.md`` and adds + lens-specific rules (cross-file allowlist mismatches, + handler-vs-validator path mismatches, info-disclosure / authz bypass, + uncommitted-artifact mismatches, secret leakage, OWASP cross-file + patterns). Falls back to a short inline placeholder when the shared + file isn't available. + """ + content = _pkg._read_shared_criteria( + "security-review-criteria.md", + user_override="security-review-rules.md", + repo_path=repo_path, + ) + if content is not None: + return content + _pkg.logger.warning("Shared security-review-criteria.md not found, using inline fallback") + return ( + "Inherits from `code-review-criteria.md`; only lens-specific rules " + "below override or extend it.\n\n" + "### Security lens (focus areas)\n" + "- **Cross-file allowlist mismatch** — handler in one file references " + "a check defined / extended in a different file (the PR #1964 " + "`^project$` pattern).\n" + "- **Handler-vs-validator path mismatch** — verify the validator's " + "regex / allowlist actually covers every code path the handler " + "reaches.\n" + "- Information-disclosure and authorization-bypass patterns at " + "trust boundaries.\n" + "- Uncommitted-artifact / Dockerfile-symlink mismatches (the PR " + "#1964 `sandbox/scripts/jira` pattern).\n" + "- Secret leakage via logs, error text, environment dumps, or " + "version-controlled config.\n" + "- OWASP top-10 patterns spanning more than one changed file.\n" + ) + + +def _get_code_review_holistic_criteria(repo_path: str | None = None) -> str: + """Return holistic-lens review criteria (issue #2126). + + The shared file inherits from ``code-review-criteria.md`` and adds + holistic-lens rules (end-to-end use-case walk, doc↔code symmetry, + synthetic-key / sentinel cross-module audit, silent-fallback hunt). + """ + content = _pkg._read_shared_criteria( + "code-review-holistic-criteria.md", + user_override="code-review-holistic-rules.md", + repo_path=repo_path, + ) + if content is not None: + return content + _pkg.logger.warning("Shared code-review-holistic-criteria.md not found, using inline fallback") + return ( + "Inherits from `code-review-criteria.md`; only holistic-lens rules " + "below override or extend it.\n\n" + "### Holistic lens (focus areas)\n" + "- Walk the primary advertised use case end-to-end across the " + "full diff. NACK silent dead-ends like the `__checkout__` bug " + "on PR #2105.\n" + "- Cross-check doc-claimed behaviour against what the code does. " + "NACK doc-claimed inference / migration paths that do not exist.\n" + "- Audit synthetic keys, sentinels, and magic values for " + "cross-module agreement.\n" + "- Hunt silent fallbacks that swallow operator-visible " + "misconfiguration.\n" + "- Defer line-by-line correctness to `reviewer_code`.\n" + ) + + +def _get_concurrency_review_criteria(repo_path: str | None = None) -> str: + """Return concurrency-lens review criteria (issue #1965). + + The shared file inherits from ``code-review-criteria.md`` and adds + lens-specific rules (race conditions, deadlocks, shared-state + mutation, async-context leakage, retry storms, resource-cleanup + ordering, BRC-protocol invariants). + """ + content = _pkg._read_shared_criteria( + "concurrency-review-criteria.md", + user_override="concurrency-review-rules.md", + repo_path=repo_path, + ) + if content is not None: + return content + _pkg.logger.warning("Shared concurrency-review-criteria.md not found, using inline fallback") + return ( + "Inherits from `code-review-criteria.md`; only lens-specific rules " + "below override or extend it.\n\n" + "### Concurrency lens (focus areas)\n" + "- Race conditions and deadlocks.\n" + "- Shared-state mutation without proper synchronization.\n" + "- Async-context leakage and retry-storm patterns.\n" + "- Resource-cleanup ordering bugs.\n" + "- BRC-protocol invariants (send→wait ordering, cursor threading " + "per #1925, heartbeat-stall windows per #2012).\n" + ) + + +def _get_review_criteria_for_type( + reviewer_type: str, phase: str, repo_path: str | None = None +) -> str: + """Dispatch to the correct criteria function based on reviewer type.""" + if reviewer_type == "agent-design": + return _get_agent_design_criteria() + elif reviewer_type == "code": + return _get_code_review_criteria(repo_path=repo_path) + elif reviewer_type == "code-holistic": + return _get_code_review_holistic_criteria(repo_path=repo_path) + elif reviewer_type == "contract": + return _get_contract_review_criteria(repo_path=repo_path) + elif reviewer_type == "refine": + return _get_refine_review_criteria() + elif reviewer_type == "first-principles-reviewer": + return _get_first_principles_review_criteria() + elif reviewer_type == "plan": + return _get_plan_review_criteria() + elif reviewer_type == "security": + return _get_security_review_criteria(repo_path=repo_path) + elif reviewer_type == "concurrency": + return _get_concurrency_review_criteria(repo_path=repo_path) + else: + raise ValueError(f"Unknown reviewer type: {reviewer_type}") + + +def _get_reviewer_scope_preamble(reviewer_type: str, phase: str) -> str: + """Return a scope preamble that tells the reviewer what to focus on.""" + if reviewer_type == "agent-design": + return ( + "This is a specialized **agent-mode design review**. Focus ONLY on " + "agent-mode design principles. Do NOT review general code quality, " + "security, or correctness — other reviewers handle those.\n\n" + "**Only flag issues if you find clear agent-mode design anti-patterns.** " + "If the output has no agent-mode concerns, a brief approval is acceptable " + "— you do not need to produce a lengthy analysis when there are no concerns." + ) + elif reviewer_type == "code": + return ( + "This is a **comprehensive code review**. Focus on security, correctness, " + "and robustness. Agent-mode design alignment is handled by another reviewer.\n\n" + "**Be direct.** Do not soften feedback. State issues clearly and explain " + "why they matter.\n\n" + "**Be thorough.** Find ALL issues on the first pass. Do not stop after " + "identifying a few problems.\n\n" + "**Analysis format:** Provide file-by-file analysis covering each changed " + "file. For each file, note what changed, whether the change is correct, " + "and any issues or observations." + ) + elif reviewer_type == "code-holistic": + return ( + "This is a CRITICAL **holistic code review** (issue #2126). " + "You run alongside `reviewer_code` — your job is the " + "cross-module coherence question line-by-line review does not " + "own. **Don't verify every line; `reviewer_code` covers " + "that.**\n\n" + "**Lens scope:** read the diff once with the whole PR in mind, " + "then run all four passes from the criteria below: (1) walk " + "the primary advertised use case end-to-end (the `__checkout__` " + "dead-end on PR #2105 is the canonical miss); (2) check that " + "every doc-claimed behaviour is actually implemented and every " + "user-facing code path is documented; (3) confirm synthetic " + "keys / sentinels / magic values are recognised by every " + "consumer in another module; (4) hunt silent fallbacks " + "(`except Exception:`, swallowed `None`s, default no-op " + "branches) where the operator would expect a signal.\n\n" + "**Distinct CRITICAL role.** Your NACK gates consensus on its " + "own — it is not averaged against `reviewer_code`'s " + "verdict. If the architectural-coherence question fails, " + "NACK even when the line-by-line review is clean.\n\n" + "**Analysis format:** Name the pass that found the issue, the " + "producer / consumer modules the asymmetry spans, and the " + "user-visible failure shape. If all four passes come back " + "clean a concise ACK is acceptable, but the BRC bus enforces " + "a minimum content length on ACK / NACK bodies, so write at " + "least a sentence or two summarising what you checked." + ) + elif reviewer_type == "contract": + return ( + "This is a **contract verification review**. Verify that the implementation " + "matches the contract and all acceptance criteria are met. Do NOT review " + "general code quality or security — other reviewers handle those.\n\n" + "**Analysis format:** Provide a criterion-by-criterion verification — for each " + "acceptance criterion, state whether it is met and cite the specific evidence." + ) + elif reviewer_type == "refine": + return ( + "This is a **refine phase review**. Focus on the quality and completeness " + "of the analysis produced during the refine phase. Evaluate problem " + "understanding, codebase research, options analysis, and the recommended " + "approach. Agent-mode design alignment is handled by another reviewer.\n\n" + "**Analysis format:** Provide section-by-section evaluation of the refine " + "output — assess each major section for depth, accuracy, and completeness." + ) + elif reviewer_type == "first-principles-reviewer": + return ( + "This is an adversarial **first-principles review**. Focus ONLY on " + "whether the premise is sound and the direction appropriate — the " + "seed and where the refiner's analysis is heading. Do NOT review " + "analysis quality, code, or implementation detail; other agents " + "own those.\n\n" + "You escalate by surfacing HITL decisions for the operator, not by " + "NACKing the refiner. If the direction is sound, a brief approval " + "and ACK is the right outcome — do not manufacture an objection." + ) + elif reviewer_type == "plan": + return ( + "This is a **plan phase review**. Focus on the quality and completeness " + "of the implementation plan. Evaluate task breakdown, acceptance criteria, " + "dependency ordering, risk assessment, and test strategy. Agent-mode " + "design alignment is handled by another reviewer.\n\n" + "**Analysis format:** Provide section-by-section evaluation of the plan — " + "assess task decomposition, acceptance criteria quality, dependency ordering, " + "and risk coverage." + ) + elif reviewer_type == "security": + return ( + "This is a CRITICAL **security-lens review** (issue #2139). " + "A NACK from this lens blocks consensus until the producer " + "re-proposes. Focus ONLY on the security lens; defer code " + "quality, performance, and non-security findings to " + "`reviewer_code`.\n\n" + "**Lens scope:** cross-file allowlist mismatches, " + "handler-vs-validator path mismatches, information-disclosure / " + "authorization-bypass patterns at trust boundaries, " + "uncommitted-artifact / Dockerfile-symlink mismatches, secret " + "leakage, and OWASP top-10 patterns that span more than one " + "changed file. Be especially alert to allowlist-mismatch " + "patterns where a handler in one file accepts traffic that a " + "validator in another file was supposed to reject.\n\n" + "**Analysis format:** Provide a finding-by-finding lens report. " + "If the diff has no security concerns, a concise approval is " + "acceptable — verbose reports without findings are not required, " + "but the BRC bus enforces a minimum content length on ACK / " + "NACK bodies, so write at least a sentence or two summarizing " + 'what you checked (not a single-word "LGTM").' + ) + elif reviewer_type == "concurrency": + return ( + "This is a CRITICAL **concurrency-lens review** (issue #2139). " + "A NACK from this lens blocks consensus until the producer " + "re-proposes. Focus ONLY on the concurrency lens; defer code " + "quality, performance, and non-concurrency findings to " + "`reviewer_code`.\n\n" + "**Lens scope:** race conditions, deadlocks, shared-state " + "mutation without synchronization, async-context leakage, " + "retry-storm patterns, resource-cleanup ordering bugs, and " + "BRC-protocol invariants (send→wait ordering, cursor " + "threading per #1925, heartbeat-stall windows per #2012).\n\n" + "**Analysis format:** Provide a finding-by-finding lens report. " + "If the diff has no concurrency concerns, a concise approval is " + "acceptable — verbose reports without findings are not required, " + "but the BRC bus enforces a minimum content length on ACK / " + "NACK bodies, so write at least a sentence or two summarizing " + 'what you checked (not a single-word "LGTM").' + ) + else: + raise ValueError(f"Unknown reviewer type: {reviewer_type}") diff --git a/orchestrator/routes/pipelines/_decisions.py b/orchestrator/routes/pipelines/_decisions.py new file mode 100644 index 0000000000..fff7e174c3 --- /dev/null +++ b/orchestrator/routes/pipelines/_decisions.py @@ -0,0 +1,298 @@ +"""HITL + divergence-reconcile decision helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _format_nack_summary(nack_details: list[dict]) -> str: + """Format unresolved NACK details into a human-readable summary string.""" + return "; ".join( + f"{n['reviewer']} NACKed {n['producer']}: {n.get('reason') or 'no reason given'}" + for n in nack_details + ) + + +def _incomplete_consensus_decision_text( + final_consensus: dict, + container_failure_count: int, + orchestrator_mode: bool = False, +) -> tuple[str, str]: + """Build (question, log_suffix) for incomplete-consensus HITL escalation. + + Distinguishes the two failure modes — unresolved NACKs vs. agents that + never confirmed — so the operator sees actionable detail in `/sdlc`. + + ``orchestrator_mode`` selects a mode-aware prefix: when the orchestrator + owns the event loop, no up-front containers ever ran, so the terminal + here is the consensus timeout, not container exit — the "All containers + exited" prefix would mislead an operator reading `/sdlc`. + """ + nacks = final_consensus.get("unresolved_nacks", []) or [] + blocking = final_consensus.get("blocking_agents", []) or [] + if container_failure_count: + prefix = f"{container_failure_count} container(s) exited with non-zero code; " + elif orchestrator_mode: + prefix = "Consensus timed out; " + else: + prefix = "All containers exited; " + # Retry semantics must match what "Retry phase" actually executes on + # resolve — the restart_phase route (#3421 dispatch, #3080 preservation + # semantics): fresh worktrees re-fork from the shared work branch tip, + # and unpushed per-role commits survive only via best-effort salvage. + retry_copy = ( + "'Retry phase' re-runs the phase from the shared work branch tip " + "(work pushed to the shared branch is preserved; unpushed per-role " + "commits are salvaged best-effort to egg/recovered/*)." + ) + if nacks: + summary = _pkg._format_nack_summary(nacks) + question = ( + f"{prefix}consensus incomplete with {len(nacks)} unresolved NACK(s): " + f"{summary}. {retry_copy} How to proceed?" + ) + log_suffix = f"\n--- INCOMPLETE CONSENSUS / UNRESOLVED NACKs ({len(nacks)}) ---\n{summary}" + else: + agent_list = ", ".join(blocking) if blocking else "unknown" + question = ( + f"{prefix}consensus incomplete; agents never confirmed: {agent_list}. " + f"{retry_copy} How to proceed?" + ) + log_suffix = ( + f"\n--- INCOMPLETE CONSENSUS / NO CONFIRMATION ---\nblocking_agents={agent_list}" + ) + return question, log_suffix + + +def _persist_hitl_decision( + pipeline_id: str, + pipeline: _pkg.Pipeline, + store: _pkg.StateStore, + *, + question: str, + options: list[str], + phase: _pkg.PipelinePhase | None = None, + context: str | None = None, +): + """Create and persist an HITL decision under the pipeline state lock. + + `pipeline.add_decision()` only mutates an in-memory object. The caller + of `_run_concurrent_phase` reloads the pipeline fresh from disk before + writing FAILED, so any in-memory decision is silently dropped — the + on-disk state (which `/sdlc` reads via `pipeline.get_pending_decisions()`) + never sees it. This helper mirrors the *persistence half* of + `DecisionQueue.queue_decision()` and the HITL-gate write at + pipelines.py:13080-13089: load → mutate → save under the reentrant + pipeline state lock. Note: it intentionally does **not** invoke + `_notify_handlers` — no production code currently registers a + `DecisionHandler` and `/sdlc` reads from disk on each request, so + notifications are not needed for the issue-2203 path. The in-memory + `pipeline` argument is also synced so callers observe consistent state. + + ``context`` is set on the persisted decision before save so dispatch + handlers in :mod:`routes.decisions` can route on a stable string + discriminator rather than the prose-y ``question`` text (see the + ``failed_role:`` pattern). + + Returns the created decision, or None if persistence failed (logged; + callers should not raise — losing an HITL decision is bad but losing + the rest of the cleanup path is worse). + """ + try: + with _pkg.get_pipeline_state_lock(pipeline_id): + disk_pipeline = store.load_pipeline(pipeline_id) + decision = disk_pipeline.add_decision( + question=question, + options=options, + phase=phase or disk_pipeline.current_phase, + ) + if context is not None: + decision.context = context + store.save_pipeline(disk_pipeline) + # Defensive copy: avoid sharing the list reference with the + # disk-loaded copy, which is local and goes out of scope. + pipeline.decisions = list(disk_pipeline.decisions) + return decision + except Exception: + _pkg.logger.warning( + "Failed to persist HITL decision", + pipeline_id=pipeline_id, + question=question[:100], + exc_info=True, + ) + return None + + +def _cancel_consensus_timeout_decisions(pipeline: _pkg.Pipeline) -> int: + """Cancel any pending consensus-timeout HITL on ``pipeline`` (#3315 facet c). + + Pure mutator (no lock / load / save): marks every pending + ``consensus_timeout_incomplete`` decision ``CANCELLED`` with an + auto-withdrawal note and returns how many it cancelled. Called from the + consensus-success path (under the pipeline state lock, on the freshly + loaded pipeline that is about to be saved) so a stale forced-choice a + *superseded* thread opened before the phase converged is withdrawn in the + same write that marks the agents COMPLETE — the operator is never left + disposing of a decision the system already obsoleted by converging. + """ + withdrawn = 0 + for decision in pipeline.get_pending_decisions(): + if decision.context != _pkg._CONSENSUS_TIMEOUT_HITL_CONTEXT: + continue + decision.status = _pkg.DecisionStatus.CANCELLED + decision.resolution = "auto-withdrawn: consensus subsequently converged" + decision.resolved_at = _pkg.datetime.now(_pkg.UTC) + withdrawn += 1 + return withdrawn + + +def _withdraw_arms_exhausted_decisions(pipeline_id: str, store: _pkg.StateStore) -> int: + """Cancel any pending arms-exhausted HITL on ``pipeline_id`` (#3496 review). + + The symmetric counterpart to :func:`_persist_hitl_decision` on the + arms-exhausted path: when the wedge clears (the blocked arms recovered by + a route other than the operator resolving this decision — a fresh key + derived, a spawn succeeded, an unrelated decision re-keyed the arms) the + pending ``event_arms_exhausted`` decision is obsolete, so this withdraws it + rather than leaving the operator to dispose of a decision the system + already resolved for them (mirrors :func:`_cancel_consensus_timeout_decisions` + on the convergence-success path). + + Unlike ``_cancel_consensus_timeout_decisions`` — a pure mutator that + piggybacks on the convergence-success write already under the state lock — + the wedge-clear path has no ambient lock/load/save, so this does its own + load → cancel → save under ``get_pipeline_state_lock``. Returns how many + decisions were withdrawn (0 when none were pending, so the caller can skip + logging on the common no-op). + """ + from concurrent_executor import ARMS_EXHAUSTED_HITL_CONTEXT + + with _pkg.get_pipeline_state_lock(pipeline_id): + disk_pipeline = store.load_pipeline(pipeline_id) + withdrawn = 0 + for decision in disk_pipeline.get_pending_decisions(): + if decision.context != ARMS_EXHAUSTED_HITL_CONTEXT: + continue + decision.status = _pkg.DecisionStatus.CANCELLED + decision.resolution = ( + "auto-withdrawn: the wedge cleared (blocked arms recovered) " + "before this decision was resolved" + ) + decision.resolved_at = _pkg.datetime.now(_pkg.UTC) + withdrawn += 1 + if withdrawn: + store.save_pipeline(disk_pipeline) + return withdrawn + + +def _find_pending_divergence_reconcile_decision(pipeline: _pkg.Pipeline): + """Return the oldest pending reconcile HITL on ``pipeline`` (or None). + + Used by the non-blocking ``populate_contract`` route to dedupe re-POSTs + against a pipeline already paused on a reconcile HITL — without this, + every retry would append a fresh decision and bloat ``pipeline.decisions`` + (the abort path still works on the most recent decision; this is a UX / + cleanliness fix, not a correctness fix). + """ + for decision in pipeline.get_pending_decisions(): + if decision.context == _pkg._DIVERGENCE_RECONCILE_HITL_CONTEXT: + return decision + return None + + +def _divergence_reconcile_is_abort(resolution: str) -> bool: + """True when a reconcile-HITL resolution selects abort (#2979). + + Accepts the canonical ``Abort pipeline`` label, a couple of forgiving + synonyms, and the JSON ``{"action": ...}`` envelope the collaborator + UI sends. Any *other* resolution — the resume label, free text, an + empty string — is treated as "Reconciled — resume", so an ambiguous + resolution errs toward re-attempting the (now non-destructive) sync + rather than failing the pipeline. + """ + r = resolution.strip() + if not r: + return False + try: + payload = _pkg.json.loads(r) + if isinstance(payload, dict) and "action" in payload: + r = str(payload["action"]) + except _pkg.json.JSONDecodeError, TypeError: + pass + return r.strip().lower() in { + _pkg._DIVERGENCE_RECONCILE_ABORT.lower(), + "abort", + "cancel", + } + + +def _divergence_reconcile_hitl_question( + *, + pipeline_id: str, + phase: _pkg.PipelinePhase | None, + backup_ref: str | None, + local_only_commit_shas: tuple[str, ...] | list[str], + rebase_category: str | None = None, + rebase_detail: str | None = None, +) -> str: + """Build the HITL question for the non-destructive divergence pause (#2979). + + The worktree diverged from origin and the rebase autoresolve could + not reconcile it. Nothing has been discarded — the autoresolve + aborted back to the clean local HEAD, so the orchestrator's committed + work is intact — and the pipeline is paused (AWAITING_HUMAN, not + FAILED). The operator reconciles the orchestrator-side worktree + manually, then either resumes (the sync re-runs and the phase's + post-processing continues from where it paused) or aborts. + + ``rebase_category`` / ``rebase_detail`` name the actual autoresolve + failure (conflicting paths, rebase argv, git output excerpt) so the + operator can judge the pause from the decision alone (#3416). + """ + phase_label = phase.value if phase is not None else "current phase" + if rebase_category or rebase_detail: + failure_label = rebase_category or "unknown failure" + failure_line = ( + f"({failure_label}: {rebase_detail})" if rebase_detail else f"({failure_label})" + ) + else: + failure_line = ( + "(failure detail unavailable — see the divergence_rebase_failed " + "log line for the rebase output)" + ) + backup_line = ( + f"A backup ref pins the current tip: {backup_ref} (inspect with `git log {backup_ref}`)." + if backup_ref + else "Backup ref write failed — see the WARN log for the inlined commit SHAs." + ) + if local_only_commit_shas: + commits_block = "Local-only commits preserved on the worktree HEAD:\n - " + "\n - ".join( + local_only_commit_shas + ) + else: + commits_block = ( + "The local-only commit list could not be enumerated; check the " + "WARN log and the backup ref for the exact set." + ) + return ( + f"Pipeline {pipeline_id}: the worktree diverged from origin at the " + f"{phase_label} boundary and the rebase autoresolve could not " + f"reconcile it {failure_line}. " + f"Nothing was discarded — the worktree is left at the local HEAD " + f"with the orchestrator's committed work intact, and the pipeline " + f"is paused (not failed) for a manual reconcile (#2979). " + f"{backup_line}\n{commits_block}\n\n" + f"Reconcile the orchestrator-side worktree manually (e.g. rebase the " + f"local commits onto origin/<branch> and resolve the conflict), then " + f"choose:\n" + f"- '{_pkg._DIVERGENCE_RECONCILE_RESUME}' — re-run the worktree sync and " + f"resume the {phase_label} phase's post-processing from where it " + f"paused (no full phase re-run).\n" + f"- '{_pkg._DIVERGENCE_RECONCILE_ABORT}' — fail the pipeline; the backup " + f"ref preserves the commits for offline inspection." + ) diff --git a/orchestrator/routes/pipelines/_drafts.py b/orchestrator/routes/pipelines/_drafts.py new file mode 100644 index 0000000000..9be73b75af --- /dev/null +++ b/orchestrator/routes/pipelines/_drafts.py @@ -0,0 +1,758 @@ +"""Drafts helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pre-split barrel. Patched barrel globals are +reached through ``import routes.pipelines as _pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import subprocess # noqa: F401 +from pathlib import Path # noqa: F401 +from typing import Any # noqa: F401 + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _verdict_path_for_type( + phase: str, + reviewer_type: str, + issue_number: int | None = None, + pipeline_id: str | None = None, +) -> str: + """Return the relative verdict file path for a given reviewer type. + + Uses issue_number as prefix when available, otherwise pipeline_id. + """ + prefix = _pkg._pipeline_identifier(issue_number, pipeline_id or "unknown") + return f".egg-state/reviews/{prefix}-{phase}-{reviewer_type}-review.json" + + +def _draft_filename(phase: str) -> str | None: + """Return the draft filename for a phase, without any prefix. + + Centralises the phase-to-filename mapping so that + ``_get_draft_path`` and ``_get_generic_draft_path`` stay in sync. + """ + if phase == "refine": + return "analysis.md" + elif phase == "implement": + return None + else: + return f"{phase}.md" + + +def _get_draft_path( + phase: str, + issue_number: int | None = None, + pipeline_id: str | None = None, +) -> str | None: + """Return relative path to the draft file for a phase. + + Spec-driven (#3077 slice-3): the registered ``refine`` and ``plan`` + phases route through :func:`egg_contracts.artifact_spec.resolve_artifact_path` + so the registry is the single source of truth that propose-time + validation (:func:`orchestrator.routes.signals._validate_producer_artifacts`) + and every draft reader in this module share. Slice-2 of #3077 pins + the equality with a mandatory consistency test + (``TestConsistencyB_GetDraftPathEquality`` in + ``shared/egg_contracts/tests/test_artifact_spec.py``); the slice-3 + rewrite below makes that equality structural rather than incidental + — refine-risk-1's "no second copy of path knowledge" ratchet. + + Phases not yet registered in the spec (currently ``pr``) keep their + legacy path via the centralised ``_draft_filename`` mapping, so + pre-existing PR-phase callers stay byte-identical. ``implement`` + has no draft and falls out as ``None`` here. + + Uses ``issue_number`` as prefix when available, otherwise + ``pipeline_id``; falls back to ``"unknown"`` when neither is supplied. + """ + _SPEC_BY_PHASE = {"refine": "analysis-draft", "plan": "plan-draft"} + spec_name = _SPEC_BY_PHASE.get(phase) + if spec_name is not None: + # Lazy import: the spec module is pure Python and has no + # orchestrator/gateway deps, but importing it at module load + # time would still pull egg_contracts into pipelines.py's + # import graph regardless of whether _get_draft_path is called + # — keep the deferral so the import cost only lands on actual + # invocations. + from egg_contracts.artifact_spec import resolve_artifact_path + + identifier = _pkg._pipeline_identifier(issue_number, pipeline_id or "unknown") + return resolve_artifact_path(spec_name, identifier) + + filename = _draft_filename(phase) + if not filename: + return None + prefix = _pkg._pipeline_identifier(issue_number, pipeline_id or "unknown") + return f".egg-state/drafts/{prefix}-{filename}" + + +_HUMAN_SPEC_BY_PHASE = {"refine": "analysis-draft-human", "plan": "plan-draft-human"} + + +def _get_human_draft_path( + phase: str, + issue_number: int | None = None, + pipeline_id: str | None = None, +) -> str | None: + """Return the relative path to the human-focused companion draft. + + Returns ``None`` for phases without a registered human companion + (currently only ``refine`` and ``plan`` have one). + """ + spec_name = _HUMAN_SPEC_BY_PHASE.get(phase) + if spec_name is None: + return None + from egg_contracts.artifact_spec import resolve_artifact_path + + identifier = _pkg._pipeline_identifier(issue_number, pipeline_id or "unknown") + return resolve_artifact_path(spec_name, identifier) + + +def _cleanup_stale_generic_drafts(worktree_path: Path) -> bool: + """Remove unprefixed generic draft files from a worktree. + + Legacy pipelines left behind ``analysis.md`` and ``plan.md`` (without + an issue-number or pipeline-id prefix) in ``.egg-state/drafts/``. + These stale files can confuse downstream draft-reading logic. This + helper deletes only the exact unprefixed filenames; prefixed files + (e.g. ``1553-analysis.md``) are left untouched. + + Uses ``git rm`` so the deletions are staged and can be committed + immediately. Falls back to ``os.unlink`` if the file is untracked. + + Safe to call when the drafts directory does not exist (no-op). + + Returns ``True`` if a commit was made (i.e. tracked files were removed + and committed), ``False`` otherwise. + """ + drafts_dir = worktree_path / ".egg-state" / "drafts" + if not drafts_dir.is_dir(): + return False + + git_base = [ + "git", + "-c", + "core.hooksPath=/dev/null", + "-c", + f"safe.directory={worktree_path}", + "-C", + str(worktree_path), + ] + removed = False + + stale_names = ("analysis.md", "plan.md") + for name in stale_names: + stale = drafts_dir / name + if stale.exists(): + _pkg.logger.info( + "Removing stale generic draft", + path=str(stale), + ) + try: + subprocess.run( + [*git_base, "rm", "-f", str(stale.relative_to(worktree_path))], + capture_output=True, + text=True, + check=True, + timeout=10, + ) + removed = True + except subprocess.CalledProcessError as exc: + # File may be untracked — just delete it from disk. + # Warn so that unexpected git rm failures (e.g. index + # lock) are diagnosable. + _pkg.logger.warning( + "git rm failed for stale draft, falling back to unlink", + path=str(stale), + error=str(exc), + ) + stale.unlink(missing_ok=True) + + if removed: + try: + subprocess.run( + [ + *git_base, + "commit", + "--no-verify", + "-m", + "Remove stale generic draft files", + ], + capture_output=True, + text=True, + check=True, + timeout=30, + ) + return True + except subprocess.CalledProcessError as commit_err: + _pkg.logger.debug( + "No changes to commit after stale draft cleanup", + error=str(commit_err), + ) + + return False + + +def _get_generic_draft_path(phase: str) -> str | None: + """Return the generic (unprefixed) draft path for a phase. + + Used as a fallback when the issue-specific draft file is missing. + """ + filename = _draft_filename(phase) + if not filename: + return None + return f".egg-state/drafts/{filename}" + + +def _git_show_draft( + repo_path: Path, + branch: str, + rel_path: str, + timeout: int = 15, +) -> str | None: + """Read a file from ``origin/{branch}`` via ``git show``. + + Returns the file content as a string, or ``None`` if the file does + not exist on the remote ref or the git command fails. This is a + read-only operation that does not modify the worktree. + + Note: this function does **not** ``git fetch`` itself. The caller is + responsible for ensuring ``origin/{branch}`` is fresh (e.g., by + running ``git fetch origin {branch}`` before calling this helper). + """ + git_base = [ + "git", + "-c", + "core.hooksPath=/dev/null", + "-c", + f"safe.directory={repo_path}", + "-C", + str(repo_path), + ] + try: + result = subprocess.run( + [*git_base, "show", f"origin/{branch}:{rel_path}"], + capture_output=True, + text=True, + timeout=timeout, + check=False, + ) + if result.returncode == 0 and result.stdout: + return result.stdout + if result.returncode != 0: + _pkg.logger.debug( + "git show returned non-zero", + branch=branch, + rel_path=rel_path, + returncode=result.returncode, + stderr=result.stderr.strip()[:200], + ) + except Exception as exc: + _pkg.logger.debug( + "git show failed for draft", + branch=branch, + rel_path=rel_path, + error=str(exc), + ) + return None + + +def _read_source_branch_artifacts( + repo_path: Path, + source_branch: str, + issue_number: int | None, + pipeline_id: str, + store: Any, + pipeline: Any, + source_artifact_prefix: str | None = None, + spawner: Any | None = None, + gateway_mode: str = "public", +) -> bool: + """Read plan and analysis artifacts from a source branch. + + Reads draft files from ``origin/<source_branch>`` via ``git show``. + Only populates ``pipeline.plan`` and ``pipeline.analysis`` when they + are not already set (inline values take precedence). + + Prefix resolution order for the exact-path lookup: + + 1. ``source_artifact_prefix`` (explicit override, e.g. ``"issue-1570-v3"``) + 2. ``pipeline_id`` (includes qualifier, e.g. ``"issue-1570-v7"``) + 3. ``issue_number`` (bare issue number, e.g. ``1570``) + + Falls back to listing available files via ``git ls-tree`` when none + of the prefixes match. + + Args: + repo_path: Path to the repository (worktree or main). + source_branch: Branch name to read artifacts from. + issue_number: Pipeline issue number (for deriving prefix). + pipeline_id: Pipeline ID (includes qualifier when present). + store: StateStore instance for saving updated pipeline. + pipeline: Pipeline model instance to populate. + source_artifact_prefix: Explicit prefix override for draft + filenames on the source branch (e.g. ``"issue-1570-v3"``). + When set, only this prefix is tried before the ls-tree + fallback. + spawner: ContainerSpawner instance for gateway-authenticated git + operations. When provided, the fetch uses the gateway API + (which injects GitHub credentials) instead of a raw + ``git fetch`` that lacks auth in the sandboxed environment. + gateway_mode: Network mode for the gateway session (``"public"`` + or ``"private"``). + + Returns: + True if any artifacts were read, False otherwise. + """ + git_base = [ + "git", + "-c", + "core.hooksPath=/dev/null", + "-c", + f"safe.directory={repo_path}", + "-C", + str(repo_path), + ] + # Bare prefix is the issue number when available — used as a fallback + # after the full pipeline_id prefix. Do NOT use _pipeline_identifier() + # here because it returns pipeline_id for qualifier-tagged pipelines, + # which defeats the fallback chain (pipeline_id → bare issue number). + bare_prefix: int | str = issue_number if issue_number is not None else pipeline_id + updated = False + + # Fetch the source branch so origin/{source_branch} is up-to-date. + # Without this, git show fails because the remote ref isn't cached + # locally. Use the gateway-authenticated fetch when available — + # raw git commands in the sandboxed environment lack GitHub + # credentials (the gateway sidecar injects them). + if spawner is not None: + try: + spawner.gateway.fetch_branch( + pipeline_id=pipeline_id, + repo_path=str(repo_path), + args=[source_branch], + mode=gateway_mode, + ) + except Exception: + _pkg.logger.warning( + "Gateway fetch of source branch failed (will try git show anyway)", + source_branch=source_branch, + pipeline_id=pipeline_id, + exc_info=True, + ) + else: + # Fallback for tests or environments without a gateway. + try: + subprocess.run( + [*git_base, "fetch", "origin", source_branch], + capture_output=True, + text=True, + timeout=30, + check=False, + ) + except Exception: + _pkg.logger.debug( + "Failed to fetch source branch (will try git show anyway)", + source_branch=source_branch, + exc_info=True, + ) + + # Build ordered list of prefixes to try. Duplicates are removed so + # we don't hit git show twice for the same path. + if source_artifact_prefix is not None: + # Explicit override — try only this prefix before ls-tree fallback. + prefixes: list[str | int] = [source_artifact_prefix] + else: + # Default: try pipeline_id first (includes qualifier), then bare + # issue number. When pipeline_id == bare_prefix (e.g. no qualifier + # and no issue number), the dedup below collapses them. + prefixes = [] + if pipeline_id and str(pipeline_id) != str(bare_prefix): + prefixes.append(pipeline_id) + prefixes.append(bare_prefix) + + for field_name, suffix in [("analysis", "-analysis.md"), ("plan", "-plan.md")]: + # Skip if already populated (inline values take precedence). + # Use ``is not None`` so empty strings are not silently overwritten. + if getattr(pipeline, field_name) is not None: + continue + + drafts_prefix = ".egg-state/drafts/" + content = None + + # Try each prefix in order (exact path lookup). + for pfx in prefixes: + expected_path = f"{drafts_prefix}{pfx}{suffix}" + content = _pkg._git_show_draft(repo_path, source_branch, expected_path) + if content: + _pkg.logger.info( + "Read artifact from source branch (exact prefix)", + field=field_name, + source_branch=source_branch, + path=expected_path, + ) + break + + if content is None: + # Fallback: list available files and find a match + try: + result = subprocess.run( + [ + *git_base, + "ls-tree", + "--name-only", + f"origin/{source_branch}:{drafts_prefix.rstrip('/')}", + ], + capture_output=True, + text=True, + timeout=15, + check=False, + ) + if result.returncode == 0 and result.stdout.strip(): + matches = [f for f in result.stdout.strip().splitlines() if f.endswith(suffix)] + # Filter by issue number to avoid picking up artifacts + # from other issues on the same branch (#1654). + if issue_number is not None: + issue_matches = [f for f in matches if f.startswith(f"{issue_number}-")] + if issue_matches: + matches = issue_matches + else: + _pkg.logger.warning( + "No fallback match for issue number — skipping", + field=field_name, + issue_number=issue_number, + source_branch=source_branch, + available=matches, + ) + continue + if len(matches) > 1: + _pkg.logger.warning( + "Multiple fallback matches for artifact — using first", + field=field_name, + source_branch=source_branch, + matches=matches, + ) + for filename in matches: + fallback_path = f"{drafts_prefix}{filename}" + content = _pkg._git_show_draft(repo_path, source_branch, fallback_path) + if content: + _pkg.logger.info( + "Read artifact from source branch via fallback", + field=field_name, + source_branch=source_branch, + path=fallback_path, + ) + break + except Exception as exc: + _pkg.logger.debug( + "git ls-tree failed for source branch drafts", + source_branch=source_branch, + error=str(exc), + ) + + if content: + setattr(pipeline, field_name, content) + updated = True + _pkg.logger.info( + "Read artifact from source branch", + field=field_name, + source_branch=source_branch, + pipeline_id=pipeline_id, + length=len(content), + ) + + if updated: + # Clear source_branch after successful read to avoid re-reading on + # pipeline restart (same pattern as plan/analysis clearing after + # draft files are pushed). + pipeline.source_branch = None + pipeline.source_artifact_prefix = None + store.save_pipeline( + pipeline, message=f"Populate artifacts from source branch {source_branch}" + ) + else: + _pkg.logger.warning( + "No artifacts found on source branch", + source_branch=source_branch, + pipeline_id=pipeline_id, + source_artifact_prefix=source_artifact_prefix, + ) + + return updated + + +def _pull_contract_from_source_branch( + repo_path: Path, + source_branch: str, + issue_number: int | None, + pipeline_id: str, + spawner: Any | None = None, + gateway_mode: str = "public", + task_description: str | None = None, +) -> bool: + """Load a persisted contract from ``origin/<source_branch>`` into the worktree. + + When ``submit_task`` is called with ``source_branch``, the source branch + carries ``.egg-state/contracts/<pipeline>.json`` (with any resolved HITL + decisions). Without this helper, ``_run_pipeline`` calls + ``create_contract()`` unconditionally and overwrites those decisions with + a zero-state contract (#2035). This helper fetches the source branch, + reads the contract via ``git show``, rebinds its pipeline_id to the new + pipeline, and writes it into the worktree so the caller can skip + ``create_contract()`` and proceed to commit+push the pulled contract. + + ``task_description`` is the NEW submit's composed task statement + (``compose_task_description`` at the call site — identity anchor + + resubmit prompt, #3163). The pulled contract carries the SOURCE + pipeline's ``task_description``, but the new submit's statement is + authoritative for THIS pipeline and is where operators put binding + resume directives (e.g. "adopt prior branch X, do not reimplement" + — #3123). When non-empty it replaces the pulled value; the source + value stays recoverable from the source branch's git history. This + replacement is also what keeps a fork from leaking the source + pipeline's task into the new pipeline's per-event prompts: issue + and JIRA pipelines always compose a non-empty anchor, so the pulled + cross-pipeline text never survives. Only a free-text resume with a + blank prompt preserves the pulled value (a plain resume of the same + task). + + Returns True when a contract was successfully pulled, False otherwise. + Best-effort: missing, invalid, or unreachable source contracts all yield + False so the caller falls back to ``create_contract()``. + """ + from egg_contracts.loader import ( + ContractNotFoundError, + ContractValidationError, + load_contract_from_branch, + save_contract, + ) + + # Fetch the source branch so origin/<source_branch> is current. Mirrors + # the pattern in _read_source_branch_artifacts — use the gateway when + # available, fall back to raw git for tests / non-sandboxed callers. + if spawner is not None: + try: + spawner.gateway.fetch_branch( + pipeline_id=pipeline_id, + repo_path=str(repo_path), + args=[source_branch], + mode=gateway_mode, + ) + except Exception: + _pkg.logger.warning( + "Gateway fetch of source branch failed (will try git show anyway)", + source_branch=source_branch, + pipeline_id=pipeline_id, + exc_info=True, + ) + else: + try: + subprocess.run( + [ + "git", + "-c", + "core.hooksPath=/dev/null", + "-c", + f"safe.directory={repo_path}", + "-C", + str(repo_path), + "fetch", + "origin", + source_branch, + ], + capture_output=True, + text=True, + timeout=30, + check=False, + ) + except Exception: + _pkg.logger.debug( + "Failed to fetch source branch for contract pull", + source_branch=source_branch, + exc_info=True, + ) + + identifier: int | str = issue_number if issue_number is not None else pipeline_id + + try: + contract = load_contract_from_branch( + identifier, + repo_path, + branch=f"origin/{source_branch}", + ) + except ContractNotFoundError: + _pkg.logger.debug( + "No contract on source branch", + pipeline_id=pipeline_id, + source_branch=source_branch, + ) + return False + except ContractValidationError as e: + _pkg.logger.warning( + "Contract on source branch failed validation, falling back to fresh contract", + pipeline_id=pipeline_id, + source_branch=source_branch, + error=str(e), + ) + return False + except Exception: + _pkg.logger.warning( + "Failed to load contract from source branch", + pipeline_id=pipeline_id, + source_branch=source_branch, + exc_info=True, + ) + return False + + # Rebind to the new pipeline_id so save_contract writes under the new + # canonical key when the pipeline was forked with a qualifier + # (e.g. source=issue-1965, new=issue-1965-v2). + contract.pipeline_id = pipeline_id + # Refresh the task statement from the new submit (#3123/#3163): + # without this, the resubmit's composed statement — identity anchor + # plus any operator resume directives — never reaches any + # agent-visible surface, because the caller skips create_contract() + # (the only other writer of ``task_description``) whenever the pull + # succeeds. Issue/JIRA pipelines always compose non-blank (the + # anchor at minimum), so the replace also prevents a fork from + # carrying the SOURCE pipeline's task text into this pipeline's + # per-event prompts. A blank/None value (free-text resume with no + # new prompt) preserves the pulled value so the source pipeline's + # task statement still drives the resumed run. + if task_description is not None and task_description.strip(): + contract.task_description = task_description + save_contract(contract, repo_path) + + _pkg.logger.info( + "Loaded contract from source branch", + pipeline_id=pipeline_id, + source_branch=source_branch, + decision_count=len(contract.decisions), + phase_count=len(contract.slices), + ) + return True + + +def _read_phase_draft( + repo_path: Path, + phase: str, + issue_number: int | None = None, + pipeline_id: str | None = None, + max_chars: int = 32000, + branch: str | None = None, +) -> str | None: + """Read draft file contents. Truncates at max_chars. + + Returns None when the draft cannot be found (no path configured or + file missing on disk). + + Attempts in order: + + 1. Primary (issue-specific) path on disk + 2. Generic (unprefixed) path on disk + 3. Primary path via ``git show origin/{branch}:`` + 4. Generic path via ``git show origin/{branch}:`` + + The ``git show`` fallback (steps 3–4) handles cases where + ``_sync_worktree_with_remote`` failed silently and the draft exists + on the remote branch but not in the local checkout. + """ + draft_rel = _pkg._get_draft_path(phase, issue_number=issue_number, pipeline_id=pipeline_id) + if not draft_rel: + return None + + def _truncate(content: str) -> str: + if len(content) > max_chars: + return content[:max_chars] + f"\n\n... (truncated, {len(content)} chars total)" + return content + + draft_path = repo_path / draft_rel + generic_rel = _get_generic_draft_path(phase) + + # Try primary (issue-specific) path first. + if draft_path.exists(): + return _truncate(draft_path.read_text(encoding="utf-8")) + + _pkg.logger.debug( + "Draft file not found", + path=str(draft_path), + phase=phase, + issue_number=issue_number, + pipeline_id=pipeline_id, + ) + + # Fallback: try the generic (unprefixed) path on disk. + if generic_rel: + generic_path = repo_path / generic_rel + if generic_path.exists(): + _pkg.logger.debug( + "Using generic fallback draft path", + primary_path=str(draft_path), + fallback_path=str(generic_path), + phase=phase, + ) + return _truncate(generic_path.read_text(encoding="utf-8")) + + # Fallback: try reading from remote tracking ref via git show. + # This handles cases where _sync_worktree_with_remote() failed + # silently (fetch failure, detached HEAD, divergence, etc.) and + # the draft exists on origin but not in the local checkout. + if branch: + content = _pkg._git_show_draft(repo_path, branch, draft_rel) + if content is None and generic_rel: + content = _pkg._git_show_draft(repo_path, branch, generic_rel) + if content is not None: + _pkg.logger.info( + "Read draft from remote tracking ref (local copy missing)", + phase=phase, + branch=branch, + ) + return _truncate(content) + + return None + + +def _read_human_phase_draft( + repo_path: Path, + phase: str, + issue_number: int | None = None, + pipeline_id: str | None = None, + max_chars: int = 32000, + branch: str | None = None, +) -> str | None: + """Read the human-focused companion draft for a phase. + + Mirrors :func:`_read_phase_draft` (disk first, then the + ``git show origin/{branch}`` fallback for a copy that only landed on + the remote branch), but resolves the path via + :func:`_get_human_draft_path` and has no generic-path variant — the + companion is always pipeline-identified. Returns ``None`` when the + companion is absent (so the gate falls back to the agent draft). + """ + human_rel = _get_human_draft_path(phase, issue_number=issue_number, pipeline_id=pipeline_id) + if not human_rel: + return None + + def _truncate(content: str) -> str: + if len(content) > max_chars: + return content[:max_chars] + f"\n\n... (truncated, {len(content)} chars total)" + return content + + human_path = repo_path / human_rel + if human_path.exists(): + return _truncate(human_path.read_text(encoding="utf-8")) + + if branch: + content = _pkg._git_show_draft(repo_path, branch, human_rel) + if content is not None: + _pkg.logger.info( + "Read human companion draft from remote tracking ref (local copy missing)", + phase=phase, + branch=branch, + ) + return _truncate(content) + + return None diff --git a/orchestrator/routes/pipelines/_drivers.py b/orchestrator/routes/pipelines/_drivers.py new file mode 100644 index 0000000000..8f7c9bfd08 --- /dev/null +++ b/orchestrator/routes/pipelines/_drivers.py @@ -0,0 +1,263 @@ +"""pipeline-driver lifecycle helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _spawn_pipeline_run_thread( + pipeline_id: str, + repo_path: _pkg.Path, + run_epoch: _pkg.datetime, +) -> _pkg.threading.Thread: + """Spawn a fresh ``_run_pipeline`` driver thread. + + Callers (all use the ``pipeline-{id}-{epoch}`` naming scheme): + + - ``advance_phase`` (manual phase advance via REST) + - ``restart_phase`` (manual phase restart via REST) + - the auto-advance block in ``_run_pipeline`` (#2165) + + The other ``_run_pipeline`` thread spawn sites — ``start_pipeline``'s + initial-spawn and AWAITING_HUMAN-recovery paths, plus the spurious-PNFE + respawn inside ``_run_pipeline`` — use different naming or take extra + kwargs (e.g. ``_respawn_attempt``) and are deliberately left inline. + + Without a fresh thread per phase, a mid-execution exception in the new + phase's first iteration takes down the whole pipeline (#2165). + """ + thread = _pkg.threading.Thread( + target=_pkg._run_pipeline, + args=(pipeline_id, repo_path), + daemon=True, + name=f"pipeline-{pipeline_id}-{int(run_epoch.timestamp())}", + ) + thread.start() + return thread + + +def has_live_pipeline_driver(pipeline_id: str) -> bool: + """Return True if a live ``_run_pipeline`` driver thread owns this pipeline. + + Driver threads are named ``pipeline-{id}`` (``start_pipeline``'s initial + and AWAITING_HUMAN-recovery spawns), ``pipeline-{id}-{epoch}`` + (``_spawn_pipeline_run_thread``), or ``pipeline-{id}-respawn-...`` (the + spurious-PNFE recovery). Every variant is either exactly ``pipeline-{id}`` + or carries a ``pipeline-{id}-`` prefix, so the literal-hyphen boundary + keeps a pipeline whose id is a prefix of another (``issue-3`` vs + ``issue-32``) from matching. + + After an orchestrator restart the process holds no driver threads, which + is precisely the orphaned-parked condition behind #3233: a pipeline left + AWAITING_HUMAN with a pending decision has no thread polling + ``wait_for_decision``, so a later resolution is recorded with no consumer + and the pipeline hangs silently. + """ + exact = f"pipeline-{pipeline_id}" + prefix = exact + "-" + for t in _pkg.threading.enumerate(): + if not t.is_alive(): + continue + if t.name == exact or t.name.startswith(prefix): + return True + return False + + +def relaunch_driverless_running_pipelines(store) -> int: + """Relaunch drivers for RUNNING pipelines orphaned by a restart (#3469). + + Called once per repo store at orchestrator startup, after + ``startup_reconciliation.reconcile_stale_containers`` has settled each + pipeline's status. A pipeline still RUNNING at that point was mid-flight + when the previous orchestrator process died: its consensus state is fully + reconciled at boot, but its ``_run_pipeline`` driver thread — and the BRC + event loop the driver owns — died with the old process, and no other code + path revives it. ``restart_agent`` delegates the respawn to the (dead) + event loop and returns success, while ``start_pipeline`` rejects + status=RUNNING with a 409, so without this sweep the pipeline is + permanently driverless and never spawns another pod (#3469). + + Relaunching reuses the proven resume path (the same one + ``restart_agent``'s inactive-pipeline branch relies on, #3244): + ``_run_pipeline`` re-enters ``pipeline.current_phase``, re-syncs the + worktree with the remote, and restarts the event loop, which respawns + one-shot agent Jobs within one poll. The persisted ``run_epoch`` is + deliberately NOT bumped: the old process is gone so no stale thread can + contend for the epoch, and the relaunched thread derives its own epoch + from persisted state exactly as the original did. + + AWAITING_HUMAN pipelines are out of scope — their drivers are revived on + decision resolution by ``maybe_revive_orphaned_awaiting_human_driver`` + (#3233). + + The sweep iterates ``store.get_active_pipelines()`` rather than the full + ``list_pipelines()`` so terminal/historical records (COMPLETE, FAILED, + CANCELLED) are skipped without a redundant load — reconciliation already + walked every pipeline immediately before this, and re-scanning the whole + store would double the boot-time git reads on repos with many historical + pipelines. + + Returns the number of drivers relaunched. Failures are isolated at two + layers: a record that fails to load with ``StateStoreError`` (corruption) + is skipped inside ``get_active_pipelines()``, and a per-pipeline failure + during the relaunch itself (driver probe or thread spawn) is logged and + skipped so one bad pipeline cannot strand the rest. The one case that is + *not* isolated: a load failure other than ``StateStoreError`` propagates out + of ``get_active_pipelines()`` and the outer ``except`` aborts the sweep + (returns 0) — an accepted trade for using the canonical active-pipeline + accessor, matching how ``get_active_pipelines()`` behaves for its other + callers. + """ + try: + active_pipelines = store.get_active_pipelines() + except Exception as e: # noqa: BLE001 - startup sweep must not raise + _pkg.logger.warning( + "Driver relaunch sweep skipped: could not list active pipelines", + error=str(e), + ) + return 0 + + relaunched = 0 + for pipeline in active_pipelines: + try: + if pipeline.status != _pkg.PipelineStatus.RUNNING: + continue + if _pkg.has_live_pipeline_driver(pipeline.id): + continue + run_epoch = pipeline.run_epoch or pipeline.created_at + _pkg._spawn_pipeline_run_thread(pipeline.id, store.repo_path, run_epoch) + relaunched += 1 + _pkg.logger.warning( + "Relaunched _run_pipeline driver for RUNNING pipeline with no " + "live driver thread (orchestrator restart recovery, #3469)", + pipeline_id=pipeline.id, + phase=pipeline.current_phase.value, + run_epoch=run_epoch.isoformat(), + ) + except Exception as e: # noqa: BLE001 - per-pipeline isolation + _pkg.logger.warning( + "Failed to relaunch driver for RUNNING pipeline (continuing sweep)", + pipeline_id=getattr(pipeline, "id", "unknown"), + error=str(e), + ) + return relaunched + + +def _broadcast_orphaned_driver_alert(pipeline_id: str, pipeline: _pkg.Pipeline) -> None: + """Surface an orphaned-driver revival as an overseer alert (#3233). + + A resolved decision on a driver-less pipeline used to return ``success`` + and hang invisibly. Emit an OVERSEER_ALERT alongside the WARNING log so + the recovery is visible on the bus, not just in orchestrator logs. + Best-effort: a broadcast failure never blocks the revival itself. + """ + try: + from message_store import Message, MessageType + + store_fn = _pkg._get_message_store() + if store_fn is None: + return + msg_store = store_fn() + phase = pipeline.current_phase.value if pipeline.current_phase else None + msg_store.add_message( + Message( + pipeline_id=pipeline_id, + from_role="orchestrator", + to_role="all", + message_type=MessageType.OVERSEER_ALERT, + subject="orphaned_driver_revived: orchestrator [medium]", + body=( + "A HITL decision was resolved on a pipeline whose " + "_run_pipeline driver thread did not survive an " + "orchestrator restart. The driver is being re-launched so " + "the resolution is acted on (no manual start_pipeline " + "needed). See #3233." + ), + metadata={"reason": "restart_orphaned_awaiting_human"}, + phase=phase, + ) + ) + except Exception as alert_err: # noqa: BLE001 + _pkg.logger.warning( + "Failed to broadcast orphaned-driver revival alert (non-fatal)", + pipeline_id=pipeline_id, + error=str(alert_err), + ) + + +def maybe_revive_orphaned_awaiting_human_driver(pipeline_id: str, repo_path: _pkg.Path) -> bool: + """Re-launch the driver for an AWAITING_HUMAN pipeline orphaned by a restart. + + Called from the decision-resolve path (#3233). When the orchestrator + restarts while a pipeline is parked AWAITING_HUMAN at a phase gate, the + in-memory ``_run_pipeline`` driver blocked on ``wait_for_decision`` is + gone and startup reconciliation deliberately leaves the still-pending + decision as-is (``startup_reconciliation.py``). Resolving the decision + then flips it to RESOLVED with no consumer and the pipeline hangs + silently — the operator sees ``success`` and nothing happens. + + This detects that no live driver owns the pipeline and, once the queue + has no remaining pending decisions, routes through ``start_pipeline``'s + proven AWAITING_HUMAN recovery branch (advance-or-rerun + driver respawn) + so the resolution self-heals without a manual ``start_pipeline``. + + No-ops (returns ``False``) when a live driver is already polling — the + normal in-process path consumes the resolution — or when the pipeline + isn't in the orphaned-parked state. Must be called from a Flask request + context (it reuses the lifecycle-secret-guarded ``start_pipeline`` route), + which the resolve-decision handler satisfies. + """ + store = _pkg.get_state_store(repo_path) + try: + pipeline = store.load_pipeline(pipeline_id) + except Exception: + return False + + if pipeline.status != _pkg.PipelineStatus.AWAITING_HUMAN: + return False + # A multi-decision batch (e.g. the contract-decision bridge) is only + # ready to resume once every decision is resolved; leave it parked while + # siblings are still pending. + if pipeline.get_pending_decisions(): + return False + if _pkg.has_live_pipeline_driver(pipeline_id): + return False + + _pkg.logger.warning( + "Decision resolved on AWAITING_HUMAN pipeline with no live driver " + "thread (orphaned by an orchestrator restart); reviving via " + "start_pipeline recovery so the resolution is acted on (#3233)", + pipeline_id=pipeline_id, + current_phase=pipeline.current_phase.value if pipeline.current_phase else None, + ) + _pkg._broadcast_orphaned_driver_alert(pipeline_id, pipeline) + + try: + _resp, status_code = _pkg.start_pipeline(pipeline_id) + except Exception as revive_err: # noqa: BLE001 + _pkg.logger.warning( + "Orphaned-driver revival raised (decision is still resolved; an " + "operator can recover manually via start_pipeline) (#3233)", + pipeline_id=pipeline_id, + error=str(revive_err), + ) + return False + + if status_code != 200: + _pkg.logger.warning( + "Orphaned-driver revival did not start the pipeline (#3233)", + pipeline_id=pipeline_id, + status_code=status_code, + ) + return False + + _pkg.logger.info( + "Orphaned AWAITING_HUMAN pipeline revived after decision resolution (#3233)", + pipeline_id=pipeline_id, + ) + return True diff --git a/orchestrator/routes/pipelines/_first_principles.py b/orchestrator/routes/pipelines/_first_principles.py new file mode 100644 index 0000000000..b31ee65c80 --- /dev/null +++ b/orchestrator/routes/pipelines/_first_principles.py @@ -0,0 +1,247 @@ +"""first-principles redirect + refine restart helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def apply_first_principles_redirect( + pipeline_id: str, + new_task_description: str, + *, + reason: str, +) -> list[str]: + """Adopt a first-principles redirect: rewrite the seed and re-run refine. + + Called in-process from the decision-resolve hook when an operator adopts a + redirect raised by the ``first_principles_reviewer``. Two durable steps: + + 1. **Rewrite the seed** via the operator-grade + ``rewrite_task_description_as_operator`` (audited, ``Role.HUMAN``), then + commit+push the worktree to the work branch so the refine restart's + re-fork (which forks fresh worktrees from ``origin/<branch>``) sees the + rewritten ``task_description`` rather than the old one. + 2. **Re-run refine** via :func:`_restart_refine_phase`. + + Returns the role values respawned. Raises on failure; the caller logs and + leaves the decision resolved (the operator's intent is recorded regardless). + """ + from operator_actions import rewrite_task_description_as_operator + + repo_path = _pkg.get_repo_path() + store, pipeline = _pkg._resolve_pipeline(pipeline_id, repo_path) + issue_number = getattr(pipeline, "issue_number", None) + gateway_mode, _ = _pkg._compute_gateway_mode(pipeline) + spawner = _pkg._get_spawner() + + rewrite = rewrite_task_description_as_operator( + pipeline_id, + new_task_description, + reason=reason, + actor="operator:first-principles-redirect", + issue_number=issue_number, + ) + + # Durably land the rewritten seed on the work branch. The refine restart + # below deletes per-agent worktrees and re-forks fresh ones from + # ``origin/<branch>``; without this push the re-fork would re-materialise + # the OLD seed and the redirect would be silently lost (#3080 re-fork + # semantics). + worktree = _pkg.Path(rewrite["worktree"]) + identifier = _pkg._pipeline_identifier(issue_number, pipeline_id) + try: + committed = _pkg._commit_statefiles_to_worktree( + worktree, + f"first-principles redirect: rewrite seed — {reason}"[:200], + identifier, + pipeline_id=pipeline_id, + ) + if committed and pipeline.branch: + spawner.gateway.push_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(worktree), + branch=pipeline.branch, + mode=gateway_mode, + base_branch=pipeline.base_branch, + ) + except Exception as exc: # noqa: BLE001 — best-effort; restart still proceeds + _pkg.logger.warning( + "Failed to push rewritten seed to work branch; refine restart may " + "re-fork the prior seed (first-principles redirect)", + pipeline_id=pipeline_id, + error=str(exc), + ) + + return _pkg._restart_refine_phase( + pipeline_id, store, reason=reason, spawner=spawner, gateway_mode=gateway_mode + ) + + +def _restart_refine_phase( + pipeline_id: str, + store: _pkg.Any, + *, + reason: str, + spawner: _pkg.Any, + gateway_mode: str, +) -> list[str]: + """Re-run the refine phase in-process (non-route sibling of ``restart_phase``). + + Mirrors ``restart_phase``'s essential steps for the refine phase so the + first-principles accept-path can re-run refine from the decision-resolve + hook (no Flask request). Refine has no slices, so the per-slice tracker + loop in ``restart_phase`` is intentionally omitted. Raises ``ValueError`` + if the pipeline is not currently parked at the refine phase. + """ + phase = _pkg.PipelinePhase.REFINE.value + lock = _pkg.get_pipeline_state_lock(pipeline_id) + with lock: + pipeline = store.load_pipeline(pipeline_id) + if pipeline.current_phase.value != phase: + raise ValueError( + f"_restart_refine_phase: pipeline {pipeline_id} is not at the " + f"refine phase (current: {pipeline.current_phase.value})" + ) + phase_exec = pipeline.phases.get(phase) + if phase_exec is None: + raise ValueError(f"Refine phase not found in pipeline {pipeline_id}") + + agent_roles: list[_pkg.AgentRole] = [] + for agent in phase_exec.agents: + if hasattr(agent, "role"): + role = ( + agent.role + if isinstance(agent.role, _pkg.AgentRole) + else _pkg.AgentRole(agent.role) + ) + agent_roles.append(role) + if not agent_roles: + from egg_contracts.agent_roles import get_roles_for_phase as _grfp + + for r in _grfp( + phase, + include_reviewers=True, + repo=pipeline.repo, + has_contract=getattr(pipeline, "has_contract", True), + ): + try: + agent_roles.append(_pkg.AgentRole(r.value)) + except ValueError: + continue + + old_container_ids = [c.container_id for c in phase_exec.containers] + phase_exec.containers = [] + phase_exec.agents = [] + phase_exec.review_cycles = 0 + phase_exec.hitl_review_cycles = 0 + phase_exec.status = _pkg.PipelineStatus.PENDING + phase_exec.started_at = None + phase_exec.work_started_at = None + phase_exec.completed_at = None + phase_exec.error = None + phase_exec.cycle_timings = [] + pipeline.status = _pkg.PipelineStatus.RUNNING + pipeline.error = None + pipeline.run_epoch = _pkg.datetime.now(_pkg.UTC) + store.update_pipeline(pipeline_id, pipeline.model_dump(mode="json")) + + # --- Outside the lock: slow, idempotent, best-effort teardown --- + for container_id in old_container_ids: + try: + spawner.stop_agent_container(container_id, cleanup_session=True) + except Exception as e: # noqa: BLE001 + _pkg.logger.warning( + "Failed to stop container during refine redirect restart", + container_id=container_id[:12] if container_id else "?", + error=str(e), + ) + try: + spawner.remove_agent_container(container_id, force=True, cleanup_session=False) + except Exception as e: # noqa: BLE001 + _pkg.logger.warning( + "Failed to remove container during refine redirect restart", + container_id=container_id[:12] if container_id else "?", + error=str(e), + ) + + restart_role_values = {role.value for role in agent_roles} + try: + all_worktrees = _pkg.agent_salvage.enumerate_agent_worktrees( + pipeline_id, validate_git=False + ) + except (OSError, ImportError, RuntimeError) as e: + _pkg.logger.warning( + "Failed to enumerate per-agent worktrees during refine redirect restart", + pipeline_id=pipeline_id, + error=str(e), + ) + all_worktrees = [] + worktrees_to_delete = [wt for wt in all_worktrees if wt.agent_role in restart_role_values] + if worktrees_to_delete: + try: + _pkg.agent_salvage.auto_salvage_pipeline( + spawner.gateway, + pipeline_id, + worktree_filter={wt.worktree_id for wt in worktrees_to_delete}, + mode=gateway_mode, + base_branch=pipeline.base_branch, + ) + except Exception as e: # noqa: BLE001 + _pkg.logger.warning( + "Auto-salvage failed during refine redirect restart; proceeding", + pipeline_id=pipeline_id, + error=str(e), + ) + for wt in worktrees_to_delete: + try: + spawner.gateway.delete_worktrees(container_id=wt.worktree_id, force=True) + except Exception as e: # noqa: BLE001 + _pkg.logger.warning( + "Failed to delete per-agent worktree during refine redirect restart", + agent_worktree_id=wt.worktree_id, + pipeline_id=pipeline_id, + error=str(e), + ) + + try: + from peer_consensus import get_peer_consensus_tracker + + tracker = get_peer_consensus_tracker(pipeline_id) + if tracker: + tracker.clear() + except Exception as e: # noqa: BLE001 + _pkg.logger.warning( + "Failed to clear peer consensus during refine redirect restart", + pipeline_id=pipeline_id, + error=str(e), + ) + + spawner.reset_restart_counts(pipeline_id) + try: + from health_monitor import get_health_monitor + + _hm = get_health_monitor() + if _hm is not None: + for role in agent_roles: + _hm.reset_agent(role.value) + except Exception as e: # noqa: BLE001 + _pkg.logger.warning( + "Failed to reset health-monitor state during refine redirect restart", + pipeline_id=pipeline_id, + error=str(e), + ) + + _pkg._spawn_pipeline_run_thread(pipeline_id, store.repo_path, pipeline.run_epoch) + agents_to_restart = [role.value for role in agent_roles] + _pkg.logger.info( + "Refine phase re-run for first-principles redirect", + pipeline_id=pipeline_id, + reason=reason, + agents_to_restart=agents_to_restart, + ) + return agents_to_restart diff --git a/orchestrator/routes/pipelines/_hitl_rerun.py b/orchestrator/routes/pipelines/_hitl_rerun.py new file mode 100644 index 0000000000..45a30cc870 --- /dev/null +++ b/orchestrator/routes/pipelines/_hitl_rerun.py @@ -0,0 +1,337 @@ +"""HITL phase-rerun + iteration context helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _build_phase_iteration_context( + operator_directives: list[_pkg.OperatorDirective] | None, + iteration_history: list[_pkg.IterationSummary] | None, +) -> str: + """Render operator directives + prior iteration history as a prompt section. + + Issued in iteration N+1 prompts (for **both** producers and reviewers) + after one or more HITL phase-gate kickbacks. Replaces the unstructured + ``## Review Feedback`` rendering that previously squatted on the + agentic-cycle feedback channel — operator directives now have their own + section with explicit precedence prose so reviewers cannot faithfully + NACK a directive-driven change against a stale default rubric (#2795). + + Returns an empty string when there are no directives and no history + so the caller can unconditionally append the result. + """ + directives = operator_directives or [] + history = iteration_history or [] + if not directives and not history: + return "" + + lines: list[str] = ["## Phase Iteration Context\n"] + if directives: + lines.append( + "The operator has kicked this phase back through HITL one or " + "more times. The directives below **override prompt-template " + "defaults**. If a rubric item in your role's instructions " + "conflicts with a directive, the directive wins. Later " + "directives override earlier ones.\n" + ) + lines.append("### Operator Directives (chronological)\n") + for idx, directive in enumerate(directives, start=1): + ts = directive.created_at.isoformat() + lines.append(f"**Directive {idx}** (iteration {directive.iteration_n}, {ts}):") + lines.append("") + lines.append(directive.feedback_text.rstrip()) + lines.append("") + + if history: + lines.append("### Prior Iteration History\n") + lines.append( + "Each entry below is a frozen snapshot of a previously kicked-" + "back iteration's BRC outcome — what the reviewers concluded " + "and why. Use it to see which rubric items tripped last round " + "so you do not repeat the same NACKs.\n" + ) + for summary in history: + ts = summary.completed_at.isoformat() + lines.append(f"**Iteration {summary.iteration_n}** (completed {ts}):") + if summary.final_proposal_commit: + # SHAs are pre-filtered by _build_iteration_summary_from_tracker + # (empty + RECONSTRUCTED_NO_SHA dropped before the dict is + # populated), so every value here is a real commit. + commit_parts = [ + f"{producer}={sha[:12]}" + for producer, sha in sorted(summary.final_proposal_commit.items()) + ] + lines.append(f"- Final proposal commits: {', '.join(commit_parts)}") + if summary.verdict_matrix: + verdicts = "; ".join( + f"{edge}: {state}" for edge, state in sorted(summary.verdict_matrix.items()) + ) + lines.append(f"- Verdict matrix: {verdicts}") + if summary.nack_reasons: + lines.append(f"- NACK reasons ({len(summary.nack_reasons)}):") + for reason in summary.nack_reasons: + lines.append(f" - {reason}") + if summary.artifacts_snapshot: + arts = ", ".join(sorted(summary.artifacts_snapshot.keys())) + lines.append(f"- Artifacts at iteration close: {arts}") + lines.append("") + + return "\n".join(lines) + + +def _build_iteration_summary_from_tracker( + tracker: _pkg.Any, + iteration_n: int, + artifacts: dict[str, str] | None = None, + completed_at: _pkg.datetime | None = None, +) -> _pkg.IterationSummary: + """Capture an :class:`IterationSummary` from a live BRC tracker. + + Called by the HITL kickback handler **before** ``_clear_concurrent_state`` + wipes the tracker so the iteration N+1 prompt can render what tripped + iteration N. Tolerates a ``None`` tracker — returns a summary with only + the iteration index + completion timestamp populated, which still lets + downstream prompts mention that a kickback occurred without claiming + false verdict detail. + """ + completion = completed_at or _pkg.datetime.now(_pkg.UTC) + summary = _pkg.IterationSummary( + iteration_n=iteration_n, + completed_at=completion, + artifacts_snapshot=dict(artifacts or {}), + ) + if tracker is None: + return summary + + try: + matrix = getattr(tracker, "matrix", None) + if matrix is None: + return summary + # Snapshot the matrix entries + commit SHAs under the tracker's + # lock so concurrent mutations from a still-live tracker can't + # tear the read. RLock means re-entry is safe if callers already + # hold it. Iteration below runs on the local copies. + lock = getattr(tracker, "_lock", None) + commits_snapshot: dict[str, str] = {} + if lock is not None: + with lock: + entries_snapshot = list(getattr(matrix, "_entries", {}).items()) + commits_snapshot = dict(getattr(tracker, "_proposal_commit_shas", {})) + else: + entries_snapshot = list(getattr(matrix, "_entries", {}).items()) + commits_snapshot = dict(getattr(tracker, "_proposal_commit_shas", {})) + + verdict_matrix: dict[str, str] = {} + nack_reasons: list[str] = [] + for (reviewer, producer), entry in entries_snapshot: + state = getattr(entry, "state", None) + state_val = state.value if state is not None else "unknown" + verdict_matrix[f"{reviewer}->{producer}"] = state_val + if state_val == "nacked" and getattr(entry, "reason", ""): + nack_reasons.append(f"{reviewer}→{producer}: {entry.reason}") + summary.verdict_matrix = verdict_matrix + summary.nack_reasons = nack_reasons + + producers = {producer for _, producer in (k for k, _ in entries_snapshot)} + commits: dict[str, str] = {} + for producer in producers: + sha = commits_snapshot.get(producer, "") + if sha and sha != "RECONSTRUCTED_NO_SHA": + commits[producer] = sha + summary.final_proposal_commit = commits + except Exception as e: # noqa: BLE001 + _pkg.logger.debug( + "Failed to snapshot iteration summary from tracker", + iteration_n=iteration_n, + error=str(e), + ) + return summary + + +def _apply_inline_hitl_kickback_to_phase( + phase_execution: _pkg.PhaseExecution, + revision_feedback: str, + tracker: _pkg.Any = None, +) -> list[_pkg.ContainerInfo]: + """Apply the inline HITL kickback's phase-state mutations. + + Extracted from the inline ``request_changes`` handler so tests can + drive the assertion through production code rather than constructing + a fixture by hand (#2795 review). The caller is still responsible for + the wrapping concerns: clearing the message store + consensus tracker + via ``_clear_concurrent_state``, persisting the pipeline via + ``store.save_pipeline``, and stopping the stale containers returned + here (the K8s delete is asynchronous so an explicit stop is required + to avoid iteration N+1 racing iteration N's still-terminating pods). + + Returns the snapshot of containers that were running at kickback + time, for the caller to issue the defensive stop on. + """ + # Monotone across the legacy-hitl_feedback migration boundary: a + # pre-#2795 phase migrates with iteration_history empty but a + # synthetic OperatorDirective carrying iteration_n derived from + # hitl_review_cycles. ``len(iteration_history)`` alone would + # restart at 0 and label two distinct iterations identically; use + # one past the maximum existing directive index as the floor so + # the displayed "iteration X" labels stay monotone. + iteration_n = max( + len(phase_execution.iteration_history), + max( + (d.iteration_n for d in phase_execution.operator_directives), + default=-1, + ) + + 1, + ) + phase_execution.operator_directives.append( + _pkg.OperatorDirective( + iteration_n=iteration_n, + feedback_text=revision_feedback, + ) + ) + phase_execution.iteration_history.append( + _pkg._build_iteration_summary_from_tracker( + tracker, + iteration_n=iteration_n, + artifacts=phase_execution.artifacts, + ) + ) + stale_containers = list(phase_execution.containers) + phase_execution.containers = [] + phase_execution.agents = [] + phase_execution.artifacts = {} + phase_execution.review_cycles = 0 + return stale_containers + + +def _broadcast_hitl_nonconvergence_alert( + pipeline_id: str, + pipeline: _pkg.Pipeline, + current_phase: _pkg.PipelinePhase, + cycles: int, + threshold: int, +) -> None: + """Non-fatal overseer alert when the HITL converge loop runs long (#3392). + + The converge-before-advance loop is human-gated every round (the + operator resolves decisions before each re-run), so a long-running loop + cannot burn compute silently and is never force-advanced. After + ``threshold`` rounds we surface an ``OVERSEER_ALERT`` so a pathological + non-convergence — a real carry-forward bug, or a genuinely churning + design — is visible. Best-effort: a broadcast failure never blocks the + re-run. + """ + try: + from message_store import Message, MessageType + + store_fn = _pkg._get_message_store() + if store_fn is None: + return + msg_store = store_fn() + phase = current_phase.value if current_phase else None + msg_store.add_message( + Message( + pipeline_id=pipeline_id, + from_role="orchestrator", + to_role="all", + message_type=MessageType.OVERSEER_ALERT, + subject="hitl_nonconvergence: orchestrator [medium]", + body=( + f"The {phase} phase HITL converge-before-advance loop has run " + f"{cycles} rounds (>= {threshold}) without reaching a fixpoint. " + f"Each round is human-gated, so this is surfaced for visibility, " + f"not force-advanced. Investigate whether a decision keeps " + f"re-surfacing (carry-forward bug) or the design is genuinely " + f"churning. See #3392." + ), + metadata={"reason": "hitl_nonconvergence", "cycles": cycles}, + phase=phase, + ) + ) + except Exception as alert_err: # noqa: BLE001 + _pkg.logger.warning( + "Failed to broadcast HITL non-convergence alert (non-fatal)", + pipeline_id=pipeline_id, + error=str(alert_err), + ) + + +def _perform_hitl_phase_rerun( + *, + store: _pkg.Any, + spawner: _pkg.Any, + pipeline: _pkg.Pipeline, + phase_execution: _pkg.PhaseExecution, + pipeline_id: str, + current_phase: _pkg.PipelinePhase, + feedback_text: str, + event_message: str, +) -> None: + """Tear down the current phase iteration and arm a re-run (#3392). + + Shared by the two re-run triggers in the converge-before-advance HITL + loop: the operator-feedback kickback (``request_changes`` / + ``change_approach``) and the decision-driven re-run that folds resolved + HITL answers back into the phase documents. Snapshots the BRC tracker + for the next iteration's prompt, appends the operator directive + + iteration summary (#2795), clears concurrent state so the re-run does + not short-circuit on stale ``CONSENSUS_CONFIRMED`` messages (#1296), + persists, and stops the stale containers (the K8s delete is async, so an + explicit idempotent stop prevents iteration N+1 racing iteration N's + still-terminating pods). + + The caller must already hold the pipeline state lock, have set the + pipeline/phase status back to RUNNING, and incremented + ``phase_execution.hitl_review_cycles``. The caller issues the + ``continue`` that re-enters the outer loop. + """ + # Capture the BRC tracker state BEFORE _clear_concurrent_state drops + # it — that's our only chance to snapshot this iteration's verdicts for + # the next iteration's prompt. + rerun_tracker = None + try: + from peer_consensus import get_peer_consensus_tracker as _gpct + + rerun_tracker = _gpct(pipeline_id) + except Exception as tracker_err: # noqa: BLE001 + _pkg.logger.debug( + "Tracker lookup failed during HITL re-run snapshot", + pipeline_id=pipeline_id, + error=str(tracker_err), + ) + + stale_containers = _pkg._apply_inline_hitl_kickback_to_phase( + phase_execution, + feedback_text, + tracker=rerun_tracker, + ) + + from routes.phases import _clear_concurrent_state + + _clear_concurrent_state(pipeline_id) + + store.save_pipeline(pipeline) + + for _ctr in stale_containers: + if _ctr.container_id and _ctr.status == _pkg.ContainerStatus.RUNNING: + try: + spawner.backend.stop_container(_ctr.container_id, timeout=10) + except Exception as stop_err: # noqa: BLE001 + _pkg.logger.debug( + "Best-effort HITL re-run teardown failed", + pipeline_id=pipeline_id, + container_id=_ctr.container_id, + error=str(stop_err), + ) + + _pkg.report_pipeline_status( + pipeline, + event_type="phase.revision_requested", + message=event_message, + ) + _pkg._emit_pipeline_event(pipeline, "phase.revision_requested") diff --git a/orchestrator/routes/pipelines/_ledger.py b/orchestrator/routes/pipelines/_ledger.py new file mode 100644 index 0000000000..4ab88ca39a --- /dev/null +++ b/orchestrator/routes/pipelines/_ledger.py @@ -0,0 +1,1364 @@ +"""decision-ledger + gap-gate + apply-handoff helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _sync_pipeline_decisions_to_contract( + repo_path: _pkg.Path, + worktree_repo_path: _pkg.Path, + pipeline_id: str, +) -> None: + """Sync resolved non-phase-gate pipeline decisions to the contract. + + Converts HITLDecision objects from pipeline state into contract Decision + objects so that implement-phase agents can see what was decided during + refine/plan phases. + + Only syncs decisions with decision_type != "phase_gate" (substantive + choices, not process-control gates). Skips decisions already present + in the contract (matched by question text) to avoid duplicates on + re-runs after HITL revision cycles. + + Args: + repo_path: Orchestrator's main repo path — root for the state + store that owns pipeline records. + worktree_repo_path: Pipeline's per-run worktree path — root for + the contract under ``<worktree>/.egg-state/contracts/``. + """ + try: + from egg_contracts.loader import load_contract, save_contract + from egg_contracts.models import Decision, DecisionOption, DecisionType + except ImportError: + _pkg.logger.warning("egg_contracts not available, skipping decision sync") + return + + # Load pipeline from the orchestrator's state store, NOT the per-run + # worktree. Pipeline records live under ``repo_path``'s persistent + # state-store worktree; the per-run worktree has none. Conflating + # the two silently no-op'd this helper for every issue-mode pipeline + # since #950 (#2345). + store = _pkg.get_state_store(repo_path) + try: + pipeline = store.load_pipeline(pipeline_id) + except Exception as exc: + _pkg.logger.warning( + "decision_sync_pipeline_load_failed", + pipeline_id=pipeline_id, + state_store_repo_path=str(repo_path), + error=str(exc), + ) + return + + # Filter to resolved, non-phase-gate decisions + substantive_decisions = [ + d + for d in pipeline.decisions + if d.decision_type != "phase_gate" and d.status == _pkg.DecisionStatus.RESOLVED + ] + + if not substantive_decisions: + _pkg.logger.debug("No substantive decisions to sync", pipeline_id=pipeline_id) + return + + try: + contract = load_contract(pipeline_id, worktree_repo_path) + except Exception: + _pkg.logger.warning( + "Contract not found, skipping decision sync", + pipeline_id=pipeline_id, + ) + return + + # Build set of existing contract decision questions for deduplication + existing_questions = {d.question for d in contract.decisions} + + # Determine next decision ID (continue numbering after existing ones) + max_existing_id = 0 + for d in contract.decisions: + # Extract numeric suffix from "decision-N" + try: + num = int(d.id.split("-")[1]) + max_existing_id = max(max_existing_id, num) + except IndexError, ValueError: + pass + + synced_count = 0 + for pipeline_decision in substantive_decisions: + if pipeline_decision.question in existing_questions: + continue + + max_existing_id += 1 + decision_id = f"decision-{max_existing_id}" + + # Convert pipeline options (list[str]) to contract DecisionOption objects + contract_options = [ + DecisionOption(id=f"opt-{i + 1}", label=opt) + for i, opt in enumerate(pipeline_decision.options) + ] + + contract_decision = Decision( + id=decision_id, + question=pipeline_decision.question, + type=DecisionType.HITL, + options=contract_options, + resolved=True, + resolution=pipeline_decision.resolution, + resolved_by="human", + resolved_at=pipeline_decision.resolved_at, + ) + contract.decisions.append(contract_decision) + existing_questions.add(pipeline_decision.question) + synced_count += 1 + + if synced_count > 0: + save_contract(contract, worktree_repo_path) + _pkg.logger.info( + "Synced pipeline decisions to contract", + pipeline_id=pipeline_id, + synced_count=synced_count, + total_contract_decisions=len(contract.decisions), + ) + + +def _ledger_attestation_question(role: str, rationale: str, phase_value: str) -> str: + """Compose the explicit-none confirmation question (#3462). + + A producer's claim that a phase raises no operator decisions is itself + a judgment call about what *is* a judgment call — exactly the class of + decision the HITL contract assigns to the operator. It therefore + surfaces as its own confirmable decision, not a sentence embedded in + the phase_gate question. + """ + return ( + f"The {role} attests the {phase_value} phase deliberately raises " + f"no operator decisions (#3462):\n\n" + f"> {rationale}\n\n" + f"Confirm to proceed to the phase gate, or choose " + f"“{_pkg._LEDGER_BACKSTOP_RERUN_OPTION}” to send the phase back so its " + f"agents register the decisions as first-class contract entries " + f"(cq-N). Any free-text reply is treated as a re-run directive and " + f"forwarded to the agents." + ) + + +def _unwrap_choice_resolution(resolution: str) -> str: + """Unwrap the ``{"action":"select","selected":<label>}`` envelope. + + The SDLC HITL CLI resolves a ``choice`` decision with that structured + envelope (mirrors ``routes.decisions._normalize_choice_resolution``); + a bare string / non-JSON resolution passes through unchanged. + """ + try: + payload = _pkg.json.loads(resolution) + if isinstance(payload, dict) and payload.get("action") == "select": + selected = payload.get("selected") + if isinstance(selected, str): + return selected + except ValueError, TypeError: + pass + return resolution + + +def _ledger_attestation_confirmed(resolution: str) -> bool: + """Return True when ``resolution`` confirms the explicit-none attestation. + + Conservative on purpose (#3462): only the bare keyword ``confirm`` or + the full confirm-option label counts. Anything else — the re-run + option, or free text naming decisions the operator expected — kicks + the phase back, with the text riding along as the directive. + """ + normalized = _pkg._unwrap_choice_resolution(resolution).strip().lower() + return normalized in ("confirm", _pkg._LEDGER_ATTESTATION_CONFIRM_OPTION.lower()) + + +def _ledger_attestation_rerun_directive(phase_value: str, rationale: str, resolution: str) -> str: + """Compose the re-run directive for a rejected explicit-none attestation (#3462). + + The operator declined to confirm that the phase raises no operator + decisions, so the phase re-runs with an instruction to register each + decision — including ones the producer believes prior context already + resolves. Any free-text resolution (i.e. not the bare re-run option) is + an operator note and rides along verbatim so the agents see the specific + concern. + """ + directive = ( + f"The operator declined to confirm the {phase_value} phase's " + f"no-decisions attestation (#3462). The phase claimed: " + f"“{rationale}”. Register each operator-grade decision via " + f"`egg-contract add-decision` — including decisions you believe " + f"prior context already resolves: register those with your " + f"recommended answer as the first option and cite the resolving " + f"context in its description. Belief about resolution is a " + f"recommended disposition, not a reason to skip registration." + ) + if resolution.strip().lower() != _pkg._LEDGER_BACKSTOP_RERUN_OPTION.lower(): + directive += f"\n\nOperator note: {resolution.strip()}" + return directive + + +def _handle_explicit_none_attestation_gate( + *, + pipeline, + pipeline_id: str, + repo_path, + current_phase: _pkg.PipelinePhase, + ledger_note: str, + explicit_none: tuple[str, str], + store, + spawner, +): + """Surface an explicit-none attestation as a confirmable HITL decision (#3462). + + A producer's claim that a refine/plan phase raises no operator decisions + bypasses the entire register → bridge → resolve chain, and the claim is + itself a judgment call the HITL contract assigns to the operator. Rather + than folding it into the phase_gate question as prose, surface it as its + own confirmable ``choice`` decision: confirming records the operator's + endorsement on the ledger note; rejecting re-runs the phase so producers + register the decisions as first-class ``cq-N`` entries. + + Returns ``(rerun_requested, ledger_note, pipeline)``: + + - ``rerun_requested`` — True when the operator rejected the attestation + and the phase has already been re-run here; the caller must ``continue`` + its poll loop. False when the attestation was confirmed (or fail-open on + a cancelled/non-RESOLVED terminal state); the caller proceeds to the + phase gate. + - ``ledger_note`` — the note to thread into the phase_gate question, + annotated with the confirmation outcome. + - ``pipeline`` — the (possibly reloaded) pipeline the caller must rebind, + since queuing the confirmation decision reloads and mutates state. + """ + attest_role, attest_rationale = explicit_none + attest_question = _pkg._ledger_attestation_question( + attest_role, attest_rationale, current_phase.value + ) + # A converge-loop round (or a resume) re-enters this gate with the same + # attestation — do not re-ask a question the operator already answered, + # and reuse a pending one instead of queueing a duplicate (mirrors the + # phase_gate's #1152 guard). + prior_confirm = next( + ( + d + for d in reversed(pipeline.decisions) + if d.decision_type == "choice" + and d.phase == current_phase + and d.question == attest_question + and d.status == _pkg.DecisionStatus.RESOLVED + and _pkg._ledger_attestation_confirmed(str(d.resolution or "")) + ), + None, + ) + if prior_confirm is not None: + return False, ledger_note + " Operator confirmed the attestation.", pipeline + + dq = _pkg.get_decision_queue(pipeline_id, repo_path) + pending_attest = next( + ( + d + for d in reversed(pipeline.decisions) + if d.decision_type == "choice" + and d.phase == current_phase + and d.question == attest_question + and d.status == _pkg.DecisionStatus.PENDING + ), + None, + ) + if pending_attest is not None: + attest_decision = pending_attest + newly_created = False + else: + attest_decision = dq.queue_decision( + question=attest_question, + context=ledger_note, + options=[ + _pkg._LEDGER_ATTESTATION_CONFIRM_OPTION, + _pkg._LEDGER_BACKSTOP_RERUN_OPTION, + ], + decision_type="choice", + phase=current_phase, + ) + newly_created = True + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + pipeline.status = _pkg.PipelineStatus.AWAITING_HUMAN + phase_execution = pipeline.get_phase_execution(current_phase) + phase_execution.status = _pkg.PipelineStatus.AWAITING_HUMAN + store.save_pipeline(pipeline) + # Only announce a freshly-created decision. Reusing a pending decision + # across polls must not re-emit ``decision.created`` — a duplicate event + # for a decision the operator is already looking at (#3462 review). + if newly_created: + _pkg.report_pipeline_status( + pipeline, + event_type="decision.created", + message=( + f"{current_phase.value} phase attests no operator " + f"decisions — awaiting operator confirmation (#3462)" + ), + ) + _pkg._emit_pipeline_event(pipeline, "decision.created") + + attest_resolved = dq.wait_for_decision(attest_decision.id) + attest_resolution = _pkg._unwrap_choice_resolution( + str(getattr(attest_resolved, "resolution", None) or "") + ).strip() + resolved_ok = attest_resolved.status == _pkg.DecisionStatus.RESOLVED + confirmed = resolved_ok and _pkg._ledger_attestation_confirmed(attest_resolution) + + if resolved_ok and not confirmed: + # Rejected — re-run the phase so producers register the decisions as + # first-class cq-N entries. + rerun_directive = _pkg._ledger_attestation_rerun_directive( + current_phase.value, attest_rationale, attest_resolution + ) + _pkg.logger.info( + "Explicit-none attestation rejected: re-running phase (#3462)", + pipeline_id=pipeline_id, + phase=current_phase.value, + ) + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + pipeline.status = _pkg.PipelineStatus.RUNNING + phase_execution = pipeline.get_phase_execution(current_phase) + phase_execution.status = _pkg.PipelineStatus.RUNNING + phase_execution.completed_at = None + phase_execution.hitl_review_cycles += 1 + _alert_threshold = pipeline.config.max_hitl_review_cycles + if phase_execution.hitl_review_cycles >= _alert_threshold: + _pkg._broadcast_hitl_nonconvergence_alert( + pipeline_id, + pipeline, + current_phase, + phase_execution.hitl_review_cycles, + _alert_threshold, + ) + _pkg._perform_hitl_phase_rerun( + store=store, + spawner=spawner, + pipeline=pipeline, + phase_execution=phase_execution, + pipeline_id=pipeline_id, + current_phase=current_phase, + feedback_text=rerun_directive, + event_message=( + f"Re-running {current_phase.value}: no-decisions attestation rejected (#3462)" + ), + ) + return True, ledger_note, pipeline + + if confirmed: + return False, ledger_note + " Operator confirmed the attestation.", pipeline + # Fail open to the phase gate on a non-RESOLVED terminal state (cancel): + # the gate itself still blocks for approval, mirroring the missing-ledger + # backstop's posture. Record the outcome accurately — do not claim a + # confirmation the operator never gave (#3462 review). + return ( + False, + ledger_note + " Attestation confirmation was cancelled; deferring to the phase gate.", + pipeline, + ) + + +def _find_explicit_none_attestation( + pipeline_id: str, + phase_value: str, +) -> tuple[str, str] | None: + """Find a producer's explicit-none decision-ledger attestation (#3390). + + Scans the phase's ``CONSENSUS_PROPOSE`` messages (newest first) for a + proposal whose attestation carries a non-empty + ``no_decisions_rationale`` — the durable record that a producer + *deliberately* registered no decisions this phase (propose-time + validation guarantees the field was well-formed when accepted). + Returns ``(role, rationale)`` or ``None``; message-store outages + degrade to ``None`` (the caller fails closed into the backstop HITL, + which the operator can resolve either way — never a silent pass). + """ + try: + from message_store import MessageType, get_message_store + + messages = get_message_store().get_messages(pipeline_id, limit=500) + except Exception as exc: # noqa: BLE001 + _pkg.logger.warning( + "Decision-ledger attestation scan failed (treating as not found)", + pipeline_id=pipeline_id, + phase=phase_value, + error=str(exc), + ) + return None + + for message in reversed(messages): + if message.message_type != MessageType.CONSENSUS_PROPOSE: + continue + if message.phase is not None and message.phase != phase_value: + continue + payload = (message.metadata or {}).get("payload") + if not isinstance(payload, dict): + continue + attestation = payload.get("attestation") + if not isinstance(attestation, dict): + continue + rationale = attestation.get("no_decisions_rationale") + if isinstance(rationale, str) and rationale.strip(): + return message.from_role, rationale.strip() + return None + + +def _collect_decision_ledger_status( + worktree_repo_path: _pkg.Path, + pipeline_id: str, + pipeline_identifier: int | str, + phase: _pkg.PipelinePhase, +) -> tuple[str, bool, tuple[str, str] | None]: + """Summarize the phase's decision ledger for the gate surface (#3390). + + Returns ``(note, missing, explicit_none)``: + + - ``note`` — an operator-visible one-liner appended to the phase_gate + question so "N registered" vs "explicitly none" vs "MISSING" is + readable at the gate without a ``get_contract`` round-trip. + - ``missing`` — True only when the phase registered zero decisions + AND no producer attested an explicit empty ledger. With propose-time + enforcement in place this means the gate was reached on a path that + bypassed consensus (force-advance, resume) or the producer's claim + was lost — the caller surfaces a dedicated backstop HITL rather + than silently advancing. + - ``explicit_none`` — the ``(role, rationale)`` of a producer's + explicit-none attestation when that is what stands in for a ledger + (zero registered decisions), else ``None``. The caller surfaces it + as its own confirmable decision (#3462) rather than trusting the + self-attestation. Mutually exclusive with ``missing``. + """ + phase_value = phase.value + registered_ids: list[str] = [] + try: + from egg_contracts.loader import load_contract + + contract = load_contract(pipeline_identifier, worktree_repo_path) + except Exception as exc: # noqa: BLE001 + _pkg.logger.warning( + "Decision-ledger status: contract not loadable", + pipeline_id=pipeline_id, + phase=phase_value, + error=str(exc), + ) + contract = None + + if contract is not None: + for d in contract.decisions: + d_type = getattr(d.type, "value", d.type) + if d_type != "hitl": + continue + d_phase = getattr(d.phase, "value", d.phase) if d.phase is not None else None + if d_phase is None or d_phase == phase_value: + registered_ids.append(d.id) + + if registered_ids: + resolved = 0 + for d in contract.decisions: + if d.id in registered_ids and d.resolved: + resolved += 1 + return ( + f"Decision ledger: {len(registered_ids)} decision(s) registered this " + f"phase ({', '.join(registered_ids)}), {resolved} resolved.", + False, + None, + ) + + explicit_none = _pkg._find_explicit_none_attestation(pipeline_id, phase_value) + if explicit_none is not None: + role, rationale = explicit_none + return ( + f"Decision ledger: explicitly none — {role} attested: {rationale}", + False, + explicit_none, + ) + + return ( + "⚠️ Decision ledger MISSING: this phase registered no HITL decisions " + "and no producer attested an explicit empty ledger (#3390). " + "“0 decisions” here cannot be distinguished from “failed " + "to register”.", + True, + None, + ) + + +def _queue_and_await_contract_decisions( + dq: _pkg.Any, + worktree_repo_path: _pkg.Path, + pipeline_id: str, + pipeline_identifier: int | str, + phase: _pkg.PipelinePhase, +) -> int: + """Promote unresolved contract decisions/feedback into the orchestrator queue. + + Returns the number of contract decisions/feedback this call surfaced and + the operator *resolved* this round — the converge-before-advance signal + (#3392). Decisions that were surfaced but came back non-RESOLVED (e.g. the + operator cancelled them) are **not** counted: the contract ``cq-N`` stays + open, and counting it would re-run the phase, re-surface the still-open + question (carry-forward only adopts *resolved* questions), and loop with no + termination now that the force-advance backstop is gone. A non-zero count + means the operator just answered something, so the caller re-runs the + phase to fold the resolutions into the documents; a zero count means the + round resolved nothing new and the caller may advance. + + + Agents register architectural questions via ``egg-contract add-decision`` + and ``add-feedback``. Those writes only touch ``.egg-state/contracts/ + {identifier}.json`` — the orchestrator's decision queue never sees them, + so approving the phase_gate silently drops the questions and the next + phase's agents have to guess (issue #1889). + + This helper bridges contract-scoped questions for the current phase into + the orchestrator queue after phase_gate approval, so HTTP/MCP callers + (e.g. the ``/sdlc`` skill's Phase 4 handler) surface them as individual + ``choice`` / ``feedback`` decisions. Resolutions are written back to + the contract so implement-phase agents see the human's answers. + + All pending decisions (plus the feedback entry, if any) are queued up + front before any ``wait_for_decision`` call, so ``get_status`` surfaces + them as a single batch. Callers can then prompt for up to 4 at a time + and submit answers in parallel, collapsing what was previously N prompts + and N polling cycles into ~⌈N/4⌉ prompts and one cycle (issue #1956). + + Once the batch is queued, a single ``decision.created`` event is + published to the EventBus so event-driven watchers (the ``wait-status`` + monitor long-polling ``/status/wait``) wake immediately. + ``DecisionQueue.queue_decision`` itself emits no event, so without this + the bridged decisions are created silently and the operator only + discovers them via a manual ``get_status`` (issue #2770). + """ + try: + from egg_contracts.loader import load_contract, save_contract + except ImportError: + _pkg.logger.warning( + "egg_contracts not available, skipping contract decision bridge", + pipeline_id=pipeline_id, + ) + return 0 + + try: + contract = load_contract(pipeline_identifier, worktree_repo_path) + except Exception as e: + _pkg.logger.debug( + "Contract not loadable, skipping contract decision bridge", + pipeline_id=pipeline_id, + error=str(e), + ) + return 0 + + phase_value = phase.value + pending_decisions = [ + d + for d in contract.decisions + if not d.resolved + and getattr(d.type, "value", d.type) == "hitl" + and (d.phase is None or getattr(d.phase, "value", d.phase) == phase_value) + ] + fb = contract.feedback + pending_feedback = None + if fb is not None and not fb.submitted: + fb_phase_val = getattr(fb.phase, "value", fb.phase) if fb.phase is not None else None + if fb_phase_val is None or fb_phase_val == phase_value: + pending_feedback = fb + + if not pending_decisions and pending_feedback is None: + return 0 + + _pkg.logger.info( + "Bridging contract decisions/feedback into orchestrator queue", + pipeline_id=pipeline_id, + phase=phase_value, + decision_count=len(pending_decisions), + has_feedback=pending_feedback is not None, + ) + + def _save_contract_update(mutator: _pkg.Callable[[_pkg.Any], bool]) -> None: + try: + latest = load_contract(pipeline_identifier, worktree_repo_path) + except Exception as e: + _pkg.logger.warning( + "Could not reload contract to persist bridged resolution", + pipeline_id=pipeline_id, + error=str(e), + ) + return + if not mutator(latest): + return + try: + save_contract(latest, worktree_repo_path) + except Exception as e: + _pkg.logger.warning( + "Failed to save contract after bridged resolution", + pipeline_id=pipeline_id, + error=str(e), + ) + + # Pass 1: queue every pending decision + feedback up front. + queued_decisions: list[tuple[str, _pkg.Any]] = [] + for contract_decision in pending_decisions: + options_labels = [opt.label for opt in contract_decision.options] + queued = dq.queue_decision( + question=contract_decision.question, + context=( + f"Open contract question {contract_decision.id}, " + f"registered by an agent during the {phase_value} phase." + ), + options=options_labels, + decision_type="choice", + phase=phase, + ) + queued_decisions.append((contract_decision.id, queued)) + + queued_feedback: _pkg.HITLDecision | None = None + if pending_feedback is not None: + questions_payload = [ + {"id": q.id, "question": q.question, "answer": ""} for q in pending_feedback.questions + ] + queued_feedback = dq.queue_decision( + question=f"Open feedback request {pending_feedback.id}", + context=( + f"Open contract feedback {pending_feedback.id}, " + f"registered by an agent during the {phase_value} phase." + ), + options=[], + decision_type="feedback", + questions=questions_payload, + phase=phase, + ) + + # Surface the freshly-queued batch to event-driven watchers before + # blocking on resolution. ``DecisionQueue.queue_decision`` emits no + # EventBus event, so without this the bridged decisions are created + # silently — the operator's ``wait-status`` monitor never wakes and + # only finds them via a manual ``get_status`` (#2770). The phase_gate + # decision emits ``decision.created`` the same way. + if _pkg._emit_event is not None: + _pkg._emit_event( + _pkg.EventType.DECISION_CREATED, + pipeline_id, + data={"phase": phase_value}, + ) + + # Pass 2: wait for each to resolve and persist back to the contract. + # Count only decisions whose queue resolution was RESOLVED — a + # CANCELLED / non-resolved outcome leaves the contract ``cq-N`` open and + # must NOT count toward the convergence signal, or the caller would re-run + # the phase, re-surface the still-open question (carry-forward only adopts + # *resolved* questions), and loop without the operator ever being able to + # break out (#3392 review). + resolved_count = 0 + for contract_id, queued in queued_decisions: + resolved = dq.wait_for_decision(queued.id) + if resolved.status != _pkg.DecisionStatus.RESOLVED: + continue + resolved_count += 1 + resolution_str = (resolved.resolution or "").strip() + + def _apply(latest: _pkg.Any, _cd_id: str = contract_id, _res: str = resolution_str) -> bool: + for d in latest.decisions: + if d.id == _cd_id: + d.resolved = True + d.resolution = _res + d.resolved_by = "human" + d.resolved_at = _pkg.datetime.now(_pkg.UTC) + return True + return False + + _save_contract_update(_apply) + + feedback_resolved = False + if queued_feedback is not None and pending_feedback is not None: + resolved = dq.wait_for_decision(queued_feedback.id) + if resolved.status == _pkg.DecisionStatus.RESOLVED: + feedback_resolved = True + answers: dict[str, str] = {} + try: + payload = _pkg.json.loads(resolved.resolution or "") + if isinstance(payload, dict): + raw_answers = payload.get("answers") + if isinstance(raw_answers, dict): + answers = {str(k): str(v) for k, v in raw_answers.items()} + except _pkg.json.JSONDecodeError, TypeError: + pass + + fb_id = pending_feedback.id + + def _apply_fb( + latest: _pkg.Any, _fb_id: str = fb_id, _answers: dict[str, str] = answers + ) -> bool: + if latest.feedback is None or latest.feedback.id != _fb_id: + return False + for q in latest.feedback.questions: + if q.id in _answers: + q.answer = _answers[q.id] + # Always mark submitted after resolution — even if + # individual answers didn't parse, the human responded + # and shouldn't be asked again. + latest.feedback.submitted = True + latest.feedback.submitted_by = "human" + latest.feedback.submitted_at = _pkg.datetime.now(_pkg.UTC) + return True + + _save_contract_update(_apply_fb) + + # Convergence signal (#3392): the number of decisions + feedback this + # round the operator actually *resolved* (not merely surfaced). Non-zero ⇒ + # the operator answered something ⇒ caller re-runs the phase to fold the + # resolutions in. A surfaced-but-cancelled decision is deliberately + # excluded: counting it would re-run the phase, re-surface the still-open + # question, and loop indefinitely now that the force-advance backstop is + # gone. + return resolved_count + (1 if feedback_resolved else 0) + + +def _await_unresolved_gap_gate( + store: _pkg.Any, + pipeline_id: str, + repo_path: _pkg.Path, + worktree_repo_path: _pkg.Path, + pipeline_identifier: int | str, + phase: _pkg.PipelinePhase, + hitl_gates: bool = True, +) -> bool: + """Block phase finalize while the contract carries unresolved TaskGaps. + + A tester→coder :class:`TaskGap` left ``resolved == False`` ships into + the committed contract snapshot and fails ``test_models_gaps.py`` red + in CI on the already-open PR (#3298 class 4). The implement phase is + **not** in ``_HITL_GATE_PHASES``, so the autonomous run loop would + otherwise mark it complete and finalize with the gap open and no + human in the loop. This surfaces a blocking ``phase_gate`` HITL + decision listing the open gaps and waits — mirroring the + unresolved-HITL guard in ``complete_phase`` (#1788). See #3300. + + The operator resolves the gap (set the gap's ``resolved=true`` via + the contract-mutate path, e.g. by re-running/kicking the coder) and + approves, or picks the override option to ship with the gap open. The + contract is re-read after each approval so a stale ``approve`` cannot + advance with a gap still open. Returns ``True`` when a gate was + surfaced and the contract may have changed (resolved or overridden), + ``False`` when the contract was already clean (the common path) or + the escalation could only be logged (autonomous run, below). + + **Autonomous runs.** ``wait_for_decision`` polls indefinitely and + both options require a human, so a fully-autonomous pipeline + (``hitl_gates is False``) has no path forward — blocking here would + convert a red-but-progressing PR into an indefinite stall (the + health monitor would eventually tear it down). When ``hitl_gates is + False`` we therefore *surface* the escalation (event + warning) but + do **not** block: the reactive ``test_models_gaps.py`` CI check + remains the backstop, exactly as it was before this gate existed. + + A best-effort scan: contract load failures fail open (log + return) + so a transient read error can never strand the pipeline. + """ + try: + from egg_contracts.loader import load_contract + except ImportError: + _pkg.logger.warning( + "egg_contracts not available, skipping unresolved-gap gate", + pipeline_id=pipeline_id, + ) + return False + + def _load_open_gaps() -> list[tuple[str, _pkg.Any]] | None: + try: + contract = load_contract(pipeline_identifier, worktree_repo_path) + except Exception as e: # noqa: BLE001 + _pkg.logger.warning( + "Could not load contract for unresolved-gap gate (skipping)", + pipeline_id=pipeline_id, + error=str(e), + ) + return None + return contract.unresolved_gaps() + + open_gaps = _load_open_gaps() + if not open_gaps: + return False + + if not hitl_gates: + # No human in the loop — do not block forever. Both options need a + # human, so blocking would convert a red-but-progressing PR into an + # indefinite stall. Surface the escalation (so observers still see + # it) + log loudly, and let the reactive CI backstop catch the open + # gap on the PR, exactly as before this gate existed. + _pkg.report_pipeline_status( + store.load_pipeline(pipeline_id), + event_type="phase.gap_gate", + message=f"{phase.value} phase has unresolved coverage gaps", + ) + _pkg.logger.warning( + "Unresolved-gap gate: open gaps on an autonomous pipeline " + "(hitl_gates=False); surfacing but not blocking", + pipeline_id=pipeline_id, + phase=phase.value, + open_gap_ids=[f"{t}/{g.id}" for t, g in open_gaps], + ) + return False + + def _set_status(status: _pkg.PipelineStatus) -> _pkg.Pipeline: + # Mirror the phase_gate block: drive both pipeline and the phase + # box so the DAG visualization renders the gate on the right + # phase, and the operator's wait-status monitor wakes. + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + pipeline.status = status + phase_execution = pipeline.get_phase_execution(phase) + if phase_execution is not None: + phase_execution.status = status + store.save_pipeline(pipeline) + return pipeline + + dq = _pkg.get_decision_queue(pipeline_id, repo_path) + gated = False + + while open_gaps: + gated = True + gap_lines = "\n".join( + f"- `{task_id}` / `{gap.id}` ({gap.from_role}→{gap.to_role}): {gap.description}" + for task_id, gap in open_gaps + ) + question = ( + f"The {phase.value} phase has {len(open_gaps)} unresolved coverage " + f"gap{'s' if len(open_gaps) != 1 else ''}. Resolve " + f"{'them' if len(open_gaps) != 1 else 'it'} (mark the gap resolved " + "via the contract) and approve, or choose 'override' to finalize " + "with the gap open." + ) + context = ( + "These tester→coder coverage gaps are still open on the contract. " + "Finalizing with an open gap ships it into the committed contract " + "and fails CI (test_models_gaps.py) red on the PR.\n\n" + f"{gap_lines}" + ) + decision = dq.queue_decision( + question=question, + context=context, + options=["approve", "override"], + decision_type="phase_gate", + phase=phase, + ) + + # Mark AWAITING_HUMAN + surface to event watchers, mirroring the + # phase_gate block so the operator's wait-status monitor wakes. + pipeline = _set_status(_pkg.PipelineStatus.AWAITING_HUMAN) + _pkg.report_pipeline_status( + pipeline, + event_type="phase.gap_gate", + message=f"{phase.value} phase has unresolved coverage gaps", + ) + if _pkg._emit_event is not None: + _pkg._emit_event( + _pkg.EventType.DECISION_CREATED, + pipeline_id, + data={"phase": phase.value}, + ) + + resolved = dq.wait_for_decision(decision.id) + # Restore RUNNING now the gate cleared (re-set to AWAITING_HUMAN + # above on the next loop if gaps remain). + _set_status(_pkg.PipelineStatus.RUNNING) + + if resolved.status != _pkg.DecisionStatus.RESOLVED: + # Cancelled / abandoned — don't spin; let the loop proceed so + # a cancel can tear the pipeline down. + _pkg.logger.warning( + "Unresolved-gap gate ended without resolution; proceeding", + pipeline_id=pipeline_id, + phase=phase.value, + decision_status=getattr(resolved.status, "value", resolved.status), + ) + return gated + + resolution = (resolved.resolution or "").strip().lower() + if "override" in resolution: + _pkg.logger.warning( + "Unresolved-gap gate overridden — finalizing with open gaps", + pipeline_id=pipeline_id, + phase=phase.value, + open_gap_ids=[f"{t}/{g.id}" for t, g in open_gaps], + ) + # Record the override on the frozen phase artifacts for audit + # parity with the complete_phase endpoint's ``force`` path + # (otherwise the load-bearing run-loop override left only a + # transient log line). Values must be strings — + # PhaseExecution.artifacts is dict[str, str]. + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + phase_execution = pipeline.get_phase_execution(phase) + if phase_execution is not None: + merged = dict(phase_execution.artifacts) + merged["force_completed_gaps"] = _pkg.json.dumps( + [f"{t}/{g.id}" for t, g in open_gaps] + ) + phase_execution.artifacts = merged + store.save_pipeline(pipeline) + return gated + + # Approval path: re-read the contract. If the operator actually + # marked the gaps resolved, the gate clears; otherwise re-surface. + reloaded = _load_open_gaps() + if reloaded is None: + # Load failed — fail open rather than strand the pipeline. + return gated + open_gaps = reloaded + if open_gaps: + _pkg.logger.info( + "Unresolved-gap gate approved but gaps still open; re-surfacing", + pipeline_id=pipeline_id, + phase=phase.value, + remaining=len(open_gaps), + ) + + return gated + + +def _next_phases_for_epic( + pipeline: _pkg.Pipeline, + current_phase: _pkg.PipelinePhase, + default_next_phases: list[_pkg.PipelinePhase], +) -> list[_pkg.PipelinePhase]: + """Reroute auto-advance through ``APPLY`` for Jira-epic pipelines. + + Issue #1557: when ``pipeline.is_epic`` is true the orchestrator + inserts the new ``APPLY`` phase between ``PLAN`` and ``IMPLEMENT`` + so the ``APPLIER`` role can drive Jira mutations (epic-Description + write, child create / link / Won't-Do) on HITL approval. Non-epic + pipelines see ``default_next_phases`` returned unchanged so the + pre-#1557 scheduling is preserved bit-for-bit. + + The orchestrator-side scheduler is the authoritative gate per the + architecture's "VALID_TRANSITIONS lists APPLY but the scheduler + decides whether to actually pick it" design (see the comment on + :data:`gateway.phase_transition.VALID_TRANSITIONS`). Returns a + single-element list so the call site's ``next_phases[0]`` indexing + works without change. + """ + if not getattr(pipeline, "is_epic", False): + return default_next_phases + if current_phase == _pkg.PipelinePhase.PLAN: + return [_pkg.PipelinePhase.APPLY] + if current_phase == _pkg.PipelinePhase.APPLY: + return [_pkg.PipelinePhase.IMPLEMENT] + return default_next_phases + + +def _drain_wontdo_batch_after_apply( + pipeline: _pkg.Pipeline, + worktree_repo_path: _pkg.Path, +) -> None: + """Run the orchestrator-only Won't-Do drain after ``APPLY`` consensus. + + Trigger chain (issue #1557 task-2-7): the HITL operator approves + the plan-gate → ``_persist_phase_gate_resolution`` flips state → + the scheduler routes through ``APPLY`` → the applier writes a + handoff JSON at ``.egg-state/agent-outputs/<pipeline>-wontdo.json`` + listing every obsolete child key it could not transition itself + (decision-15: agent-facing routes deny Jira transitions) → the + APPLIER's CONSENSUS_PROPOSE → REVIEWER_CONTRACT ACK confirms → + this hook fires from the auto-advance block, iterates the handoff, + and POSTs to ``/api/v1/jira/ticket/transition`` with the launcher- + secret bearer token. + + Runs **out of band** from ``_persist_phase_gate_resolution`` so a + slow Jira API does not extend the HITL approve POST's latency SLA + (task-2-7 acceptance). Fail-open: a missing handoff file means + "no Won't-Dos to drain" and returns silently; a per-transition + failure surfaces as a logger warning but does not block the + pipeline from advancing to ``IMPLEMENT``. + + Naming note (reviewer_code v1 non-blocking): the handoff file + this function READS is the applier's *output* + (``<pipeline.id>-wontdo.json``), distinct from the applier's + *input* handoff (``<pipeline.id>-apply-handoff.json``) written + by :func:`_write_apply_phase_handoff` just before APPLY spawns. + + Per-Task lifecycle (reviewer_contract v1 finding #3 / task-2-7): + the drain registers an ``on_entry_result`` callback with + ``run_wontdo_drain``. After each transition attempt, the callback + loads the contract via ``egg_contracts.loader.load_contract``, + locates the corresponding Task (by ``task_id`` when the applier + included one in the handoff entry, otherwise by ``jira_key`` + match), and writes ``Task.jira_action_status = 'applied'`` / + ``'failed'`` plus the failure reason into ``Task.notes``. The + write is best-effort: contract-load / save failures surface as a + logger warning so a brittle contract state never breaks the + drain — the operator can re-run later with the same handoff JSON + (the gateway's idempotency cache absorbs the duplicate transition + calls within the 5-minute window). + """ + handoff_path = ( + _pkg.Path(worktree_repo_path) + / ".egg-state" + / "agent-outputs" + / f"{pipeline.id}-wontdo.json" + ) + if not handoff_path.exists(): + _pkg.logger.debug( + "Won't-Do drain skipped — no handoff file produced by applier", + pipeline_id=pipeline.id, + handoff_path=str(handoff_path), + ) + return + + # Per-entry contract writeback callback (reviewer_contract v1 #3). + # Each invocation looks up the task by ``task_id`` (when the + # applier set it on the handoff entry) or by ``jira_key`` match + # otherwise, flips ``jira_action_status`` to ``'applied'`` / + # ``'failed'`` and records the failure reason in ``Task.notes``. + def _on_entry_result(entry: _pkg.Any, ok: bool, reason: str) -> None: + try: + try: + from egg_contracts.loader import load_contract, save_contract + except ImportError: # pragma: no cover - defensive + _pkg.logger.warning( + "Won't-Do drain: egg_contracts loader unavailable; " + "skipping per-Task lifecycle writeback", + pipeline_id=pipeline.id, + ) + return + try: + contract = load_contract(pipeline.id, worktree_repo_path) + except Exception as load_err: # noqa: BLE001 + _pkg.logger.warning( + "Won't-Do drain: contract load failed; skipping per-Task lifecycle writeback", + pipeline_id=pipeline.id, + error=str(load_err), + ) + return + target_task = None + entry_task_id = getattr(entry, "task_id", None) + entry_key = getattr(entry, "jira_key", None) + for sl in getattr(contract, "slices", []) or []: + for tsk in getattr(sl, "tasks", []) or []: + if entry_task_id and tsk.id == entry_task_id: + target_task = tsk + break + if ( + not entry_task_id + and entry_key + and getattr(tsk, "jira_key", None) == entry_key + ): + target_task = tsk + break + if target_task is not None: + break + if target_task is None: + # No matching task — applier-written handoff may have + # entries for keys outside the contract's task list + # (e.g. consolidate-into "obsolete-only" rows). Log + # at DEBUG since this is expected for split / consolidate + # patterns. + _pkg.logger.debug( + "Won't-Do drain: no contract task matches handoff entry; " + "skipping lifecycle writeback for this row", + pipeline_id=pipeline.id, + entry_task_id=entry_task_id, + entry_key=entry_key, + ) + return + target_task.jira_action_status = "applied" if ok else "failed" + if not ok: + existing_notes = target_task.notes or "" + failure_note = f"wontdo drain failed: {reason}" + target_task.notes = existing_notes + ("\n" if existing_notes else "") + failure_note + try: + save_contract(contract, worktree_repo_path) + except Exception as save_err: # noqa: BLE001 + _pkg.logger.warning( + "Won't-Do drain: contract save failed after lifecycle writeback", + pipeline_id=pipeline.id, + error=str(save_err), + ) + except Exception as cb_err: # noqa: BLE001 - defensive + _pkg.logger.warning( + "Won't-Do drain: per-Task callback raised (continuing)", + pipeline_id=pipeline.id, + error=str(cb_err), + ) + + # Contract-state idempotency gate. The drain consults this predicate + # before posting each transition so a benign re-run (orchestrator + # restart, manual re-drain, re-entry of the apply phase) does not + # double-POST transitions whose outcomes the gateway's 5-minute + # idempotency cache has long since forgotten — and does not flip an + # ``'applied'`` Task back to ``'failed'`` when Jira returns 400 for + # an already-transitioned ticket. + def _entry_already_applied(entry: _pkg.Any) -> bool: + try: + try: + from egg_contracts.loader import load_contract + except ImportError: # pragma: no cover - defensive + _pkg.logger.warning( + "Won't-Do drain idempotency gate disarmed: egg_contracts.loader not importable", + pipeline_id=pipeline.id, + ) + return False + try: + contract = load_contract(pipeline.id, worktree_repo_path) + except Exception as load_err: # noqa: BLE001 - defensive + # Contract unreadable / corrupted: idempotency gate is + # disarmed for this drain run. The drain re-POSTs every + # entry, Jira returns 400 for already-transitioned ones, + # and ``_on_entry_result`` flips ``'applied'`` → + # ``'failed'`` — surface this loudly so the operator can + # repair the contract before the next re-run. + _pkg.logger.warning( + "Won't-Do drain idempotency gate disarmed: load_contract failed", + pipeline_id=pipeline.id, + error=str(load_err), + ) + return False + entry_task_id = getattr(entry, "task_id", None) + entry_key = getattr(entry, "jira_key", None) + for sl in getattr(contract, "slices", []) or []: + for tsk in getattr(sl, "tasks", []) or []: + matches_task = bool(entry_task_id and tsk.id == entry_task_id) + matches_key = bool( + not entry_task_id + and entry_key + and getattr(tsk, "jira_key", None) == entry_key + ) + if matches_task or matches_key: + return getattr(tsk, "jira_action_status", None) == "applied" + return False + except Exception as predicate_err: # noqa: BLE001 - defensive + _pkg.logger.warning( + "Won't-Do drain idempotency gate raised; treating entry as not-yet-applied", + pipeline_id=pipeline.id, + error=str(predicate_err), + ) + return False + + try: + # Reviewer_code v1 non-blocking note: mirror the dual-import + # pattern used elsewhere in this module (e.g. ``from + # jira_epic import resolve_epic_mode``) so the helper still + # resolves when ``orchestrator/`` is imported as a package + # rather than treated as ``sys.path`` root. + try: + from wontdo_drain import run_wontdo_drain + except ImportError: # pragma: no cover — packaged-import fallback + from orchestrator.wontdo_drain import run_wontdo_drain # type: ignore[no-redef] + + result = run_wontdo_drain( + handoff_path=handoff_path, + on_entry_result=_on_entry_result, + is_already_applied=_entry_already_applied, + ) + except Exception as exc: # noqa: BLE001 — defensive: drain must not crash auto-advance + _pkg.logger.warning( + "Won't-Do drain failed after APPLY phase (continuing)", + pipeline_id=pipeline.id, + error=str(exc), + ) + return + _pkg.logger.info( + "Won't-Do drain complete after APPLY phase", + pipeline_id=pipeline.id, + succeeded=len(result.succeeded), + failed=len(result.failed), + skipped=len(result.skipped), + ) + + +def _write_apply_phase_handoff( + pipeline: _pkg.Pipeline, + worktree_repo_path: _pkg.Path, + approved_phase: str, +) -> None: + """Write the applier handoff JSON before the ``APPLY`` phase spawns. + + The applier prompt consumes a one-line JSON identifying which + artifact was just approved so it can branch between refine-apply + (writing the analysis to the epic Description) and plan-apply + (walking ``Task.jira_action`` + driving the Jira CLI per task). + + The handoff lands at + ``.egg-state/agent-outputs/<pipeline-id>-apply-handoff.json`` + inside the per-pipeline worktree so the applier (running in a + sandbox container with the same worktree mounted) reads from a + deterministic path. Fail-open: I/O errors surface as a logger + warning but never abort phase advancement. + """ + handoff_dir = _pkg.Path(worktree_repo_path) / ".egg-state" / "agent-outputs" + try: + handoff_dir.mkdir(parents=True, exist_ok=True) + except OSError as exc: + _pkg.logger.warning( + "Failed to create agent-outputs dir for applier handoff (continuing)", + pipeline_id=pipeline.id, + error=str(exc), + ) + return + contract_path = ( + _pkg.Path(worktree_repo_path) / ".egg-state" / "contracts" / f"{pipeline.id}.json" + ) + draft_path = ( + _pkg.Path(worktree_repo_path) + / ".egg-state" + / "brc-history" + / f"{pipeline.id}-{approved_phase}.md" + ) + payload = { + "approved_phase": approved_phase, + "contract_path": str(contract_path), + "draft_path": str(draft_path), + } + handoff_path = handoff_dir / f"{pipeline.id}-apply-handoff.json" + try: + handoff_path.write_text(_pkg.json.dumps(payload, indent=2) + "\n", encoding="utf-8") + except OSError as exc: + _pkg.logger.warning( + "Failed to write applier handoff JSON (continuing)", + pipeline_id=pipeline.id, + handoff_path=str(handoff_path), + error=str(exc), + ) + return + _pkg.logger.info( + "Applier handoff JSON written for APPLY phase", + pipeline_id=pipeline.id, + approved_phase=approved_phase, + handoff_path=str(handoff_path), + ) + + +def _persist_phase_gate_resolution( + repo_path: _pkg.Path, + pipeline_id: str, + decision: _pkg.HITLDecision, + phase: str, + issue_number: int | None = None, +) -> None: + """Persist a phase-gate resolution to the contract and draft. + + After a human approves a phase gate, the resolution context needs to be + visible to agents in the next phase. This function: + + 1. Adds the resolution as a HITL decision in the contract so next-phase + agents see it when they load the contract. + 2. Appends a ``## HITL Resolution`` section to the phase draft file so + agents reading the draft also see the human's decisions. + + See: #1295 + """ + # Extract structured context from JSON resolution, or use raw string + resolution_context: str = "" + raw = (decision.resolution or "").strip() + if raw: + try: + payload = _pkg.json.loads(raw) + if isinstance(payload, dict): + resolution_context = payload.get("context", "") or payload.get("feedback", "") + if not resolution_context: + _pkg.logger.debug( + "Phase gate approved without context, nothing to persist", + pipeline_id=pipeline_id, + phase=phase, + ) + return + else: + resolution_context = raw + except _pkg.json.JSONDecodeError, TypeError: + resolution_context = raw + + if not resolution_context: + _pkg.logger.debug( + "Phase gate resolution has no context to persist", + pipeline_id=pipeline_id, + phase=phase, + ) + return + + # --- 1. Sync to contract --- + try: + from egg_contracts.loader import load_contract, save_contract + from egg_contracts.models import Decision, DecisionOption, DecisionType + + contract = load_contract(pipeline_id, repo_path) + + existing_questions = {d.question for d in contract.decisions} + question_text = f"[Phase gate: {phase}] {decision.question}" + + if question_text not in existing_questions: + # Determine next decision ID + max_existing_id = 0 + for d in contract.decisions: + try: + num = int(d.id.split("-")[1]) + max_existing_id = max(max_existing_id, num) + except IndexError, ValueError: + pass + + contract_options = [ + DecisionOption(id=f"opt-{i + 1}", label=opt) + for i, opt in enumerate(decision.options) + ] + + contract_decision = Decision( + id=f"decision-{max_existing_id + 1}", + question=question_text, + type=DecisionType.HITL, + options=contract_options, + resolved=True, + resolution=resolution_context, + resolved_by="human", + resolved_at=decision.resolved_at, + ) + contract.decisions.append(contract_decision) + save_contract(contract, repo_path) + _pkg.logger.info( + "Persisted phase gate resolution to contract", + pipeline_id=pipeline_id, + phase=phase, + ) + except ImportError: + _pkg.logger.warning("egg_contracts not available, skipping phase gate contract sync") + except Exception: + _pkg.logger.warning( + "Failed to persist phase gate resolution to contract (continuing)", + pipeline_id=pipeline_id, + phase=phase, + exc_info=True, + ) + + # --- 2. Append to draft --- + try: + draft_rel = _pkg._get_draft_path(phase, issue_number, pipeline_id) + if draft_rel: + draft_path = repo_path / draft_rel + if draft_path.exists(): + existing = draft_path.read_text(encoding="utf-8") + if "## HITL Resolution" not in existing: + section = ( + f"\n\n## HITL Resolution\n\n" + f"The following was approved by a human reviewer at the " + f"{phase} phase gate:\n\n{resolution_context}\n" + ) + draft_path.write_text(existing + section, encoding="utf-8") + _pkg.logger.info( + "Appended HITL resolution to draft", + pipeline_id=pipeline_id, + phase=phase, + draft=draft_rel, + ) + except Exception: + _pkg.logger.warning( + "Failed to append phase gate resolution to draft (continuing)", + pipeline_id=pipeline_id, + phase=phase, + exc_info=True, + ) diff --git a/orchestrator/routes/pipelines/_lifecycle_helpers.py b/orchestrator/routes/pipelines/_lifecycle_helpers.py new file mode 100644 index 0000000000..5de24ae559 --- /dev/null +++ b/orchestrator/routes/pipelines/_lifecycle_helpers.py @@ -0,0 +1,339 @@ +"""pipeline lifecycle mutation helpers helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +from typing import Literal # noqa: F401 + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _normalize_submission_repos( + repos_arg: _pkg.Any, +) -> tuple[str | None, list[dict[str, str | None]], str | None, str | None]: + """Validate + normalize a multi-repo submission list (#3393). + + Accepts the ``repos`` payload from ``POST /api/v1/pipelines`` — a list of + ``{repo, base_branch?, primary?}`` entries (a bare ``"owner/name"`` string + is tolerated as ``{repo: ...}``). Returns + ``(error, entries, primary_repo, primary_base_branch)``: + + * ``error`` — a human-readable message when validation fails (the other + fields are meaningless in that case), else ``None``. + * ``entries`` — normalized ``{"repo", "base_branch"}`` dicts, reordered so + the primary is ``entries[0]`` (the ``Pipeline`` validator mirrors + ``repos[0]`` onto the legacy singleton and ``primary_repo``). + + Per-entry repo/base_branch formats are validated with the same regexes the + single-repo path uses. Same-name repos under different owners are NOT + rejected here — they are distinct full ``owner/name`` slugs (operator + ruling #6; the owner/repo re-key lands in slice 3). + """ + if not isinstance(repos_arg, list) or not repos_arg: + return ("repos must be a non-empty list of {repo, base_branch} entries", [], None, None) + entries: list[dict[str, str | None]] = [] + primary_index = 0 + seen_primary = False + for idx, raw in enumerate(repos_arg): + entry = {"repo": raw} if isinstance(raw, str) else raw + if not isinstance(entry, dict) or not entry.get("repo"): + return (f"repos[{idx}] must be an object with a 'repo' field", [], None, None) + repo_val = entry["repo"] + if not _pkg.re.match(r"^[a-zA-Z0-9._-]+/[a-zA-Z0-9._-]+$", repo_val): + return ( + f"Invalid repo format in repos[{idx}]: {repo_val!r} (expected owner/name)", + [], + None, + None, + ) + base_val = entry.get("base_branch") + if base_val is not None and ( + not _pkg.re.match(r"^[a-zA-Z0-9_./-]+$", base_val) or ".." in base_val + ): + return (f"Invalid base_branch in repos[{idx}]: {base_val!r}", [], None, None) + entries.append({"repo": repo_val, "base_branch": base_val}) + if entry.get("primary"): + if seen_primary: + return ("At most one repos entry may set 'primary'", [], None, None) + seen_primary = True + primary_index = idx + # Reorder so the primary is first: the Pipeline model mirrors repos[0] + # onto the legacy repo/base_branch singleton and exposes it as + # ``primary_repo``. + if primary_index != 0: + entries.insert(0, entries.pop(primary_index)) + primary = entries[0] + return (None, entries, primary["repo"], primary["base_branch"]) + + +def _assert_repo_set_uniform(repos: list[str]) -> str | None: + """Reject mixed-visibility / mixed-auth repo sets at submission (#3393, task-2-2). + + A pipeline-wide private-mode posture (context filtering, egress rules) + requires every repo in one run to be uniformly private or uniformly public, + and — for v1 — to share a single auth mode. Returns an actionable, + repo-naming error string when the set diverges on either dimension, or + ``None`` when it is uniform. A single repo (after de-duplication) is + trivially uniform and short-circuits before any lookup, so N=1 pipelines + pay no cost and make no gateway round-trip. + + Runtime note (container boundary): the orchestrator image bundles + ``config/repo_config.py`` but NOT ``gateway/``, so the per-repo lookups are + reached the way the orchestrator already reaches them — auth via + ``repo_config.assert_uniform_auth`` (imported directly, the same callable the + gateway's ``validate_auth_mode_uniformity`` delegates to) and visibility via + ``GatewayClient.get_repo_visibility`` over HTTP (the gateway holds the + tokens; mirrors ``_compute_gateway_mode``). ``internal`` counts as private. + The visibility comparison below is the HTTP-boundary twin of + ``gateway.repo_visibility.validate_visibility_uniformity`` (which the + orchestrator cannot import); keep the two in step. + """ + unique = list(dict.fromkeys(repos)) + if len(unique) <= 1: + return None + + # Auth-mode uniformity — repo_config is bundled into the orchestrator image. + try: + from repo_config import assert_uniform_auth + + assert_uniform_auth(unique) + except ValueError as exc: + return str(exc) + except Exception as exc: # pragma: no cover - defensive (config read failure) + # Fail CLOSED for consistency with the visibility boundary below + # (reviewer_security v1): a config-read failure means we cannot prove a + # uniform auth mode, so we must not admit the set. repo_config is a + # local, bundled read — this path is genuinely exceptional, not a + # transient network hiccup. + _pkg.logger.warning("Auth-mode uniformity check errored; failing closed", error=str(exc)) + return ( + "Could not determine the auth mode for the pipeline's repos, so a " + "uniform bot/user auth mode cannot be verified. Resubmit once repo " + "configuration is resolvable." + ) + + # Visibility uniformity — resolved via the gateway (the orchestrator's only + # visibility source). FAIL CLOSED on an indeterminate lookup (reviewer_security + # v1): for a multi-repo set (we only reach here when len(unique) > 1) a repo + # whose visibility cannot be resolved to a known bucket means the uniform + # private/public posture cannot be PROVEN — and this is a confidentiality + # boundary (a mixed set that slips through would let private-repo content + # flow through shared plan/contract/PR surfaces into a public repo, with no + # downstream re-check: _compute_gateway_mode derives the network mode from + # the PRIMARY repo only). N=1 short-circuits above, so the common case pays + # nothing. This mirrors gateway.repo_visibility.validate_visibility_uniformity; + # keep the two in step. Unrecognized (non-None) labels are treated as + # indeterminate too — only the known {public|private|internal} contract admits. + gw = _pkg.get_gateway_client() + posture: dict[str, list[str]] = {} + for repo in unique: + vis = gw.get_repo_visibility(repo) + if vis in ("private", "internal"): + bucket = "private" + elif vis == "public": + bucket = "public" + else: + return ( + f"Could not determine repository visibility for {repo!r}; cannot " + "verify a uniform private/public posture across the pipeline's " + "repos (a run must be uniformly private or uniformly public so " + "private-repo content cannot leak through shared plan/contract/PR " + "surfaces). Resubmit once the repo's visibility is resolvable." + ) + posture.setdefault(bucket, []).append(repo) + if len(posture) > 1: + groups = "; ".join(f"{b}: {', '.join(sorted(rs))}" for b, rs in sorted(posture.items())) + return ( + "Mixed repository visibility across the pipeline's repos is not allowed " + "(a run must be uniformly private or uniformly public, so private-repo " + f"content cannot leak through shared plan/PR surfaces). Diverging repos — {groups}." + ) + return None + + +def _clear_pipeline_runtime_state(pipeline_id: str, *, reason: str) -> None: + """Evict per-pipeline runtime state that is keyed by pipeline_id alone. + + The peer-consensus tracker, the legacy consensus evaluator, and the + inter-agent message store are all keyed by pipeline_id. Without a + matching ``run_epoch`` namespace, a fresh pipeline that reuses an id + from a prior terminal run (same branch, e.g. ``issue-1965``) will + inherit the prior run's CONFIRMED consensus and message history. The + leak surfaces in the ``/status/wait`` route's Path-B envelope, which + would report ``concurrent.consensus.is_complete: true`` for a + pipeline that has not spawned any agents yet (#2053). + + Called when a pipeline transitions to a terminal status, when its + state file is deleted, and immediately after a fresh pipeline is + created (covers paths that bypass PATCH/DELETE — auto-FAILED, and + Redis-backed message-store entries that survived an orchestrator + restart between cancel and resubmit). + """ + try: + try: + from peer_consensus import remove_peer_consensus_tracker + except ImportError: + from ..peer_consensus import ( # type: ignore[no-redef] + remove_peer_consensus_tracker, + ) + remove_peer_consensus_tracker(pipeline_id) + except ImportError: + pass + except Exception as e: + _pkg.logger.warning( + "Failed to clear peer consensus tracker", + pipeline_id=pipeline_id, + reason=reason, + error=str(e), + ) + + # Reconstruct-from-messages would otherwise replay the prior run's + # CONSENSUS_* messages and rebuild a CONFIRMED tracker, defeating the + # tracker eviction above. + try: + try: + from message_store import get_message_store + except ImportError: + from ..message_store import get_message_store # type: ignore[no-redef] + get_message_store().clear(pipeline_id) + except ImportError: + pass + except Exception as e: + _pkg.logger.warning( + "Failed to clear message store", + pipeline_id=pipeline_id, + reason=reason, + error=str(e), + ) + + +def _mark_pipeline_records_terminated( + store: _pkg.StateStore, + pipeline_id: str, +) -> _pkg.Pipeline: + """Mark all running containers and agents as stopped after pipeline termination. + + Called when a pipeline transitions to a terminal state (cancelled or failed). + After Docker containers are force-removed, the pipeline state still shows + them as "running". This reloads the latest state from the store (to avoid + overwriting updates made between the status change and container + cleanup), marks running records as stopped, and saves. + + Returns the updated pipeline so the caller can use it in the response. + """ + pipeline = store.load_pipeline(pipeline_id) + now = _pkg.datetime.now(_pkg.UTC) + changed = False + + for phase_exec in pipeline.phases.values(): + for container in phase_exec.containers: + if container.status in ( + _pkg.ContainerStatus.PENDING, + _pkg.ContainerStatus.CREATING, + _pkg.ContainerStatus.RUNNING, + ): + container.status = _pkg.ContainerStatus.REMOVED + container.exited_at = now + changed = True + + for agent in phase_exec.agents: + if agent.status in ( + _pkg.AgentExecutionStatus.PENDING, + _pkg.AgentExecutionStatus.RUNNING, + ): + agent.status = _pkg.AgentExecutionStatus.FAILED + agent.completed_at = now + agent.error = f"Pipeline {pipeline.status.value}" + changed = True + + if changed: + store.save_pipeline(pipeline) + _pkg.logger.info( + "Synced pipeline state after termination", + pipeline_id=pipeline_id, + ) + + return pipeline + + +def _compute_gateway_mode( + pipeline: _pkg.Pipeline, +) -> tuple[Literal["public", "private"], str | None]: + """Compute gateway session mode from pipeline config and repo visibility. + + Uses the explicit ``network_mode`` if set, otherwise auto-detects from + repository visibility via the gateway. Defaults to ``"public"``. + + Returns: + A ``(mode, visibility)`` tuple. ``visibility`` is ``None`` when + ``network_mode`` is explicit, the pipeline has no repo, or the + gateway query failed. + """ + if pipeline.network_mode: + return pipeline.network_mode, None + if pipeline.repo: + vis = _pkg.get_gateway_client().get_repo_visibility(pipeline.repo) + if vis in ("private", "internal"): + return "private", vis + return "public", vis + return "public", None + + +def _cleanup_remote_branches( + pipeline_id: str, + pipeline: _pkg.Pipeline, + repo_path: _pkg.Path, +) -> None: + """Best-effort cleanup of remote branches for a pipeline. + + Deletes the pipeline's shared branch (``pipeline.branch``, typically + ``egg/{pipeline_id}/work`` since #2399) and every per-container + worktree branch (``egg/{container_id}/work``). Slice integration + branches at ``egg/{pipeline_id}/slice-N`` are siblings of the + pipeline tip and are NOT deleted here — see follow-up tracking on + #2399 for full namespace cleanup. Failures are logged as warnings + and do not block pipeline deletion. + """ + branches: set[str] = set() + if pipeline.branch: + branches.add(pipeline.branch) + for phase_exec in pipeline.phases.values(): + for container in phase_exec.containers: + branches.add(f"egg/{container.container_id}/work") + + if not branches: + return + + gateway_client = _pkg.get_gateway_client() + repo_path_str = str(repo_path) + mode, _vis = _pkg._compute_gateway_mode(pipeline) + + deleted = 0 + for branch in sorted(branches): + result = gateway_client.delete_remote_branch(pipeline_id, repo_path_str, branch, mode=mode) + # ``already_deleted`` means the desired state (branch absent on + # remote) is satisfied — count it as success rather than churning a + # warning every time a pipeline is cleaned up before any branch was + # ever pushed. + if result or result.category == "already_deleted": + deleted += 1 + else: + _pkg.logger.warning( + "Remote branch deletion failed during pipeline cleanup", + pipeline_id=pipeline_id, + branch=branch, + category=result.category, + detail=result.detail, + ) + + if deleted: + _pkg.logger.info( + "Cleaned up remote branches", + pipeline_id=pipeline_id, + branches_deleted=deleted, + branches_total=len(branches), + ) diff --git a/orchestrator/routes/pipelines/_overseer.py b/orchestrator/routes/pipelines/_overseer.py new file mode 100644 index 0000000000..1253177756 --- /dev/null +++ b/orchestrator/routes/pipelines/_overseer.py @@ -0,0 +1,740 @@ +"""overseer detection-plane helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Any # noqa: F401 + +import routes.pipelines as _pkg # noqa: E402,F401 +from models import Pipeline, PipelinePhase, PipelineStatus # noqa: F401 + +if TYPE_CHECKING: + from overseer.corrective import CorrectiveExecutor # noqa: F401 + from overseer.decision_maker import AdjudicationVerdict # noqa: F401 + + try: + from ..container_spawner import ContainerSpawner # noqa: F401 + except ImportError: # pragma: no cover + from container_spawner import ContainerSpawner # type: ignore # noqa: F401 + + try: + from ..kubernetes_spawner import SpawnedContainer # noqa: F401 + except ImportError: # pragma: no cover + from kubernetes_spawner import SpawnedContainer # type: ignore # noqa: F401 + + +def _spawn_overseer_agent( + *, + spawner: "ContainerSpawner", # noqa: UP037 + pipeline_id: str, + issue_number: int | None, + gateway_mode: str, + pipeline_repos: list | None, + max_turns: int, + decision_model: str = "sonnet", + prompt_override: str | None = None, +) -> "SpawnedContainer": # noqa: UP037 + """Spawn the overseer as a normal agent (#2270 §1.5). + + The overseer is just a particular agent — it goes through the generic + :meth:`ContainerSpawner.spawn_agent_job` path with a command built by + ``build_agent_command`` exactly like every other role. There is no bespoke + spawn method, no ``EGG_OVERSEER_*`` env, and no baked-in + ``overseer_monitor.py`` bootstrap (that trust-and-run script was the direct + cause of the §1 self-injection loop). Monitoring arrives via the agent's + normal MCP tools / the ``egg-orch`` CLI. + + The overseer's model tier resolves through ``resolve_overseer_model`` (Opus + by default, #2270 §1 / folds #2813); the deprecated + ``overseer_decision_maker_model`` (passed as ``decision_model``) is inert and + only warns. A resolver regression degrades to the built-in opus/anthropic + default rather than crashing spawn. + """ + from agent_model_resolution import ( + DEFAULT_AGENT_MODEL, + UPSTREAM_ANTHROPIC, + classify_model, + resolve_overseer_model, + ) + from egg_agent import build_agent_command + + try: + from ..models import AgentRole + except ImportError: + from models import AgentRole # type: ignore[no-redef] + + # The overseer resolves its model from the pipeline's PRIMARY repo. + # ``pipeline_repos`` is canonically primary-first (#3393 slices 1-2), so + # take the first (primary) entry via ``next(iter(...))`` rather than a + # positional ``[0]`` collapse (#3393 slice-3). + overseer_repo = next(iter(pipeline_repos or []), None) + try: + overseer_decision = resolve_overseer_model( + "adversarial", + pipeline_config=None, + repo=overseer_repo, + ) + except Exception as resolve_err: # noqa: BLE001 — degrade, don't crash + _pkg.logger.warning( + "Failed to resolve overseer model decision for spawn; " + "falling back to built-in opus / anthropic default", + error=str(resolve_err), + ) + overseer_decision = classify_model(DEFAULT_AGENT_MODEL) + + # The bespoke ``overseer_decision_maker_model`` no longer drives the spawn. + # Warn if an operator still sets it to a non-default value (#2270 §1 / #2813). + if decision_model and decision_model != "sonnet": + _pkg.logger.warning( + "overseer_decision_maker_model=%r is deprecated and no longer " + "drives the overseer spawn; the base model now resolves via " + "resolve_agent_model(OVERSEER) -> %s. Set agent_models['overseer'] " + "to override. See #2270 §1 / #2813.", + decision_model, + overseer_decision.claude_code_alias, + ) + + # #2270 §1.5: no bespoke ``EGG_OVERSEER_*`` env — only the generic + # ``BASH_COMMAND_TIMEOUT`` (long-poll CLI calls) and the resolved-decision + # model env (custom-model registration + context guardrails, #2832/#3175). + extra_env = { + "BASH_COMMAND_TIMEOUT": "0", + **overseer_decision.env_vars(), + } + + # Monitoring arrives via MCP tools / the ``egg-orch`` CLI, not a baked-in + # script the agent is told to trust and run. The prompt describes the + # observe→classify→alert loop and leaves the mechanics to the agent's tools. + # When ``prompt_override`` is set (the #2270 slice-4 on-demand adjudicator), + # use it verbatim — a single-shot adjudication of one finding, not the + # continuous monitoring loop. + default_overseer_prompt = ( + f"You are the overseer agent for pipeline {pipeline_id}. You are a " + "normal egg agent with read-only monitoring permissions: there is no " + "baked-in script to run and no pre-built monitoring loop to trust. " + "Observe pipeline health using your MCP tools and the `egg-orch` CLI, " + "and surface only genuine anomalies.\n\n" + "Loop until the pipeline reaches a terminal state (complete, failed, or " + "cancelled):\n" + "1. Read the live pipeline state (`mcp__progress__query_status` or " + "`egg-orch pipeline status`), the BRC consensus matrix " + "(`mcp__brc__get_state`), and recent agent messages.\n" + "2. Classify what you see. The overwhelming majority of observations " + "are normal — only a wedged phase transition, a real consensus " + "deadlock, repeated agent crashes, or similar genuine failures warrant " + "action.\n" + "3. When (and only when) you find a real problem, broadcast a single " + "OVERSEER_ALERT with `mcp__progress__overseer_alert`, setting priority " + "by severity and naming the anomaly, the evidence, and a recommended " + "operator action.\n" + "4. Otherwise wait briefly and repeat.\n\n" + "Be conservative: a false alarm trains operators to ignore you, so " + "prefer silence over a low-confidence alert. When the pipeline ends, " + "emit a final health summary." + ) + overseer_prompt = prompt_override or default_overseer_prompt + command = build_agent_command( + prompt=overseer_prompt, + model=overseer_decision.claude_code_alias, + max_turns=max_turns, + effort=overseer_decision.effort, + ) + + spawn_kwargs: dict[str, _pkg.Any] = { + "pipeline_id": pipeline_id, + "agent_role": AgentRole.OVERSEER, + "issue_number": issue_number, + "repo_volumes": None, + "mode": gateway_mode, + "extra_env": extra_env, + "repos": pipeline_repos if pipeline_repos else None, + "command": command, + } + # Forward per-agent upstream routing only when it would change behavior, so + # the default Anthropic overseer keeps the pre-#2769 call signature (mirrors + # ``concurrent_executor._spawn_agent``). + if ( + overseer_decision.upstream != UPSTREAM_ANTHROPIC + or overseer_decision.upstream_model is not None + ): + spawn_kwargs["upstream"] = overseer_decision.upstream + spawn_kwargs["upstream_model"] = overseer_decision.upstream_model + + return spawner.spawn_agent_job(**spawn_kwargs) + + +def _consume_adjudicator_verdict(spawned: _pkg.Any, finding: _pkg.Any) -> "AdjudicationVerdict": # noqa: UP037 + """Consume the structured verdict an on-demand adjudicator produced. + + Best-effort and defensive (#2270 slice-4). The adjudicator is a NORMAL + spawned agent; its structured verdict reaches the orchestrator either inline + on the spawn result (when a synchronous runner surfaces it as + ``adjudication_verdict`` / ``result_text``) or out-of-band. When no verdict + is available yet, we degrade to a conservative *defer-to-operator* verdict so + a genuine deadlock is never silently dropped — the slice-6 authority plane + executes on whatever this returns. + """ + from overseer.decision_maker import parse_adjudication_verdict + + raw: _pkg.Any = None + for attr in ("adjudication_verdict", "result_text", "stdout"): + value = getattr(spawned, attr, None) + if value: + raw = value + break + return parse_adjudication_verdict(raw, finding=finding) + + +def _overseer_should_be_present( + *, running_agent_count: int, pipeline_status: PipelineStatus +) -> bool: + """Gate overseer presence on agents actually running (#2270 slice-5, §3). + + Decisive rules (the tester contract pins these exactly): + + * ``running_agent_count <= 0`` ⇒ ``False`` regardless of status — the §3 + guarantee that a multi-hour *zero-agent* HITL park spawns no overseer. + * a terminal pipeline status (``COMPLETE`` / ``FAILED`` / ``CANCELLED``) ⇒ + ``False`` regardless of the count — nothing left to monitor. + * otherwise (agents in flight, non-terminal) ⇒ ``True``. + + The overseer is only useful while a phase is actively executing agents, so + presence tracks "are there agents to watch", not the phase calendar. + """ + if running_agent_count <= 0: + return False + if pipeline_status in ( + PipelineStatus.COMPLETE, + PipelineStatus.FAILED, + PipelineStatus.CANCELLED, + ): + return False + return True + + +def _count_phase_agents(pipeline: Pipeline, phase: PipelinePhase) -> int: + """Count the agents a phase is about to run (#2270 slice-5 roster source). + + Prefers the runtime roster cached on the phase execution (populated once + the phase has spawned); falls back to the deterministic + ``get_roles_for_phase`` source the concurrent executor itself consults, so + a not-yet-spawned phase still reports its imminent cohort. A derivation + failure returns 0 — conservatively *no* overseer rather than guessing, + which keeps the §3 "no overseer with zero agents" invariant safe. + """ + phase_exec = pipeline.phases.get(phase) + if phase_exec is not None and getattr(phase_exec, "agents", None): + return len(phase_exec.agents) + try: + from egg_contracts.agent_roles import get_roles_for_phase + + roles = get_roles_for_phase( + phase.value, + include_reviewers=True, + repo=pipeline.repo, + has_contract=getattr(pipeline, "has_contract", True), + ) + return len(list(roles)) + except Exception as exc: # noqa: BLE001 - roster derivation is best-effort + _pkg.logger.debug( + "Could not derive phase roster for overseer presence gate", + pipeline_id=getattr(pipeline, "id", None), + phase=getattr(phase, "value", str(phase)), + error=str(exc), + ) + return 0 + + +def _escalate_finding_to_adjudicator( + finding: _pkg.Any, + *, + spawner: "ContainerSpawner", # noqa: UP037 + pipeline_id: str, + issue_number: int | None, + gateway_mode: str, + pipeline_repos: list | None, + max_turns: int = 3, + spawn_overseer: _pkg.Any = None, + consume_verdict: _pkg.Any = None, +) -> "AdjudicationVerdict | None": # noqa: UP037 + """Escalate a finding to an on-demand OVERSEER adjudicator (#2270 slice-4). + + The escalation→adjudicator path is the ONLY thing the orchestrator-side + overseership spends an agent on. The gate is strict: + + * a finding **without** ``requires_adjudication`` returns ``None`` and NEVER + spawns an adjudicator — the routine majority is handled deterministically; + * a finding **with** ``requires_adjudication`` spawns a NORMAL on-demand + OVERSEER agent (the slice-3 normalized spawn, Opus via the slice-2 + resolver) with a one-shot adjudication prompt, and the orchestrator + consumes its structured verdict in-process. + + ``spawn_overseer`` / ``consume_verdict`` are injectable seams so the path is + unit-testable without a live container; they default to + :func:`_spawn_overseer_agent` and :func:`_consume_adjudicator_verdict`. + """ + if not getattr(finding, "requires_adjudication", False): + return None # routine finding — deterministic handling, no agent spend + + from overseer.decision_maker import build_adjudication_prompt + + spawn = spawn_overseer or _pkg._spawn_overseer_agent + consume = consume_verdict or _pkg._consume_adjudicator_verdict + + prompt = build_adjudication_prompt(finding) + spawned = spawn( + spawner=spawner, + pipeline_id=pipeline_id, + issue_number=issue_number, + gateway_mode=gateway_mode, + pipeline_repos=pipeline_repos if pipeline_repos else None, + max_turns=max_turns, + prompt_override=prompt, + ) + verdict = consume(spawned, finding) + _pkg.logger.info( + "Overseer adjudicated finding", + pipeline_id=pipeline_id, + finding_class=getattr(finding, "finding_class", "?"), + confirmed=getattr(verdict, "confirmed", None), + recommended_action=getattr(verdict, "recommended_action", None), + ) + return verdict + + +def _run_overseer_detection_plane( + snapshot: _pkg.Any, + *, + spawner: "ContainerSpawner", # noqa: UP037 + pipeline_id: str, + issue_number: int | None, + gateway_mode: str, + pipeline_repos: list | None, + plane: _pkg.Any = None, + max_turns: int = 3, +) -> "list[tuple[Any, AdjudicationVerdict | None]]": # noqa: UP037 + """Evaluate the detection plane and escalate only findings that need it. + + The orchestrator-side overseership spine (#2270 Option C, slice-4): run the + deterministic detectors over ``snapshot`` (no LLM), then escalate ONLY the + findings carrying ``requires_adjudication`` to the on-demand adjudicator. + Returns ``(finding, verdict)`` pairs — ``verdict`` is ``None`` for routine + findings that were handled deterministically without an agent. + + The default plane already carries the slice-8 §5 coverage-gap detectors + (registered in :meth:`DetectionPlane.default`), so production runs the full + detector set without any wiring here. + """ + from health_checks.detection_plane import default_detection_plane, escalate_findings + + active_plane = plane or default_detection_plane() + findings = active_plane.evaluate(snapshot) + + results: list[tuple[_pkg.Any, _pkg.Any]] = [] + + def _spawn_adjudicator(finding: _pkg.Any) -> _pkg.Any: + verdict = _pkg._escalate_finding_to_adjudicator( + finding, + spawner=spawner, + pipeline_id=pipeline_id, + issue_number=issue_number, + gateway_mode=gateway_mode, + pipeline_repos=pipeline_repos, + max_turns=max_turns, + ) + results.append((finding, verdict)) + return verdict + + # The canonical gate (health_checks.detection_plane.escalate_findings) calls + # the spawn callback exactly once per requires_adjudication finding and never + # for routine ones — a single source of truth shared with the tester contract. + escalate_findings(findings, spawn_adjudicator=_spawn_adjudicator) + return results + + +def _send_brc_confirmation_nudge( + escalation: dict[str, _pkg.Any], + pipeline_id: str, + phase: str | None, +) -> bool: + """Wake a producer stuck post-ACK with a directed OVERSEER_ALERT (#2079). + + Wired as an escalation callback for HealthMonitor's + ``brc_confirmation_timeout`` alert. The deterministic detector in + ``check_brc_progress`` knows the exact remediation, so we deliver + it directly to the stuck producer rather than relying on the + overseer agent's discretion. + + Uses ``OVERSEER_ALERT`` (not ``STATUS`` or ``NUDGE``) because it + appears in **both** the producer's pre-confirm wait_loop filter + (``CONSENSUS_ACK,CONSENSUS_NACK,CONSENSUS_RE_REVIEW,STATUS,OVERSEER_ALERT``, + post-#2531) and post-confirm wait_loop filter + (``CONSENSUS_CONFIRMED,CONSENSUS_RE_REVIEW,OVERSEER_ALERT``) and has + no protocol-specific semantics that would conflict with a producer + nudge — ``CONSENSUS_RE_REVIEW`` is also in both filters but means + "a peer re-proposed; re-review their artifact," not "you are + wedged; confirm." ``STATUS`` is in the pre-confirm filter (it + carries the orchestrator's *Ready to confirm* nudge) but not the + post-confirm filter, so it wouldn't reach a producer wedged after + a successful confirm. A wedged producer is in the + ``fully_acked but not confirmed`` set, which means they are most + likely blocked on the pre-confirm wait. The subject calls out + that the alert originated from the orchestrator's deterministic + detector rather than the overseer agent. + + Returns True when a message was posted, False otherwise (wrong + alert type, missing fields, message store unavailable, send error). + """ + if escalation.get("alert_type") != "brc_confirmation_timeout": + return False + + producer = escalation.get("agent_id") + if not producer: + return False + + elapsed = escalation.get("elapsed_seconds") + # check_brc_progress always populates elapsed_seconds; treat + # missing or non-positive values as a malformed escalation rather + # than rendering "have not confirmed in 0s" in the body. + if elapsed is None or elapsed <= 0: + return False + + store_fn = _pkg._get_message_store() + if store_fn is None: + return False + + # _get_message_store already verified the package is importable; + # Message/MessageType live in the same module so a defensive + # try/except here would only add per-call import overhead. + from message_store import Message, MessageType + + body = ( + f"You are PROPOSED and fully ACKed but have not confirmed in " + f"{elapsed}s. Call `mcp__brc__confirm` now. If it returns " + "`status='pending_acks'`, read `message` for the guard reason and " + "wait on the prerequisite events instead: `CONSENSUS_PROPOSE` if a " + "producer hasn't proposed (`zero_proposal_producers`), " + "`CONSENSUS_ACK` / `CONSENSUS_RE_REVIEW` if a reviewer's ACK is " + "stale or unresolved. Then retry confirm." + ) + + try: + msg_store = store_fn() + # Bypass the POST /messages/send route on purpose: this is an + # orchestrator-internal nudge, and we do not want HealthMonitor's + # MESSAGE_SENT handler (rate-limit + HEARTBEAT tracking) to see it. + # Future audit/observability subscribers should be aware this path + # does not emit EventType.MESSAGE_SENT. + msg_store.add_message( + Message( + pipeline_id=pipeline_id, + from_role="orchestrator", + to_role=producer, + message_type=MessageType.OVERSEER_ALERT, + subject="BRC confirmation timeout — call mcp__brc__confirm", + body=body, + phase=phase, + metadata={ + "alert_type": "brc_confirmation_timeout", + "elapsed_seconds": elapsed, + "source": "health_monitor", + }, + ) + ) + _pkg.logger.info( + "Sent BRC confirmation-timeout nudge", + pipeline_id=pipeline_id, + producer=producer, + elapsed_seconds=elapsed, + ) + return True + except Exception as send_err: + _pkg.logger.warning( + "Failed to send BRC confirmation-timeout nudge (non-fatal)", + pipeline_id=pipeline_id, + producer=producer, + error=str(send_err), + ) + return False + + +def _corrective_open_operator_hitl( + *, + pipeline_id: str, + issue_number: int | None = None, + repo_path: _pkg.Any = None, + question: str | None = None, + options: _pkg.Any = None, + finding: _pkg.Any = None, + phase: str | None = None, + **_: _pkg.Any, +) -> str: + """``open_operator_hitl`` seam: open a HITL contract decision (orchestrator id). + + The decision is written via :func:`apply_mutation` under ``Role.IMPLEMENTER`` + — the same ``decisions.*`` owner the ``register_open_question`` MCP tool and + the impasse router use — with an orchestrator-side actor so the audit trail + stays distinct from agent-authored decisions. This is the REAL enforcement + point: the contract write runs as the control plane (which has no gateway + agent pattern), while agents — incl. the overseer — stay blocked from + ``.egg-state/contracts/``. Returns the new decision id. + """ + from egg_contracts.decisions import next_cq_id + from egg_contracts.loader import load_contract, save_contract + from egg_contracts.models import Decision, DecisionOption, DecisionType + from egg_contracts.roles import Role + from egg_contracts.validator import apply_mutation + + identifier = _pkg._pipeline_identifier(issue_number, pipeline_id) + resolved_repo = repo_path or _pkg.get_repo_path() + contract = load_contract(identifier, resolved_repo) + existing = contract.decisions or [] + next_idx = len(existing) + decision_id = next_cq_id(existing) + + finding_class = str(getattr(finding, "finding_class", "") or "") + severity = str(getattr(finding, "severity", "") or "medium") + + if question: + question_text = question + else: + lines = [ + f"The overseer detection plane flagged ``{finding_class or 'an anomaly'}`` " + f"(severity ``{severity}``) in pipeline ``{pipeline_id}`` and the on-demand " + "adjudicator escalated it for operator judgement.", + ] + evidence = getattr(finding, "evidence", None) + if evidence: + lines.append(f"**Evidence**: {evidence}") + question_text = "\n".join(lines) + + if options: + decision_options = [ + DecisionOption(id=f"opt-{i + 1}", label=str(label)) for i, label in enumerate(options) + ] + else: + decision_options = [ + DecisionOption(id="opt-1", label="Intervene now (operator will act manually)"), + DecisionOption(id="opt-2", label="Dismiss — detector over-fired (calibration data)"), + DecisionOption(id="opt-3", label="Other (explain in reply)"), + ] + + decision = Decision( + id=decision_id, + question=question_text, + type=DecisionType.HITL, + phase=contract.current_phase, + options=decision_options, + ) + result = apply_mutation( + contract, + role=Role.IMPLEMENTER, + actor="orchestrator-overseer-corrective", + field_path=f"decisions.{next_idx}", + new_value=decision, + reason=f"Overseer corrective: open operator HITL for {finding_class or 'finding'}", + ) + if not result.success: + raise RuntimeError(f"failed to open operator HITL decision: {result.message}") + save_contract(contract, resolved_repo) + # NOTE(#3427): like ``route_impasses``, this overseer-corrective writer + # lands the ``cq-N`` decision with a bare ``save_contract`` and no + # write-time ``persist_contract_statefiles`` — so a HITL opened between + # checkpoints shares the same phase-restart volatility window (the + # ``git reset --hard origin/<work>`` can revert it). The append-only + # guard protects it from id reuse, but not from reversion. Not persisted + # here because the corrective seam runs against ``get_repo_path()`` (the + # base repo), not a pushable pipeline worktree — wiring a worktree-scoped + # persist through the CorrectiveExecutor is the residual follow-up. + return decision_id + + +def _corrective_nudge_agent( + *, + pipeline_id: str, + target_role: str | None = None, + phase: str | None = None, + finding: _pkg.Any = None, + escalation: dict[str, _pkg.Any] | None = None, + **_: _pkg.Any, +) -> bool: + """``nudge_agent`` seam: deliver the deterministic BRC-confirmation nudge. + + Wires to :func:`_send_brc_confirmation_nudge` (the #2079 directed wake), which + posts an ``OVERSEER_ALERT`` the stuck producer's wait-loop filters admit. An + explicit ``escalation`` dict is used when present, otherwise synthesized in + the ``brc_confirmation_timeout`` shape that helper requires. Returns whether + the nudge was delivered. + """ + payload = dict(escalation or {}) + payload.setdefault("alert_type", "brc_confirmation_timeout") + payload.setdefault("agent_id", target_role) + elapsed = payload.get("elapsed_seconds") + payload["elapsed_seconds"] = elapsed if (elapsed and elapsed > 0) else 1 + return _pkg._send_brc_confirmation_nudge(payload, pipeline_id, phase) + + +def _corrective_respawn_cohort( + *, + pipeline_id: str, + target_role: str | None = None, + reason: str | None = None, + **_: _pkg.Any, +) -> bool: + """``respawn_cohort`` seam: restart the target role(s) via the general path. + + Delegates to the orchestrator's public restart endpoint + (``POST /agents/<role>/restart``) — the same general-restart machinery the + overseer monitor's ``_execute_restart_agent`` uses — so restart-budget + enforcement, consensus reset, and one-shot Job teardown all happen + server-side, with no bespoke respawn plumbing. ``target_role`` may be a single + role or a comma-separated cohort. Returns whether every role restarted. + """ + import urllib.request + from urllib.parse import quote + + roles = [r.strip() for r in str(target_role or "").split(",") if r.strip()] + if not roles: + raise RuntimeError("respawn_cohort: empty target cohort") + + orchestrator_url = _pkg.os.environ.get("EGG_ORCHESTRATOR_URL", "http://localhost:9849") + restart_reason = (reason or "overseer corrective respawn")[:500] + for role in roles: + restart_url = ( + f"{orchestrator_url}/api/v1/pipelines/" + f"{quote(pipeline_id, safe='')}/agents/{quote(role, safe='')}/restart" + ) + req = urllib.request.Request( + restart_url, + data=_pkg.json.dumps({"reason": restart_reason}).encode(), + headers={"Content-Type": "application/json"}, + method="POST", + ) + opener = urllib.request.build_opener(urllib.request.ProxyHandler({})) + with opener.open(req, timeout=60) as resp: + result = _pkg.json.loads(resp.read().decode()) + if not result.get("success"): + raise RuntimeError( + f"restart of {role!r} failed: {result.get('message', 'unknown error')}" + ) + return True + + +def _build_overseer_corrective_executor( + *, + issue_number: int | None = None, + repo_path: _pkg.Any = None, + config: _pkg.Any = None, + audit_sink: _pkg.Any = None, + open_operator_hitl: _pkg.Any = None, + nudge_agent: _pkg.Any = None, + respawn_cohort: _pkg.Any = None, +) -> "CorrectiveExecutor": # noqa: UP037 + """Construct the §4 :class:`CorrectiveExecutor` wired to the production seams. + + Seams are injectable so the path stays unit-testable without a live + orchestrator. The default ``open_operator_hitl`` seam is bound to the + pipeline's ``issue_number`` / ``repo_path`` so it can resolve the contract. + The rate-limit window derives from the overseer config when present, falling + back to the executor default. + """ + from overseer.corrective import CorrectiveExecutor + + def _default_open_hitl(**kwargs: _pkg.Any) -> str: + kwargs.setdefault("issue_number", issue_number) + kwargs.setdefault("repo_path", repo_path) + return _pkg._corrective_open_operator_hitl(**kwargs) + + kwargs: dict[str, _pkg.Any] = {} + window = getattr(config, "overseer_infra_error_dedup_window_seconds", None) + if isinstance(window, int) and window > 0: + kwargs["window_seconds"] = float(window) + + return CorrectiveExecutor( + open_operator_hitl=open_operator_hitl or _default_open_hitl, + nudge_agent=nudge_agent or _pkg._corrective_nudge_agent, + respawn_cohort=respawn_cohort or _pkg._corrective_respawn_cohort, + audit_sink=audit_sink, + **kwargs, + ) + + +def _execute_overseer_verdicts( + results: list[tuple[_pkg.Any, _pkg.Any]], + *, + pipeline_id: str, + issue_number: int | None, + running_agent_count: int, + phase: str | None = None, + executor: _pkg.Any = None, +) -> list[_pkg.Any]: + """Run the §4 authority plane over adjudicated ``(finding, verdict)`` pairs. + + For each pair carrying a verdict, dispatch the recommended action through the + :class:`CorrectiveExecutor`. The executor enforces the closed vocabulary (a + ``none`` recommendation is skipped here as the non-executable no-op), the + zero-agent-park bar, rate-limiting, idempotency, and audit logging. Returns + the per-verdict :class:`CorrectiveOutcome` list (empty when nothing was + adjudicated or actioned). + """ + active = executor or _pkg._build_overseer_corrective_executor(issue_number=issue_number) + outcomes: list[_pkg.Any] = [] + for finding, verdict in results: + if verdict is None: + continue # routine finding — handled deterministically, no action + action = str(getattr(verdict, "recommended_action", "") or "").strip() + if action in ("", "none"): + continue # adjudicator advised no action — nothing to execute + evidence = getattr(finding, "evidence", None) or {} + target_role = str(getattr(verdict, "target", "") or "") or str( + evidence.get("agent_role") or evidence.get("agent_id") or "" + ) + finding_class = str(getattr(finding, "finding_class", "") or "") + outcomes.append( + active.execute( + action, + pipeline_id=pipeline_id, + running_agent_count=running_agent_count, + phase=phase, + target_role=target_role, + finding=finding, + idempotency_key=f"{finding_class}:{target_role}" if finding_class else None, + ) + ) + return outcomes + + +def _teardown_phase_overseer( + spawner: "ContainerSpawner", # noqa: UP037 + container_id: str, + pipeline_id: str, + phase_label: str, + reason: str, +) -> None: + """Stop the phase-scoped overseer container. + + Caller is responsible for holding ``overseer_lock`` and setting + ``phase_overseer_active = False`` before this call. + """ + try: + spawner.stop_agent_container( + container_id, + cleanup_session=True, + timeout=10, + ) + _pkg.logger.info( + f"Overseer container stopped ({reason})", + pipeline_id=pipeline_id, + phase=phase_label, + container_id=container_id[:12], + ) + except Exception as overseer_err: + _pkg.logger.debug( + f"Failed to stop overseer container ({reason})", + pipeline_id=pipeline_id, + error=str(overseer_err), + ) diff --git a/orchestrator/routes/pipelines/_pod_liveness.py b/orchestrator/routes/pipelines/_pod_liveness.py new file mode 100644 index 0000000000..967326517d --- /dev/null +++ b/orchestrator/routes/pipelines/_pod_liveness.py @@ -0,0 +1,228 @@ +"""live-pod guarding helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _get_spawner(): + """Get the appropriate spawner for the current runtime. + + Returns KubernetesSpawner when EGG_RUNTIME=kubernetes, otherwise + ContainerSpawner (Docker). + """ + if _pkg._RUNTIME == "kubernetes": + return _pkg.get_kubernetes_spawner() + return _pkg.get_container_spawner() + + +def _count_live_pods_for_pipeline(pipeline_id: str, *, quiet: bool = False) -> int | None: + """Count live pods labeled to this pipeline (#2420). + + Live = ``ContainerStatus`` in :data:`_LIVE_POD_STATUSES` (Pending / + Creating / Running). Pods in terminal phases (``Failed`` / ``Succeeded`` + → ``ContainerStatus.FAILED`` / ``EXITED``) are excluded — they have + already exited and the start_pipeline reset orphans no work tied to + them. + + Returns the number of live pods, or ``None`` if the label query failed — + callers must distinguish "verified zero" from "unknown" because the + start_pipeline reset would orphan any pods we couldn't see. + + ``quiet=True`` suppresses the helper-level warning when the label query + fails. The guard's ``force=true`` branch passes this flag because it + emits its own structured audit log on the ``live is None`` path; the + helper's warning would just duplicate it. + """ + try: + spawner = _pkg._get_spawner() + pods = spawner.backend.list_containers( + labels={_pkg.LABEL_PIPELINE_ID: pipeline_id}, + ) + return sum(1 for p in pods if p.status in _pkg._LIVE_POD_STATUSES) + except Exception as e: + if not quiet: + _pkg.logger.warning( + "start_pipeline live-pod check failed", + pipeline_id=pipeline_id, + error=str(e), + ) + return None + + +def _live_event_agents(pipeline_id: str, slice_id: str | None) -> list[dict[str, _pkg.Any]]: + """Running-agent view reconstructed from live Job labels (#3230). + + Under the orchestrator-owned BRC event loop (#3164, now unconditional) + each role's pod is an on-demand one-shot the loop deliberately does NOT + persist into ``phase_exec.agents`` — ``event_loop.py`` treats the + consensus tracker plus live-Job labels as the only sources of truth. So + the persisted agent list is empty even while role pods are ``Running``, + which the dashboard (``get_status.running_agents``) and the overseer + (``concurrent.agents`` stall-duration math) both read as "0 running + agents" — a blind dashboard and false ``phase stalled`` alerts. + + This reconstructs the running-pod cohort from the labels that ARE + authoritative. Live = ``status`` in :data:`_LIVE_POD_STATUSES` + (Pending / Creating / Running); terminal pods lingering in the + ``ttlSecondsAfterFinished`` window are excluded so between-spawn + quiescence reads as "no running agents" (the normal idle state, not a + stall). Scoped to ``slice_id`` when supplied so a slice-DAG implement + phase reports its own slice's pods rather than a cross-slice union; + refine/plan phases are unsliced and query by pipeline label alone. + + Entry shape mirrors the persisted ``agents`` entries (``role`` / + ``status`` / ``started_at`` / ``elapsed_seconds`` / ``container_id``) + so consumers need no special-casing. ``status`` is reported as + ``"running"`` for every live pod — Pending/Creating pods are agents + spinning up, and the dashboard's running-agent filter keys on that + literal. + + Best-effort: an absent/failed label query yields ``[]`` (callers treat + that identically to "no persisted agents", so there is no regression + versus the pre-fix behavior). + """ + try: + spawner = _pkg._get_spawner() + labels = {_pkg.LABEL_PIPELINE_ID: pipeline_id} + if slice_id: + labels[_pkg.LABEL_SLICE_ID] = slice_id + pods = spawner.backend.list_containers(labels=labels) + except Exception as e: # noqa: BLE001 — observability backfill is best-effort + _pkg.logger.debug( + "Live event-agent backfill query failed (#3230)", + pipeline_id=pipeline_id, + slice_id=slice_id, + error=str(e), + ) + return [] + + now = _pkg.datetime.now(_pkg.UTC) + entries: list[dict[str, _pkg.Any]] = [] + for pod in pods: + if pod.status not in _pkg._LIVE_POD_STATUSES: + continue + role = pod.agent_role.value if pod.agent_role is not None else None + if not role: + continue + entry: dict[str, _pkg.Any] = {"role": role, "status": "running"} + if isinstance(pod.container_id, str) and pod.container_id: + entry["container_id"] = pod.container_id + started_at = pod.started_at + if isinstance(started_at, _pkg.datetime): + started_dt = started_at if started_at.tzinfo else started_at.replace(tzinfo=_pkg.UTC) + entry["started_at"] = started_dt.isoformat() + entry["elapsed_seconds"] = max(0, int((now - started_dt).total_seconds())) + entries.append(entry) + return entries + + +def _slice_agents_alive(spawner: _pkg.Any, pipeline_id: str, slice_id: str) -> bool: + """Check if any live agents exist for a slice (#2914). + + Returns ``True`` if at least one pod labeled with the pipeline and + slice IDs is in a live state (Pending/Creating/Running). Returns + ``False`` if zero live pods or if the label query fails — the + conservative default forces re-spawn rather than risking a wedge. + + Caller contract: callers must have already torn down stale cohorts + with foreground propagation (e.g. ``restart_phase`` step 4 calls + ``remove_agent_container(force=True)``). A pod whose Job is being + deleted but is still in its termination grace period still reports + ``phase=Running`` (``kubernetes_client.py`` maps Running → RUNNING + without a Terminating-specific status), so without foreground + teardown the helper can false-positive against terminating pods + and wedge again. The ``spawner`` is taken as a parameter (rather + than fetched via ``_get_spawner``) so tests can inject a stub + directly, paralleling how ``_classify_non_complete_slice`` + receives ``gateway``. + """ + try: + pods = spawner.backend.list_containers( + labels={ + _pkg.LABEL_PIPELINE_ID: pipeline_id, + _pkg.LABEL_SLICE_ID: slice_id, + }, + ) + live_count = sum(1 for p in pods if p.status in _pkg._LIVE_POD_STATUSES) + return live_count > 0 + except Exception as e: # noqa: BLE001 + _pkg.logger.warning( + "Slice liveness check failed; treating as not-alive to force re-spawn (#2914)", + pipeline_id=pipeline_id, + slice_id=slice_id, + error=str(e), + ) + return False + + +def _guard_live_pods_or_force( + pipeline_id: str, + force: bool, + force_reason: str | None, +) -> tuple[_pkg.Response, int] | None: + """Refuse a phase reset that would orphan live pods (#2420). + + Returns ``None`` when the reset is safe to proceed (zero live pods, or + ``force=true``). Returns a 409 ``(response, status)`` when live pods are + present (or the label query failed) and the caller did not pass + ``force=true``. + """ + if force: + # ``quiet=True`` because the ``live is None`` branch below emits + # its own structured audit log; the helper-level warning would + # just duplicate it on the override path. + live = _pkg._count_live_pods_for_pipeline(pipeline_id, quiet=True) + # Template the audit log so the static message reflects what the + # override actually did. ``live == 0`` means the override was a + # no-op — log at ``info`` so it doesn't read like a near-miss. + if live is None: + _pkg.logger.warning( + "start_pipeline force=true override; live-pod check failed, " + "phase reset will proceed regardless", + pipeline_id=pipeline_id, + live_pod_count=None, + force_reason=force_reason, + ) + elif live > 0: + _pkg.logger.warning( + "start_pipeline force=true override; phase reset will proceed " + "and orphan live pods labeled to the pipeline", + pipeline_id=pipeline_id, + live_pod_count=live, + force_reason=force_reason, + ) + else: + _pkg.logger.info( + "start_pipeline force=true override applied (no live pods present)", + pipeline_id=pipeline_id, + live_pod_count=0, + force_reason=force_reason, + ) + return None + + live = _pkg._count_live_pods_for_pipeline(pipeline_id) + if live is None: + return _pkg.make_error_response( + f"Could not verify live pod count for pipeline {pipeline_id}; " + "the start_pipeline reset would orphan any pods labeled to it. " + "Cancel them first via cancel_task(cleanup=true) or pass " + "force=true to override.", + status_code=409, + reason="live_pod_check_failed", + ) + if live > 0: + return _pkg.make_error_response( + f"Pipeline {pipeline_id} has {live} live pod(s); the " + "start_pipeline reset would orphan them. Cancel them first via " + "cancel_task(cleanup=true) or pass force=true to override.", + status_code=409, + details={"live_pod_count": live}, + reason="live_pods_present", + ) + return None diff --git a/orchestrator/routes/pipelines/_populate.py b/orchestrator/routes/pipelines/_populate.py new file mode 100644 index 0000000000..8d5a121e3c --- /dev/null +++ b/orchestrator/routes/pipelines/_populate.py @@ -0,0 +1,1460 @@ +"""plan-draft synthesis + contract population helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Literal # noqa: F401 + +import routes.pipelines as _pkg # noqa: E402,F401 + +if TYPE_CHECKING: + from egg_contracts.models import Slice as ContractSlice # noqa: F401 + + +def _synthesize_plan_draft( + repo_path: _pkg.Path, + pipeline_id: str, + pipeline_mode: str = "issue", + issue_number: int | None = None, +) -> None: + """Synthesize a plan draft from multi-agent plan outputs. + + In multi-agent plan mode, ARCHITECT and RISK_ANALYST write to + .egg-state/agent-outputs/. TASK_PLANNER writes the plan draft + directly to .egg-state/drafts/{id}-plan.md. This function combines + the remaining agent outputs into the plan draft (if the task_planner + has not already written one) so that _populate_contract_from_plan() + and the HITL gate can find it. + """ + draft_rel = _pkg._get_draft_path("plan", issue_number=issue_number, pipeline_id=pipeline_id) + if not draft_rel: + _pkg.logger.debug( + "No draft path for plan phase, skipping synthesis", + pipeline_id=pipeline_id, + ) + return + + draft_path = repo_path / draft_rel + if draft_path.exists(): + # Draft already written (e.g. by a single-agent run) — don't overwrite. + return + + outputs_dir = repo_path / ".egg-state" / "agent-outputs" + if not outputs_dir.is_dir(): + _pkg.logger.warning( + "No agent-outputs directory, cannot synthesize plan draft", + pipeline_id=pipeline_id, + ) + return + + # Derive the pipeline identifier for namespaced output filenames. + _synth_id = _pkg._pipeline_identifier(issue_number, pipeline_id) + + from egg_contracts.artifact_spec import resolve_artifact_path + + sections: list[str] = [] + # Spec *names* — not bare filenames — so the agent-output path knowledge + # lives only in egg_contracts.artifact_spec (the slice-2 single-source-of- + # truth ratchet covers this reader, not just the prompt builder). + # ``resolve_artifact_path("<name>", id)`` yields the namespaced + # ``.egg-state/agent-outputs/{id}-<file>`` path; the old un-namespaced + # global filename (basename minus the ``{id}-`` prefix) stays the fallback. + agent_specs = [ + ("architect-output", "Architecture Analysis"), + ("architect-slices", "Slice Scaffold"), + ("risk-analyst-output", "Risk Assessment"), + ] + + for spec_name, heading in agent_specs: + prefixed_rel = resolve_artifact_path(spec_name, _synth_id) + global_filename = _pkg.Path(prefixed_rel).name.removeprefix(f"{_synth_id}-") + # Try prefixed filename first, fall back to old global filename + prefixed_file = repo_path / prefixed_rel + if prefixed_file.exists(): + output_file = prefixed_file + else: + output_file = outputs_dir / global_filename + if not output_file.exists(): + continue + try: + raw = output_file.read_text() + data = _pkg.json.loads(raw) + # Agent outputs may contain a "content" or "output" key with + # the main text, or may be the full JSON blob. + content = data.get("content") or data.get("output") or _pkg.json.dumps(data, indent=2) + except _pkg.json.JSONDecodeError: + # Fall back to raw text if not valid JSON + content = raw + except Exception as e: + _pkg.logger.warning( + "Failed to read agent output for plan draft", + pipeline_id=pipeline_id, + file=global_filename, + error=str(e), + ) + continue + + # Skip empty or whitespace-only outputs + if not content or not content.strip(): + _pkg.logger.warning( + "Agent output is empty, skipping from plan draft", + pipeline_id=pipeline_id, + file=global_filename, + ) + continue + + sections.append(f"## {heading}\n\n{content}") + + if not sections: + _pkg.logger.warning( + "No agent outputs found to synthesize plan draft", + pipeline_id=pipeline_id, + ) + return + + draft_content = "\n\n".join(sections) + "\n" + + # Guard against a draft that has section headings but no real content. + stripped = draft_content + for _, heading in agent_specs: + stripped = stripped.replace(f"## {heading}", "") + if len(stripped.strip()) < _pkg._MIN_PLAN_DRAFT_CONTENT_LENGTH: + _pkg.logger.warning( + "Synthesized plan draft has insufficient content, not writing", + pipeline_id=pipeline_id, + content_length=len(stripped.strip()), + ) + return + + draft_path.parent.mkdir(parents=True, exist_ok=True) + draft_path.write_text(draft_content, encoding="utf-8") + _pkg.logger.info( + "Synthesized plan draft from agent outputs", + pipeline_id=pipeline_id, + path=str(draft_path), + sections=len(sections), + ) + + +def _slice_gate_block_monolithic_demotion( + worktree_repo_path: _pkg.Path, + pipeline_id: str, + issue_number: int | None, +) -> "SliceGateMonolithicBlock | None": # noqa: UP037 — forward ref; see docstring + """#2337 defensive recheck for the slice-loop gate. + + Called only when ``contract.slices`` is empty at implement-phase entry. + Returns a :class:`SliceGateMonolithicBlock` when the on-disk plan + draft parses to N>1 slices — the exact contract+plan mismatch that + demoted issue-2261's 15-slice plan to a monolithic slice-1 PR + (#2337). When this fires the implement phase should be marked + FAILED rather than silently routed through ``_run_concurrent_phase``. + + The returned tuple carries the human-readable ``message`` plus the + parsed ``draft_slice_count`` so the caller can emit a dedicated HITL + naming the divergence inline without having to re-parse the message + (#2627 follow-up). The annotation is quoted because + ``SliceGateMonolithicBlock`` is declared further down the module to + keep it grouped with the other #2627 follow-up types. + + Returns ``None`` when: + * The plan draft is missing on local — there's nothing to parse, and + the populator's own ``plan_draft_missing`` warning already covers + that case (with ``source="plan_complete"`` it raises so we wouldn't + reach this gate at all). + * The plan parses to 0 or 1 slice — single-slice/no-slice contracts + legitimately use the monolithic path. + * Plan parsing fails — defensive: don't block on a parser regression, + just log and let the gate fall through to monolithic. + """ + draft_rel = _pkg._get_draft_path("plan", issue_number=issue_number, pipeline_id=pipeline_id) + if not draft_rel: + return None + draft_path = worktree_repo_path / draft_rel + if not draft_path.exists(): + return None + try: + from egg_contracts.plan_parser import parse_plan as _parse_plan_for_gate + + plan_text = draft_path.read_text() + parsed = _parse_plan_for_gate(plan_text) + if not parsed.success: + return None + draft_slice_count = len(parsed.to_contract_slices()) + except Exception as parse_err: # noqa: BLE001 + _pkg.logger.debug( + "Slice-loop gate: draft re-parse failed", + pipeline_id=pipeline_id, + error=str(parse_err), + ) + return None + if draft_slice_count <= 1: + return None + return _pkg.SliceGateMonolithicBlock( + message=( + f"plan draft parses to {draft_slice_count} slices but contract.slices " + f"is empty — populator silently failed earlier (#2337); refusing to " + f"demote to monolithic implement" + ), + draft_slice_count=draft_slice_count, + ) + + +class PlanDraftMissingOnLocalError(RuntimeError): + """Raised by the natural plan-completion populator path when the plan + draft is missing from the local worktree but present on origin. + + This is the silent-failure mode behind #2337: a multi-slice plan-phase + pipeline whose populator returned without slices because + ``_sync_worktree_with_remote`` left agents' plan-phase commits on + origin. Surfacing as an exception lets the natural call site mark + the pipeline FAILED instead of silently demoting to monolithic + implement. The force-advance call site (#1941) keeps swallowing. + """ + + +class PlanDraftMissingOnLocalAndOriginError(RuntimeError): + """Raised by the natural plan-completion populator path when the plan + draft is missing from BOTH the local worktree and origin. + + Symmetric to :class:`PlanDraftMissingOnLocalError` (#2337) for the + case where the draft was deleted-and-not-replaced rather than left + on origin only. Observed in the wild on issue-1557-v2 (#2627): the + orchestrator's pre-sync state-write commit deleted the draft and + the consolidated-write step never replaced it, leaving the pipeline + to advance to implement with an empty contract and 8 agents + spinning ``WAITING_FOR_EVENT`` for ~45 min. Surfacing as an + exception lets the natural call site mark the pipeline FAILED so + the operator can intervene. The force-advance call site (#1941) + keeps swallowing. + """ + + +class PopulateOutcome(_pkg.StrEnum): + """Structured discriminator for :func:`_populate_contract_from_plan` outcomes. + + Added in #2627 follow-up: previously the populator returned ``None`` + on every branch (success, draft-missing, parse-failed, etc.), so + callers couldn't tell "populated N>0 tasks" from "silently produced + an empty contract" without re-loading the contract and counting. + The slice-gate guard at implement-phase entry catches the empty + contract case after the orchestrator has already transitioned to + implement, leaving a generic Retry/Accept/Abort HITL that respawns + into the same broken state. A structured outcome lets the + plan-complete and start_phase=implement call sites fail-fast at + the boundary with an actionable HITL inline. + """ + + POPULATED = "populated" + DRAFT_MISSING = "draft_missing" + NO_DRAFT_PATH = "no_draft_path" + PARSE_FAILED = "parse_failed" + EMPTY_RESULT = "empty_result" + CONTRACT_LOAD_FAILED = "contract_load_failed" + EGG_CONTRACTS_UNAVAILABLE = "egg_contracts_unavailable" + FOREST_VIOLATION = "forest_violation" + # #3046 — two slices touch overlapping files with no dependency edge + # between them; rejected at ingestion like a forest violation. + SLICE_OVERLAP_VIOLATION = "slice_overlap_violation" + UNEXPECTED_EXCEPTION = "unexpected_exception" + + +class PopulateProducedEmptyContractError(RuntimeError): + """Raised at the natural plan-completion call site when + :func:`_populate_contract_from_plan_safe` returns a ``PopulateResult`` + whose outcome indicates the populate step did not produce a contract + with tasks the implement-phase agents can act on. + + Two shapes: + + * ``outcome != POPULATED`` — the populator returned a non-success + outcome (``EMPTY_RESULT``, ``PARSE_FAILED``, ``CONTRACT_LOAD_FAILED``, + ``EGG_CONTRACTS_UNAVAILABLE``, ``FOREST_VIOLATION``, + ``UNEXPECTED_EXCEPTION``, ``NO_DRAFT_PATH``). ``DRAFT_MISSING`` at + ``source="plan_complete"`` is pre-empted by + :class:`PlanDraftMissingOnLocalError` / + :class:`PlanDraftMissingOnLocalAndOriginError` so it never reaches + this exception via that path. + * ``outcome == POPULATED`` with ``slice_count == 0`` — the populator + considered itself "changed" (PR metadata populated, or + ``current_phase`` advanced) but produced no slices/tasks, so + implement-phase agents would have nothing to do. This is the + orthogonal silent-corruption shape flagged in #2627's "Additionally + — and orthogonally" paragraph (#2627 review). + + Orthogonal to :class:`PlanDraftMissingOnLocalError` / + :class:`PlanDraftMissingOnLocalAndOriginError` (which fire when the + draft is missing from one or both refs). The slice-gate at + implement-phase entry would catch most of these later, but failing at + the boundary lets the same dedicated HITL fire for both paths. + Force-advance call sites (#1941) keep swallowing — they inspect the + return value but never raise. + """ + + def __init__(self, outcome: _pkg.PopulateOutcome, slice_count: int = 0) -> None: + if outcome == _pkg.PopulateOutcome.POPULATED: + # Populator returned "changed=True" but produced no slices/tasks + # (only PR metadata or current_phase advance changed). #2627 + # review's "POPULATED with slice_count == 0" case. + message = ( + "plan populate completed but produced 0 slices/tasks — " + "refusing to advance plan phase with empty contract" + ) + else: + message = ( + f"plan populate produced {outcome.value} outcome — refusing to " + f"advance plan phase with empty contract" + ) + super().__init__(message) + self.outcome = outcome + self.slice_count = slice_count + + +class PopulateResult(_pkg.NamedTuple): + """Return type of :func:`_populate_contract_from_plan` and its safe wrapper. + + ``slice_count`` and ``task_count`` are populated only on + ``POPULATED`` (zero on every failure outcome). ``FOREST_VIOLATION`` + is observed at the wrapper after catching the inner raise — the + inner function continues to ``raise ForestValidationError`` so + HTTP callers keep their 422 contract. + """ + + outcome: _pkg.PopulateOutcome + slice_count: int = 0 + task_count: int = 0 + + +class SliceGateMonolithicBlock(_pkg.NamedTuple): + """Return type of :func:`_slice_gate_block_monolithic_demotion`. + + Carries the human-readable failure message plus the parsed slice + count so callers can emit a structured HITL naming the divergence + inline (#2627 follow-up). Previously the helper returned a bare + ``str`` and the slice count had to be re-parsed from the message, + making the dedicated HITL payload awkward to build. + """ + + message: str + draft_slice_count: int + + +def _populate_result_is_empty_contract(result: _pkg.PopulateResult) -> bool: + """Return True if a ``PopulateResult`` means the contract is empty/broken. + + Centralizes the fail-fast condition used by the natural plan-complete + handler and the ``start_phase=implement`` safety net. The two + branches it discriminates: + + * ``outcome != POPULATED`` — the populator reported any non-success + outcome (``EMPTY_RESULT``, ``PARSE_FAILED``, ``CONTRACT_LOAD_FAILED``, + ``EGG_CONTRACTS_UNAVAILABLE``, ``FOREST_VIOLATION``, + ``UNEXPECTED_EXCEPTION``, ``DRAFT_MISSING``, ``NO_DRAFT_PATH``). + ``DRAFT_MISSING`` at ``source="plan_complete"`` is pre-empted by + the ``PlanDraftMissing*`` raises in the safe wrapper so it does + not reach this check via that path; the safety net (which calls + the inner directly with no source) does see it here. + * ``outcome == POPULATED`` with ``slice_count == 0`` — the populator + considered itself "changed" (PR metadata populated, or + ``current_phase`` advanced) but produced no slices/tasks, so the + implement-phase agents would have nothing to do. Flagged in the + "Additionally — and orthogonally" paragraph on #2627 and the + review's "POPULATED with slice_count == 0 still silently advances" + observation. + + Extracted so the call-site check is unit-testable without standing + up the full ``_run_pipeline`` integration setup, and so the two + call sites can't drift out of agreement. Re #2627 review. + """ + return result.outcome != _pkg.PopulateOutcome.POPULATED or result.slice_count == 0 + + +def _empty_contract_hitl_question( + *, + pipeline_id: str, + reason: str, + draft_slice_count: int | None, + gate: str, +) -> str: + """Build the HITL question text naming the empty-contract root cause inline. + + ``pipeline_id`` is interpolated into the recovery URL so operators can + copy it verbatim instead of substituting a literal ``{id}`` placeholder + by hand (#2627 review). ``reason`` is the operator-visible identifier + (typically a :class:`PopulateOutcome` value or the slice-gate's own + discriminator). ``draft_slice_count`` is None when the plan draft + itself could not be parsed (so we can't quote a count). ``gate`` names + the call site that detected the divergence — ``slice_gate`` / + ``start_phase_implement_safety_net`` / ``plan_complete`` — so the + operator sees which guard fired. + + The opening phrase is "Pipeline blocked at {gate}" rather than + "Implement-phase blocked at {gate}": ``gate=plan_complete`` fires while + the *plan* phase is being marked FAILED, before the implement phase is + spawned, so the implement-specific phrasing would read oddly against + ``pipeline.error`` and the phase-execution status (#2627 review). + """ + if draft_slice_count is not None: + divergence_line = ( + f"contract.slices is empty but the on-disk plan draft parses " + f"to {draft_slice_count} slices" + ) + elif reason in _pkg._DIVERGENCE_LINE_BY_REASON: + # Reason-aware wording for outcomes whose root cause isn't + # "draft missing/unparseable/empty" — the widened + # :func:`_populate_result_is_empty_contract` check now routes + # ``FOREST_VIOLATION`` / ``CONTRACT_LOAD_FAILED`` / + # ``EGG_CONTRACTS_UNAVAILABLE`` / ``UNEXPECTED_EXCEPTION`` / + # ``POPULATED``-with-zero-slices through this same HITL, where + # the generic "draft missing, unparseable, or yielded no tasks" + # prose would contradict the ``reason=`` field (#2627 review). + divergence_line = _pkg._DIVERGENCE_LINE_BY_REASON[reason] + else: + divergence_line = ( + "contract.slices is empty and the plan draft is missing, " + "unparseable, or yielded no tasks" + ) + return ( + f"Pipeline blocked at {gate}: {divergence_line} " + f"(reason={reason}). The sync helper's auto-reconcile path " + f"(#2792) tried to bring the worktree forward before the " + f"populator ran; if you're seeing this, that reconcile either " + f"didn't fire or didn't restore the draft, so pipeline state " + f"and the contract have diverged. Plain restart_phase implement " + f"will respawn into the same broken state. How to proceed?\n" + f"- 'Repopulate contract from plan draft and retry' — run " + f"POST /pipelines/{pipeline_id}/phase/populate-contract, then " + f"restart_phase implement.\n" + f"- 'Restart plan phase' — restart_phase plan to regenerate the " + f"draft from scratch.\n" + f"- 'Abort pipeline' — cancel_task." + ) + + +def _populate_outcome_to_hitl_reason(outcome: _pkg.PopulateOutcome) -> str: + """Return the empty-contract HITL ``reason`` for a populate outcome. + + Maps a :class:`PopulateOutcome` to the operator-visible ``reason`` + string used by the dedicated empty-contract HITL: + + * ``POPULATED`` → ``"populated_but_empty_slices"`` — the populator + ran but yielded 0 slices/tasks (the orthogonal "draft existed, + populator ran, but produced nothing" case so the HITL doesn't + claim a bare ``"populated"`` reason that contradicts the empty + contract — #2627 review). + * every other outcome → ``outcome.value`` (e.g. ``forest_violation``, + ``contract_load_failed``, ``empty_result``). + + Extracted so both empty-contract call sites — the plan-complete + handler (via :func:`_empty_contract_hitl_reason`) and the + ``start_phase=implement`` safety net — share a single dispatch and + can't drift if a new outcome needs special-cased reason handling + (#2627 review follow-up). + """ + if outcome == _pkg.PopulateOutcome.POPULATED: + return "populated_but_empty_slices" + return outcome.value + + +def _forest_error_to_outcome(err: _pkg.ForestValidationError) -> _pkg.PopulateOutcome: + """Map a :class:`ForestValidationError` to the matching populate outcome.""" + return _pkg._FOREST_REASON_TO_OUTCOME.get(err.reason, _pkg.PopulateOutcome.FOREST_VIOLATION) + + +def _plan_preflight_hitl_question( + *, + missing_fields: list[str], + plan_draft_rel: str, +) -> str: + """Build the HITL question for an implement-start pre-flight rejection (#3100). + + Names the missing plan-draft fields inline and maps each recovery + option to its concrete operator action, mirroring + :func:`_empty_contract_hitl_question`'s shape so operators see the + same actionable-decision pattern at both implement-start gates. + """ + fields = ", ".join(missing_fields) + return ( + f"Pipeline blocked at start_phase_implement_plan_preflight: the " + f"plan draft ({plan_draft_rel}) is missing required field(s) " + f"{fields}. The context-PR opener reads contract.pr metadata " + f"from the plan's top-level ``pr:`` block; without it the " + f"work-branch context PR can never open — both runner-side " + f"openers soft-fail with missing_pr_metadata on every slice, " + f"and no advance_phase call runs on the implement-start path " + f"to enforce the #2777 hard-require (#3100). How to proceed?\n" + f"- 'Fix the plan draft's pr: block and restart implement' — add " + f"a top-level ``pr:`` block (title, description, test_plan, " + f"manual_steps) to the draft's ``# yaml-tasks`` fence on the " + f"work branch, then restart_phase implement.\n" + f"- 'Restart plan phase' — restart_phase plan to regenerate the " + f"draft from scratch.\n" + f"- 'Abort pipeline' — cancel_task." + ) + + +def _enforce_implement_start_plan_preflight( + pipeline_id: str, + pipeline: _pkg.Pipeline, + store: _pkg.StateStore, + worktree_repo_path: _pkg.Path, + plan_draft_rel: str, +) -> bool: + """Enforce the #2777 plan pre-flight at the implement-start boundary (#3100). + + The natural plan→implement path runs + :func:`egg_contracts.plan_parser.validate_plan_preflight` at the + ``advance_phase`` REST/MCP site (``routes/phases.py``) and rejects + with a typed 422 when the plan draft lacks the ``pr:`` metadata the + context-PR opener needs. ``start_phase=implement`` submits never + traverse ``advance_phase``, so before #3100 a draft without a + ``pr:`` block sailed straight into the implement phase: every + runner-side opener backstop soft-failed with + ``missing_pr_metadata`` at WARNING level, the slice stack ran with + no context PR, and the operator discovered the gap only by noticing + the PR was absent (observed on pipeline-da68d70c and + pipeline-2d9cc50d, Khan/webapp). + + Runs AFTER the empty-contract gate at the call site, so the + established empty-contract HITL routing (#2627) is unchanged — this + gate fires only when the populate succeeded but the draft lacks the + PR metadata. + + Scope: + + * Remote pipelines only (``pipeline.repo`` or ``pipeline.base_branch`` + set). Local-mode pipelines never open a context PR (the opener's + own local-mode skip in + :func:`_open_context_pr_at_implement_start`), so requiring ``pr:`` + metadata there would fail test pipelines over a PR that would + never exist. + * Infra failures log a WARNING and return False — the gate must + not add a new hard-fail mode for transient errors, and the + populate path's own outcomes already cover an unreadable draft. + Only the two named infra-class exceptions are caught: an + :class:`ImportError` from the ``plan_parser`` import (validator + unavailable on this host) and an :class:`OSError` from the draft + ``read_text`` (file vanished, permission flake). Any other + exception out of :func:`validate_plan_preflight` propagates to + the outer ``_run_pipeline`` Exception handler, mirroring the + ``advance_phase`` site's behaviour — the goal is to never + swallow a real parser bug under a generic "infra" umbrella. + + Returns True when the pipeline was marked FAILED (the caller must + return without spawning implement-phase agents), False when the + pre-flight passed or was legitimately skipped. + """ + if not (pipeline.repo or pipeline.base_branch): + return False + + try: + from egg_contracts.plan_parser import ( + PlanPreflightError, + validate_plan_preflight, + ) + except ImportError as imp_err: + _pkg.logger.warning( + "Implement-start plan pre-flight: plan_parser import failed " + "(continuing without the gate) (#3100)", + pipeline_id=pipeline_id, + error=str(imp_err), + ) + return False + + try: + plan_text = (worktree_repo_path / plan_draft_rel).read_text() + except OSError as read_err: + _pkg.logger.warning( + "Implement-start plan pre-flight: failed to read plan draft " + "(continuing without the gate) (#3100)", + pipeline_id=pipeline_id, + error=str(read_err), + ) + return False + + try: + validate_plan_preflight(plan_text) + return False + except PlanPreflightError as preflight_err: + error_msg = ( + "start_phase=implement plan pre-flight failed — plan draft is " + f"missing required field(s) " + f"{', '.join(preflight_err.missing_fields)}: refusing to run " + "the implement phase with no openable context PR (#2777 " + "pre-flight, #3100 implement-start enforcement)" + ) + _pkg.logger.error( + "OVERSEER_ALERT start_phase_implement_plan_preflight_failed", + pipeline_id=pipeline_id, + missing_fields=preflight_err.missing_fields, + ) + with _pkg.get_pipeline_state_lock(pipeline_id): + disk_pipeline = store.load_pipeline(pipeline_id) + disk_pipeline.status = _pkg.PipelineStatus.FAILED + disk_pipeline.error = error_msg + store.save_pipeline(disk_pipeline) + _pkg._persist_hitl_decision( + pipeline_id, + disk_pipeline, + store, + question=_pkg._plan_preflight_hitl_question( + missing_fields=preflight_err.missing_fields, + plan_draft_rel=plan_draft_rel, + ), + options=list(_pkg._PLAN_PREFLIGHT_HITL_OPTIONS), + phase=disk_pipeline.current_phase, + ) + _pkg.report_pipeline_status( + disk_pipeline, + event_type="pipeline.failed", + message=f"Pipeline failed: {error_msg[:100]}", + ) + _pkg._emit_pipeline_event(disk_pipeline, "pipeline.failed") + return True + + +def _empty_contract_hitl_reason( + err: _pkg.PlanDraftMissingOnLocalError + | _pkg.PlanDraftMissingOnLocalAndOriginError + | _pkg.PopulateProducedEmptyContractError, +) -> str: + """Return the ``reason`` field for the empty-contract HITL. + + Dispatches the operator-visible HITL ``reason`` from one of the + three plan-complete fail-loud exceptions: + + * :class:`PlanDraftMissingOnLocalError` → ``plan_draft_missing_on_local`` + * :class:`PlanDraftMissingOnLocalAndOriginError` → + ``plan_draft_missing_on_local_and_origin`` + * :class:`PopulateProducedEmptyContractError` — delegates to + :func:`_populate_outcome_to_hitl_reason` so the outcome → reason + mapping is shared with the ``start_phase=implement`` safety net + (#2627 review). + + Extracted so the plan-complete call site's HITL-reason dispatch + is unit-testable without standing up the full ``_run_pipeline`` + integration setup (#2627 review). + """ + if isinstance(err, _pkg.PlanDraftMissingOnLocalError): + return "plan_draft_missing_on_local" + if isinstance(err, _pkg.PlanDraftMissingOnLocalAndOriginError): + return "plan_draft_missing_on_local_and_origin" + return _pkg._populate_outcome_to_hitl_reason(err.outcome) + + +def _empty_contract_failure_metadata( + err: _pkg.PlanDraftMissingOnLocalError + | _pkg.PlanDraftMissingOnLocalAndOriginError + | _pkg.PopulateProducedEmptyContractError, +) -> tuple[str, str]: + """Return ``(teardown_reason, log_event)`` for the plan-complete + fail-loud handler in :func:`_run_pipeline`. + + Dispatches on the three #2627 fail-loud exception classes: + + * :class:`PlanDraftMissingOnLocalError` — draft missing from the local + worktree but present on origin (the #2337 silent-failure). + * :class:`PlanDraftMissingOnLocalAndOriginError` — draft missing from + both refs (the #2627 silent-failure). + * :class:`PopulateProducedEmptyContractError` — draft existed but the + populator yielded an empty/broken contract (the orthogonal "draft + existed but populate yielded nothing" failure mode #2627 also + called out). + + Extracted so the dispatch is unit-testable without standing up the + full ``_run_pipeline`` integration setup — a typo that swapped the + branches would otherwise pass the existing populator-helper + tests. Re #2627 review. + + ``log_event`` uses the ``"OVERSEER_ALERT <discriminator>"`` event-name + convention so the plan-complete fail-loud path is visible to the same + log filters operators use for the slice-gate and start_phase + safety-net (#2627 review). The matching pre-raise OVERSEER_ALERTs + (emitted by :func:`_populate_contract_from_plan_safe` for the two + ``PlanDraftMissing*`` cases, and by ``_run_pipeline``'s plan-complete + synthesis for :class:`PopulateProducedEmptyContractError`) use the + same event names so the pre-raise log and the FAILED-cleanup log + share a single discriminator on every branch. + """ + if isinstance(err, _pkg.PlanDraftMissingOnLocalError): + return ( + "plan draft missing on local", + "OVERSEER_ALERT plan_draft_missing_on_local_but_present_on_origin", + ) + if isinstance(err, _pkg.PlanDraftMissingOnLocalAndOriginError): + return ( + "plan draft missing on local and origin", + "OVERSEER_ALERT plan_draft_missing_on_local_and_origin", + ) + return ( + f"populate produced {err.outcome.value} outcome", + "OVERSEER_ALERT plan_populate_produced_empty_contract", + ) + + +def _origin_has_plan_draft(repo_path: _pkg.Path, branch: str, draft_rel: str) -> bool: + """Return True if ``origin/{branch}:{draft_rel}`` resolves locally. + + Uses ``git cat-file -e`` against the local refs to origin (the + immediately preceding ``_sync_worktree_with_remote`` call has already + fetched), so this is a cheap on-disk check, not a network round-trip. + + A False return collapses two cases: origin really doesn't have the + draft, or the ``cat-file`` probe itself failed (transient git error, + timeout, etc.). The natural plan-completion call site treats False + as "definitively missing on origin" and, when local is also missing, + raises :class:`PlanDraftMissingOnLocalAndOriginError` so the pipeline + is marked FAILED rather than advancing to implement with an empty + contract (#2627). This is a deliberate fail-loud choice: a transient + probe failure combined with a missing local draft will fail the + pipeline rather than silently advance. Operators can re-run the + pipeline; silently shipping an empty contract has no recovery path. + """ + try: + result = _pkg.subprocess.run( + [ + "git", + "-c", + "core.hooksPath=/dev/null", + "-c", + f"safe.directory={repo_path}", + "-C", + str(repo_path), + "cat-file", + "-e", + f"origin/{branch}:{draft_rel}", + ], + capture_output=True, + text=True, + timeout=10, + check=False, + ) + return result.returncode == 0 + except Exception: + return False + + +def _auto_populate_contract_at_implement_start( + worktree_repo_path: _pkg.Path, + pipeline_id: str, + pipeline_mode: str, + issue_number: int | None, + current_phase: _pkg.PipelinePhase, + pipeline_branch: str, + *, + gateway: _pkg.Any, + gateway_mode: str, + base_branch: str | None, +) -> int: + """Attempt to auto-populate an empty contract at implement start (#2915). + + When a pipeline enters the implement phase with zero slices in the + contract, this helper tries to populate it from the plan draft. On + success, commits and pushes the populated contract; on failure, logs + and returns 0 (still empty). + + Returns the number of slices in the contract after the attempt. + + NOTE: restored in slice-4 v4 of #2908 — the slice-4 base merge + (commit 06c5a6cb0) accidentally dropped this function when bringing + slice-1/2/3 work into the coder branch. The orphan import in + ``orchestrator/tests/test_auto_populate_contract.py`` broke + ``pytest --collect-only`` and blocked ``make test`` from running + any tests at all (per tester v3 NACK blocker #1). The function + body matches ``origin/main`` verbatim; the call site at + ``_run_pipeline`` is unchanged. + """ + _pkg.logger.info( + "Attempting to auto-populate empty contract at implement start (#2915)", + pipeline_id=pipeline_id, + issue_number=issue_number, + ) + try: + _populate_result = _pkg._populate_contract_from_plan( + worktree_repo_path, + pipeline_id, + pipeline_mode, + issue_number, + current_phase=current_phase, + ) + except _pkg.ForestValidationError as _forest_err: + _pkg.logger.warning( + "Auto-populate contract failed: slice-DAG validation error", + pipeline_id=pipeline_id, + reason=_forest_err.reason, + errors=_forest_err.errors, + ) + return 0 + except Exception as _populate_err: # noqa: BLE001 + _pkg.logger.warning( + "Auto-populate contract failed at implement start", + pipeline_id=pipeline_id, + error=str(_populate_err), + exc_info=True, + ) + return 0 + + if ( + _populate_result.outcome != _pkg.PopulateOutcome.POPULATED + or _populate_result.slice_count == 0 + ): + _pkg.logger.warning( + "Auto-populate contract returned empty or failed", + pipeline_id=pipeline_id, + outcome=_populate_result.outcome.value, + slice_count=_populate_result.slice_count, + ) + return 0 + + # Commit the populated contract + try: + _committed = _pkg._commit_statefiles_to_worktree( + worktree_repo_path, + "Auto-populate contract at implement start (#2915)", + _pkg._pipeline_identifier(issue_number, pipeline_id), + pipeline_id=pipeline_id, + ) + if not _committed: + _pkg.logger.warning( + "Auto-populate: commit returned False (nothing to commit)", + pipeline_id=pipeline_id, + ) + return 0 + except Exception as _commit_err: # noqa: BLE001 + _pkg.logger.warning( + "Auto-populate: commit failed", + pipeline_id=pipeline_id, + error=str(_commit_err), + ) + return 0 + + # Push the populated contract. Failure is non-fatal — the contract is + # already committed locally — but mirror the canonical pattern from + # agent_salvage._push_recovery (try/except for transport, then check + # push_result.ok for gateway-reported rejections like non_fast_forward + # / auth_failed / gateway_unreachable). Thread gateway_mode and + # base_branch so private-mode pipelines route correctly and non-FF + # reconcile uses --onto and doesn't replay base-branch commits. + push_succeeded = False + try: + push_result = gateway.push_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(worktree_repo_path), + branch=pipeline_branch, + mode=gateway_mode, + base_branch=base_branch, + ) + except Exception as _push_err: # noqa: BLE001 + _pkg.logger.warning( + "Auto-populate: push transport failure (non-fatal, contract committed locally)", + pipeline_id=pipeline_id, + error=str(_push_err), + ) + else: + if not push_result.ok: + _pkg.logger.warning( + "Auto-populate: push rejected by gateway (non-fatal, contract committed locally)", + pipeline_id=pipeline_id, + category=push_result.category, + detail=push_result.detail, + ) + else: + push_succeeded = True + + _pkg.logger.info( + "Auto-populate contract succeeded" + if push_succeeded + else "Auto-populate contract succeeded locally only (push did not land)", + pipeline_id=pipeline_id, + slice_count=_populate_result.slice_count, + push_succeeded=push_succeeded, + ) + return _populate_result.slice_count + + +def _populate_contract_from_plan_safe( + repo_path: _pkg.Path, + pipeline_id: str, + pipeline_mode: str = "issue", + issue_number: int | None = None, + *, + source: Literal[ + "plan_complete", + "advance_phase_force", + "hitl_plan_gate_approval", + ] = "advance_phase_force", + branch: str | None = None, + current_phase: _pkg.PipelinePhase | None = None, +) -> _pkg.PopulateResult: + """Run :func:`_populate_contract_from_plan` without propagating failures. + + Shared call path for the three code sites that run the populate step + when a pipeline leaves the ``plan`` phase: ``_run_pipeline``'s + post-complete block (``source="plan_complete"``), ``advance_phase`` + (used by the MCP ``advance_phase`` tool, especially with + ``force=true`` — ``source="advance_phase_force"``), and the HITL + plan-gate approval path in :func:`start_pipeline` + (``source="hitl_plan_gate_approval"`` — operator approved the + plan_gate while the pipeline was AWAITING_HUMAN, recovery + re-spawns ``_run_pipeline``). Blocking the phase transition on a + populate failure would defeat the purpose of the advance hammer + or recovery path — see #1941 — so all non-natural call sites + keep the swallow-everything behaviour. + + The natural plan-completion call site (``source="plan_complete"``) + additionally raises: + + * :class:`PlanDraftMissingOnLocalError` when the draft is missing + from local but present on origin — the silent-failure mode + behind #2337. + * :class:`PlanDraftMissingOnLocalAndOriginError` when the draft is missing from + BOTH local and origin — the silent-failure mode behind #2627 + (orchestrator-side delete with no consolidated re-write). + + Caller is expected to mark the pipeline FAILED so the operator can + intervene rather than advancing to implement with an empty + contract. + + Returns a :class:`PopulateResult` so non-raising failure modes are + still inspectable: callers that need to fail-fast on + ``EMPTY_RESULT`` / ``PARSE_FAILED`` (#2627 follow-up) can branch on + the outcome. ``ForestValidationError`` raised by the inner is + caught and translated to ``PopulateResult(FOREST_VIOLATION, 0, 0)``; + any other unexpected exception translates to + ``PopulateResult(UNEXPECTED_EXCEPTION, 0, 0)``. + """ + if source == "plan_complete" and branch is not None: + draft_rel = _pkg._get_draft_path("plan", issue_number=issue_number, pipeline_id=pipeline_id) + if draft_rel is not None: + local_path = repo_path / draft_rel + on_local = local_path.exists() + on_origin = _pkg._origin_has_plan_draft(repo_path, branch, draft_rel) + if not on_local and on_origin: + _pkg.logger.error( + "OVERSEER_ALERT plan_draft_missing_on_local_but_present_on_origin", + pipeline_id=pipeline_id, + branch=branch, + draft_rel=draft_rel, + note=( + "_sync_worktree_with_remote returned without bringing " + "agents' plan-phase commits into the local worktree; " + "blocking phase advance to avoid silent demotion to " + "monolithic implement (#2337)" + ), + ) + raise _pkg.PlanDraftMissingOnLocalError( + f"plan draft {draft_rel} missing on local but present on " + f"origin/{branch} — refusing to advance plan phase" + ) + if not on_local and not on_origin: + _pkg.logger.error( + "OVERSEER_ALERT plan_draft_missing_on_local_and_origin", + pipeline_id=pipeline_id, + branch=branch, + draft_rel=draft_rel, + note=( + f"plan draft is missing from both the local worktree " + f"and origin/{branch}; advancing would produce an " + f"empty contract and strand implement-phase agents " + f"with nothing to do (#2627)" + ), + ) + raise _pkg.PlanDraftMissingOnLocalAndOriginError( + f"plan draft {draft_rel} missing on local and " + f"origin/{branch} — refusing to advance plan phase" + ) + + try: + return _pkg._populate_contract_from_plan( + repo_path, + pipeline_id, + pipeline_mode, + issue_number, + current_phase=current_phase, + ) + except _pkg.ForestValidationError as forest_err: + # Slice-DAG structural rejection is the expected #2137 / #3046 + # NACK path — log structurally so the discriminator shows up in + # operator audit, but don't propagate to the wrapper's + # caller (the populator already stashed the structured + # errors on contract.plan_review_feedback so the plan + # reviewer prompt can NACK the architect). The exception's + # ``reason`` selects the matching outcome so operators see an + # accurate discriminator (forest shape vs file-overlap order). + _pkg.logger.warning( + "contract_phases_ingest_failed", + pipeline_id=pipeline_id, + reason=forest_err.reason, + source="safe_wrapper", + errors=forest_err.errors, + ) + return _pkg.PopulateResult(_pkg._forest_error_to_outcome(forest_err)) + except Exception as pop_err: + _pkg.logger.warning( + "contract_phases_ingest_failed", + pipeline_id=pipeline_id, + reason="unexpected_exception", + source="safe_wrapper", + error=str(pop_err), + exc_info=True, + ) + return _pkg.PopulateResult(_pkg.PopulateOutcome.UNEXPECTED_EXCEPTION) + + +def _merge_preserved_slice_runtime( + new_slices: "list[ContractSlice]", # noqa: UP037 + old_slices: "list[ContractSlice]", # noqa: UP037 +) -> None: + """Carry runtime slice/task state from ``old_slices`` onto ``new_slices`` in place. + + ``_populate_contract_from_plan`` re-parses the plan markdown into a + fresh set of slices on every call — and its safety-net caller fires + on *every* ``start_phase=implement`` restart (deliberately outside + the ``contract_synced`` guard). The plan is the source of truth for + slice/task STRUCTURE (names, descriptions, dependencies, acceptance + criteria); it always parses back as ``PENDING`` with the runtime + bookkeeping fields unset. Blindly assigning ``contract.slices = + <freshly parsed>`` therefore wipes every slice the slice loop had + already advanced — resetting COMPLETE slices to PENDING and dropping + the ``parent_branch_at_creation`` / ``integration_base_sha`` a real + run stamped — so a restarted pipeline re-runs slice-1 forever and can + never reach slice-2 (#2908). + + Mirroring the PR-metadata preservation a few lines down in the + caller, this merges by slice id (and by task id within a slice): the + plan supplies STRUCTURE while RUNTIME state survives a re-populate. + Unmatched ids (a re-plan that adds or removes slices/tasks) simply + keep the plan's fresh ``PENDING`` defaults. + + Task-level runtime fields covered (each is durably written by a + runtime path that the plan parser cannot reconstruct): + + - ``status``, ``commit``, ``checkpoint_id``, ``review_cycles``, + ``escalated``, ``gaps`` — slice-loop / reviewer / tester + bookkeeping. + - ``role`` + ``delegation_attempts`` — paired SYSTEM-owned + impasse-delegation state. ``impasse_routing.py`` flips + ``task.role`` to the suggested alternative and bumps + ``delegation_attempts`` in the same ``apply_mutation`` cycle + under ``Role.SYSTEM`` (only SYSTEM owns these two fields); the + slice-loop dispatcher then routes the task to the new role. + Preserving the counter without the role would re-spawn the + original producer on restart and trip ``DELEGATION_LIMIT`` on + the next impasse, escalating to HITL even though no delegation + visibly happened — so both fields must survive together. + - ``notes`` — APPLIER writes Won't-Do drain failure reasons here + (``pipelines.py`` Won't-Do path) and agents write implementation + narrative via ``mcp__task__update_notes`` / ``egg-contract + update-notes``; the plan parser always emits ``""``. + - ``jira_action_status`` — APPLIER advances ``pending`` → + ``in_flight`` → ``applied``/``failed`` (#1557 risk_analyst R7); + idempotency depends on ``applied`` surviving re-populate so the + next apply skips it instead of re-creating the Jira issue. + - ``jira_key`` — APPLIER writes the freshly-allocated key back after + a ``create`` action so re-runs skip the create; plan parser emits + ``None`` on ``create`` actions, so re-populate would otherwise + strand the applier into creating duplicate tickets. + """ + old_by_id = {s.id: s for s in (old_slices or [])} + for new_slice in new_slices: + old_slice = old_by_id.get(new_slice.id) + if old_slice is None: + continue + # Slice-level runtime state stamped by ``_run_one_slice_inner`` + # and the bootstrap reconciler — never re-derivable from the plan. + new_slice.status = old_slice.status + new_slice.parent_branch_at_creation = old_slice.parent_branch_at_creation + new_slice.integration_base_sha = old_slice.integration_base_sha + new_slice.commit = old_slice.commit + new_slice.review_cycles = old_slice.review_cycles + # Defensive copy so post-merge mutations of the discarded ``old`` + # contract don't alias-leak into the live ``new`` contract. + new_slice.review_feedback = list(old_slice.review_feedback) + new_slice.escalated = old_slice.escalated + new_slice.escalation_reason = old_slice.escalation_reason + # Task-level runtime state: match by task id so a re-plan that + # adds/removes tasks still preserves completion of the survivors. + old_tasks_by_id = {t.id: t for t in old_slice.tasks} + for new_task in new_slice.tasks: + old_task = old_tasks_by_id.get(new_task.id) + if old_task is None: + continue + new_task.status = old_task.status + new_task.commit = old_task.commit + new_task.checkpoint_id = old_task.checkpoint_id + new_task.review_cycles = old_task.review_cycles + new_task.escalated = old_task.escalated + # Paired SYSTEM-owned impasse-delegation state — preserving + # the counter without the role would silently undo the + # delegation on restart (see docstring). + new_task.role = old_task.role + new_task.delegation_attempts = old_task.delegation_attempts + new_task.gaps = list(old_task.gaps) + # Runtime narrative + applier idempotency anchors. The + # plan parser cannot reconstruct any of these — see the + # docstring for the per-field invariants. + new_task.notes = old_task.notes + new_task.jira_action_status = old_task.jira_action_status + new_task.jira_key = old_task.jira_key + + +def _populate_contract_from_plan( + repo_path: _pkg.Path, + pipeline_id: str, + pipeline_mode: str = "issue", + issue_number: int | None = None, + *, + current_phase: _pkg.PipelinePhase | None = None, +) -> _pkg.PopulateResult: + """Read the plan draft and populate the contract with tasks. + + Extracts task structure from markdown headers in the plan draft + and writes tasks + acceptance criteria to the contract. + + Returns a :class:`PopulateResult` whose ``outcome`` discriminates + success from each silent-failure mode (#2627 follow-up). Callers + that need to fail-fast on an empty contract — natural plan-complete + and the ``start_phase=implement`` safety net — branch on ``outcome`` + to surface a dedicated HITL instead of advancing into an implement + phase with nothing to do. ``ForestValidationError`` continues to + raise so HTTP callers keep their structured-422 contract; the + wrapper translates that raise into + ``PopulateResult(FOREST_VIOLATION, 0, 0)``. + + When ``current_phase`` is provided, the contract's + ``current_phase`` is advanced to that value **only if it would move + the phase forward** (REFINE → PLAN → IMPLEMENT → PR). Backward + transitions are silently ignored so a respawn of the safety-net + populator (e.g. when a ``start_phase=implement`` pipeline progresses + to PR and re-enters ``_run_pipeline``) cannot demote the contract. + The advance also appends a ``create_transition_entry`` audit log + entry so operators inspecting the audit trail see the transition. + + This parameter is needed because the natural plan-completion path + advances ``pipeline.current_phase`` (orchestrator-side) but leaves + ``contract.current_phase`` for the reviewer agent / gateway phase + API to advance via ``apply_mutation``. When ``start_phase=implement`` + no plan reviewer runs, so the populator nudges the contract itself + (#2427 sub-bug). + """ + try: + from egg_contracts.loader import load_contract, save_contract + except ImportError: + _pkg.logger.warning( + "contract_phases_ingest_failed", + pipeline_id=pipeline_id, + reason="egg_contracts_unavailable", + ) + return _pkg.PopulateResult(_pkg.PopulateOutcome.EGG_CONTRACTS_UNAVAILABLE) + + # Resolve draft path + draft_rel = _pkg._get_draft_path("plan", issue_number=issue_number, pipeline_id=pipeline_id) + if not draft_rel: + _pkg.logger.warning( + "contract_phases_ingest_failed", + pipeline_id=pipeline_id, + reason="no_draft_path", + ) + return _pkg.PopulateResult(_pkg.PopulateOutcome.NO_DRAFT_PATH) + + plan_path = repo_path / draft_rel + if not plan_path.exists(): + _pkg.logger.warning( + "contract_phases_ingest_failed", + pipeline_id=pipeline_id, + reason="plan_draft_missing", + path=str(plan_path), + ) + return _pkg.PopulateResult(_pkg.PopulateOutcome.DRAFT_MISSING) + + try: + contract = load_contract(pipeline_id, repo_path) + except Exception as load_err: + _pkg.logger.warning( + "contract_phases_ingest_failed", + pipeline_id=pipeline_id, + reason="contract_load_failed", + error=str(load_err), + ) + return _pkg.PopulateResult(_pkg.PopulateOutcome.CONTRACT_LOAD_FAILED) + + try: + from egg_contracts.plan_parser import parse_plan + + plan_text = plan_path.read_text() + result = parse_plan(plan_text) + + if not result.success: + _pkg.logger.warning( + "contract_phases_ingest_failed", + pipeline_id=pipeline_id, + reason="parse_failed", + error=result.error, + ) + return _pkg.PopulateResult(_pkg.PopulateOutcome.PARSE_FAILED) + + for warning in result.warnings: + _pkg.logger.warning( + "Plan parse warning", + pipeline_id=pipeline_id, + warning_message=warning.message, + warning_context=warning.context, + ) + + contract_slices = result.to_contract_slices() + changed = False + + if contract_slices: + # Forest validation (#2137 TASK-2-2): the slice DAG must be + # a forest (every slice has ≤1 DAG parent). Multi-parent + # slices break the stacked-PR invariant and are rejected + # at ingestion so the plan reviewer NACKs the planner. + # + # ``parse_plan`` was already imported unconditionally above, + # so we don't guard ``validate_forest`` import — if the + # parser module is unavailable the populator has already + # failed; silently defaulting ``forest_errors = []`` would + # let a broken-import multi-parent contract slip past the + # gate (reviewer_code_holistic v2 finding #5). + from egg_contracts.plan_parser import ( + validate_forest, + validate_slice_file_overlap, + ) + + forest_errors = validate_forest(contract_slices) + + if forest_errors: + # Stash the structured errors onto the contract's + # ``plan_review_feedback`` so the plan reviewer's + # prompt picks them up and NACKs the planner with the + # error verbatim. The slices are NOT written to the + # contract — leaving ``contract.slices`` empty makes + # downstream phases visibly broken so the violation + # cannot silently leak through. + _pkg.logger.warning( + "contract_phases_ingest_failed", + pipeline_id=pipeline_id, + reason="forest_violation", + errors=forest_errors, + ) + feedback_lines = [ + "Plan ingestion REJECTED: the slice DAG is not a forest.", + "", + "Each slice must have at most one DAG parent. The " + "implement phase ships every slice as a stacked PR with " + "exactly one base branch — multi-parent slices break " + "this invariant. Re-emit the plan with " + "``serialized_chain_order`` populated on the downstream " + "slice (see issue #2137 plan TASK-2-3 for the rule).", + "", + "Structured errors:", + ] + feedback_lines.extend(f"- {e}" for e in forest_errors) + contract.plan_review_feedback = "\n".join(feedback_lines) + save_contract(contract, repo_path) + # Raise a structured ForestValidationError so any + # caller running this in an HTTP context (e.g. a + # plan-ingestion API endpoint) can surface a 422 with + # the inlined errors. Internal callers + # (``_populate_contract_from_plan_safe`` and the + # pipeline run-loop) catch and log instead — the + # ``plan_review_feedback`` stash above is the durable + # signal the reviewer prompt picks up either way. + raise _pkg.ForestValidationError("slice DAG is not a forest", errors=forest_errors) + + # File-overlap ordering validation (#3046). The forest is + # valid (≤1 parent per slice), but the implement phase cuts + # each slice's integration branch off its dependency parent + # (roots off ``work``) — so two slices that touch the same + # file MUST be ordered along a dependency chain, or their + # branches fork independently off the shared base and their + # edits collide at integration (the guaranteed modify/delete + # conflict observed on #3023). Reject overlapping-but-unordered + # slices here, with the SAME NACK-the-architect handling as a + # forest violation: stash the structured errors on + # ``plan_review_feedback`` and leave ``contract.slices`` empty + # so the defect cannot silently leak into the implement phase. + overlap_errors = validate_slice_file_overlap(contract_slices) + if overlap_errors: + _pkg.logger.warning( + "contract_phases_ingest_failed", + pipeline_id=pipeline_id, + reason="slice_overlap_violation", + errors=overlap_errors, + ) + feedback_lines = [ + "Plan ingestion REJECTED: slices touch overlapping files " + "without a dependency ordering.", + "", + "The implement phase cuts each slice's integration branch " + "off its dependency parent (root slices off the ``work`` " + "branch) and ships it as a stacked PR. Two slices that " + "touch the same file must be ordered along a single " + "dependency chain so the later slice's branch is forked " + "from a base that already contains the earlier slice's " + "commits — otherwise both branches fork independently off " + "the shared base and their edits collide at integration " + "(a guaranteed modify/delete conflict). The forest " + "constraint means the fix is always to serialise the " + "overlapping cluster into ONE linear ``dependencies`` " + "chain (you cannot depend on two parents) — or merge the " + "slices into one.", + "", + "Structured errors:", + ] + feedback_lines.extend(f"- {e}" for e in overlap_errors) + contract.plan_review_feedback = "\n".join(feedback_lines) + save_contract(contract, repo_path) + raise _pkg.ForestValidationError( + "slices share files without a dependency ordering", + errors=overlap_errors, + reason="slice_overlap_violation", + ) + # Preserve runtime slice/task progress across re-populates so + # the safety-net populator (which fires on every + # ``start_phase=implement`` restart) cannot reset COMPLETE + # slices to PENDING and strand the pipeline on slice-1 (#2908). + _pkg._merge_preserved_slice_runtime(contract_slices, contract.slices) + contract.slices = contract_slices + changed = True + + # Populate PR metadata from plan if available + if result.pr_title: + from egg_contracts.models import PRMetadata + + # Preserve orchestrator-populated runtime fields on + # ``PRMetadata`` across re-populates. The planner-emitted + # title/description/test_plan/manual_steps flow in fresh + # from the parsed plan; the fields below are populated by + # orchestrator code paths (the up-front context-PR opener + # in ``_open_context_pr_at_implement_start``, the + # conditional-ACK gate at ``complete_phase``) and would + # otherwise be silently dropped when this safety-net + # populator re-runs (e.g. on a ``start_phase=implement`` + # re-entry where ``deferred_actions`` was already populated + # during implement-phase close). + # + # ``deferred_actions`` is the merge-blocking *Pre-merge + # Obligations* handoff written by ``decisions.py`` after a + # conditional-ACK gate resolves; losing it here erases the + # reviewer's only durable handoff for git-mv / migration / + # cross-repo flips. See test + # ``test_populate_contract_from_plan_preserves_deferred_actions``. + preserved_pr_number = contract.pr.context_pr_number if contract.pr is not None else None + preserved_deferred_actions = ( + list(contract.pr.deferred_actions) if contract.pr is not None else [] + ) + contract.pr = PRMetadata( + title=result.pr_title, + description=result.pr_description or "", + test_plan=result.pr_test_plan or "", + manual_steps=result.pr_manual_steps or "", + context_pr_number=preserved_pr_number, + deferred_actions=preserved_deferred_actions, + ) + changed = True + + if current_phase is not None and contract.current_phase != current_phase: + # Forward-only: never demote. Without this guard a respawn + # of _run_pipeline (e.g. when a start_phase=implement pipeline + # progresses past the implement boundary and re-enters the + # safety-net call site) would silently roll + # contract.current_phase back from IMPLEMENT to whatever the + # call site hardcoded. The PR phase was removed in #2777 + # (cq-4); IMPLEMENT is now terminal. + _phase_order = ( + _pkg.PipelinePhase.REFINE, + _pkg.PipelinePhase.PLAN, + _pkg.PipelinePhase.IMPLEMENT, + ) + if ( + contract.current_phase in _phase_order + and current_phase in _phase_order + and _phase_order.index(current_phase) > _phase_order.index(contract.current_phase) + ): + from egg_contracts.audit import create_transition_entry + from egg_contracts.models import AuditRole + + old_phase = contract.current_phase + contract.audit_log.append( + create_transition_entry( + actor="orchestrator", + role=AuditRole.SYSTEM, + from_phase=old_phase.value, + to_phase=current_phase.value, + reason=( + "populator advanced contract.current_phase " + "(no apply_mutation caller for this pipeline; #2427)" + ), + ) + ) + contract.current_phase = current_phase + changed = True + + if changed: + save_contract(contract, repo_path) + slice_count = len(contract.slices) + task_count = sum(len(s.tasks) for s in contract.slices) + _pkg.logger.info( + "contract_phases_populated", + pipeline_id=pipeline_id, + phase_count=slice_count, + task_count=task_count, + has_pr_metadata=contract.pr is not None, + ) + return _pkg.PopulateResult( + _pkg.PopulateOutcome.POPULATED, + slice_count=slice_count, + task_count=task_count, + ) + else: + # Parse succeeded but yielded neither phases nor PR metadata — + # this is the #1931 failure mode (empty contract with no error). + # Emit a discriminator so the gap is visible in audit logs. + _pkg.logger.warning( + "contract_phases_ingest_failed", + pipeline_id=pipeline_id, + reason="empty_result", + warning_count=len(result.warnings), + ) + return _pkg.PopulateResult(_pkg.PopulateOutcome.EMPTY_RESULT) + + except _pkg.ForestValidationError: + # Re-raise so callers with HTTP context (or the safe wrapper) + # can surface the structured errors. The populator already + # stashed feedback on contract.plan_review_feedback before + # raising. + raise + except Exception as e: + _pkg.logger.warning( + "contract_phases_ingest_failed", + pipeline_id=pipeline_id, + reason="unexpected_exception", + source="parse_save", + error=str(e), + exc_info=True, + ) + return _pkg.PopulateResult(_pkg.PopulateOutcome.UNEXPECTED_EXCEPTION) + + +# Single source of truth for ForestValidationError.reason → PopulateOutcome +# mapping. Both ``_populate_contract_from_plan_safe`` and the +# ``start_phase=implement`` safety net translate a structural NACK into an +# outcome the empty-contract HITL prose dispatcher (#3046) can key off, so +# centralising the table here keeps the two catch sites from drifting if a +# third reason is added to :class:`ForestValidationError` (forest-shape vs. +# file-overlap-ordering today). Unknown reasons fall back to +# ``FOREST_VIOLATION`` — that's the conservative choice because the operator +# prose for forest violations names the slice DAG generally rather than the +# specific defect, so a new reason without a dedicated outcome still routes to +# actionable (if generic) HITL prose. Lives here (not the barrel) because it +# references the PopulateOutcome enum at definition time; it re-exports through +# the barrel so ``_pkg._FOREST_REASON_TO_OUTCOME`` resolves (#3312 slice-4). +_FOREST_REASON_TO_OUTCOME: dict[str, PopulateOutcome] = { + "slice_overlap_violation": PopulateOutcome.SLICE_OVERLAP_VIOLATION, + "forest_violation": PopulateOutcome.FOREST_VIOLATION, +} diff --git a/orchestrator/routes/pipelines/_prompt_agent.py b/orchestrator/routes/pipelines/_prompt_agent.py new file mode 100644 index 0000000000..b8624f74d1 --- /dev/null +++ b/orchestrator/routes/pipelines/_prompt_agent.py @@ -0,0 +1,1346 @@ +"""agent-prompt assembly helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _build_file_boundary_section(role_value: str, repo: str | None = None) -> str: + """Build a file boundary section for an agent prompt. + + Sources the role's allowed/blocked patterns from + ``egg_restrictions.patterns.build_agent_patterns`` so the prompt + matches what the gateway will actually enforce on push — including + per-repo ``role_patterns:`` overrides from ``repositories.yaml`` + (#2528). The legacy ``egg_contracts.agent_roles`` patterns were + Python-only and didn't honour the per-repo knobs, which created a + contradictory message for non-Python repos: the gateway would + enforce Go conventions while the prompt told the agent the boundary + was Python. + + Returns an empty string when no patterns are defined for the role. + """ + try: + from egg_restrictions.patterns import get_agent_pattern_for_repo + except ImportError: + return "" + + pattern = get_agent_pattern_for_repo(role_value, repo=repo) + if pattern is None: + return "" + + if ( + not pattern.allowed_patterns + and not pattern.blocked_patterns + and not pattern.hard_blocked_patterns + ): + return "" + + lines = [ + "## File Boundaries (Gateway-Enforced)\n", + f"Your role ({role_value.upper()}) can only push changes to files " + "matching these patterns. The gateway will **reject your push** if it " + "includes files outside your boundaries. Only create and modify files " + "you are allowed to push.\n", + ] + if pattern.allowed_patterns: + lines.append("**Allowed:** " + ", ".join(f"`{p}`" for p in pattern.allowed_patterns)) + if pattern.blocked_patterns: + lines.append("**Blocked:** " + ", ".join(f"`{p}`" for p in pattern.blocked_patterns)) + # Hard blocks are a stricter tier: they are rejected even when they would + # otherwise match your allow list or a docs/fixture exemption (#3396). The + # agent must see them, or it will author a hard-blocked path (e.g. + # `.egg-state/contracts/fixtures/x.json`, `.github/actions/x/testdata/`), + # hit a gateway 403, and have no way to understand why. + if pattern.hard_blocked_patterns: + hard_line = "**Hard-blocked (never pushable, no exemption applies):** " + ", ".join( + f"`{p}`" for p in pattern.hard_blocked_patterns + ) + if pattern.hard_block_exempt_patterns: + hard_line += " — except " + ", ".join( + f"`{p}`" for p in pattern.hard_block_exempt_patterns + ) + lines.append(hard_line) + + # `.github/` staging-dir convention (issue #2508). Surfaced for the + # coder role specifically because it's the producer that's expected + # to initiate `.github/` work. The role-pattern check + # (``startswith(".github/")``) doesn't match `.github-staging/`, so + # autofixer / conflict_resolver allowlists technically reach the + # staging path too — but those roles are reactive and aren't asked + # to plan new `.github/` changes, so the convention's planning-time + # guidance only needs to land for coder. + if role_value == "coder": + lines.append("") + lines.append( + "**`.github/` changes**: `.github/` is blocked above. If your " + "task requires modifying CI workflows, CODEOWNERS, dependabot " + "config, or anything else under `.github/`, write the proposed " + "end-state to top-level `.github-staging/` instead, mirroring " + "the `.github/` structure (e.g. stage " + "`.github/workflows/test-e2e.yml` as " + "`.github-staging/workflows/test-e2e.yml`). Call out the " + "staged files explicitly in your PR body so the human reviewer " + "knows to move them into `.github/` before merge — see issue " + "#2508." + ) + lines.append("") + return "\n".join(lines) + + +def _build_agent_prompt( + role_value: str, + phase: str, + pipeline_id: str, + pipeline_mode: str, + prompt: str | None = None, + issue_number: int | None = None, + repo: str | None = None, + branch: str | None = None, + base_branch: str | None = None, + review_feedback: str | None = None, + review_cycle: int = 0, + repo_path: str | None = None, + phase_obj=None, + all_phases=None, + concurrent: bool = False, + network_mode: str | None = None, + operator_directives: list[_pkg.OperatorDirective] | None = None, + iteration_history: list[_pkg.IterationSummary] | None = None, +) -> str: + """Build a role-specific prompt for multi-agent execution. + + For the CODER role, delegates to the existing _build_phase_prompt(). + Other roles (TESTER, DOCUMENTER, ARCHITECT, etc.) get + role-specific instructions. + + Execution roles (tester, documenter) receive a summarized + background with structured task information instead of the full issue + body. Analysis roles (architect, task_planner, risk_analyst) receive + the full issue body. + + Note: Handoff data is passed via the EGG_HANDOFF_DATA environment + variable, not via the prompt — prompts are built once before + execution starts. + + Args: + role_value: Agent role string (e.g. "coder", "tester") + phase: Pipeline phase name + pipeline_id: Pipeline ID + pipeline_mode: "issue" or "local" + prompt: Original task prompt + issue_number: GitHub issue number + repo: Repository name + branch: Branch name + review_feedback: Feedback from prior review cycle + review_cycle: Current review cycle number + repo_path: Filesystem path to repository (for user override lookup) + phase_obj: Current plan phase object (optional) + all_phases: All contract phases (optional) + concurrent: Whether agent runs in concurrent multi-agent mode. + When True, adds consensus lifecycle preamble instructing the + agent to stay alive, poll messages, and participate in consensus. + network_mode: Pipeline network mode ("public", "private", or None). + When "private", injects warnings about blocked package downloads. + + Returns: + Complete prompt string for the agent + """ + # CODER and REFINER use the existing phase prompt (phase-specific + # instructions are already tailored for refine vs implement etc.) + if role_value in ("coder", "refiner"): + base_prompt = _pkg._build_phase_prompt( + phase=phase, + pipeline_id=pipeline_id, + pipeline_mode=pipeline_mode, + prompt=prompt, + issue_number=issue_number, + repo=repo, + branch=branch, + review_feedback=review_feedback, + review_cycle=review_cycle, + repo_path=repo_path, + operator_directives=operator_directives, + iteration_history=iteration_history, + ) + # Surface file boundaries so agent knows what it can push (#1431). + # Pass repo so the rendered patterns match per-repo overrides + # (#2528) the gateway will enforce on push. + boundary_section = _pkg._build_file_boundary_section(role_value, repo=repo) + if boundary_section: + base_prompt += "\n" + boundary_section + # Producer escape hatch (#2529) — coder is one of the impassing + # producer roles, so it must see the actionable + # check_file_restriction / report_impasse guidance instead of + # inventing workarounds. Refiner runs in the refine phase and + # never owns implement-phase tasks, so it doesn't need this. + if role_value == "coder": + base_prompt += "\n" + _pkg._build_impasse_escape_hatch_section() + # In concurrent mode, inject BRC consensus preamble so the coder/refiner + # knows to propose, respond to reviews, confirm, and stay alive. + if concurrent: + base_prompt += _pkg._build_brc_preamble( + role_value, + phase, + repo=repo, + branch=branch, + base_branch=base_branch, + ) + return base_prompt + + if role_value.startswith("reviewer_"): + # Reviewer prompts are fully built by _build_review_prompt with its + # own criteria/verdict format + iteration-context wiring; we don't + # accumulate the role-shared ``lines`` block for them. Dispatching + # here (rather than mid-function with an early return) prevents + # future drift where a "must always be included" line is added to + # the accumulation and silently never reaches reviewers (#2795). + reviewer_type = role_value.replace("reviewer_", "", 1).replace("_", "-") + review_prompt = _pkg._build_review_prompt( + phase=phase, + pipeline_id=pipeline_id, + pipeline_mode=pipeline_mode, + reviewer_type=reviewer_type, + issue_number=issue_number, + review_cycle=review_cycle + 1, + prior_feedback=review_feedback, + repo_path=repo_path, + base_branch=base_branch, + concurrent=concurrent, + operator_directives=operator_directives, + iteration_history=iteration_history, + ) + if concurrent: + review_prompt += "\n" + _pkg._build_brc_preamble( + role_value, + phase, + repo=repo, + branch=branch, + base_branch=base_branch, + ) + return review_prompt + + # Build context header (shared across all roles) + lines = [f"You are the **{role_value.upper()}** agent in the **{phase}** phase.\n"] + lines.append("## Context\n") + lines.append(f"Pipeline ID: {pipeline_id}") + lines.append(f"Phase: {phase}") + lines.append(f"Mode: {pipeline_mode}") + lines.append(f"Agent Role: {role_value}") + if repo: + lines.append(f"Repository: {repo}") + if branch: + lines.append(f"Branch: {branch}") + if issue_number is not None: + lines.append(f"Issue: #{issue_number}") + lines.append("") + + # Concurrent mode: add BRC consensus lifecycle preamble so agents understand + # they must stay alive and participate in Broadcast-Review-Converge consensus. + if concurrent: + lines.append( + _pkg._build_brc_preamble( + role_value, + phase, + repo=repo, + branch=branch, + base_branch=base_branch, + ) + ) + + # Include role-appropriate context instead of the raw issue body. + # Analysis roles (architect, task_planner, risk_analyst) receive the full + # issue body. Execution roles (tester, documenter) receive a + # brief summary with structured task information and context pointers. + role_context = _pkg._build_role_context( + role_value=role_value, + prompt=prompt, + issue_number=issue_number, + phase_obj=phase_obj, + all_phases=all_phases, + base_branch=base_branch, + ) + if role_context: + lines.append(role_context) + + # Phase iteration context: operator directives + prior iteration history. + # Rendered for all roles (producers AND reviewers) so reviewers cannot + # NACK a directive-driven change against a stale default rubric (#2795). + iteration_context = _pkg._build_phase_iteration_context(operator_directives, iteration_history) + if iteration_context: + lines.append(iteration_context) + + # Review feedback from prior agentic cycles (scoped to agentic NACKs + # since #2795 — HITL kickbacks render via the iteration context above). + if review_feedback: + lines.append("## Review Feedback\n") + lines.append(review_feedback) + lines.append("") + + # Derive the pipeline identifier for namespaced output filenames. + _identifier = _pkg._pipeline_identifier(issue_number, pipeline_id) + + # Spec-driven agent-output paths (#3077 slice-3): resolve each path + # via the artifact registry so the prompt prose, the propose-time + # validator (signals._validate_producer_artifacts), and the gateway + # artifact-read endpoint (slice-4) all share one source of truth. + # The slice-2 mandatory consistency test + # (TestConsistencyC in shared/egg_contracts/tests/test_artifact_spec.py) + # pins these call sites to the registry; a future row rename + # surfaces here as a missing prompt path instead of as #3016-style + # drift between spec and rendered prose. + from egg_contracts.artifact_spec import resolve_artifact_path as _resolve_artifact_path + + _architect_output_path = _resolve_artifact_path("architect-output", _identifier) + _architect_slices_path = _resolve_artifact_path("architect-slices", _identifier) + _risk_analyst_output_path = _resolve_artifact_path("risk-analyst-output", _identifier) + # Human-focused companion drafts the simplifier produces (one per phase). + _analysis_human_path = _resolve_artifact_path("analysis-draft-human", _identifier) + _plan_human_path = _resolve_artifact_path("plan-draft-human", _identifier) + + # Role-specific instructions + lines.append("## Your Task\n") + + if role_value == "tester": + # Look up per-repo check commands from repositories.yaml + repo_checks: list[dict[str, str]] = [] + if repo: + try: + repo_checks = _pkg.get_repo_checks(repo) + except FileNotFoundError: + repo_checks = [] + + lines.extend( + [ + "**ROLE BOUNDARY: You are the TESTER, not the CODER.** " + "Do NOT implement application logic, create source files, write configuration, " + "or set up project infrastructure. Your job is to write tests for the CODER's " + "implementation, run checks, and report gaps. If the coder hasn't committed yet, " + "wait — do not implement the solution yourself.", + "", + "**Your mandate is two-fold**:", + "", + "1. **Comprehensive coverage** — write tests that prevent " + "regressions, covering the happy path and realistic alternative " + "paths through every changed area. New behavior gets new tests; " + "modified behavior gets updated tests; nothing the coder changed " + "should silently lose coverage.", + "2. **Adversarial probing** — actively probe the coder's " + "implementation for bugs and edge cases they missed. Treat the " + "implementation as suspect until you have tried to break it. " + "Write tests that target suspected weaknesses. When a test " + "fails because of a coder-side bug, **the committed failing " + "test is evidence — the NACK is the bug report**. Pair every " + "failing test with an explicit NACK on the coder's proposal " + "that names the failing test in its rationale; otherwise the " + "bug is easy for the coder to miss. Also list the bug in " + "`gaps_found` and HANDOFF to coder with the failure output. " + "The coder owns the fix; you own surfacing the bug.", + "", + "You are also responsible for **lint/type-check validation**.", + "", + "### When the slice warrants no new tests (#3027)", + "", + "Pure refactors (symbol moves, decompositions with no behavior " + "change), doc-only slices, and other no-test-work slices still " + "require you to **propose** — BRC consensus blocks until every " + "producer has proposed at least once. **Don't just heartbeat " + "and wait for work that isn't coming.** Instead submit a " + "generic no-op propose:", + "", + "1. (Optional but encouraged) run the configured checks against " + "the coder's diff (`make lint`, `make test`, etc.) to confirm " + "the slice really is behavior-preserving.", + "2. Propose a no-op: `egg-orch consensus propose " + "--no-changes-needed --no-changes-reason '<concrete reason, " + "e.g. slice-3 is a pure decomposition: symbol moves between " + "submodules, no behavior change; existing suite covers the " + "re-exported barrel>'`. No artifacts or commit-sha are needed.", + "", + "The no-op counts as proposing (so consensus is not blocked on " + "you) and is accepted as a non-blocking no-op — reviewers do not " + "review or NACK it. If the slice **does** have new test work " + "(real behavior changes, new edge cases, modified contracts), do " + "NOT use the no-op path — author tests and propose as usual.", + "", + "### Testing", + "", + "1. Review the changed files (available in handoff data or via git diff)", + "2. Build coverage tests for the happy path and realistic " + "alternative paths in every changed area", + "3. **Adversarially probe** the implementation: identify " + "suspected bugs and untested edge cases, then write tests that " + "target them", + "4. Run all tests. Tests that pass demonstrate coverage; " + "**tests that fail demonstrate bugs you have found** — keep them", + "5. For every failing test caused by a coder-side bug: " + "commit the failing test AND **NACK the coder's proposal, " + "explicitly naming the failing test in the NACK rationale**. " + "The committed test alone is not sufficient — the NACK is " + "what surfaces the bug to the coder. Also list the bug in " + "`gaps_found` and HANDOFF to the coder with the failure " + "output. Your `test` configured check will fail until the " + "coder pushes a fix — that is expected; do NOT propose " + "consensus until every configured check passes per the " + "*Configured Checks* section below", + "6. Commit all test files with descriptive messages", + "", + "Adversarial probing — actively try to break the implementation:", + "- Missing error handling and input validation", + "- Boundary conditions, off-by-one, empty/null/oversized inputs", + "- Uncovered code paths and branches (especially error paths)", + "- Concurrency: races, partial failures, retry behavior, ordering assumptions", + "- Contract violations: does the code actually match the " + "acceptance criteria, or just the happy path of them?", + "- Integration gaps between components and unstated interface assumptions", + "", + "Gap-finding focus (still report these in `gaps_found` even " + "when you cannot write a test for them):", + "- Logic errors that would require design changes to fix", + "- Inconsistencies between the implementation and the plan/contract", + "- Missing test infrastructure that prevents adequate coverage", + "", + "### Configured Checks (MANDATORY)", + "", + "You MUST run **ALL** configured checks below and fix any failures " + "before proposing consensus. Skipping checks (e.g., running tests but " + "not lint) is a common failure mode — do not skip any.", + "", + ] + ) + + if repo_checks: + # Inject explicit check commands from repositories.yaml + lines.extend( + [ + "The following check commands are configured for this repository. " + "Run **every one** of them **in order**:", + "", + ] + ) + for i, check in enumerate(repo_checks, 1): + name = check["name"].replace("\n", " ").strip() + cmd = check["command"].replace("\n", " ").strip() + lines.append(f"{i}. **{name}**: `{cmd}`") + lines.extend( + [ + "", + "If ANY check fails in test files you wrote, fix the issue and re-run. " + "If failures are in source code, do NOT fix them — report them to the coder.", + "", + "After running all checks:", + ] + ) + else: + # Fall back to auto-discovery + lines.extend( + [ + "1. **Discover commands**: Look for Makefile, pyproject.toml, package.json, " + "setup.cfg, tox.ini, or similar build/test configuration files", + "2. **Run linters**: Execute linters (ruff, eslint, golangci-lint, etc.)", + "3. **Run type checkers**: Execute type checkers (mypy, pyright, tsc, etc.)", + "", + "After running all checks:", + ] + ) + + lines.extend( + [ + "- **Auto-fix test files only**: Fix auto-fixable issues in test files you wrote " + "(formatting, import order, simple type errors)", + "- **Repeat**: Re-run checks to verify fixes. Repeat up to 3 times.", + "- **Commit test fixes**: Commit all test-file fixes together with a descriptive message", + "", + "Auto-fixable (in test files only — commit fixes directly):", + "- Lint errors in test files (formatting, import order, code style)", + "- Type errors in test files with clear fixes", + "", + "Report only (do NOT modify source code — NACK the coder and explain what's needed):", + "- Lint or type errors in source code — tell the coder to fix these", + "- Test failures caused by bugs in the coder's implementation — tell the coder to fix", + "- Complex logic errors requiring design decisions", + "- Security issues requiring architectural changes", + "", + "When testing third-party library integrations or unfamiliar frameworks, " + "use WebSearch and WebFetch (when available) to look up testing patterns, " + "known edge cases, and recommended test approaches for those libraries.", + "", + "## Parallel Execution with Subagents\n", + "If the changes span multiple independent components or modules, you can use " + "Claude Code's **Agent tool** to parallelize test writing. Launch one subagent " + "per component to write and run tests concurrently. Each subagent should work " + "on non-overlapping test files. Subagents should only write files — do NOT " + "stage or commit from subagents. After all subagents complete, run the full " + "test suite to verify everything passes together, then stage and commit yourself.", + "", + *_pkg._EXPLORATION_SUBAGENT_GUIDANCE, + ] + ) + + # Test execution verification — prevents proposing consensus with + # unverified tests (issue #1359). + test_verify_lines = [ + "### Test Execution Verification (CRITICAL)\n", + "You MUST actually execute the test suite (`go test`, `pytest`, `jest`, etc.). " + "Passing gofmt, syntax checks, or linting alone does NOT count as tests run.\n", + "If tests cannot run (e.g., dependency downloads blocked in private network mode, " + "missing build tools), you MUST:", + "1. Set `tests_execution_blocked: true` and provide `tests_execution_blocked_reason` " + "in your attestation when proposing consensus", + '2. Include an explicit **"TESTS UNVERIFIED"** warning in your proposal summary', + '3. Do NOT claim your work is "complete" — state that tests are written but unverified', + "", + "**Distinguish `tests_execution_blocked` from a no-op propose** " + "(see the no-op section above): set `tests_execution_blocked=true` " + "when you DID author / intend tests but the configured checks could " + "not run (blocked downloads, missing tools) — that is a real " + "proposal with lower confidence. Use the generic no-op propose " + "(`--no-changes-needed`) only when the slice genuinely warrants no " + "new tests at all. Don't conflate the two.", + "", + ] + if network_mode == "private": + test_verify_lines.extend( + [ + "**WARNING: Private network mode is active** — external package downloads " + "(go mod download, npm install, pip install, etc.) may be blocked. " + "If dependency installation fails, you cannot verify tests. " + "Follow the instructions above to flag tests as unverified.", + "", + ] + ) + lines.extend(test_verify_lines) + + # Check execution verification — prevents proposing consensus without + # running all configured checks (issue #1414). + check_verify_lines = [ + "### Check Execution Verification (CRITICAL)\n", + "You MUST run **every** configured check command and ensure they **pass** " + "before proposing consensus. Running tests alone is NOT sufficient — " + "lint, type-check, and security checks must also pass. If you skip a " + "check or propose with a failing check, the server will reject your " + "proposal.\n", + "Before proposing, verify:", + "- [ ] All configured check commands have been executed", + "- [ ] All checks pass (or failures have been auto-fixed and re-verified)", + "- [ ] Any auto-fix commits have been pushed", + "", + # Source-failure handling — without this, agents have rationalised + # inventing ad-hoc check names so their attestation passes, masking + # red CI on the initial push (issue #1966). + "### When Source-Code Checks Fail (CRITICAL)\n", + "If a configured check fails because of the **coder's source code** " + "(not test files you wrote), you have a binding choice: " + "**do NOT propose consensus**. The role boundary above forbids you " + "from fixing source code, and the rules below forbid you from " + "papering over the failure. Instead:\n", + "1. **Do NOT fix it yourself** — that crosses the tester role boundary.", + "2. **Do NOT invent a narrower or renamed check** " + "(e.g. `pytest-<your-suite>`, `ruff-check-tester-files`) and attest to " + "*that* in `checks_passed`. Only the literal names from " + "`repositories.yaml` (`lint`, `test`, `security`, etc.) are valid; " + "the server will reject anything else, and substituting narrower names " + "hides real CI failures from reviewers.", + "3. **Send a HANDOFF message to the coder** describing the failing " + "check, the command, and the diagnostic output, e.g.:", + " ```", + " egg-orch message send --to coder --type HANDOFF \\", + ' --subject "lint failing on src/foo.py" \\', + ' --body "make lint exits 1: mypy errors in src/foo.py:42 ' + '(incompatible types). Please fix and push; I will re-run lint."', + " ```", + " If you are also reviewing the coder's own consensus proposal, " + "NACK it for the same reason — the two channels reinforce each other.", + "4. **Wait** for the coder to push a fix, then **re-run every " + "configured check** from scratch. Use `egg-orch message wait-loop` " + "(see Producer Lifecycle) — do not spin in a shell `for` loop or " + "prefix with `sleep`.", + "5. **Only propose consensus once every configured check passes " + "literally**, with the configured names in `checks_passed`.", + "", + "If the coder is unresponsive or the failure genuinely cannot be " + "fixed within this phase, document it in `gaps_found` and let the " + "orchestrator escalate via `OVERSEER_ALERT`. Do NOT work around the " + "block by proposing with a partial or renamed `checks_passed` list.", + "", + "### Attestation: `checks_passed` (REQUIRED)\n", + "When proposing consensus, your attestation MUST include a `checks_passed` " + "list containing the **name** of every configured check that **passed**. " + "Do NOT include checks that failed, and do NOT invent ad-hoc names " + "(e.g. `pytest-<scope>`, `ruff-check-tester-files`) — only the literal " + "names from `repositories.yaml`. " + "For example, if the repo has `lint` and `test` checks and both pass, " + 'your attestation must include `"checks_passed": ["lint", "test"]`. ' + "The server will reject your proposal if any configured check is missing " + "from this list (i.e. did not pass).", + "", + ] + lines.extend(check_verify_lines) + + elif role_value == "documenter": + lines.extend( + [ + "Document the CURRENT STATE of the code after this change. " + "Write as if the code has always worked this way — the " + "slice/pipeline machinery that produced the change does not " + "belong in the documentation:", + "", + "1. Review the changed files (available in handoff data or via git diff)", + "2. Update relevant documentation (READMEs, docstrings, API docs) so it " + "describes how the system works now", + "3. Add or update inline code comments where they clarify current behavior", + "4. Commit documentation changes with descriptive messages", + "", + "Write snapshots, not changelogs:", + "- Describe what the code does now, not what changed or when it changed.", + "- NEVER reference SDLC artifacts — slice numbers, TASK-N ids, phase or " + "HITL iteration numbers — in any doc, docstring, or inline comment you write.", + '- Include historical context (issue links, "previously X" rationale, ' + "migration notes) ONLY when it is tangibly valuable to a reader of the " + 'current system, and prefer rationale ("why it is this way") over ' + 'chronology ("what it used to be / when it changed").', + "- When updating an existing doc, fold the new state into the snapshot and " + "REMOVE now-stale ledger or historical entries rather than appending " + "another layer.", + "", + "When documenting third-party integrations or external APIs, use WebSearch " + "and WebFetch (when available) to verify current API signatures, link to " + "official documentation, and confirm usage examples are up to date.", + "", + "### When the slice warrants no doc updates (#3027)", + "", + "Pure refactors (symbol moves, decompositions with no " + "surfaced API change), test-only slices, and internal-only " + "slices that don't touch any documented surface still " + "require you to **propose** — BRC consensus blocks until " + "every producer has proposed at least once. **Don't just " + "heartbeat and wait for work that isn't coming.** Instead " + "submit a generic no-op propose:", + "", + "1. Walk the coder's diff and confirm there is no " + "documented-surface impact: no public API signature " + "changes, no behavior changes a user-facing doc describes, " + "no new feature or flag mentioned in README / docs/, no " + "docstring contracts that drift.", + "2. Propose a no-op: `egg-orch consensus propose " + "--no-changes-needed --no-changes-reason '<concrete reason, " + "e.g. a pure decomposition: symbol moves between " + "submodules, no surfaced API change; no README / docs/ / " + "docstring surface impacted>'`. No artifacts or commit-sha " + "are needed.", + "", + "The no-op counts as proposing (so consensus is not blocked " + "on you) and is accepted as a non-blocking no-op — reviewers " + "do not review or NACK it. If the slice **does** have doc " + "impact (any of the bullets above), do NOT use the no-op " + "path — author doc changes and propose as usual.", + "", + *_pkg._EXPLORATION_SUBAGENT_GUIDANCE, + ] + ) + elif role_value == "architect": + lines.extend( + [ + "Analyze the task and produce an architecture analysis:", + "", + "1. Understand the problem or feature request from the issue", + "2. Research the current codebase to understand existing patterns", + "3. Research externally when the task involves third-party libraries, APIs, " + "or frameworks — use WebSearch and WebFetch (when available) to verify " + "assumptions, check current documentation, review architectural patterns, " + "and look up current best practices. Skip external research for purely " + "internal changes.", + "4. Identify key files, constraints, and dependencies", + "5. Consider multiple implementation approaches", + "6. Recommend an approach with justification and document technical decisions", + "7. **Surface runtime-primitive assumptions explicitly (see #2594).** " + "When your analysis mentions a class, function, HTTP route, env var, " + "ConfigMap key, test fixture, CLI flag, or decorator, cite it with " + "`file:line` evidence (`grep -rn` is enough). Call out scope on " + "**both** of the following orthogonal axes when either matters: " + "(a) **purpose** — is the primitive unit-test-only (e.g. a test " + "double like `ScriptedProvider`) vs deployed-pod / production " + "code; (b) **execution context** — does the consumer run as " + "`in-sandbox-agent` (agent pod, reaches gateway via `GATEWAY_URL`) " + "vs `trusted-CI-runner` (pytest from outside the cluster, sees " + "`orchestrator_url` / lifecycle-secret-gated routes / kubectl). A " + "primitive can be unit-test-only but invoked from either runner, " + "or deployed-pod-only but called from either runner — these are " + "independent dimensions, so spell out whichever applies. Buried " + "runtime assumptions are the dominant cause of expensive " + "implement-phase NACKs; surfacing them here makes the plan-phase " + "audit cheap.", + "", + f"Write your analysis to `{_architect_output_path}`.", + "", + # ---------------------------------------------------- + # #2809 — architect owns slice composition + # ---------------------------------------------------- + "## Slice composition authority (#2809)", + "", + "**You are the sole authority for slice composition in the " + "plan phase.** ``task_planner`` enumerates tasks within the " + "slices you define; ``risk_analyst`` surfaces risks that " + "feed your design. Neither owns slice shape — you do. " + "Specifically, you own:", + "", + "- **Slice count.** Treat the operator's ``cq-1`` (or " + "equivalent refine-phase complexity answer) as a coarse " + "top-level hint, not a literal slice count. Subdivide " + "further when the natural slice DAG calls for it.", + "- **Slice boundaries.** Which work goes into which slice, " + "anchored on design seams.", + "- **Slice DAG shape.** Parent/child dependencies between " + "slices. The forest constraint (every slice has at most " + "ONE DAG parent) is HARD — multi-parent slices break the " + "stacked-PR invariant. If a slice would naturally have >1 " + "parents, serialise the upstream slices into a linear " + "chain and record the chosen ordering on the downstream " + "slice's ``serialized_chain_order`` field. See " + "``docs/architecture/slice-dag.md``.", + "- **File-overlap ⇒ dependency edge (HARD — #3046).** Any two " + "slices that touch the same file MUST be ordered on one " + "dependency chain (express the order in ``dependencies`` — a " + "single-parent id per slice — not just in " + "``serialized_chain_order``, which the scheduler does not read " + "for branch topology). Slices that edit a shared file but are " + "left as parallel roots/siblings fork independently off the " + "shared base and collide at integration — plan ingestion " + "hard-rejects this. A slice that deletes or retires a file " + "must depend on every slice that modifies it. Keep slices with " + "disjoint file sets parallel so they still run concurrently.", + "- **Test co-location (HARD — #3411).** A slice that removes, " + "renames, or rewrites code must carry the matching updates to " + "the tests exercising that code — skip-guards, deletions, " + "rewrites — in the SAME slice, never a later one. Every " + "cumulative slice tip must be independently green: the " + "per-slice green gate (#3398) runs the repo's checks at each " + "slice tip and blocks the PR while any check is red, so a " + "plan that parks test obsolescence in a later slice " + "guarantees a blocked slice and repair-loop churn on slices " + "whose only sin is plan topology. In repos that ship the " + "changeset-aware selector (this repo's " + "``scripts/select_tests``), the affected tests are " + "statically discoverable with the same import graph ``make " + "test`` narrowing uses: ``python3 " + "scripts/select_tests/__main__.py --impacted-tests " + "<file>...`` prints every test file that transitively " + "imports the named files (exit 2 = closure unavailable — " + "fall back to grepping the removed symbols in the test " + "trees). Write the removing slice's ``goal`` so it " + "explicitly includes those test updates; ``task_planner`` " + "enumerates them as tasks in that slice.", + "- **Sub-slicing.** When one slice would be too coarse, " + "subdivide it. Right-size slices for a single BRC cycle: " + "avoid bundling distinct file-category groups (e.g. " + "orchestrator + gateway + schema + tests + docs all in " + "one slice), avoid bundling deletion-heavy work with " + "new-API-introduction work, and avoid bundling task " + "groups that have no internal dependency — those are " + "natural seams for parallel sub-slices. If a slice would " + "require the implementing producer to " + "commit-propose-revise more than 3–4 times to converge, " + "subdivide it.", + "", + "Emit the slice scaffold as a YAML file alongside your " + "JSON analysis. ``task_planner`` will copy this scaffold " + "**verbatim** into the plan document's ``# yaml-tasks`` " + "appendix and fill in ``tasks:`` under each slice — the " + "scaffold is binding. If ``reviewer_plan`` NACKs on " + "``slice_size`` or the structural lens calls a " + "sub-division, you re-propose with the updated scaffold; " + "task_planner re-consumes the new scaffold on the next " + "BRC cycle.", + "", + f"Write the slice scaffold to `{_architect_slices_path}`:", + "", + "```yaml", + "slices:", + " - id: 1", + " name: |-", + " <slice name>", + " goal: |-", + " <what this slice achieves>", + " # root slice — omit ``dependencies``", + " - id: 2", + " name: |-", + " <slice name>", + " goal: |-", + " <what this slice achieves>", + " dependencies: slice-1", + "```", + "", + "Omit ``dependencies`` for root slices; for every non-root " + "slice set ``dependencies`` to its single parent's " + "``slice-<id>`` (e.g. ``slice-1``). ``dependencies`` is the " + "canonical ordering key the plan parser reads (per " + "`.egg/schemas/yaml-tasks.schema.json`) — the slice DAG is a " + "forest, so each slice has at most one parent (one id, not a " + "list). Do NOT include ``tasks:`` in the scaffold — that is " + "task_planner's job. Keep ``name`` and ``goal`` concise " + "enough that task_planner can copy them without rewording.", + "", + "### File Restrictions", + "", + "You MUST only write to:", + f"- `{_architect_output_path}`", + f"- `{_architect_slices_path}`", + "", + "Do NOT create or modify any other files. Specifically:", + "- Do NOT modify analysis drafts (`.egg-state/drafts/*-analysis.md`) — " + "these are finalized in the refine phase and are read-only", + "- Do NOT create or modify contracts (`.egg-state/contracts/`)", + "- Do NOT create or modify reviews (`.egg-state/reviews/`)", + "- Do NOT create or modify plan drafts (`.egg-state/drafts/*-plan.md`)", + "", + *_pkg._EXPLORATION_SUBAGENT_GUIDANCE, + ] + ) + elif role_value == "task_planner": + draft_path = _pkg._get_draft_path( + "plan", issue_number=issue_number, pipeline_id=pipeline_id + ) + # Spec-driven (#3077 slice-3) — reuses the helper-resolved path above + # so the task_planner prose and the architect prompt cannot drift. + architect_slices_path = _architect_slices_path + lines.extend( + [ + "Decompose the architecture analysis into a slice-DAG implementation " + "plan. The implement-phase pipeline ships each slice as its own " + "stacked PR.", + "", + "**Slice composition is NOT your call (#2809).** ``architect`` owns " + "slice count, slice boundaries, slice DAG shape, and sub-slicing — " + f"and emits the binding scaffold at `{architect_slices_path}`. Your job " + "is to enumerate ``tasks:`` within those slices, **not to re-shape " + "them**. Copy the architect's scaffold verbatim into the " + "``# yaml-tasks`` appendix (preserving slice ``id``, ``name``, " + "``goal``, and ``dependencies``) and add ``tasks:`` under each " + "slice with task IDs of the form ``TASK-<slice_id>-<n>``.", + "", + "If a slice has too many tasks for one BRC cycle, or you discover a " + "natural sub-seam the architect missed, that is a **slicing problem " + "the architect must fix** — surface it as NACK pressure (your peer " + "reviewer ``risk_analyst`` and the structural reviewer " + "``reviewer_plan`` will NACK ``architect`` on ``slice_size`` when " + "evidence supports it; you can also flag the concern in your plan " + "prose so the reviewers pick it up). **Do NOT silently re-shape " + "slices.** Re-propose against the architect's revised scaffold " + "once it lands.", + "", + "**Test co-location (HARD — #3411).** When a slice removes, " + "renames, or rewrites code, enumerate the matching test " + "updates (skip-guard, deletion, rewrite) as tasks IN THAT " + "SLICE — never in a later slice — and list the test files in " + "those tasks' ``files:``. Every cumulative slice tip must be " + "independently green: the per-slice green gate (#3398) " + "blocks a slice PR while any repo check is red at its tip, " + "so a test that still imports a symbol removed two slices " + "earlier blocks the whole stack. Discover the affected " + "tests with the same import graph ``make test`` narrowing " + "uses, where the repo ships it (this repo: ``python3 " + "scripts/select_tests/__main__.py --impacted-tests " + "<file>...``; exit 2 = closure unavailable — fall back to " + "grepping the removed symbols in the test trees).", + "", + "Steps:", + f"1. Read the architecture analysis AND the slice scaffold at `{architect_slices_path}`", + "2. Copy the architect's slice scaffold verbatim into the " + "``# yaml-tasks`` appendix (same ``id`` / ``name`` / ``goal`` / " + "``dependencies`` values, in the same order)", + "3. Enumerate ``tasks:`` under each slice — discrete, " + "actionable, with clear acceptance criteria and dependency ordering " + "between tasks", + "4. Identify the test strategy — what automated tests cover the " + "changes, and what manual verification is needed", + "5. Identify any manual pre-merge or post-merge steps " + "(migrations, config changes, deployments)", + "", + "## Output Format", + "", + "Write a markdown plan document with a **yaml-tasks** structured", + "appendix at the end. The prose section should explain the approach;", + "the appendix is machine-parsed for contract population.", + "", + *_pkg._PR_DESCRIPTION_GUIDANCE, + "", + "End your document with a fenced YAML block like this:", + "", + "````", + "```yaml", + "# yaml-tasks", + "pr:", + ' title: "Short imperative summary (≤70 chars)"', + " description: |", + *_pkg._PR_DESCRIPTION_YAML_EXAMPLE, + " test_plan: |", + " - Automated: describe which tests cover the changes", + " - Manual: specific steps a reviewer should take to verify", + " manual_steps: |", + " Pre-merge: any required steps before merging", + " Post-merge: any required steps after merging", + "slices:", + " - id: 1", + " name: |-", + " Slice Name", + " goal: |-", + " What this slice achieves, written for a reviewer of the", + " target repo. This text is rendered verbatim as the lead", + " paragraph of the slice's PR body (#3115), so keep it 1-3", + " plain-language sentences with no plan-internal", + " cross-references (reviewer codes, section numbers, draft", + " version markers).", + " tasks:", + " - id: TASK-1-1", + " description: |-", + " What to do — safe to include `code: type` snippets,", + " URLs, and other punctuation inside a block scalar.", + " acceptance: |-", + " How to verify it is done", + " role: coder # optional: coder (default), tester, or documenter", + " files:", + " - path/to/file.py", + "```", + "````", + "", + *_pkg._YAML_TASKS_SAFETY_GUIDANCE, + "", + "Do NOT use a `pr_plan` key — slice packaging is owned by the " + "slice-DAG section below, not by an ad-hoc PR list.", + "", + "The `test_plan` field is **required** — describe both automated test " + "coverage and any manual verification steps. The `manual_steps` field " + "should list any pre-merge or post-merge actions required by the reviewer " + "or deployer; use an empty string if none.", + "", + # ---------------------------------------------------- + # #2594 — primitives audit (cheap plan-phase NACK) + # ---------------------------------------------------- + "## Primitives audit (#2594)", + "", + "Plan-phase NACKs are cheap; implement-phase NACKs on missing " + "primitives are expensive (8+ pod spawns per slice, ~60–90 min " + "per cycle). Make the audit cheap by **pre-citing every " + "primitive your tasks depend on**. For each named class, " + "function, HTTP route, env var, ConfigMap key, test fixture, " + "CLI flag, or decorator your plan references:", + "", + "1. **Cite existence** with `file:line` (use `grep -rn` to " + "verify *before* writing the task). If the primitive does not " + "exist yet because the task itself will create it, mark it " + "`(NEW — task TASK-X-Y)` so the plan reviewer doesn't NACK on " + "missing-primitive evidence. When you mark a primitive " + "`(NEW — task TASK-X-Y)`, you MUST also: (a) ensure the " + "referenced task's acceptance criteria actually produce that " + "primitive in the form the plan uses (right kind, right " + 'module, right scope — not just "adds the feature"), and ' + "(b) order downstream tasks that consume the primitive " + "**after** the creating task in the slice DAG. The plan " + "reviewer's §9 exception verifies both; mismatches NACK.", + "2. **Cite trust-boundary scope.** Some primitives exist but " + "are unavailable in the execution context the task assumes. " + "Canonical example: `ScriptedProvider` is unit-test-only; " + "deployed agent pods run the real provider. Likewise the " + "`integration_tests/` fixture tiering — the only " + "`gateway_url` pytest fixture lives at " + "`integration_tests/local_pipeline/conftest.py:261` and is " + "kubectl-gated via `local_pipeline_stack`. The parent " + "`integration_tests/conftest.py` does **not** expose " + "`gateway_url` as a fixture; it exposes `gateway_url` as an " + "attribute on the `EggStack` dataclass " + "(`integration_tests/conftest.py:78`), accessed as " + "`egg_stack.gateway_url`, not as a fixture-injectable " + "parameter. `orchestrator_url` and lifecycle-secret-gated " + "routes are also `local_pipeline/`-only. **No pytest fixture " + "in `integration_tests/` is `in-sandbox-agent`-runnable " + "today** — every fixture transitively depends on `egg_stack` " + "or `local_pipeline_stack`, both of which `pytest.skip` when " + "`_kubectl_available()` returns `False`. Tasks that need any " + "of `gateway_url` / `orchestrator_url` as a pytest fixture " + "MUST live under (or below) `local_pipeline/` or an " + "equivalent trusted directory. Verify with " + "`grep -rn 'def gateway_url' integration_tests/` — exactly " + "one hit. The agent-runtime `GATEWAY_URL` env is a " + "**separate surface** from pytest fixtures; production code " + "an agent writes can reach the gateway sidecar through it, " + "but that is not a pytest test. See " + "`docs/architecture/integration-test-trust-boundary.md`.", + "", + "Recommended shape: a short `## Primitives` section in the " + "prose with one row per primitive (name, `file:line`, " + "execution-context scope). The plan reviewer will run the " + "Primitive-Existence Audit (criteria §9) and Trust-Boundary " + "Audit (criteria §10) against this table; both are hard " + "NACKs when a named primitive has no grep hit or is used " + "outside its scope.", + "", + # ---------------------------------------------------- + # #2137 — slice-DAG planner guidance + # ---------------------------------------------------- + "## Slice-DAG guidance (#2137)", + "", + "The implement-phase pipeline now ships each plan **slice** " + "(formerly **phase**) as its own stacked PR. The plan you " + "emit drives that DAG; the planner rules below are mandatory.", + "", + "**Yaml key swap**: prefer the canonical ``slices:`` key in " + "your ``# yaml-tasks`` block (the parser also accepts " + "``phases:`` for backward compatibility with already-shipped " + "planner prompts). New plans should use ``slices:``.", + "", + "**Slice sizing is the architect's call (#2809).** Slice " + "count, boundaries, and DAG shape come from the architect's " + "scaffold — copy them verbatim. ``reviewer_plan`` will hard " + "NACK ``architect`` on ``slice_size`` when a slice is " + "oversized for one BRC cycle (judgment-based — see the " + "reviewer's §11 rubric); do NOT silently re-shape slices " + "to dodge a size concern. Raise it as NACK pressure on " + "architect instead (see the surfacing guidance above).", + "", + "**Forest constraint (HARD, enforced at plan ingestion)**: " + "every slice must have at most ONE DAG parent — the " + "implement-phase pipeline ships every slice as a stacked " + "PR with exactly one base branch. The architect's scaffold " + "encodes this via a single-parent ``dependencies`` id " + "(``slice-<N>``); preserve it.", + "", + "**Auto-serialization for would-be multi-parent slices**: " + "the architect is responsible for serialising would-be " + "multi-parent slices and populating " + "``serialized_chain_order`` on the downstream slice. " + "Preserve that field verbatim from the scaffold.", + "", + "**File-overlap ⇒ ordering (HARD — #3046)**: you fill in each " + "slice's tasks and their ``files_affected``, so you see the " + "file sets first. If you find yourself assigning the SAME file " + "to two slices that the architect left unordered (parallel " + "roots or siblings), do NOT silently proceed — plan ingestion " + "hard-rejects overlapping slices with no dependency edge, " + "because their branches fork independently off the shared base " + "and collide at integration. Raise NACK pressure on the " + "architect (via the plan prose) to serialise the overlapping " + "cluster into one ``dependencies`` chain — or to merge the " + "slices. Do not re-shape the slice DAG yourself.", + "", + "Worked example: if ``slice-3`` would naturally have " + "parents ``[slice-1, slice-2]``, instead emit:", + "", + "```yaml", + " - id: 1", + " name: |-", + " Foundations", + " # ... (root)", + " - id: 2", + " name: |-", + " Middle", + " dependencies:", + " - slice-1", + " - id: 3", + " name: |-", + " Downstream", + " dependencies:", + " - slice-2 # serialised — slice-2 is the only DAG parent", + " serialized_chain_order:", + " - slice-1", + " - slice-2 # records that you deliberately picked", + " # slice-1 → slice-2 → slice-3", + "```", + "", + "Your judgement is the source of truth. The fallback " + "heuristic when you have no preference is: cluster " + "would-be parents by ``files_affected`` Jaccard overlap " + "(>0.3), then order by descending downstream fan-out.", + "", + f"Write your plan to `{draft_path}`.", + "", + *_pkg._EXPLORATION_SUBAGENT_GUIDANCE, + ] + ) + # Append role file restriction info so the planner assigns tasks correctly. + # Pass the pipeline's repo so per-repo role_patterns from + # repositories.yaml are rendered (#2528) — keeps planner-prompt + # boundaries in sync with the gateway's push-time enforcement. + lines.append(_pkg._build_role_restrictions_section(repo=repo or None)) + elif role_value == "risk_analyst": + lines.extend( + [ + "**You are dual-role (producer AND reviewer) in this phase " + "(#2809).** You produce the risk register AND you review " + "``architect`` and ``task_planner`` through the risk lens — " + "your NACK blocks plan-phase consensus until the upstream " + "producer re-proposes addressing the concern. This mirrors " + "the implement-phase ``tester`` dual-role pattern (#2749); " + "the *Dual-Role Execution Order* banner in your BRC " + "preamble is the authoritative ordering — read it first.", + "", + "## Producer role (risk register)", + "", + "Assess technical risks for the proposed implementation:", + "", + "1. Review the architecture analysis from the ARCHITECT agent", + "2. Identify technical risks (security, performance, compatibility)", + "3. Research externally when the change involves third-party dependencies — " + "use WebSearch and WebFetch (when available) to check for known " + "vulnerabilities, deprecation notices, and compatibility issues. " + "Skip external research for purely internal changes.", + "4. Assess impact and likelihood of each risk", + "5. Propose mitigation strategies and rollback plans", + "6. Flag areas that need human review", + "7. **Flag runtime-primitive and trust-boundary risks (see " + "#2594).** Plans that depend on classes, fixtures, routes, " + "or env vars which don't exist in the form the plan assumes " + "— or which exist but only in a different execution context " + "than the task uses (e.g. unit-test-only `ScriptedProvider` " + "vs deployed agent pods; `orchestrator_url` fixture defined " + "only in `integration_tests/local_pipeline/conftest.py` vs " + "in-sandbox-agent tests) — are a recurring high-impact " + "failure mode (see #2474). Call these out explicitly so the " + "plan reviewer can audit them.", + "", + f"Write your risk assessment to `{_risk_analyst_output_path}`.", + "", + "## Reviewer role (risk lens on architect + task_planner)", + "", + "When ``architect`` or ``task_planner`` proposes (their " + "``CONSENSUS_PROPOSE`` will wake you via the dual-role " + "augmentation on your producer waits — see the banner), " + "review their work through the risk lens and emit ACK or " + "NACK. ``blocking_concerns`` are NACK-shaped: they block " + "plan-phase consensus and force the upstream producer to " + "re-propose addressing them.", + "", + "Use this verdict shape in your producer artifact " + "(risk-register JSON) **and** mirror the verdict / " + "feedback in your ``egg-orch consensus ack`` / " + "``egg-orch consensus nack`` ``--reason`` body so the " + "upstream producer can act on it:", + "", + "```json", + "{", + ' "verdict": "ACK" | "NACK",', + ' "risks": [...],', + ' "top_3_risks": [...],', + ' "blocking_concerns": [...],', + ' "feedback": "concrete revision instructions for architect / task_planner (empty on ACK)"', + "}", + "```", + "", + "NACK when a risk is severe enough that shipping the plan " + "as-proposed would invite a known-class failure (security " + "regression, data loss, compliance break, runtime-primitive " + "or trust-boundary mismatch that would surface as an " + "expensive implement-phase NACK). ACK when risks are real " + "but mitigated, or low enough that the plan can ship and " + "the risks belong in the register as forward-looking " + "notes. Be specific in ``feedback`` — name the file, " + "the slice, the missing mitigation — so the upstream " + "producer's re-propose is actionable.", + "", + *_pkg._EXPLORATION_SUBAGENT_GUIDANCE, + ] + ) + elif role_value == "simplifier": + if phase == "plan": + _upstream = "task_planner" + _upstream_draft = "the implementation plan" + _human_path = _plan_human_path + _essence = ( + "what will be built, the major steps/phases, the test strategy " + "in brief, and the key risks" + ) + else: # refine + _upstream = "refiner" + _upstream_draft = "the refine analysis" + _human_path = _analysis_human_path + _essence = "the problem, the recommended approach, and the key trade-offs" + lines.extend( + [ + "**You are a producer only in this phase.** You produce a " + f"human-focused companion to {_upstream_draft}. You do NOT " + f"review **{_upstream}**'s draft, and you issue no ACK or NACK " + "on it: an internal wake-wire re-invokes you when it proposes " + "so you know its draft is ready, and consensus never waits on a " + "verdict from you. The *Execution Order* banner in your BRC " + "preamble is the authoritative ordering — read it first.", + "", + "## Producer role (human-focused companion)", + "", + f"Your WORK depends on **{_upstream}**'s draft existing. ORIENT " + f"now, then start producing only once **{_upstream}** issues " + "`CONSENSUS_PROPOSE` (the event pump re-invokes you carrying that " + "proposal). On that invocation:", + "", + f"1. Read **{_upstream}**'s draft of {_upstream_draft}.", + f"2. Write a HUMAN-FOCUSED companion to `{_human_path}`. This is a " + "simplified, higher-level summary for a **broad audience — " + "engineers, PMs, and managers** — not a peer review. Capture the " + f"essence: {_essence}.", + "", + " Rules:", + " - **Broad, mixed audience.** Write so a non-engineer " + "(PM, manager) can follow *what is changing and why it matters*, " + "while staying accurate enough for an engineer. Explain any " + "unavoidable technical term in plain language.", + " - **No egg-internal jargon.** Do not mention BRC, consensus, " + "propose/ACK/NACK, slices / slice-DAG, contracts, phases, " + "`serialized_chain_order`, Jaccard, or agent-role names. Describe " + "independently-shippable pieces in plain terms if you must " + "reference them at all.", + " - **No implementation minutiae.** No `file:line` references, " + "no function / struct / field / type names or other code " + "identifiers, no per-field enumerations. Describe behaviour and " + "impact, not the code.", + " - **This is NOT a review.** Do not critique, score, or gate " + 'the upstream draft. No ACK/NACK language, no "the draft should ' + 'commit to …", no "anti-pattern to reject", no constraint ' + "lists. You have no critique to record anywhere — your only " + "output is this plain-language summary.", + f" - **Exactly one file.** Commit ONLY `{_human_path}`. Do " + "NOT create any other `.egg-state/drafts/` file — no separate " + "`*-simplifier-*.md` constraints/guardrails/verification " + "companion. Any review reasoning goes in the BRC channel " + "(your verdict), never a second persisted document. A " + "proposal that introduces a second draft is rejected at " + "propose time.", + " - **Much shorter and more digestible** than the upstream " + "draft — plain prose and short lists, not exhaustive enumeration.", + " - **Faithful** — reflect the upstream draft accurately; " + "introduce no new scope, claims, or recommendations.", + "", + f"3. Commit and push `{_human_path}`, then PROPOSE it via " + "`egg-orch consensus propose`. The companion is **mandatory** — " + "always write at least a one-paragraph summary; do NOT take the " + "no-op propose path. That completes your work for this phase.", + "", + *_pkg._EXPLORATION_SUBAGENT_GUIDANCE, + ] + ) + else: + lines.extend( + [ + f"Execute your role as {role_value} for this phase.", + "", + ] + ) + + # Phase restrictions + _recovery_base_ref = _pkg._resolve_origin_ref(base_branch) + lines.append("## Phase Restrictions\n") + if phase == "implement": + lines.extend( + [ + "- You CAN push code changes to git (git push)", + "- You CAN link commits to tasks (egg-contract add-commit)", + "- You CANNOT push .egg-state/ files (except checkpoints)", + "- You CANNOT create PRs (the pipeline manages the PR)", + "", + "### Push Recovery", + "", + "If your push is rejected due to restricted files on the branch, " + f"create a clean branch from {_recovery_base_ref} and cherry-pick " + "only your code commits:", + "```", + f"git checkout -b egg/<new-branch> {_recovery_base_ref}", + "git cherry-pick <your-commit-hash>", + "git push origin egg/<new-branch>", + "```", + "Do NOT retry the same push — fix the branch first.", + "After pushing to the new branch, use `egg-contract add-commit` to " + "link your commits so the pipeline can track them on the new branch.", + "", + ] + ) + elif phase in ("refine", "plan"): + lines.extend( + [ + "- You CAN write to `.egg-state/drafts/` and `.egg-state/agent-outputs/`", + "- You CAN push these state files to git (git push)", + "- You CAN create HITL decisions (egg-contract add-decision)", + "- You CAN create feedback requests (egg-contract add-feedback)", + "- You CANNOT modify production code (src/, lib/, gateway/, sandbox/, " + "action/, docs/, tests/, test/)", + "- You CANNOT modify contracts (.egg-state/contracts/) or CI config (.github/)", + "- You CANNOT create PRs (gh pr create)", + "", + "### Push Recovery", + "", + "If your push is rejected due to restricted files on the branch, " + f"create a clean branch from {_recovery_base_ref} and cherry-pick " + "only your state file commits:", + "```", + f"git checkout -b egg/<new-branch> {_recovery_base_ref}", + "git cherry-pick <your-commit-hash>", + "git push origin egg/<new-branch>", + "```", + "Do NOT retry the same push — fix the branch first.", + "After pushing to the new branch, use `egg-contract add-commit` to " + "link your commits so the pipeline can track them on the new branch.", + "", + ] + ) + + # File boundaries (#1431) — surface allowed/blocked patterns so + # the agent avoids creating files the gateway will reject on push. + # Pass repo so the rendered patterns match per-repo overrides + # (#2528) the gateway will enforce on push. + boundary_section = _pkg._build_file_boundary_section(role_value, repo=repo) + if boundary_section: + lines.append(boundary_section) + + # Producer escape hatch (#2529) — tester/documenter are the other + # two impassing producer roles (coder is handled in the early-return + # branch above). They need the actionable + # check_file_restriction / report_impasse guidance so they don't + # invent workarounds when their assigned task is structurally + # impossible. + if role_value in ("tester", "documenter"): + lines.append(_pkg._build_impasse_escape_hatch_section()) + + lines.append("## Phase Completion\n") + if concurrent: + lines.extend( + [ + "When you have completed your primary work:\n", + "1. Commit all changes", + '2. Run: `egg-orch signal readiness --state READY --reason "Work complete"`', + "3. Enter an **event-driven** stay-alive wait (issue #1897). " + "Do NOT wrap `egg-orch` in a shell `for i in 1..N` loop, " + "and do NOT `sleep N` — use the server-side blocking primitive:", + "```bash", + "egg-orch message wait-loop \\", + " --for CONSENSUS_CONFIRMED \\", + " --for CONSENSUS_RE_REVIEW \\", + " --for OVERSEER_ALERT \\", + " --timeout 60", + "```", + "`wait-loop` blocks server-side and loops forever until a " + "NEW matching BRC event arrives (exit 0) or a permanent error " + "occurs (exit 1). There is no outer timeout — the wrapper " + "owns the 0/1 contract. Events that predate the call " + "(including your own just-sent CONSENSUS_CONFIRMED) are " + "skipped (issue #1925); if you need zero-drop semantics " + "across a send→wait boundary, capture the ID of your " + "send and pass `--since <id>`. See " + "`docs/reference/agent-wait-patterns.md` for the full " + "exit-code contract and the five anti-patterns to avoid.", + "4. If `wait-loop` returns with a message that affects your work, " + "transition back to WORKING, address it, then signal READY again. " + "**In particular, if you receive a `CONSENSUS_RE_REVIEW` message, " + "you MUST re-confirm via `egg-orch consensus confirmed` (or " + "re-review and ACK/NACK if you are a reviewer of the re-proposing " + "producer). Ignoring this message will stall the pipeline.**", + "5. **Do NOT exit.** The orchestrator will stop your container when consensus " + "is reached.", + ] + ) + else: + lines.append( + "When you have completed your work, ensure everything is committed and exit successfully." + ) + + return "\n".join(lines) diff --git a/orchestrator/routes/pipelines/_prompt_phase.py b/orchestrator/routes/pipelines/_prompt_phase.py new file mode 100644 index 0000000000..03e085fbd4 --- /dev/null +++ b/orchestrator/routes/pipelines/_prompt_phase.py @@ -0,0 +1,1407 @@ +"""phase-prompt + BRC preamble helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _build_phase_prompt( + phase: str, + pipeline_id: str, + pipeline_mode: str, + prompt: str | None = None, + issue_number: int | None = None, + repo: str | None = None, + branch: str | None = None, + review_feedback: str | None = None, + review_cycle: int = 0, + repo_path: str | None = None, + operator_directives: list[_pkg.OperatorDirective] | None = None, + iteration_history: list[_pkg.IterationSummary] | None = None, +) -> str: + """Build a phase-specific prompt for the sandbox Claude invocation. + + Follows a structured prompt format: + Context → Task → Restrictions → Completion. + """ + # --- Context header --- + lines = [f"You are in the **{phase}** phase of the SDLC pipeline.\n"] + lines.append("## Context\n") + lines.append(f"Pipeline ID: {pipeline_id}") + lines.append(f"Phase: {phase}") + if repo: + lines.append(f"Repository: {repo}") + if branch: + lines.append(f"Branch: {branch}") + if issue_number is not None: + lines.append(f"Issue: #{issue_number}") + lines.append("") + + # --- Phase iteration context (HITL kickbacks) --- + # Operator directives have their own section with explicit precedence + # prose so reviewers cannot faithfully NACK a directive-driven change + # against a stale default rubric. See issue #2795. + iteration_context = _pkg._build_phase_iteration_context(operator_directives, iteration_history) + if iteration_context: + lines.append(iteration_context) + + # --- Prior review feedback (agentic revision cycles only) --- + # Scoped to agentic-cycle review feedback since #2795 — HITL kickback + # feedback now flows through ``operator_directives`` / the iteration + # context section above. + if review_feedback: + if review_cycle > 0: + lines.append(f"## Prior Review Feedback (Cycle {review_cycle})\n") + else: + lines.append("## Prior Review Feedback\n") + has_tester_findings = _pkg.TESTER_FINDINGS_HEADER in review_feedback + if phase == "implement": + revision_action = "Address the feedback below and revise your implementation." + else: + revision_action = ( + "Address the feedback below and revise your draft **in-place** " + "(overwrite the same file)." + ) + if review_cycle == 0: + consensus_override = ( + " Even if an existing draft appears " + "to have reached consensus previously, that consensus is " + "superseded — you must revise to address this feedback before " + "proposing a new consensus." + ) + else: + consensus_override = "" + if has_tester_findings: + lines.append( + "The reviewer and tester found issues with your previous work. " + f"{revision_action}{consensus_override}\n" + ) + else: + preamble_noun = "implementation" if phase == "implement" else "draft" + lines.append( + f"The reviewer found issues with your previous {preamble_noun}. " + f"{revision_action}{consensus_override}\n" + ) + lines.append(review_feedback) + lines.append("") + + # --- Task description --- + # Skip re-embedding the full task description on revision cycles for + # implement phase — the coder already knows the task from cycle 0. + if prompt and not (phase == "implement" and review_cycle > 0): + lines.append("## Task Description\n") + lines.append(prompt) + lines.append("") + + # --- Phase-specific instructions --- + lines.append("## Your Task\n") + + # Get the correct draft path based on mode + analysis_path = _pkg._get_draft_path( + "refine", issue_number=issue_number, pipeline_id=pipeline_id + ) + plan_path = _pkg._get_draft_path("plan", issue_number=issue_number, pipeline_id=pipeline_id) + + if phase == "refine": + lines.extend( + [ + "Analyze this issue and produce a structured analysis document. Your goal is to:\n", + "1. Understand the problem or feature request", + "2. Research the current codebase to understand existing patterns", + "3. Research externally when the task involves third-party libraries, APIs, " + "or integrations — use WebSearch and WebFetch (when available) to look up " + "current documentation, best practices, and known issues. Skip external " + "research for purely internal changes where codebase context is sufficient.", + "4. Identify constraints and dependencies", + "5. Consider multiple implementation approaches", + "6. Recommend an approach with justification", + "7. Surface the questions and uncertainties that genuinely need a " + "human to answer — see `## How to Populate Open Questions` below for " + "the filter (slice/PR packaging, implementation strategy, and " + "API/schema details belong to the planner, not the refiner)", + "", + "**IMPORTANT**: Do NOT create an implementation plan, task breakdown, " + "or phased rollout. That is the **plan** phase's job. Stay focused on " + "**analysis**: understanding the problem, researching the codebase, " + "evaluating options, and surfacing decisions for the human.", + "", + "## Output Format\n", + "Create an analysis document following the template below. The " + "fenced block is the **template literal** — copy it as-is and fill " + "in the bracketed placeholders. The unfenced sections that follow " + "(`## How to Populate Open Questions`, `## Complexity Assessment`) " + "are **meta-guidance** — do **not** transcribe them into your " + "analysis document.\n", + "````markdown", + "# Analysis: [Issue Title]\n", + "> Issue: #[number] | Phase: refine\n", + "## Problem Statement\n", + "[Describe the problem or feature request. " + "What is the current state? What is the desired outcome?]\n", + "## Current Behavior\n", + "[Describe how the system currently works in the relevant area. " + "Include code references where helpful.]\n", + "## Constraints\n", + "- [Technical constraints (compatibility, performance, security)]", + "- [Business constraints (timeline, scope)]", + "- [Dependencies on other systems or features]\n", + "## Options Considered\n", + "### Option A: [Name]\n", + "**Approach**: [Brief description]\n", + "**Pros**:", + "- [Advantage 1]\n", + "**Cons**:", + "- [Disadvantage 1]\n", + "### Option B: [Name]\n", + "**Approach**: [Brief description]\n", + "**Pros**:", + "- [Advantage 1]\n", + "**Cons**:", + "- [Disadvantage 1]\n", + "## Recommended Approach\n", + "[Which option is recommended and why. Reference the option above.]\n", + "## Open Questions\n", + "[Register every open question by following the protocol in " + "`## How to Populate Open Questions` below the template, then paste " + "the markdown output of each registration command into this section. " + "Do **not** copy the protocol instructions themselves into this " + "document.]\n", + "---\n", + "*Authored-by: egg*", + "````\n", + "", + "## How to Populate Open Questions\n", + "These instructions tell you how to handle the `## Open Questions` " + "section of the template above. They are **meta-guidance**, not " + "template content — do **not** transcribe this section into the " + "analysis document you write.\n", + "**Every open question MUST be registered as a contract decision or " + "feedback item using `egg-contract`.** Do not just write questions " + "as prose — they will not be seen by the human unless registered.\n", + "**Skip already-resolved questions.** If the Task Description above " + "includes an `## Additional Context` section, treat anything addressed " + "there as already decided by the operator (those came from a pre-refine " + "HITL round). Do NOT call `egg-contract add-decision` or " + "`egg-contract add-feedback` for questions whose answers are already " + "captured in `## Additional Context` — re-registering them wastes turns " + "and produces no-op decisions. Read that section first; if it settles " + "anything, list those items in a `### Resolved in Pre-Refine` " + "subsection at the top of `## Open Questions` (one bullet per resolved " + "item, citing the answer). Only register questions that go beyond what " + "`## Additional Context` covers. This skip rule is NARROW: it covers " + "only answers THIS pipeline's operator recorded in " + "`## Additional Context`. It never covers decisions the task " + "description names as operator-owned, and never answers inherited " + "from a prior or cancelled run's seeded context — register those " + "(see the next rule).\n", + "**Task-named decisions are non-optional (#3462).** If the task " + "description or contract names specific decisions as the operator's " + "to make — or contains any directive to surface decisions as HITL " + "questions — you MUST register each one via " + "`egg-contract add-decision`, even when you believe prior context " + "already resolves it, or that it is non-blocking or deferred. " + "Belief about resolution is a *recommended disposition*, not a " + "reason to skip registration: make your recommended answer the " + "first option (suffix its label with `(recommended)`) and cite the " + "resolving context in that option's description, so the operator " + "can confirm in one click while retaining the authority to choose " + "differently. Documenting a decision in draft prose is a " + "supplement to registration, never a substitute — unregistered " + "decisions never reach the operator's decision surface.\n", + "Surface uncertainties, ambiguities, and assumptions **that genuinely " + "need a human to answer**. Filter ruthlessly: a good open question is " + "one the operator must answer because the answer changes what we're " + "building. A bad open question is one the planner phase will decide on " + "its own once it sees the analysis — those waste the operator's " + "attention and pre-anchor the planner. Err toward registering questions " + "about *what the problem actually is* and *what's in or out of scope* " + "rather than *how to build it*.\n", + "**Out of scope for refine open questions** — do NOT register decisions " + "about:\n" + "- **Work decomposition / slice-DAG shape / PR packaging** — " + "**Slice / PR packaging is NOT a refine-phase decision.** The " + "plan phase owns slice-DAG construction (see " + "`docs/architecture/slice-dag.md`) and the operator approves the " + "proposed slice shape at the plan HITL gate. Do not register " + "`add-decision` items asking how the work should be sliced, how " + "many PRs to ship, or which parts should run in parallel. If " + "the task obviously spans multiple parts, name them in Problem " + "Statement or Constraints — the planner will propose a shape " + "from the analysis it reads.\n" + "- **Implementation strategy choices** that the planner can decide " + 'from Problem Statement + Constraints (e.g. "which migration ' + 'approach", "which fallback design", "which detector shape"). ' + "Surface these as Options Considered / Recommended Approach in the " + "analysis prose, not as `add-decision` items.\n" + "- **API / schema details** the planner phase will work out once it " + "starts designing. If the operator must constrain the API shape, " + "frame it as a *constraint* in `## Constraints`, not an open question.\n" + "Register questions when the answer is a fact only the human knows " + "(product intent, scope boundaries, external commitments, " + "user-visible behavior) — not when the answer is a design call the " + "planner will make.\n", + "**Multiple-choice questions** — RUN this command for each question " + "where the human must pick from discrete options:", + "```bash", + 'egg-contract add-decision --question "Which approach should we use?" \\', + ' --options "Option A" "Option B" "Option C" --format markdown', + "```", + "Copy the markdown output into your analysis. The human can check " + 'a checkbox to select an option. An "Other (explain in reply)" ' + "option is auto-appended.\n", + "**Open-ended questions** — EXECUTE this command for free-form " + "questions where you need the human to provide text answers:", + "```bash", + "egg-contract add-feedback \\", + ' --question "What is the expected request volume?" \\', + ' --question "Are there any constraints on third-party dependencies?" \\', + " --format markdown", + "```", + "This creates a dedicated comment for the human to fill in answers. " + 'They edit the comment to add their responses and check "Submit ' + 'feedback" when done. The pipeline will resume with the feedback ' + "available in the contract.\n", + "**Advisory seam-listing is fine** — if the task obviously spans " + "independently-implementable parts, you MAY name them in Problem " + 'Statement or Constraints (e.g. "the change touches the gateway, ' + 'the orchestrator, and the sandbox") so the planner has the seam ' + "information. Make it **explicitly advisory**: the planner is free " + "to slice differently if it sees a better seam. Do not pre-number " + "parts as `slice-1 / slice-2`, do not draw a DAG, and do not pick " + "a 1-PR-vs-3-PR shape — those choices belong to the planner.\n", + "**DO NOT:**", + "- Write questions as plain markdown text without running " + "`egg-contract add-decision` or `egg-contract add-feedback`", + "- Use custom HTML comment markers like " + "`<!-- DECISION: ... -->` instead of the contract CLI", + "- Skip registration because you think the questions are minor — " + "register every question", + "- Skip registration because you believe a decision is already " + "resolved, non-blocking, or deferred — register it with your " + "recommended disposition instead (#3462)", + "- Attest `no_decisions_rationale` when the task names decisions " + "to surface — the attestation is presented to the operator as its " + "own confirmable decision, and a rejected 'none' sends the phase " + "back for a re-run (#3462)", + "- Transcribe this `## How to Populate Open Questions` section " + "into your analysis document — it is meta-guidance, not template " + "content\n", + "**Attest your decision ledger when proposing (#3390).** Your " + "consensus propose is REJECTED unless its attestation carries " + "the ledger: `--decisions-registered cq-1 cq-2 ...` (every id " + "you registered this phase) or " + '`--no-decisions-rationale "<why>"` when you deliberately ' + "registered none. Attested ids must exist on the contract for " + "this phase, and the draft must cite each one — the " + "`--format markdown` output you copied above embeds the id, so " + "the registration flow satisfies the citation automatically. " + "If your only open questions went into an `add-feedback` " + "request (no `cq-N` decisions), attest the rationale form and " + "name the feedback request in it. This is what lets the " + "operator trust that an empty gate means *deliberately no " + "decisions*, not *forgot to register*. The explicit-none form " + "is not a shortcut (#3462): the orchestrator surfaces it to " + "the operator as its own confirmable decision before the " + "phase gate, and it is only valid when the phase genuinely " + "raises no meaningful decision — never when the task names " + "decisions to surface, and never as a substitute for " + "registering a decision you believe is already resolved.\n", + "", + ] + ) + lines.extend( + [ + "## Complexity Assessment\n", + "After completing your analysis, assess the task complexity:", + "- **low**: Single-file change, straightforward bug fix, small config update, typo fix", + "- **medium**: Multi-file change with clear scope, feature addition with known patterns", + "- **high**: Architectural change, new subsystem, cross-cutting concern, " + "many independent phases that could be parallelized", + "", + ] + ) + lines.extend(_pkg._EXPLORATION_SUBAGENT_GUIDANCE) + lines.extend( + [ + f"Write your analysis to `{analysis_path}`.", + "Commit and push the draft when done.\n", + "**IMPORTANT**: Do NOT post your analysis directly to the issue. " + "The pipeline will have an internal reviewer check your analysis. " + "If revisions are needed, you'll be re-invoked with feedback. " + "Only after internal review passes will the analysis be posted " + "for human approval.", + "", + ] + ) + + elif phase == "plan": + lines.extend( + [ + "Create a detailed implementation plan, decomposing the work into " + "slices per the slice-DAG guidance at the end of this section. The " + "implement-phase pipeline ships each slice as its own stacked PR. " + "**Slice shape is your call.** A single-slice plan is fine when the " + "work is cohesive; pick a multi-slice shape when the work has clean " + "seams that ship independently. If the refine analysis sketched a " + "decomposition (e.g. naming the components touched), treat it as " + "**advisory context** — you are free to slice differently if a " + "better seam exists. The only thing that binds your slice shape is " + "an explicit slice-DAG HITL decision recorded by the operator on " + "the contract; if you believe such a decision is wrong, raise it as " + "an open question in your plan rather than silently overriding.", + "", + "Steps:", + "1. Review any prior analysis", + "2. Break down the work into phases with discrete tasks", + "3. Define clear acceptance criteria for each task", + "4. Identify test strategy — what automated tests cover the changes, " + "and what manual verification is needed", + "5. Identify any manual pre-merge or post-merge steps " + "(migrations, config changes, deployments)", + "6. Consider rollback and risks", + "", + "## Output Format", + "", + "Write a markdown plan with a **yaml-tasks** structured appendix at the end.", + "The prose section explains the approach; the appendix is machine-parsed.", + "", + *_pkg._PR_DESCRIPTION_GUIDANCE, + "", + "End your document with a fenced YAML block like this:", + "", + "````", + "```yaml", + "# yaml-tasks", + "pr:", + ' title: "Short imperative summary (≤70 chars)"', + " description: |", + *_pkg._PR_DESCRIPTION_YAML_EXAMPLE, + " test_plan: |", + " - Automated: describe which tests cover the changes", + " - Manual: specific steps a reviewer should take to verify", + " manual_steps: |", + " Pre-merge: any required steps before merging", + " Post-merge: any required steps after merging", + "slices:", + " - id: 1", + " name: |-", + " Slice Name", + " goal: |-", + " What this slice achieves, written for a reviewer of the", + " target repo. This text is rendered verbatim as the lead", + " paragraph of the slice's PR body (#3115), so keep it 1-3", + " plain-language sentences with no plan-internal", + " cross-references (reviewer codes, section numbers, draft", + " version markers).", + " tasks:", + " - id: TASK-1-1", + " description: |-", + " What to do — safe to include `code: type` snippets,", + " URLs, and other punctuation inside a block scalar.", + " acceptance: |-", + " How to verify it is done", + " files:", + " - path/to/file.py", + "```", + "````", + "", + *_pkg._YAML_TASKS_SAFETY_GUIDANCE, + "", + "Do NOT use a `pr_plan` key — slice packaging is owned by the " + "slice-DAG section below, not by an ad-hoc PR list.", + "", + "The `test_plan` field is **required** — describe both automated test " + "coverage and any manual verification steps. The `manual_steps` field " + "should list any pre-merge or post-merge actions required by the reviewer " + "or deployer; use an empty string if none.", + "", + # ---------------------------------------------------- + # #2137 — slice-DAG planner guidance (mirrors the + # concurrent task_planner block; keep the two paths + # aligned so the slice-shape rules behave the same way + # regardless of which planner runs). + # ---------------------------------------------------- + "## Slice-DAG guidance (#2137)", + "", + "The implement-phase pipeline ships each plan **slice** (formerly " + "**phase**) as its own stacked PR. The plan you emit drives that " + "DAG; the rules below are mandatory.", + "", + "**Yaml key swap**: prefer the canonical ``slices:`` key in your " + "``# yaml-tasks`` block (the parser also accepts ``phases:`` for " + "backward compatibility). New plans should use ``slices:``.", + "", + "**Slice-sizing NACK (hard, judgment-based — #2809)**: the plan " + "reviewer will hard-NACK an oversized slice. Use judgment when " + "shaping — no fixed LOC budget, but avoid bundling more than ~3 " + "distinct file-categories in one slice, avoid combining " + "deletion-heavy work with new-API-introduction work, avoid " + "slices that would require >3–4 commit-propose-revise cycles, " + "and avoid bundling independent task groups with no internal " + "dependency. Subdivide along those seams up front rather than " + "earning a NACK.", + "", + "**Forest constraint (HARD)**: every slice must have at most ONE " + "DAG parent — the implement-phase pipeline ships every slice as a " + "stacked PR with exactly one base branch. Multi-parent slices " + "break the stacking invariant and are rejected at plan ingestion.", + "", + "**Auto-serialization rule for would-be multi-parent slices**: " + "when a slice would naturally have >1 parents, serialise the " + "upstream slices into a linear chain and record the chosen " + "ordering on the downstream slice's ``serialized_chain_order`` " + "field. The list names the upstream slice IDs in their chosen " + "serialization order.", + "", + "**File-overlap rule (HARD, enforced at plan ingestion — " + "#3046)**: two slices that touch the SAME file must be ordered " + "on one dependency chain — one a transitive ``dependencies`` " + "ancestor of the other — never left as parallel roots or " + "siblings. The implement phase cuts each slice's branch off " + "its dependency parent, so an unordered overlapping pair forks " + "independently off the shared base and its edits to the shared " + "file collide at integration (a guaranteed modify/delete " + "conflict). Deletion/retirement slices are the classic trap: a " + "slice that removes a file must depend on every slice that " + "modifies it. Slices with disjoint file sets stay parallel.", + "", + "**Test co-location rule (HARD — #3411)**: a slice that " + "removes, renames, or rewrites code carries the matching " + "test updates (skip-guards, deletions, rewrites) in the SAME " + "slice — never a later one — with the test files listed in " + "that slice's task ``files:``. Each cumulative slice tip " + "must be independently green: the per-slice green gate " + "(#3398) runs the repo's checks at the slice tip before the " + "PR opens and blocks while any check is red, so deferring " + "test obsolescence to a later slice guarantees a blocked " + "slice. Discover the tests that statically reach the " + "changed files with the changeset-aware selector where the " + "repo ships it (this repo: ``python3 " + "scripts/select_tests/__main__.py --impacted-tests " + "<file>...``; exit 2 = closure unavailable — fall back to " + "grepping the removed symbols in the test trees).", + "", + "Worked example: if ``slice-3`` would naturally have " + "parents ``[slice-1, slice-2]``, instead emit:", + "", + "```yaml", + " - id: 1", + " name: |-", + " Foundations", + " # ... (root)", + " - id: 2", + " name: |-", + " Middle", + " dependencies:", + " - slice-1", + " - id: 3", + " name: |-", + " Downstream", + " dependencies:", + " - slice-2 # serialised — slice-2 is the only DAG parent", + " serialized_chain_order:", + " - slice-1", + " - slice-2 # records that you deliberately picked", + " # slice-1 → slice-2 → slice-3", + "```", + "", + "Your judgement is the source of truth. The fallback heuristic " + "when you have no preference is: cluster would-be parents by " + "``files_affected`` Jaccard overlap (>0.3), then order by " + "descending downstream fan-out.", + "", + f"Write your plan to `{plan_path}`.", + "Commit and push the draft when done.", + "", + ] + ) + + elif phase == "implement": + # Embed plan or analysis text directly on first cycle + # (avoids file-I/O turns inside the sandbox). + draft_embedded = False + if repo_path and review_cycle == 0: + draft_text = _pkg._read_phase_draft( + _pkg.Path(repo_path), + "plan", + issue_number=issue_number, + pipeline_id=pipeline_id, + branch=branch, + ) + if draft_text: + lines.append("## Plan\n") + lines.append(f"```markdown\n{draft_text}\n```\n") + draft_embedded = True + + # Embed contract task checklist on first cycle + contract_tasks = _pkg._render_contract_tasks( + repo_path, pipeline_id, pipeline_mode, issue_number + ) + if contract_tasks: + lines.append(contract_tasks) + lines.append("") + + if review_cycle == 0: + # Build numbered step list; only include the "review" step + # when the draft wasn't already embedded above. + lines.append("Implement the changes described in the task and plan:") + lines.append("") + + steps: list[str] = [] + if not draft_embedded: + steps.append("Review the plan (check `.egg-state/drafts/`)") + steps.extend( + [ + "Implement the required changes — when working with third-party " + "libraries or APIs, use WebSearch and WebFetch (when available) to " + "look up current documentation, usage examples, and best practices", + "After completing each plan phase or task group, commit and push " + "immediately — do not batch all work into a final commit. Mark " + "tasks done: `egg-contract complete-task --task <id> --commit <sha>`", + "Run tests to verify correctness, then commit any fixes", + ] + ) + for i, step in enumerate(steps, 1): + lines.append(f"{i}. {step}") + lines.append("") + + lines.append("## Parallel Execution with Subagents\n") + lines.append( + "You have access to Claude Code's **Agent tool** for spawning subagents. " + "Use it to parallelize independent work:\n" + ) + lines.append( + "- If the plan has multiple independent phases or task groups that don't touch " + "overlapping files, implement them in parallel by launching one subagent per " + "phase/group." + ) + lines.append( + "- Each subagent gets a clear, self-contained prompt describing its scope " + "(files to modify, tasks to complete, acceptance criteria)." + ) + lines.append( + "- Subagents share your working directory and git state. Ensure parallel " + "subagents work on **non-overlapping files** to avoid conflicts." + ) + lines.append( + "- Subagents should only edit files — do NOT stage or commit from subagents. " + "After each group of parallel subagents completes, **immediately** commit and " + "push their combined changes before launching the next group." + ) + lines.append( + "- After subagents complete, verify the combined changes compile, pass tests, " + "and integrate correctly. Do NOT defer all commits to the end." + ) + lines.append( + "- For small or sequential tasks, just implement directly — don't over-parallelize." + ) + lines.append("") + lines.extend(_pkg._EXPLORATION_SUBAGENT_GUIDANCE) + else: + # Revision cycle: slim delta-focused prompt. + # Guard: if review_feedback is unexpectedly missing, fall + # back to including the task description so the coder isn't + # left with a nearly empty prompt. + if not review_feedback: + if prompt: + lines.append("## Task Description\n") + lines.append(prompt) + lines.append("") + + lines.append("## Revision Instructions\n") + if review_feedback: + has_tester_findings = _pkg.TESTER_FINDINGS_HEADER in review_feedback + if has_tester_findings: + lines.extend( + [ + "The reviewer and tester found issues with your implementation. " + "Focus on addressing the specific feedback above.\n", + "1. Review the feedback in the **Prior Review Feedback** section above", + "2. Check `git diff` to understand the current state of changes", + f"3. Check `.egg-state/agent-outputs/" + f"{_pkg._pipeline_identifier(issue_number, pipeline_id)}" + f"-tester-output.json` for test failures and gaps", + "4. Fix the specific issues raised", + "5. Run tests to verify your fixes", + "6. Commit with descriptive messages", + "", + ] + ) + else: + lines.extend( + [ + "The reviewer found issues with your implementation. " + "Focus on addressing the specific feedback above.\n", + "1. Review the feedback in the **Prior Review Feedback** section above", + "2. Check `git diff` to understand the current state of changes", + "3. Fix the specific issues raised by the reviewer", + "4. Run tests to verify your fixes", + "5. Commit with descriptive messages", + "", + ] + ) + else: + lines.extend( + [ + "A revision was requested but no specific feedback was provided. " + "Review the task description above and check `git diff` for the current state.\n", + "1. Review the task description above and check `git diff`", + "2. Verify the implementation meets the requirements", + "3. Run tests to verify correctness", + "4. Commit with descriptive messages", + "", + ] + ) + + # Contract CLI instructions for both local and issue mode + lines.extend( + [ + "Use the contract CLI to track progress incrementally — update after " + "each commit, not in a batch at the end:", + "- `egg-contract show` — View current contract state", + "- `egg-contract complete-task --task <id> --commit <sha>` — Mark task done and link commit", + "- `egg-contract complete-phase --phase <id> --commit <sha>` — Mark phase done and link commit", + "- `egg-contract add-commit --task <id> --commit <sha>` — Link commit to task without marking done", + "", + ] + ) + + else: + lines.append(f"Execute the {phase} phase.\n") + + # --- Phase restrictions --- + lines.append("## Phase Restrictions\n") + if issue_number is None and phase in ("refine", "plan"): + lines.extend( + [ + "In this phase:", + "- You CAN push state files to git (contracts, drafts, checkpoints)", + "- You CAN create HITL decisions (egg-contract add-decision)", + "- You CAN create feedback requests (egg-contract add-feedback)", + "- You CANNOT push code changes", + "- You CANNOT create PRs (gh pr create)", + "- You CANNOT post comments to the GitHub issue (gh issue comment) — write reviews to `.egg-state/reviews/` instead", + "- You CANNOT edit the GitHub issue (gh issue edit)", + "- You CAN read and modify local files", + "- You CAN run tests", + "- You CAN commit locally", + "", + ] + ) + elif issue_number is None and phase == "implement": + lines.extend( + [ + "In this phase:", + "- You CAN push code changes to git", + "- You CANNOT push .egg-state/ files (except checkpoints)", + "- You CANNOT create PRs (gh pr create)", + "- You CANNOT post comments to the GitHub issue (gh issue comment)", + "- You CANNOT edit the GitHub issue (gh issue edit)", + "- You CAN read and modify local files", + "- You CAN run tests", + "- You CAN commit locally", + "", + ] + ) + else: + if phase in ("refine", "plan"): + lines.extend( + [ + "- You CAN write drafts to `.egg-state/drafts/`", + "- You CAN push draft files (git push)", + "- You CAN create HITL decisions (egg-contract add-decision)", + "- You CAN create feedback requests (egg-contract add-feedback)", + "- You CANNOT post comments to the GitHub issue (gh issue comment) — write reviews to `.egg-state/reviews/` instead", + "- You CANNOT edit the GitHub issue (gh issue edit)", + "- You CANNOT create PRs (gh pr create)", + "", + ] + ) + elif phase == "implement": + lines.extend( + [ + "- You CAN push code (git push)", + "- You CAN link commits to tasks (egg-contract add-commit)", + "- You CANNOT create PRs (the pipeline manages the PR)", + "- You CANNOT post comments to the GitHub issue (gh issue comment)", + "- You CANNOT edit the GitHub issue (gh issue edit)", + "", + ] + ) + # --- Completion --- + lines.append("## Phase Completion\n") + if phase in ("refine", "plan"): + lines.append( + "When your draft is complete, commit and push it. " + "The pipeline will have an internal reviewer evaluate your work. " + "If revisions are needed, you'll be re-invoked with feedback. " + "Only after internal review passes will the output be posted " + "for human approval." + ) + else: + lines.append( + "When you have completed your work for this phase, " + "ensure everything is committed and exit successfully." + ) + + return "\n".join(lines) + + +def _contract_enforcer_role_names() -> frozenset[str]: + """Roles whose ACK/CONFIRM is gated on contract-task completeness (#3114). + + Lazy wrapper so the preamble builder keys its enforcer-specific + instructions off the same capability set the orchestrator's signal + gate enforces (``egg_contracts.agent_roles.CONTRACT_ENFORCER_ROLES``) + — prose and enforcement stay in lockstep. + """ + from egg_contracts.agent_roles import CONTRACT_ENFORCER_ROLE_NAMES + + return CONTRACT_ENFORCER_ROLE_NAMES + + +def _build_brc_preamble( + role_value: str, + phase: str, + repo: str | None = None, + branch: str | None = None, + base_branch: str | None = None, +) -> str: + """Build the BRC consensus lifecycle preamble for an agent. + + Returns a formatted string block that can be appended to any agent prompt + to inject BRC protocol instructions. Used by both the coder/refiner path + (which delegates to _build_phase_prompt) and the generic multi-agent path. + + Includes: + - Agent roster showing all active agents and what they produce + - Role-specific proactive preparation instructions + - Full BRC lifecycle steps (including the generic no-op propose path, + #3027, for a producer that finds it has no work in this slice) + """ + try: + from review_graph import get_review_graph_for_phase + + graph = get_review_graph_for_phase(phase, repo=repo) + is_producer = graph.is_producer(role_value) + is_reviewer = graph.is_reviewer(role_value) + reviewers = graph.reviewers_for(role_value) if is_producer else [] + producers = graph.producers_for(role_value) if is_reviewer else [] + wake_only_producers = graph.wake_only_producers_for(role_value) + all_roles = sorted(graph.all_roles()) + graph_available = True + except Exception: + is_producer = role_value in ( + "coder", + "tester", + "documenter", + "refiner", + "architect", + "task_planner", + "risk_analyst", + "simplifier", + ) + is_reviewer = role_value in ( + "reviewer_code", + "reviewer_code_holistic", + "reviewer_contract", + "tester", + "reviewer_refine", + "reviewer_agent_design", + # first_principles_reviewer is a genuine refine-phase reviewer: it + # casts a real ACK verdict on the refiner (CRITICAL edge), so the + # degraded fallback keeps its Reviewer Lifecycle block. It never + # NACKs (redirects go to the operator as HITL decisions), but it + # DOES vote, so — unlike the simplifier — it is a real verdict and + # ``casts_real_verdicts`` (raw ``is_reviewer`` here) stays True. + "first_principles_reviewer", + "reviewer_plan", + # risk_analyst is a genuine dual-role reviewer in the plan graph + # (CRITICAL reviewer of architect + task_planner, #2809) as well as + # a producer of the risk register. Listed here so the degraded + # fallback path keeps its Reviewer Lifecycle / "As a reviewer" + # block instead of stripping it to producer-only — mirroring the + # live plan graph. Unlike the simplifier its edges are real + # verdicts, so ``casts_real_verdicts`` (raw ``is_reviewer`` in the + # degraded path) correctly stays True for it. + "risk_analyst", + # simplifier retains a wake_only advisory edge over the upstream + # refine/plan producer, so the graph reports it as a reviewer — + # but it casts no verdict (#3381) and is rendered PRODUCER-ONLY + # below (the wake_only edge is excluded from the real-reviewer + # determination). Listed here so the degraded fallback path still + # recognizes it; producer-only rendering is handled uniformly. + "simplifier", + ) + reviewers = [] + producers = [] + wake_only_producers = set() + all_roles = [] + graph_available = False + + lines: list[str] = [ + "\n\n## CRITICAL: BRC Consensus Protocol\n", + "You are running in CONCURRENT mode with the Broadcast-Review-Converge " + "(BRC) protocol. Your job is NOT just your task — it is the **full " + "BRC lifecycle**.\n", + ] + + is_dual_role = is_producer and is_reviewer + + # A role whose only reviewed producers are reached via wake_only edges + # (the de-roled simplifier, #3381) casts no verdict on anyone, so it is a + # PRODUCER in every behavioural sense — render it as one. We keep the + # graph-level ``is_dual_role`` flag intact for the banner dispatch below; + # only the rendered role-type label and the "assigned producers" line + # exclude wake_only producers, so the preamble does not contradict the + # producer-only execution banner the simplifier receives. + real_producers = [p for p in producers if p not in wake_only_producers] + if graph_available: + casts_real_verdicts = bool(real_producers) + else: + # Degraded path: the graph load failed, so ``producers == []`` for every + # role and we cannot distinguish wake_only edges from real ones. Fall + # back to raw ``is_reviewer`` so we don't silently strip the Reviewer + # Lifecycle / "As a reviewer" coordination block from a *genuine* + # reviewer (reviewer_code/refine/plan) when the graph is unavailable — + # pre-#3381 this path gated those blocks on raw ``is_reviewer``. The + # simplifier — the only wake_only role — stays producer-only: it is + # excluded here, and is independently rendered producer-only by the + # ``is_dual_role and role_value == "simplifier"`` banner dispatch, which + # still fires in the fallback. + casts_real_verdicts = is_reviewer and role_value != "simplifier" + + if is_producer and casts_real_verdicts: + role_type_desc = "PRODUCER and REVIEWER (dual role)" + elif is_producer: + role_type_desc = "PRODUCER" + elif casts_real_verdicts: + role_type_desc = "REVIEWER" + else: + role_type_desc = "PARTICIPANT" + + lines.append(f"Your role type: **{role_type_desc}**") + if reviewers: + lines.append(f"Your reviewers: {', '.join(reviewers)}") + if real_producers: + lines.append(f"Your assigned producers: {', '.join(real_producers)}") + lines.append("") + + # Agent roster: show all active agents and what they do + if all_roles: + roster = _pkg._build_agent_roster(all_roles, role_value, phase) + if roster: + lines.append(roster) + + # Dual-role ordering banner (#2749, updated for coder-owns-tests). A + # dual-role agent (today: only TESTER in the implement graph) receives + # both the Producer and Reviewer Lifecycle blocks below. The coder now + # authors its own tests; the tester's job is to review-and-harden them + # after the coder proposes. So the tester's producer WORK legitimately + # depends on the coder's ``CONSENSUS_PROPOSE`` — it orients up-front, + # exits after ORIENT, and is re-invoked by the event-pump wrapper when + # the coder proposes, at which point it hardens + proposes + ACK/NACKs + # in one pass. This does not reintroduce the f4c7d780 / 8b81ed32 + # self-block (where the tester idled on a reviewer wait-loop before + # proposing its own scaffolded work): the coder proposes independently + # and does not wait on the tester, so the coder's propose is the + # trigger, and the tester proposes right after. The tester therefore + # has TWO reviewer rendezvous points, both surfaced as fresh wrapper + # invocations under the event-pump model: (a) the coder's first + # ``CONSENSUS_PROPOSE`` re-invokes the tester so it has something to + # harden; (b) subsequent re-proposes and peer-producer proposals + # (after the tester has proposed) likewise re-invoke the tester to + # handle the Reviewer Lifecycle for those events. + if is_dual_role and role_value == "simplifier": + # The simplifier is a PRODUCER ONLY (#3381). It is woken to write the + # companion by the ordinary producer propose-arm (it self-gates on the + # upstream draft existing), NOT by its advisory edge over the upstream + # — that edge is wake_only and casts no verdict, so it is inert in + # consensus derivation (see review_graph.ReviewEdge.wake_only). It is + # NOT a reviewer in any behavioural sense: it issues no verdict, casts + # no ACK/NACK, and never critiques the draft. Treating it as a reviewer + # is what made the companion come out as a review/critique memo instead + # of a plain-language summary. So it gets a PRODUCER-ONLY banner here + # and must NOT inherit the tester's review-and-harden banner below. + lines.append( + "### Execution Order (READ FIRST — simplifier)\n\n" + "You are a **producer only**: your single job is to write a " + "plain-language, human-focused companion to the upstream " + "producer's draft. You do **not** review, critique, score, or " + "vote on that draft — you never issue an ACK or a NACK. (An " + "internal wake-wire re-invokes you when the upstream proposes so " + "you know its draft is ready; consensus never waits on a verdict " + "from you, so there is nothing to respond to.)\n\n" + "**Execute in this order:**\n\n" + "1. **ORIENT (FIRST).** Read the contract and orient. Your work " + "depends on the upstream producer's draft existing, so you begin " + "writing only once that producer issues `CONSENSUS_PROPOSE` — the " + "event-pump wrapper re-invokes you carrying that proposal. Do not " + "race ahead before the draft exists.\n" + "2. **On the upstream producer's PROPOSE**, the wrapper re-invokes " + "you with the proposal in your event payload. SYNC the worktree, " + "read the draft, then write and PROPOSE the human-focused " + "companion (see Producer role below). That is the whole job — " + "the companion is a simplified summary written *for humans to " + "read*, never a review of the draft, a list of constraints the " + "draft should satisfy, or an ACK/NACK rationale.\n" + ) + elif is_dual_role: + lines.append( + "### Dual-Role Execution Order (READ FIRST — #2749, updated for " + "coder-owns-tests)\n\n" + "You are both PRODUCER and REVIEWER (TESTER). **The BRC round " + "cannot close until every producer (including you) has issued " + "`mcp__brc__propose` / `egg-orch consensus propose`** — so you " + "MUST eventually propose, and if you never propose your own " + "hardening you self-block the round. But your producer WORK " + "(reviewing and **hardening the coder's tests**) genuinely " + "depends on the coder's proposed tests existing, so unlike a " + "normal producer you start that work at the coder's PROPOSE, " + "not before. This does not deadlock: the coder proposes " + "independently and does **not** wait on you, so its " + "`CONSENSUS_PROPOSE` is the trigger that unblocks your work; " + "the event-pump wrapper re-invokes you carrying that PROPOSE " + "in your event payload, and you propose right after.\n\n" + "**Execute the lifecycles in this strict order:**\n\n" + "1. **Producer ORIENT (step 1) comes FIRST.** Run ORIENT now " + "to load context. **Your role-specific orientation tells you " + "whether Producer WORK (step 2) runs immediately or is gated " + "on an upstream producer's `CONSENSUS_PROPOSE`** — e.g. the " + "implement-phase tester reviews-and-hardens the coder's tests, " + "so its WORK begins after the coder proposes (#2936). Do not " + "race ahead of the role-specific orientation. While you are in " + "ORIENT, you may *opportunistically* do the Reviewer " + "Lifecycle's `1. PREPARE` work — read the contract, scan the " + "upstream producer's commits as they land on the branch — but " + "do NOT start producing artifacts your role-specific " + "orientation gates on an upstream PROPOSE. Do NOT block on a " + "reviewer wait as your scheduling primitive: the event-pump " + "wrapper invokes you again when the upstream producer's " + "`CONSENSUS_PROPOSE` arrives, at which point you handle the " + "review AND (if your WORK was gated on it) start producing.\n" + "2. **On an upstream producer's PROPOSE**, the wrapper " + "re-invokes you with the proposal in your event payload. " + "SYNC the worktree, then do your Producer WORK (read the " + "coder's tests; add the missing regression + adversarial " + "cases yourself — you share the test scope with the coder; " + "run the tests) and **PROPOSE** your hardening. In the same " + "invocation, issue your reviewer verdict on the coder: ACK " + "if coverage is sound, or NACK naming the specific failing " + "test / coverage gap.\n" + "3. **Subsequent invocations** (re-proposes from any " + "producer — `CONSENSUS_PROPOSE` version > 1 — and " + "`CONSENSUS_RE_REVIEW` events) surface as new wrapper " + "invocations. Each one is a fresh review against the new " + "delta; the per-event prompt includes the full " + "`git log {last_reviewed_commit_sha}..HEAD --not " + "origin/{base_branch} -p` so you can audit the change. " + "Fall through to Reviewer Lifecycle step 3 (SYNC) → step 4 " + "(REVIEW) → step 5 (ACK/NACK), then exit. Do NOT skip step " + "4 (REVIEW) — reading the actual referenced files and " + "forming independent judgment from them is what keeps " + "re-reviews from becoming rubber-stamps.\n" + ) + + if is_producer: + producer_lifecycle: list[str] = ["### Producer Lifecycle"] + # The no-op propose path (#3027) is only valid in the implement + # phase. In refine/plan the producer's draft is mandatory and the + # orchestrator rejects no-op explicitly — so don't even surface the + # affordance to refine/plan producers (architect, refiner, + # task_planner, risk_analyst), keeping prose and enforcement in + # lockstep (review feedback on #3029). + propose_line = ( + "3. **PROPOSE**: When done, run: " + '`egg-orch consensus propose --summary "..." --artifacts "file1" "file2" ' + '--files-changed "f1.py" "f2.py" --tests-run "test_a" "test_b" ' + '--tasks "task-1-1" "task-1-2" --commit-sha $(git rev-parse HEAD)`. ' + "The `--summary` must be ≥50 chars of substantive content describing what was " + "built, what was tested, and which contract tasks it satisfies. " + "Boilerplate like 'looks good' or 'approved' will be rejected." + ) + if phase in ("refine", "plan") and role_value in ( + # Keep in lockstep with ``_DECISION_ATTESTING_ROLES`` in + # ``routes/signals/_validation.py`` — the enforcement side of + # this prose (#3390). + "refiner", + "task_planner", + "architect", + "risk_analyst", + ): + propose_line += ( + "\n\n" + " **Attest your decision ledger (#3390 — MANDATORY).** The " + "orchestrator REJECTS your propose unless its attestation " + "carries your HITL decision ledger. Pass " + "`--decisions-registered cq-1 cq-2 ...` listing every decision " + "you registered this phase (via `egg-contract add-decision` / " + "`mcp__sdlc__register_open_question`), or " + '`--no-decisions-rationale "<why>"` when the phase ' + "deliberately raises none — an explicit empty ledger, never an " + "omission. (Via MCP: the `attestation` arg of " + "`mcp__brc__propose`, fields `decisions_registered` / " + "`no_decisions_rationale`.) Attested ids are cross-checked " + "against the contract, and your draft must cite each attested " + "`cq-N` (copying the `--format markdown` output into the " + "draft satisfies this). A decision your draft commits to " + "without a registered `cq-N` is a reviewer NACK — register " + "it or remove the unilateral commitment. The rationale form " + "is not a shortcut (#3462): the operator is asked to confirm " + "it as its own decision before the phase gate, and a rejected " + "'none' re-runs the phase. If the task names decisions to " + "surface — or you believe a decision is already resolved by " + "prior context — register it with your recommended answer as " + "the first option instead of attesting none." + ) + if phase == "implement": + propose_line += ( + "\n\n" + " **Mark your contract tasks complete (#3114).** Record each " + "delivered task with `mcp__task__complete` (link the commit) — " + "the contract reviewer's ACK is gated on your rows being " + "`complete`, so finished-but-unrecorded work blocks the slice. " + "A task waiting on a peer's work: note it in your proposal and " + "deliver after the dependency lands; the gate holds the slice " + "open until then." + ) + propose_line += ( + "\n\n" + " **No work for you in this slice? Submit a no-op propose (#3027).** " + "If after ORIENT you find your role has no assigned task here AND " + "nothing to contribute (e.g. a documenter on a code-only slice, a " + "tester on a doc-only slice, your domain is not impacted by the " + "diff), do NOT skip silently and do NOT invent busywork — run " + "`egg-orch consensus propose --no-changes-needed --no-changes-reason " + '"<why you have no work here>"` (no artifacts or commit-sha needed). ' + "This counts as proposing, so consensus is not blocked waiting on " + "you; reviewers accept it as a non-blocking no-op (they will not " + "NACK it). Then CONFIRM (step 5) as normal once peers have proposed. " + "Reach for a real propose instead the moment you do find work " + "(e.g. the coder's diff turns out to need docs). Rejected while " + "you still own incomplete contract tasks here (#3114)." + ) + producer_lifecycle.extend( + [ + "1. **ORIENT**: Before starting work, " + + _pkg._build_producer_orientation( + role_value, + phase, + reviewers, + branch=branch, + ), + "2. **WORK**: Complete your assigned task (see Your Task below).", + propose_line, + "4. **RESPOND TO REVIEWS**: When a reviewer NACKs your " + "proposal you will be re-invoked to address it. Read every " + "NACK in the event payload, fix all named blockers, and " + "re-propose with `--changed-artifacts`. **Aggregation is " + "enforced by the orchestrator (#2142):** when two or more " + "distinct reviewers have NACKed the current version, the " + "re-propose call returns HTTP 409 with the full set " + "(reviewer, reason, artifact_refs) inline in `details`; " + "address every NACK then retry. A single-reviewer NACK " + "does not trigger the barrier — re-propose proceeds " + "normally.\n\n" + " **A NACK naming new findings on your re-propose is " + "legitimate adversarial review, not goalpost-moving.** " + "Reviewers re-review v2+ as a fresh delta; \"that's not " + "what you NACK'd last time\" is not a valid objection. " + "**You can and should push back on a NACK on its merits** — " + "if the reviewer misread the code or the concern does not " + "apply, contest it via a directed message with evidence " + "(file:line, test, doc reference). What is *not* productive " + "is contesting a NACK you know is correct — re-reviews are " + "cheap by design, so when the finding is real, fix it and " + "re-propose.", + "5. **CONFIRM**: When all reviewers ACK, run " + "`egg-orch consensus confirmed` to mark your role's " + "consensus.", + "6. **HANDLE RE-REVIEW**: When you are re-invoked with a " + "`CONSENSUS_RE_REVIEW` event" + + ( + " (or a `CONSENSUS_PROPOSE` for a re-propose — " + "version > 1, after you NACKed a prior version; " + "dual-role agents handle both — see Reviewer " + "Lifecycle step 7 for the adversarial re-review " + "framing)" + if is_dual_role and casts_real_verdicts + else "" + ) + + ", act on it — failure to respond stalls the pipeline. " + + ( + "If you are a reviewer of the re-proposing producer, " + "re-review and ACK/NACK the new proposal (dual-role " + "agents: see Reviewer Lifecycle step 7 below for the " + "adversarial re-review framing that applies to this " + "case). Otherwise, re-confirm via " + "`egg-orch consensus confirmed`." + if casts_real_verdicts + else "Re-confirm via `egg-orch consensus confirmed`." + ), + "7. **RESOLVE OBLIGATIONS YOU SATISFY (#2338)**: If you " + "land a commit that satisfies a *different* producer's " + "conditional-ACK obligation in-cycle — typical pattern: " + "the coder is gateway-blocked from a path under `tests/`, " + "you (as tester) cherry-pick the satisfying commit onto " + "the branch — call `mcp__brc__resolve_obligation " + 'reviewer_role="<reviewer>" producer_role="<other_producer>" ' + "commit_sha=$(git rev-parse HEAD)` after pushing. The " + "matrix keeps the obligation text for audit but stops " + "surfacing it on the PR body and HITL gate. Skip this " + "for obligations that genuinely require a human at " + "merge time (deploys, cross-repo flips) — those should " + "remain visible to the merger. **Resolve before " + "`complete_phase`**: once the HITL gate has fired and " + "written the obligation to `contract.pr.deferred_actions`, " + "calling `resolve_obligation` afterwards does *not* " + "retroactively unpersist the entry — the obligation will " + "still appear in the PR body until the next pipeline run. " + "Resolve early. Producers cannot self-resolve their own " + "obligations (the orchestrator rejects " + "`resolver_role == producer_role`), since that would " + "single-handedly bypass the reviewer's veto.\n", + ] + ) + lines.extend(producer_lifecycle) + + # Gate the Reviewer Lifecycle on ``casts_real_verdicts``, not raw + # ``is_reviewer`` (#3381). A role whose only reviewed producers are + # reached via ``wake_only`` edges (the de-roled simplifier) issues no + # ACK/NACK and must NOT receive the reviewer playbook — REVIEW, ACK/NACK, + # CONFIRM, adversarial re-review — which would directly contradict its + # producer-only execution banner. This mirrors the producer-only invariant + # already asserted for the coder (``test_producer_only_no_sync_step``): a + # producer-only role gets no ``### Reviewer Lifecycle`` at all. A pure + # reviewer (``reviewer_refine``) and the dual-role tester both cast real + # verdicts, so they keep it. + if is_reviewer and casts_real_verdicts: + lines.extend( + [ + "### Reviewer Lifecycle", + "1. **PREPARE** (while waiting): " + + _pkg._build_reviewer_preparation( + role_value, + phase, + branch=branch, + base_branch=base_branch, + ), + "2. **INVOKED PER EVENT**: The orchestrator's event-pump " + "wrapper invokes you one-shot per actionable event. When a " + "`CONSENSUS_PROPOSE` arrives for an assigned producer, " + "you are spawned with the proposal in your event payload. " + "Do your preparation work from step 1 on the first " + "invocation; subsequent invocations land you directly at " + "step 3 (SYNC) with the proposal already in context." + + ( + "\n\n **Dual-role agents (you)** — per the " + "*Dual-Role Execution Order* banner above (updated " + "for coder-owns-tests): your first invocation does " + "ORIENT/PREPARE only. On the coder's " + "`CONSENSUS_PROPOSE` the wrapper re-invokes you with " + "the proposal in your event payload; SYNC, do your " + "Producer WORK (review + harden the coder's tests), " + "then PROPOSE your hardening and ACK/NACK the coder " + "in the same invocation (fall through to step 3 " + "(SYNC) → step 4 (REVIEW) → step 5 (ACK/NACK) here). " + "Subsequent invocations (re-proposes — " + "`CONSENSUS_PROPOSE` version > 1 — and peer-producer " + "proposals) are fresh reviews against the new delta, " + "not continuations." + if is_dual_role + else "" + ), + "3. **SYNC**: Before reviewing, sync your worktree so you have the " + "producer's commits: `git fetch origin && git merge " + + _pkg._resolve_origin_ref(branch or base_branch) + + " --no-edit`", + "4. **REVIEW**: Once a proposal arrives, form independent judgment from " + "the referenced code artifacts. Read the actual files — do not rely " + "solely on the proposal summary.", + "5. **ACK/NACK**: Your `--reason` IS your review. Put your **full analysis** " + "there — this is what the producer reads and acts on. **Always " + "pass `--ack-version` / `--nack-version`** with the producer's " + "current proposal version (#2142) — read it from the " + "`CONSENSUS_PROPOSE` message that triggered your review (the " + "`version` field). The orchestrator rejects the verdict with " + "`stale_version` if the producer has re-proposed since you " + "started reviewing.\n" + "\n" + " **NACK format** (use when blocking issues exist):\n" + " ```\n" + ' egg-orch consensus nack <role> --files-reviewed "f1" "f2" ' + '--nack-version <N> --reason "\n' + " ### Blocking\n" + " 1. **file.py:123** — Description of the issue. Fix: suggested fix.\n" + " 2. **file.py:456** — Description of the issue. Fix: suggested fix.\n" + " ### Non-blocking\n" + " - **file.py:789** — Suggestion for improvement.\n" + ' "\n' + " ```\n" + "\n" + " **ACK format** (use when no blocking issues):\n" + " ```\n" + ' egg-orch consensus ack <role> --files-reviewed "f1" "f2" ' + '--ack-version <N> --reason "\n' + " Reviewed [N files / specific areas]. Verified [what was checked].\n" + " [Specific observations about correctness, security, etc.]\n" + " ### Non-blocking\n" + " - **file.py:123** — Optional suggestions for improvement.\n" + ' "\n' + " ```\n" + "\n" + " **Conditional ACK (#1998)** — use when the work is " + "correct but a human action is needed at merge time " + "(`git mv`, secret rotation, cross-repo flip): add " + '`--pre-merge-condition "…"` to the ACK. The obligation ' + "renders as a `Pre-merge Obligations` block in the PR " + "body. Do NOT use this to smuggle blocking issues past " + "the producer — if the producer could fix it, NACK " + "instead.\n" + "\n" + " **Drop satisfied obligations on re-ACK (#2338).** When " + "you re-ACK at a new proposal version and the conditioning " + "work has landed in-cycle (the rename is in the diff, the " + "obligation is moot), drop the obligation: re-ACK without " + "`--pre-merge-condition`. Do NOT re-attach it with a " + 'self-contradicting "satisfied" hedge — the PR body ' + "renders obligations verbatim under a `do not merge` " + "banner. To preserve the audit trail instead of dropping, " + "re-ACK with `--pre-merge-condition-resolved-in-diff <sha>` " + "alongside `--pre-merge-condition` so the renderer " + "demotes (not drops) the entry (#2336).\n" + "\n" + " `--reason` must be ≥50 chars of substantive content. " + "Boilerplate like 'lgtm' or 'no issues' will be rejected.\n" + "\n" + " **Stale-version rejection (#2142):** if the producer " + "re-proposed while your verdict was in flight, the ACK / " + "NACK is rejected with HTTP 409 inlining the current " + "proposal snapshot (version, artifacts, commit_sha). " + "`git fetch && git merge`, re-review against the new " + "commit, and re-submit — don't retry the same payload." + + ( + "\n\n" + " **Contract-enforcer gate (#3114) — applies to you.** " + "Your ACK of a producer is structurally gated on the " + "contract: the orchestrator REJECTS it (409 " + "`contract_incomplete`) while any task row owned by that " + "producer in this slice is not `status=complete`. Read " + "the live task records with `mcp__sdlc__show_contract` — " + "the `.egg-state/contracts/` copy in your checkout is an " + "init-time snapshot; do not trust it. When rows are " + "incomplete, NACK the producer citing the exact task " + "ids: either the work is missing (it must deliver) or it " + "landed unrecorded (it must run `mcp__task__complete`). " + "When all rows are complete, your ACK MUST carry " + '`attestation={"tasks_verified": ["task-…", …]}` on ' + "`mcp__brc__ack`, covering every task id the producer " + "owns in this slice — absent or non-covering lists are " + "rejected (`attestation_required` / " + "`attestation_mismatch`). Your CONFIRM is likewise " + "rejected while ANY row in the slice is incomplete. A " + "producer's declared deferral (\"will land in later " + 'proposals") is an open obligation, not an end-state — ' + "hold consensus open until the rows are delivered or a " + "human descopes them." + if phase == "implement" and role_value in _pkg._contract_enforcer_role_names() + else "" + ), + "6. **CONFIRM**: When all assigned producers reviewed: " + "`egg-orch consensus confirmed`", + "7. **HANDLE RE-REVIEW**: When you are re-invoked with a " + "`CONSENSUS_RE_REVIEW` event (or a `CONSENSUS_PROPOSE` for " + "a re-propose — version > 1, after you NACKed a prior " + "version), act on it — failure to respond stalls the " + "pipeline. Re-review the re-proposing producer's new " + "proposal and ACK/NACK it, then re-confirm via " + "`egg-orch consensus confirmed`.\n\n" + " **This is adversarial re-review, not blocker-verification.** " + "Your re-review has TWO equal-weight mandates: (1) verify the " + "blockers from your prior NACK were addressed AND (2) audit the " + "delta since your last review — the commits landed since the " + "version you last verdicted (per REVIEWER-SYNC.md: `git log " + "{last_reviewed_commit}..HEAD --not origin/{base_branch} -p`) — " + "as a fresh reviewer with no NACK history, bounded to that " + "delta, NOT the whole accumulated surface. Both must pass to " + "ACK. The orchestrator's adversarial re-prime in the event " + "body carries the full framing; this is a pointer. New issues " + "outside your prior NACK's scope are blocking; **NACK without " + "hesitance** — re-reviews are cheap by design, and the " + "downstream GitHub reviewer should find nothing in your " + "re-reviewed deltas.\n", + ] + ) + + # Directed coordination guidance — role-gated + lines.append("### Directed Coordination") + lines.append( + "In addition to the BRC consensus flow (PROPOSE/ACK/NACK), you can send " + "directed peer-to-peer messages to specific agents using " + "`egg-orch message send --to <role> --type <TYPE>`. These directed messages " + "are **supplementary** to BRC consensus — they do NOT replace the " + "PROPOSE/ACK/NACK lifecycle and are never required for consensus to proceed.\n" + ) + + if is_producer: + lines.extend( + [ + "**As a producer**, use directed messages to coordinate handoffs and " + "broadcast progress:", + "- **HANDOFF**: When your work is ready for a specific peer to act on, " + "send a HANDOFF message so they know to begin. For example, a coder " + "notifying the tester that implementation is complete.", + " ```", + ' egg-orch message send --to tester --type HANDOFF --subject "Auth module ready" ' + '--body "auth.py is complete, tests can begin"', + " ```", + "- **STATUS**: Broadcast progress updates to all agents when you reach " + "significant milestones (e.g., halfway through implementation, blocked " + "on a dependency).", + " ```", + ' egg-orch message send --to all --type STATUS --subject "Implementation 50% complete" ' + '--body "Core logic done, working on edge cases"', + " ```\n", + ] + ) + + # Same gate as the Reviewer Lifecycle above (#3381): a wake-only, + # verdict-free role (de-roled simplifier) gets no reviewer-coordination + # guidance, since it never ACK/NACKs. + if is_reviewer and casts_real_verdicts: + lines.extend( + [ + "**As a reviewer**, when you need clarification before " + "ACK/NACKing, put the question in your NACK `--reason` " + "block under `### Non-blocking`. The producer sees it " + "atomically with the review verdict and the audit " + "trail is preserved. The legacy QUESTION message " + "type was removed in issue #1897; off-protocol chatter " + "is no longer advertised. A follow-up issue will " + "introduce a structured REQUEST/REPLY subsystem that " + "names a target peer and times out.", + "", + ] + ) + + lines.extend( + [ + "**Event-handler contract (#2908):** The orchestrator's " + "event-pump wrapper drives your lifecycle. You are invoked " + "one-shot per actionable BRC event: handle the event per the " + "lifecycle above, update durable BRC memory (writes happen " + "automatically inside `egg-orch consensus ack` / `nack` " + "handlers), then exit naturally. The wrapper polls " + "`egg-orch brc next-action` and re-invokes you with the next " + "event. You do NOT block on `egg-orch message wait-loop` " + "yourself; the wrapper owns the wait and the heartbeat.\n", + "", + ] + ) + + return "\n".join(lines) diff --git a/orchestrator/routes/pipelines/_prompt_review.py b/orchestrator/routes/pipelines/_prompt_review.py new file mode 100644 index 0000000000..9b08be1932 --- /dev/null +++ b/orchestrator/routes/pipelines/_prompt_review.py @@ -0,0 +1,765 @@ +"""review-prompt + role-context helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _summarize_issue(prompt: str | None, issue_number: int | None = None) -> str: + """Extract a 1-2 sentence summary from the issue title and first paragraph. + + Used to give execution agents (tester, documenter) a brief + orientation without embedding the full issue body. Analysis agents + (architect, task_planner, risk_analyst) still receive the full issue. + + Extracts the first markdown heading (or first non-empty line) as the title, + then the first paragraph as supporting context. + """ + if not prompt or not prompt.strip(): + return f"Working on issue #{issue_number}." if issue_number else "" + + lines = prompt.strip().splitlines() + + # Extract title: first markdown heading, or first non-empty line + title = "" + body_start = 0 + for i, line in enumerate(lines): + s = line.strip() + if not s: + continue + if s.startswith("#"): + title = s.lstrip("# ").strip() + else: + title = s + body_start = i + 1 + break + + # Extract first paragraph after title (up to ~300 chars) + first_para_lines: list[str] = [] + for line in lines[body_start:]: + s = line.strip() + if not s: + if first_para_lines: + break + continue + first_para_lines.append(s) + + first_para = " ".join(first_para_lines) + if len(first_para) > 300: + first_para = first_para[:297] + "..." + + # Build summary + issue_ref = f" (issue #{issue_number})" if issue_number else "" + summary = f"**Background**: {title}{issue_ref}" + if first_para: + summary += f"\n\n{first_para}" + + return summary + + +def _extract_plan_overview(plan_text: str) -> str: + """Extract the plan overview section (before individual phase details). + + Returns the summary/overview portion of the plan, stopping before + individual phase task listings (### Phase N: ...) and the yaml-tasks + appendix. This gives the coder high-level context without the full plan. + """ + lines = plan_text.splitlines() + overview_lines: list[str] = [] + + for line in lines: + stripped = line.strip() + # Stop at individual phase headings + if stripped.startswith("### Phase ") or stripped.startswith("### phase-"): + break + # Stop at the yaml-tasks appendix + if "yaml-tasks" in stripped: + break + # Stop at structured task appendix + if stripped.startswith("## Structured Task Appendix"): + break + # Stop at issue-to-task mapping (detailed reference section) + if stripped.startswith("## Issue-to-Task Mapping"): + break + overview_lines.append(line) + + # Trim trailing blank lines + while overview_lines and not overview_lines[-1].strip(): + overview_lines.pop() + + return "\n".join(overview_lines) + + +def _build_role_context( + role_value: str, + prompt: str | None, + issue_number: int | None = None, + phase_obj=None, + all_phases=None, + base_branch: str | None = None, +) -> str: + """Build role-appropriate context to replace raw issue body embedding. + + Analysis roles (architect, task_planner, risk_analyst) receive the full + issue body since they need it for problem understanding and planning. + + Execution roles (tester, documenter) receive a brief summary + with structured task information and pointers to full context. + + Args: + role_value: Agent role string + prompt: Original task prompt (full issue body) + issue_number: GitHub issue number + phase_obj: Current plan phase object (phase context) + all_phases: All contract phases (phase context) + + Returns: + Role-appropriate context string to embed in the agent prompt + """ + from egg_contracts.agent_roles import EXECUTION_ROLE_VALUES + + # Analysis roles need the full issue body for problem understanding + if role_value in ("architect", "task_planner", "risk_analyst"): + if prompt: + return f"## Task Description\n\n{prompt}\n" + return "" + + lines: list[str] = [] + + # Brief summary for execution roles + summary = _pkg._summarize_issue(prompt, issue_number) + if summary: + lines.append(f"## Background\n\n{summary}\n") + + # Phase-specific context + if phase_obj is not None: + lines.append(f"## Phase Scope: {phase_obj.name} ({phase_obj.id})\n") + + if role_value == "tester": + lines.append( + f"Focus your testing on code changed in plan phase `{phase_obj.id}`. " + "The following tasks were implemented in this phase:\n" + ) + elif role_value == "documenter": + lines.append( + "Document the current state of the code in the areas these tasks " + "touch — a snapshot of how the system works now, not a log of what " + "changed. The following tasks were implemented in this phase:\n" + ) + else: + lines.append("The following tasks were implemented in this phase:\n") + + # Filter tasks by role for execution agents. + # Only apply role-based filtering when at least one task has a role + # assigned — legacy plans (all role=None) show all tasks to all agents, + # preserving backward compatibility. + _has_any_role = any(t.role is not None for t in phase_obj.tasks) + if role_value in EXECUTION_ROLE_VALUES and _has_any_role: + # Unassigned tasks (role=None) default to coder. + filtered_tasks = [ + task + for task in phase_obj.tasks + if task.role == role_value or (task.role is None and role_value == "coder") + ] + else: + filtered_tasks = list(phase_obj.tasks) + + for task in filtered_tasks: + lines.append(f"- **{task.id}**: {task.description}") + if getattr(task, "acceptance_criteria", None): + lines.append(f" - Acceptance: {task.acceptance_criteria}") + if getattr(task, "files_affected", None): + lines.append(f" - Files: {', '.join(task.files_affected)}") + lines.append("") + + if all_phases and phase_obj is not None and role_value in ("tester", "documenter"): + # Brief orientation about other phases for context + other_phases = [p for p in all_phases if p.id != phase_obj.id] + if other_phases: + lines.append("### Other Phases (for orientation)\n") + for phase in other_phases: + status = getattr(phase, "status", "unknown") + lines.append(f"- {phase.id}: {phase.name} [{status}]") + lines.append("") + + # Context pointers — agents can get more detail on demand + lines.append("## For More Context\n") + if issue_number: + lines.append(f"- Full issue: `gh issue view {issue_number}`") + _rc_base_ref = _pkg._resolve_origin_ref(base_branch) + lines.append(f"- Changed files: `git diff {_rc_base_ref}...HEAD` or check handoff data") + lines.append("- Coder output: check `EGG_HANDOFF_DATA` environment variable") + lines.append("") + + return "\n".join(lines) + + +def _build_role_restrictions_section(repo: str | None = None) -> str: + """Build a prompt section describing file access restrictions per execution role. + + This section is injected into the task_planner prompt so that it can + assign each task to the correct execution role (coder, tester, documenter) + based on which files the task will modify. + + Args: + repo: Optional ``owner/repo`` for per-repo pattern overrides + (#2528). When set, the rendered patterns reflect + ``role_patterns:`` from ``repositories.yaml`` for the repo + so the planner sees the same boundaries the gateway will + enforce. When ``None``, falls back to global defaults. + + Returns: + Formatted markdown string describing role file boundaries. + """ + from egg_restrictions.patterns import get_agent_patterns_for_repo + + lines: list[str] = [ + "## Execution Role File Restrictions", + "", + "Each task should include a `role` field (coder, tester, or documenter) " + "indicating which agent should execute it. Assign roles based on the file " + "access restrictions below. Tasks without a `role` field default to coder.", + "", + ] + + patterns_by_role = get_agent_patterns_for_repo(repo) + for role_name in ("coder", "tester", "documenter"): + pattern = patterns_by_role.get(role_name) + if pattern is None: + continue + lines.append(f"### {role_name}") + if pattern.allowed_patterns: + lines.append(f"- **Allowed**: {', '.join(f'`{p}`' for p in pattern.allowed_patterns)}") + if pattern.blocked_patterns: + lines.append(f"- **Blocked**: {', '.join(f'`{p}`' for p in pattern.blocked_patterns)}") + # Hard blocks are rejected even when they'd match the allow list or a + # fixture/docs exemption (#3396) — the planner must see them so it + # doesn't route a hard-blocked path (e.g. a fixture under `.github/` + # or any `.egg-state/` subdir) to this role. + if pattern.hard_blocked_patterns: + hard = f"- **Hard-blocked (never pushable)**: {', '.join(f'`{p}`' for p in pattern.hard_blocked_patterns)}" + if pattern.hard_block_exempt_patterns: + hard += ( + f" (except {', '.join(f'`{p}`' for p in pattern.hard_block_exempt_patterns)})" + ) + lines.append(hard) + lines.append("") + + lines.append( + "Assign `role: tester` to tasks that only touch test files, " + "`role: documenter` to tasks that only touch docs/README files, " + "and `role: coder` (or omit the field) for everything else. " + "If a task spans multiple roles, split it into separate tasks per role." + ) + lines.append("") + + # Staging-dir convention for `.github/` (issue #2508). + lines.append("### `.github/` changes — use the `.github-staging/` convention") + lines.append("") + lines.append( + "Every producer role is blocked from writing under `.github/` " + "(CI workflows, CODEOWNERS, dependabot config) — this is a " + "branch-protection invariant, not a planner mistake. Tasks that " + "need to modify those files must instead write the proposed " + "end-state to top-level `.github-staging/`, mirroring the " + "`.github/` structure (e.g. a proposed change to " + "`.github/workflows/ci.yml` is staged at " + "`.github-staging/workflows/ci.yml`). The producing agent must " + "call the staged files out in the PR body so the human " + "reviewer moves them into `.github/` before merge. Assign such " + "tasks to `role: coder` and make the staging path explicit in " + "the task's `files_affected`. `.github-staging/` must remain " + "tracked by git (do not add it to `.gitignore`); otherwise the " + "staged files won't be in the PR commit and the reviewer's " + "`git mv` will fail." + ) + lines.append("") + + # Runtime escape hatch — the actionable producer-side guidance (the + # "call these two tools, do not invent a workaround, exit cleanly" + # text) lives in ``_build_impasse_escape_hatch_section`` and is + # injected into producer prompts (coder/tester/documenter); see + # issue #2529. Here we tell the planner only that the post-failure + # delegation path exists, so it knows the orchestrator can rewire a + # mis-assigned task without re-planning. The planner does not emit + # impasses itself. + lines.append("### Runtime delegation (post-failure)") + lines.append("") + lines.append( + "If a producer discovers mid-execution that its assigned task " + "is structurally impossible, it emits a typed Impasse via " + "``mcp__sdlc__report_impasse`` and the orchestrator may " + "auto-delegate the task to a different producer role (see " + "issue #2529). You don't need to plan for this — it's a " + "runtime safety net for plan bugs, role-restriction " + "mismatches, and external blockers." + ) + lines.append("") + + return "\n".join(lines) + + +def _build_impasse_escape_hatch_section() -> str: + """Build the producer-facing runtime escape hatch section (#2529). + + Injected into the coder/tester/documenter prompts so producers know + to call ``mcp__sdlc__check_file_restriction`` / + ``mcp__sdlc__report_impasse`` instead of inventing workarounds when + they hit a structurally impossible task. The planner never emits + impasses, so this section is omitted from its prompt — see + ``_build_role_restrictions_section`` for the planner-facing + summary. + """ + return "\n".join( + [ + "## Impossible task? Use the runtime escape hatch — DO NOT invent workarounds", + "", + ( + "If you discover mid-execution that the task you've been " + "assigned is structurally impossible (file restrictions " + "block your role, the plan is buggy, an external " + "dependency is missing), STOP. Do not invent a " + "workaround like staging the files in another directory " + "or asking another agent to do it via a freeform handoff " + "document — past pipelines (#2474, #2529) wasted ~10+ " + "min and triggered downstream NACKs that way." + ), + "", + "Instead, use the two MCP tools:", + "", + ( + '1. `mcp__sdlc__check_file_restriction({path: "..."})` — ' + "cheap pure-local read against `shared/egg_restrictions/" + "patterns.py`. Confirms whether your role can write the " + "path and returns `alternative_role` (the producer role " + "that *can* write it, when exactly one covers it). Call " + "this BEFORE exploring a file you suspect is outside " + "your boundary." + ), + "", + ( + "2. `mcp__sdlc__report_impasse({category, reason, " + "task_id, suggested_role, blocked_files})` — emits a " + "typed Impasse signal and exits cleanly. **`task_id` is " + "required for ``wrong_role`` impasses** (look it up in " + "your spawn prompt or via `egg-contract show`); without " + "it the orchestrator cannot route precisely and " + "escalates to HITL. The orchestrator detects the " + "impasse post-phase and either delegates to " + "``suggested_role`` (first attempt) or escalates to " + "HITL (second attempt or no eligible role). Categories: " + "``wrong_role`` (file restrictions; auto-delegateable), " + "``plan_bug`` / ``external_blocker`` / ``unknown`` " + "(always HITL). Once you've called this tool, do NOT " + "commit code or call any other producer tool — just " + "exit." + ), + "", + ] + ) + + +def _render_contract_tasks( + repo_path: str, + pipeline_id: str, + pipeline_mode: str, + issue_number: int | None = None, +) -> str | None: + """Load contract and render tasks as a markdown checklist. + + Returns None if the contract cannot be loaded. + """ + try: + from egg_contracts.loader import load_contract + from egg_contracts.models import TaskStatus + except ImportError: + return None + + # Contracts are keyed by pipeline_id (loader's compat shim handles + # legacy paths for in-flight pipelines that predate key unification). + try: + contract = load_contract(pipeline_id, _pkg.Path(repo_path)) + except Exception: + return None + + if not contract.slices: + return None + + lines = ["## Contract Tasks\n"] + for slice_ in contract.slices: + if not slice_.tasks: + continue + lines.append(f"### {slice_.name}\n") + for task in slice_.tasks: + check = "x" if task.status == TaskStatus.COMPLETE else " " + lines.append(f"- [{check}] **{task.id}**: {task.description}") + if task.acceptance_criteria: + lines.append(f" - Acceptance: {task.acceptance_criteria}") + if task.files_affected: + lines.append(f" - Files: {', '.join(task.files_affected)}") + lines.append("") + + return "\n".join(lines) if len(lines) > 1 else None + + +def _build_review_prompt( + phase: str, + pipeline_id: str, + pipeline_mode: str, + reviewer_type: str = "code", + issue_number: int | None = None, + review_cycle: int = 1, + prior_feedback: str | None = None, + repo_path: str | None = None, + last_reviewed_commit: str | None = None, + base_branch: str | None = None, + concurrent: bool = False, + operator_directives: list[_pkg.OperatorDirective] | None = None, + iteration_history: list[_pkg.IterationSummary] | None = None, +) -> str: + """Build a review prompt for the reviewer agent. + + In sequential mode, tells the reviewer to write a typed verdict JSON + file to .egg-state/reviews/. In concurrent (BRC) mode, the reviewer's + ACK/NACK reason IS the review output — no verdict file is written. + """ + draft_path = _pkg._get_draft_path(phase, issue_number=issue_number, pipeline_id=pipeline_id) + + verdict_path: str | None = None + if not concurrent: + verdict_path = _pkg._verdict_path_for_type( + phase, + reviewer_type, + issue_number=issue_number, + pipeline_id=pipeline_id, + ) + + lines = [ + f"You are reviewing the **{phase}** phase output of the SDLC pipeline " + f"({reviewer_type} reviewer).\n", + "## Scope\n", + _pkg._get_reviewer_scope_preamble(reviewer_type, phase), + "", + "## Context\n", + f"Pipeline ID: {pipeline_id}", + f"Phase: {phase}", + f"Reviewer: {reviewer_type}", + f"Review cycle: {review_cycle}", + "", + "## Your Task\n", + ] + + # Delta review: for re-reviews with a known last-reviewed commit, + # instruct the reviewer to focus on the delta. + # + # Two-dot `git diff A..HEAD` would wrongly include any base-branch merges + # landed between A and HEAD. `git log A..HEAD --not origin/<base> -p` + # explicitly excludes commits reachable from the base branch, so the + # reviewer sees only PR-authored work (issue #1758). + is_delta_review = review_cycle > 1 and last_reviewed_commit and not draft_path + _base_ref = _pkg._resolve_origin_ref(base_branch) + _delta_base_branch = _base_ref.removeprefix("origin/") + diff_command = ( + f"git log {last_reviewed_commit}..HEAD --not {_base_ref} -p" + if is_delta_review + else f"git diff {_base_ref}...HEAD" + ) + + if draft_path: + lines.append(f"1. Read the draft at `{draft_path}`") + elif is_delta_review: + lines.append( + f"1. First run `git fetch origin {_delta_base_branch}`, then review " + f"the delta using `{diff_command}` (see **Delta Review** below)" + ) + else: + lines.append( + f"1. Review the implementation using `git log --oneline -10` and `{diff_command}`" + ) + + # Add procedural steps for code reviewers (matching GHA reviewer thoroughness). + # Both ``code`` and ``code-holistic`` get the same numbered procedural-step + # scaffold, but steps 2 and 8 differ by lens: ``code`` reviews every file + # systematically and evaluates against the code-review criteria, while + # ``code-holistic`` skims the diff once and runs the four cross-module + # passes from the holistic criteria file. See issue #2126 — the prior + # unified wording told the holistic reviewer to "review every changed + # file systematically", which contradicted the holistic criteria's + # "don't verify every line". + # + # The operator-copy-paste framing (step 5) and pre-existing-broken-behavior + # clause (added after step 8) are deliberately scoped to code/code-holistic + # only. The shapes generalize — a security reviewer reading a `curl | bash` + # snippet, a concurrency reviewer reading a `gunicorn` launch line, or a + # contract reviewer reading an acceptance-criterion snippet would all + # benefit from "would this command execute as written?" — but the four + # #2724 misses that motivated these additions were code-lens issues + # (`pip install -r requirements.txt`, `${ANSWER}` shell-interpolated, + # `datetime.utcnow()` deprecation, non-atomic write). Keeping these on the + # code-lens branch avoids prompt bloat for narrower-lens reviewers whose + # rubrics already cover the same ground in lens-specific shape. Expand + # scope only if observed misses in other-lens reviews motivate it. + if reviewer_type in ("code", "code-holistic") and not draft_path: + if reviewer_type == "code-holistic": + lines.append( + "2. **Skim the full diff once** to build a mental map of " + "what the PR adds, who the user is, and what the user's " + "primary path through the change looks like — do not " + "re-verify every line; that is the code reviewer's job" + ) + else: + lines.append("2. Get the full diff and **review every changed file systematically**") + lines.append( + "3. Read surrounding context — check how changed code integrates with the rest of the codebase" + ) + lines.append( + "4. Trace data flow from input to output, especially for security-sensitive paths" + ) + lines.append( + "5. Verify end-to-end functionality — for new features, trace the complete " + "execution path in the real deployment environment. Check that config files, " + "environment variables, and dependencies are actually available where the code runs. " + "**Read every documented snippet, install command, and code example " + "as an operator about to copy-paste it.** Apply this verification " + "ladder to each snippet:\n" + " - Would the command execute as written?\n" + " - Does the documented file exist (`ls` or `find` it)?\n" + " - Does the library/API the snippet calls match the actual " + "signature (use WebSearch for deprecations and version-dependent " + "behavior)?\n" + "\n" + " The four blocking findings on PR #2724 (escaped to the GitHub " + "bot) were all of this shape — `pip install -r requirements.txt` " + "against a non-existent file, `${ANSWER}` shell-interpolated as a " + "bare Python identifier, `datetime.utcnow()` deprecated since " + "Python 3.12, non-atomic file write — and would all have been " + "caught by reading the snippet as a copy-paster instead of as a " + "documentation reader." + ) + lines.append( + "6. Research when uncertain — use WebSearch and WebFetch (when available) " + "to look up library behavior, check official documentation, verify " + "API usage patterns, and confirm the code follows current best practices" + ) + lines.append("7. Consider edge cases the author may not have tested") + if reviewer_type == "code-holistic": + lines.append( + "8. Run the four mandatory passes from the criteria below " + "(end-to-end primary use case, doc ↔ code symmetry, " + "synthetic-key / sentinel coordination, silent-fallback hunt)" + ) + else: + lines.append("8. Evaluate against the criteria below") + # Procedural surfacing of the pre-existing-broken-behavior clause + # from code-review-criteria.md:71. Buried in the rubric body it's + # easy to skim past — the #2724 misses on `pip install -r + # requirements.txt` and the Python-version mismatch both lived + # in lines the PR reflowed but did not author, and the reviewer + # treated them as out-of-scope context. Promoting it to a + # numbered step (read before reviewing, not consulted mid-review) + # makes it fire on lines the PR touches by reflowing, not only + # on lines it authors fresh. + lines.append( + "**(Pre-existing broken behavior in modified code is blocking.)** " + "Any unchanged line the PR reflows, surrounds, or otherwise " + "modifies its area of belongs to this review's scope. If the " + "PR's hunks reflow an install section, a documented snippet, or " + "a config example, verify the *whole section* works as advertised " + "— not just the lines marked `+`. The code is already being " + "changed in that area; this is the natural place to fix it. " + "Pre-existing bugs in modified code are NACK-blocking — do not " + 'dismiss as "not a regression."' + ) + if concurrent: + lines.append( + "9. Deliver your full review via ACK/NACK (see BRC protocol below). " + "Your `--reason` IS your review — include all findings there." + ) + else: + lines.append(f"9. Write your verdict to `{verdict_path}` as JSON") + lines.append("10. Commit the verdict file") + lines.append("") + lines.append( + "**Find ALL issues on the first pass** — do not stop after identifying " + "a few problems. You are the last line of defense before code reaches " + "production." + ) + elif draft_path: + # Expanded procedural steps for draft-based (non-code) reviewers + lines.append("2. Read the draft thoroughly — do not skim") + lines.append( + "3. Cross-reference each section of the draft against the review criteria below" + ) + lines.append("4. Cite specific sections, quotes, or omissions as evidence in your analysis") + lines.append("5. Evaluate completeness — identify any criteria not adequately addressed") + lines.append("6. Assess overall quality and coherence of the draft") + if concurrent: + lines.append( + "7. Deliver your full review via ACK/NACK (see BRC protocol below). " + "Your `--reason` IS your review — include all findings there." + ) + else: + lines.append(f"7. Write your verdict to `{verdict_path}` as JSON") + lines.append("8. Commit the verdict file") + else: + lines.append("2. Evaluate it against the criteria below") + if concurrent: + lines.append( + "3. Deliver your full review via ACK/NACK (see BRC protocol below). " + "Your `--reason` IS your review — include all findings there." + ) + else: + lines.append(f"3. Write your verdict to `{verdict_path}` as JSON") + lines.append("4. Commit the verdict file") + lines.append("") + + # Review criteria + lines.append("## Review Criteria\n") + lines.append(_pkg._get_review_criteria_for_type(reviewer_type, phase, repo_path=repo_path)) + lines.append("") + + # Review conventions — quality standards aligned with PR reviewer thoroughness + lines.append("## Review Conventions\n") + if reviewer_type in ("code", "code-holistic"): + lines.append( + "You are a critical part of the engineering infrastructure — the last line " + "of defense before code reaches production. Your review must meet these " + "quality standards:\n" + ) + else: + lines.append("Your review must meet these quality standards:\n") + lines.append( + "1. **Be comprehensive.** Review the entire scope, not just the obvious parts. " + "Do not stop after finding the first few issues." + ) + lines.append( + "2. **Be specific.** Reference exact file paths, line numbers, function names, " + "and code snippets. Vague feedback is not actionable." + ) + lines.append( + "3. **Be direct.** State issues plainly without hedging or softening language. " + '"This will fail when X" not "you might want to consider X".' + ) + lines.append( + "4. **Suggest fixes.** When identifying a problem, include a concrete suggestion " + "for how to resolve it." + ) + lines.append( + "5. **Provide context.** Explain *why* something is an issue — the impact, " + "the risk, or the principle being violated." + ) + lines.append("") + + # Verdict classification — only for code reviewers (aligned with review-conventions.md) + # Non-code reviewers get appropriate guidance from their type-specific criteria + # (e.g., _get_plan_review_criteria() already says "flag as needs_revision") + if reviewer_type in ("code", "code-holistic"): + _nack_label = "NACK" if concurrent else "`needs_revision`" + _ack_label = "ACK" if concurrent else "`approved`" + lines.append(f"### When to {_nack_label} vs {_ack_label}\n") + lines.append( + f"**{_nack_label} for**: Security vulnerabilities, logic errors, correctness " + "issues, non-functional features (core purpose doesn't work end-to-end), missing " + "error handling, resource leaks, breaking changes, violations of codebase patterns. " + f"When in doubt, {_nack_label}." + ) + lines.append( + f"**{_ack_label} for**: No blocking issues found after thorough review. " + "Non-blocking suggestions should still be included." + ) + lines.append("") + lines.append( + "**Key distinction**: A feature that doesn't work is a correctness issue, not a " + "style issue. If the feature's core functionality is broken — not just degraded or " + f"missing edge cases — always {_nack_label}, even if the code structure looks " + "reasonable or matches an existing pattern." + ) + lines.append("") + + # Delta review directive for re-reviews + if is_delta_review: + lines.append("## Delta Review\n") + lines.append( + f"This is review cycle {review_cycle}. Focus on new changes since your " + f"last review. First run `git fetch origin {_delta_base_branch}` to " + f"ensure the base branch is available, then use " + f"`git log {last_reviewed_commit}..HEAD --not {_base_ref} -p` to see " + "the delta — this excludes any base-branch commits that were merged " + "in since your last review, so you only see PR-authored changes. " + "Verify prior feedback was addressed AND review new code thoroughly." + ) + lines.append("") + + # Phase iteration context: operator directives + prior iteration + # history. Surfaced to reviewers so they cannot faithfully NACK a + # directive-driven change against a stale default rubric (#2795). + iteration_context = _pkg._build_phase_iteration_context(operator_directives, iteration_history) + if iteration_context: + lines.append(iteration_context) + + # Prior feedback for re-reviews + if review_cycle > 1 and prior_feedback: + lines.append("## Prior Review Feedback\n") + lines.append( + "This is a re-review. The previous review found issues. " + "Verify that the following feedback was addressed:\n" + ) + lines.append(prior_feedback) + lines.append("") + + # Verdict format — only for sequential (non-concurrent) reviewers. + # In concurrent/BRC mode, the ACK/NACK reason IS the review output. + if not concurrent: + lines.append("## Verdict Format\n") + lines.append(f"Write the following JSON to `{verdict_path}`:\n") + lines.append("```json") + lines.append("{") + lines.append(f' "reviewer": "{reviewer_type}",') + lines.append(' "verdict": "approved" or "needs_revision",') + lines.append(' "summary": "Brief summary of findings (1-2 sentences)",') + lines.append(' "analysis": "Detailed analysis of the reviewed work (see below)",') + lines.append(' "suggestions": "Non-blocking suggestions for improvement",') + lines.append(' "feedback": "Blocking issues requiring revision before approval",') + lines.append(' "timestamp": "ISO 8601 timestamp"') + lines.append("}") + lines.append("```\n") + lines.append("**Field guidelines:**\n") + lines.append( + "- **analysis**: Always provide detailed analysis regardless of verdict. " + "Describe what you reviewed, what you found, and your reasoning." + ) + lines.append( + "- **suggestions**: Non-blocking observations and improvement ideas. " + "Include these even when approving — they help the team improve over time." + ) + lines.append( + "- **feedback**: Reserved for **blocking issues only** — problems that must " + "be fixed before the work can be approved. Leave empty when approving." + ) + lines.append( + "\nIf the work meets all criteria, set verdict to `approved`. " + "If significant issues remain, set verdict to `needs_revision` " + "and provide actionable feedback in the `feedback` field." + ) + + # Phase restrictions for reviewers + lines.append("") + lines.append("## Phase Restrictions\n") + lines.append("- You CAN read all source files and review artifacts") + if not concurrent: + lines.append("- You CAN write verdict files to `.egg-state/reviews/`") + if reviewer_type == "contract": + lines.append( + "- You CAN update the contract in `.egg-state/contracts/` (e.g. marking items as done)" + ) + lines.append("- You CANNOT push code (git push)") + lines.append("- You CANNOT create or update PRs") + lines.append("- You CANNOT modify source files (src/, lib/, docs/, tests/)") + lines.append("") + + return "\n".join(lines) diff --git a/orchestrator/routes/pipelines/_prompt_reviewer.py b/orchestrator/routes/pipelines/_prompt_reviewer.py new file mode 100644 index 0000000000..b08398e1b0 --- /dev/null +++ b/orchestrator/routes/pipelines/_prompt_reviewer.py @@ -0,0 +1,593 @@ +"""reviewer-preparation prompt helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _build_agent_roster(all_roles: list[str], current_role: str, phase: str) -> str: + """Build a roster of all active agents for the current phase. + + Shows each agent's role, what they do, and what they produce so that + every agent understands who else is running and what to expect. + """ + roster_lines = ["### Active Agents in This Phase\n"] + roster_lines.append( + "The following agents are running **simultaneously**. " + "Each must complete their task AND reach CONFIRMED via BRC.\n" + ) + for role in all_roles: + desc, artifacts = _pkg._ROLE_DESCRIPTIONS.get( + role, ("Executes assigned role", "role-specific artifacts") + ) + marker = " **(you)**" if role == current_role else "" + roster_lines.append(f"- **{role}**{marker}: {desc}. Produces: {artifacts}.") + roster_lines.append("") + return "\n".join(roster_lines) + + +def _build_reviewer_preparation( + role_value: str, + phase: str, + *, + branch: str | None = None, + base_branch: str | None = None, +) -> str: + """Build proactive preparation instructions for reviewer agents. + + Tells reviewers what to do while waiting for proposals — e.g., reading + the contract, familiarizing themselves with the codebase, preparing + review criteria. This avoids idle waiting and produces better reviews. + + Args: + role_value: The reviewer role (e.g. ``reviewer_code``). + phase: Pipeline phase name. + branch: The pipeline's work branch, if any. + base_branch: The resolved base branch for diff/log commands. Falls + back to ``main`` when ``None``. + """ + base_ref = _pkg._resolve_origin_ref(base_branch) + + if phase == "implement": + if role_value == "reviewer_code": + return ( + "Start reviewing immediately — do not wait idle for proposals. " + "(a) Read the contract with `egg-contract show` to understand " + "what was planned. " + "(b) Review the issue/PR description for context. " + "(c) Check for commits on the branch: run " + f"`git fetch origin && git log --oneline {base_ref}..origin/{branch or '$(git branch --show-current)'}` " + "and if changes exist, begin reviewing the diff with " + f"`git diff {base_ref}...HEAD`. " + "(d) Note existing test patterns and code conventions. " + "By the time a proposal arrives, you should already have " + "a thorough understanding of the changes and be ready to " + "ACK or NACK with specific, detailed feedback. " + "When reviewing the tester's proposal, check whether tests were " + "actually executed (look for `tests_run` and `tests_execution_blocked` " + "in the attestation). If the tester reports `tests_execution_blocked: true`, " + "this is a blocking concern — NACK unless the limitation is clearly " + "documented and the tests are syntactically valid. " + "Also scrutinize low `tests_run` counts relative to change scope — " + "a multi-file change with only 1 test run warrants investigation. " + "If a producer has no work in this slice it submits a generic " + "no-op propose (`no_changes_needed=true`, #3027): the orchestrator " + "treats that as a non-blocking no-op and will not surface it to " + "you for review — there is nothing to ACK or NACK, and it does " + "not block consensus." + ) + elif role_value == "reviewer_code_holistic": + return ( + "Start preparing immediately — do not wait idle for proposals. " + "(a) Read the contract with `egg-contract show` to extract " + "the primary advertised use case (this is the path you will " + "walk end-to-end once the producer proposes). " + "(b) Review the issue / PR description and any doc files " + "the contract names — collect the doc-claimed behaviours " + "into a checklist for the symmetry pass. " + "(c) Identify the producer / consumer module pairs the plan " + "touches; these are where synthetic-key and silent-fallback " + "asymmetries hide. " + "(d) Once commits land " + f"(`git fetch origin && git log --oneline {base_ref}..origin/{branch or '$(git branch --show-current)'}`), " + f"skim `git diff {base_ref}...HEAD` once with the whole PR " + "in mind — do not verify line-by-line; defer that to " + "`reviewer_code`. Your job is the architectural-coherence " + "question line-by-line review does not own." + ) + elif role_value == "reviewer_contract": + return ( + "While waiting for proposals, prepare by: " + "(a) reading the contract with `egg-contract show` to understand " + "every task and its acceptance criteria, " + "(b) reviewing the issue description for original requirements, " + "(c) noting which tasks are marked as must-have vs nice-to-have. " + "When proposals arrive, you will verify each task's acceptance " + "criteria is met — prepare a checklist now." + ) + elif role_value == "tester": + return ( + "While waiting for the coder's proposal, prepare by: " + "(a) reading the contract with `egg-contract show` to understand " + "what's being implemented, " + "(b) identifying edge cases and boundary conditions from the " + "requirements, " + "(c) checking the existing test infrastructure (test frameworks, " + "fixtures, test utilities). " + "Start writing test scaffolding for known requirements while " + "waiting — you can finalize once you see the actual implementation." + ) + elif phase == "plan": + if role_value == "reviewer_plan": + return ( + "While waiting for proposals, prepare by: " + "(a) reading the issue description to understand the original " + "request, " + "(b) exploring the codebase to understand the current architecture " + "and components that may be affected, " + "(c) identifying potential risks or constraints the planners " + "should address. " + "Form your own mental model of how you would approach this — " + "then compare against the proposals when they arrive. " + "\n\n" + "**#2137 slice-DAG checks (mandatory):** " + "(1) **Forest-violation NACK** — if the contract was " + "rejected at plan ingestion with a " + "``forest_violation`` log discriminator (or the contract's " + "``plan_review_feedback`` carries a 'Plan ingestion REJECTED' " + "block), NACK the architect and cite the structured errors " + "verbatim. Instruct the architect to re-emit the slice " + "scaffold with ``serialized_chain_order`` populated on the " + "downstream slice. The SAME NACK applies to a " + "``slice_overlap_violation`` rejection (#3046 — a 'Plan " + "ingestion REJECTED: slices touch overlapping files' block): " + "two or more slices touch the same file with no dependency " + "ordering, so their branches fork independently off the shared " + "base and collide at integration. Instruct the architect to " + "serialise the overlapping cluster into one linear " + "``dependencies`` chain (or merge the slices) so each later " + "slice's branch is cut from the earlier one. " + "(2) **Slice-sizing NACK (hard, judgment-based — #2809)**: " + "slice composition is owned by the **architect**, not the " + "task_planner. You ARE empowered and required to hard-NACK " + "the architect on ``slice_size`` when a slice is oversized " + "for one BRC cycle. Use judgment — no fixed tasks-per-slice " + "or LOC budget. NACK when a slice bundles more than ~3 " + "distinct file-categories, combines deletion-heavy with " + "new-API-introduction work, would require >3–4 " + "commit-propose-revise cycles, or contains independent " + "task groups with no internal dependency. Name the seam in " + "your NACK so the architect's re-propose is actionable. " + "See criteria §11 for the full rubric and examples." + "\n\n" + "**Human-focused plan companion (the simplifier's " + "``*-plan-human.md``):** the simplifier produces a simplified, " + "plain-language companion to the plan for a **broad audience — " + "engineers, PMs, and managers**. You review it (CRITICAL). " + "**Read it side-by-side with the full plan** and ACK only when " + "it (a) faithfully captures the plan's essence, (b) is " + "materially lighter and more digestible than the full plan — " + "not a near-copy, (c) is readable by a non-engineer, and (d) " + "is free of egg-internal jargon (no " + "BRC/consensus/slice-DAG/contract/role terms). NACK the " + "**simplifier** (not the task_planner) if it misrepresents the " + "plan, leaks pipeline jargon, omits a material point, merely " + "duplicates the full plan, or — critically — reads as a " + "**review/critique** of the plan rather than a summary of it " + '(ACK/NACK language, "should commit to", "anti-pattern to ' + 'reject", constraint lists) or buries the reader in ' + "implementation detail (`file:line` refs, function/struct/field " + "names). A missing or empty companion is a NACK — the companion " + "is mandatory." + ) + elif phase == "refine": + if role_value in ("reviewer_refine", "reviewer_agent_design"): + base = ( + "While waiting for the refiner's proposal, prepare by: " + "(a) reading the prior review feedback that triggered this " + "refinement cycle, " + "(b) checking the current state of the code to understand " + "what was already implemented, " + "(c) verifying which review concerns are still outstanding. " + "When the proposal arrives, focus on whether the specific " + "feedback items were addressed." + ) + if role_value == "reviewer_refine": + base += ( + "\n\n" + "**Human-focused analysis companion (the simplifier's " + "``*-analysis-human.md``):** the simplifier produces a " + "simplified, plain-language companion to the analysis for a " + "**broad audience — engineers, PMs, and managers**. You " + "review it (CRITICAL). **Read it side-by-side with the full " + "analysis** and ACK only when it (a) faithfully captures the " + "analysis's essence, (b) is materially lighter and more " + "digestible than the full draft — not a near-copy, (c) is " + "readable by a non-engineer, and (d) is free of " + "egg-internal jargon. NACK the **simplifier** (not the " + "refiner) if it misrepresents the analysis, leaks pipeline " + "jargon, omits a material point, merely duplicates the full " + "draft, or — critically — reads as a **review/critique** of " + "the analysis rather than a summary of it (ACK/NACK " + 'language, "should commit to", "anti-pattern to reject", ' + "constraint lists) or buries the reader in implementation " + "detail (`file:line` refs, function/struct/field names). A " + "missing or empty companion is a NACK — it is mandatory." + ) + return base + if role_value == "first_principles_reviewer": + return ( + "While waiting for the refiner's proposal, prepare your " + "first-principles pass: (a) read the seed — `egg-contract " + "show` and the linked issue — and restate, in your own words, " + "the problem it claims to solve and why; (b) explore the " + "codebase to test that premise against reality (does the thing " + "already exist? is the problem already handled? is there a far " + "simpler path?); (c) form your own view of whether this is the " + "right direction and what a materially better one would be. " + "When the refiner proposes, you are checking the *premise and " + "direction*, not the analysis quality — surface any concrete " + "redirect as a phase-scoped HITL decision for the operator and " + "ACK the refiner. Never NACK on first-principles grounds." + ) + + # Generic fallback + return ( + "While waiting for proposals, read the contract " + "(`egg-contract show`), explore the codebase for context, " + "and prepare your review criteria. " + "Do NOT inspect producer artifacts before proposals arrive." + ) + + +def _re_review_priming_block( + *, + version: int | None = None, + delta_range: str | None = None, +) -> str: + """Adversarial re-prime injected at the moment of every re-review. + + Counter-anchors the persistent reviewer against the "verify named + blockers got fixed" framing that long-lived context naturally + biases toward (see #2724 post-mortem: slice-1 v2 was ACK'd despite + the v2 delta introducing a non-executable inline `python3 -c` + snippet that a downstream GitHub-bot reviewer caught immediately). + + Three design choices worth flagging: + + - **Delta-scoped, not exploration-forcing.** The block tells the + reviewer to re-read *the delta since their own last review* + adversarially, not to re-traverse the codebase. The amortized + exploration from cycle-1 is the feature; re-Reading every + referenced file on every cycle would throw away BRC's cost + advantage. + - **Per-reviewer delta, not a fixed version pair (#2887).** The + block was originally hardcoded to the v1→v2 transition and took + no arguments, yet was appended verbatim to every re-review (v3, + v4, …). On N>2 cycles the stale "audit the v2 delta as a fresh + reviewer, ignore your v1 NACK history" prose read as "re-audit + the whole accumulated surface," widening scope each cycle and + blocking multi-round convergence. The block is now parameterized + by the current proposal version (``vN`` / its prior ``v(N-1)``) + and, on per-reviewer ``CONSENSUS_RE_REVIEW`` notices, anchored to + that reviewer's own ``<last_reviewed_sha>..HEAD`` ``delta_range`` + (resolved orchestrator-side from the reviewer's last-verdicted + version). When ``delta_range`` is absent (the broadcast + ``CONSENSUS_PROPOSE`` body, ``to_role=all`` — one text for + reviewers sitting at different last-reviewed versions) the block + references the reviewer-self-tracked range from REVIEWER-SYNC.md + (``git log {last_reviewed_commit}..HEAD --not origin/{base} -p``) + instead. + - **Economic framing is explicit.** "Re-reviews are cheap / NACK + without hesitance" is load-bearing — without it, persistent + reviewers naturally optimize for convergence (ACK to end the + cycle) over rigor. The orchestrator absorbs the cost of extra + cycles; the reviewer should not be carrying it. + + The block is appended to ``CONSENSUS_RE_REVIEW`` message bodies + (signals.py, both withdrawal/re-propose and push-after-propose + paths) and to ``CONSENSUS_PROPOSE`` bodies when the producer is + re-proposing (version > 1, ``changed_artifacts`` set). Reviewers + who NACK'd the prior version receive ``CONSENSUS_PROPOSE`` rather + than ``CONSENSUS_RE_REVIEW`` on a re-propose, so both surfaces need + the re-prime to reach every reviewer. + + Args: + version: The current (re-proposed) proposal version ``N``. When + ``None`` (legacy / defensive callers) the block falls back + to generic "current" / "prior" wording without numbered + anchors. + delta_range: A concrete ``<sha>..HEAD`` git range scoping this + reviewer's mandate-2 audit to the commits landed since their + own last verdict. Only available on the per-reviewer + ``CONSENSUS_RE_REVIEW`` path; omitted on the broadcast + ``CONSENSUS_PROPOSE`` body. + """ + # Adjective placed before "review"/"verdict" ("Your v6 review" / + # "Your current review"); and the prior-version qualifier placed + # before "blockers"/"NACK history" ("named v5 blockers" / "named + # prior blockers"). Both read naturally with or without a version. + vN = f"v{version}" if version is not None else "current" + vNm1 = f"v{version - 1}" if version is not None and version >= 2 else "prior" + # Mandate-2's delta anchor. On the per-reviewer path we have an + # authoritative range; on the broadcast path we point at the + # reviewer-self-tracked range REVIEWER-SYNC.md already defines, so + # each reviewer scopes to the commits since *their* last review + # rather than the whole accumulated surface. + if delta_range: + delta_clause = ( + f"the delta since your last review (`git log {delta_range} " + "--not origin/<base> -p` — the commits landed since the " + "version you last verdicted)" + ) + delta_short = f"this delta (`{delta_range}`)" + else: + # NOTE: `{last_reviewed_commit}` and `{base_branch}` here are + # *literal* braces, deliberately matching the placeholder names + # the reviewer agent already learned from REVIEWER-SYNC.md + # (shared/prompts/REVIEWER-SYNC.md:110) — the agent substitutes + # them at read-time from its own bookkeeping. Do NOT convert this + # string to an f-string: there are no Python locals named + # `last_reviewed_commit` / `base_branch` here, so f-stringifying + # would raise `NameError` at call time. The per-reviewer branch + # above uses `<base>` instead because that path embeds a + # concrete, orchestrator-resolved range — only `<base>` remains + # for the reviewer to fill in, so the angle-bracket convention + # makes the (already-resolved vs. still-to-resolve) distinction + # visible at a glance. + delta_clause = ( + "the delta since your last review (per REVIEWER-SYNC.md: " + "`git log {last_reviewed_commit}..HEAD --not " + "origin/{base_branch} -p` — the commits landed since the " + "version you last verdicted, NOT the whole accumulated " + "proposal surface)" + ) + delta_short = "this delta (the commits since your last review)" + return ( + "\n\n**Adversarial re-review**\n\n" + f"**Your {vN} review has TWO equal-weight mandates:**\n\n" + f"1. **Verify named {vNm1} blockers were addressed** — confirm " + "the producer fixed what you NACK'd.\n" + f"2. **Audit {delta_clause} as a fresh reviewer** — ignore your " + f"{vNm1} NACK history. Read that diff as if you'd never seen the " + "prior version. Apply your lens (security threat-model, " + "concurrency races, contract AC, line-by-line bugs, " + "silent-fallback shapes — whichever your role owns) to the " + "delta itself, not to whether your previous concerns were " + "satisfied. **Mandate 2 is bounded to this delta** — it does " + "NOT ask you to re-traverse the whole accumulated surface from " + "earlier cycles; that work was amortized when you first " + "reviewed those commits.\n\n" + "Both mandates have equal weight. If (1) passes but (2) finds new " + "issues, you NACK. ACK requires both pass.\n\n" + "**The named-blockers anchor is a known trap. Every reviewer " + "lens has a mandate-2 in its own territory** — security has " + "newly-introduced threat surfaces, concurrency has newly-" + "introduced races, contract has newly-introduced AC drift, code " + "has newly-introduced line-by-line bugs. The four issues that " + "escaped PR #2724 to the GitHub bot were all of code-lens shape " + "(`${ANSWER}` as bare Python, deprecated `datetime.utcnow()`, " + "non-atomic write, bare `except: pass`) — the persistent " + 'reviewer correctly answered mandate 1 ("did prior issues get ' + 'fixed? yes") and skipped mandate 2 ("does this delta introduce ' + 'new issues? actually yes"). The shape generalizes: whatever ' + "your lens, this delta can introduce issues your prior NACK " + "didn't name. Watching the producer deliver a targeted fix " + 'pulls strongly toward "verify my fix-request landed → ACK." ' + "Recognize the pull and do mandate 2 anyway.\n\n" + "**How to execute mandate 2:**\n\n" + "- Read each new hunk as an operator who's about to copy-paste / " + "run / integrate it. Would this code execute as written? Would " + "these docs send a copy-paster down a working path?\n" + "- Apply every rubric pass to the new hunks. New issues outside " + "the scope of your prior NACK are blocking; your prior NACK does " + "not bound this re-review.\n" + "- **Fresh-reviewer simulation.** Before issuing your " + f"{vN} verdict, ask: would a reviewer who has only seen " + f"{delta_short} with no NACK history ACK this? If you can't " + "argue yes from that diff alone, NACK.\n" + "- **External-bot anchor.** Imagine `egg-reviewer[bot]` reads " + f"only {delta_short} with no NACK context. What would it flag? " + "Anything it'd flag, you should NACK first.\n\n" + f"**Your {vN} verdict must enumerate both halves** so mandate 2 " + "doesn't silently disappear from the record:\n\n" + f"- (a) Which {vNm1} blockers you verified-fixed (mandate 1).\n" + "- (b) What new issues you audited-and-did-not-find (mandate 2). " + 'Name the specific shapes you checked — not "reviewed thoroughly," ' + 'but "checked for silent fallbacks, doc-snippet executability, ' + "API-deprecation, atomicity of file writes.\" If you can't " + "enumerate (b), you haven't done mandate 2.\n\n" + "**Re-reviews are cheap by design.** Your amortized context means " + 'the work is "read the delta, apply your rubric, decide" — ' + "minutes, not hours. NACK without hesitance; the orchestrator " + "absorbs cycles. Two NACKs on the same producer where the second " + "names new findings is the correct trajectory, not " + "goalpost-moving. The downstream GitHub reviewer should find " + "nothing in this delta. Anything it catches that lives in this " + "cycle's diff is a miss attributable to this re-review." + ) + + +def _build_producer_orientation( + role_value: str, + phase: str, + reviewers: list[str], + branch: str | None = None, +) -> str: + """Build orientation instructions for producer agents. + + Tells producers what to research before starting work — understanding + context, knowing what reviewers will check, and checking existing code + patterns. This produces higher-quality first proposals and fewer NACKs. + + A producer that orients and finds it has no work in this slice takes the + generic no-op propose path described in the Producer Lifecycle (#3027) — + no special orientation text is needed. + + Args: + role_value: Producer role (e.g. ``coder``). + phase: Pipeline phase name. + reviewers: Names of reviewers that will review this producer. + branch: The pipeline's working branch, used for sync instructions. + """ + reviewer_awareness = "" + if reviewers: + reviewer_names = ", ".join(reviewers) + reviewer_awareness = ( + f" Your work will be reviewed by **{reviewer_names}** — " + "keep their review criteria in mind as you work." + ) + + # The simplifier runs in both the refine and plan phases as a PRODUCER + # ONLY (the human-focused companion). It carries an advisory review edge + # over the upstream producer purely as the event-pump wake-wire — that is + # what re-invokes it on the upstream's PROPOSE — but it issues no verdict + # and never reviews the draft (#3381). Its work depends on the upstream + # producer's draft existing, so — like the implement-phase tester — it + # orients up-front and starts producing only once the upstream proposes. + if role_value == "simplifier": + if phase == "plan": + upstream, draft_desc = "task_planner", "the implementation plan" + else: # refine + upstream, draft_desc = "refiner", "the refine analysis" + sync_note = "" + if branch: + sync_note = ( + f" When re-invoked on the PROPOSE, sync your worktree first: " + f"`git fetch origin && git merge origin/{branch} --no-edit`." + ) + return ( + f"your WORK depends on **{upstream}**'s draft of {draft_desc} " + "existing — do NOT write your companion before it is pushed. ORIENT " + "now (read the contract and the issue/task description so you " + "understand the subject), then exit; the event pump re-invokes you " + f"when **{upstream}** issues `CONSENSUS_PROPOSE`, carrying that " + "proposal in your event payload. On that invocation: read the " + "upstream draft, then write a simplified, higher-level companion " + "that captures its essence in plain, jargon-free language for a " + "broad audience (engineers, PMs, and managers) — a summary, NOT a " + "review of the draft — and PROPOSE it. That is the whole job: you " + f"do NOT review **{upstream}**'s draft and you issue no ACK or NACK " + "on it." + sync_note + reviewer_awareness + ) + + if phase == "implement": + if role_value == "coder": + return ( + "read the contract (`egg-contract show`) to understand all tasks " + "and acceptance criteria. Explore the codebase to find existing " + "patterns, conventions, and the files you will modify. Check for " + "existing tests that cover the areas you will change — do not " + "break them." + reviewer_awareness + ) + elif role_value == "tester": + sync_note = "" + if branch: + sync_note = ( + f" Before starting work, sync your worktree: " + f"`git fetch origin && git merge origin/{branch} --no-edit`." + ) + return ( + "read the contract (`egg-contract show`) to understand what is " + "being implemented. Check the existing test infrastructure — " + "test frameworks, fixtures, conftest files, and naming conventions. " + "Identify edge cases from the requirements before writing tests. " + "**Your mandate is two-fold**: comprehensive regression " + "coverage AND adversarial probing for bugs the coder missed " + "— see the *Your Task* → mandate block for the full " + "instruction (including the failing-test → NACK → HANDOFF " + "workflow when you catch a coder-side bug). " + "**Scaffold-first while the coder is producing**: draft test " + "scaffolding from the plan alone — test file paths from " + "`tasks[].files`, function signatures from each task's acceptance " + "criteria, fixture imports, and mock-input scenarios from the YAML. " + "Leave assertion bodies as TODOs. Do NOT call `wait-loop` for the " + "coder's CONSENSUS_PROPOSE before drafting these scaffolds — the " + "scaffold work does not depend on coder output and recovers " + "downstream-producer time. Your propose-ready iteration should " + "start at the coder's first commit, not their first propose. " + "**You MUST propose** even when the slice warrants no new tests " + "(pure refactor / doc-only / symbol moves with no behavior " + "change): the BRC consensus blocks until every producer has " + "proposed. For that case, submit a generic no-op propose " + "(#3027) — `egg-orch consensus propose --no-changes-needed " + "--no-changes-reason '<why: e.g. pure refactor, existing tests " + "cover>'`. It is accepted as a non-blocking no-op (reviewers do " + "not review or NACK it). Do NOT just heartbeat indefinitely " + "waiting for test work that isn't there — that deadlocks the " + "slice." + sync_note + reviewer_awareness + ) + elif role_value == "documenter": + sync_note = "" + if branch: + sync_note = ( + f" Before starting work, sync your worktree: " + f"`git fetch origin && git merge origin/{branch} --no-edit`." + ) + return ( + "read the contract (`egg-contract show`) to understand what is " + "being implemented. Check existing documentation structure — " + "README files, doc directories, inline documentation patterns. " + "Identify which docs describe the surfaces this work touches, so " + "you can fold the resulting state into them as a snapshot of " + "current behavior once the implementation is complete. " + "**You MUST propose** even when the slice warrants no doc " + "updates (pure refactor / test-only / internal-only with no " + "documented-surface impact): the BRC consensus blocks until " + "every producer has proposed. For that case, submit a generic " + "no-op propose (#3027) — `egg-orch consensus propose " + "--no-changes-needed --no-changes-reason '<why: e.g. no " + "documented surface impacted by the coder's diff>'`. It is " + "accepted as a non-blocking no-op (reviewers do not review or " + "NACK it). Do NOT just heartbeat indefinitely waiting for doc " + "work that isn't there — that deadlocks the slice." + sync_note + reviewer_awareness + ) + elif phase == "plan": + if role_value == "architect": + return ( + "read the issue/task description carefully. Explore the codebase " + "to understand the current architecture, component boundaries, " + "and dependencies. Identify the areas that will be affected by " + "the proposed changes." + reviewer_awareness + ) + elif role_value == "task_planner": + return ( + "read the issue/task description carefully. Review the codebase " + "structure to understand the scope of work. Break the work into " + "tasks with clear acceptance criteria that reviewers can verify." + + reviewer_awareness + ) + elif role_value == "risk_analyst": + return ( + "read the issue/task description carefully. Research the affected " + "areas of the codebase for potential risks — security, " + "performance, backwards compatibility, and third-party " + "dependencies." + reviewer_awareness + ) + elif phase == "refine": + if role_value == "refiner": + return ( + "read the prior review feedback carefully. Understand exactly " + "what concerns were raised and what changes are expected. Check " + "the current state of the code before making modifications. " + "When the draft you are refining is an analysis or plan, " + "surface every runtime-primitive assumption explicitly at the " + "phase_gate (see #2594) — name each class, function, route, " + "env var, ConfigMap key, fixture, CLI flag, or decorator the " + "downstream plan will depend on, with `file:line` evidence " + "and execution-context scope (in-sandbox-agent vs " + "trusted-CI-runner vs human-operator). This makes the " + "plan-phase Primitive-Existence and Trust-Boundary audits " + "cheap." + reviewer_awareness + ) + + # Generic fallback + return ( + "read the contract (`egg-contract show`) and explore the codebase " + "to understand context, patterns, and conventions before starting." + reviewer_awareness + ) diff --git a/orchestrator/routes/pipelines/_resolve.py b/orchestrator/routes/pipelines/_resolve.py new file mode 100644 index 0000000000..bdaa1b4a29 --- /dev/null +++ b/orchestrator/routes/pipelines/_resolve.py @@ -0,0 +1,196 @@ +"""pipeline resolution + identifiers + event emit helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _ensure_pipeline_work_ref(branch: str | None) -> str | None: + """Return the actual remote ref for an orchestrator-managed pipeline branch. + + The orchestrator pushes the pipeline tip to ``<branch>/work`` so the + ``<branch>/`` namespace can hold slice integration branches as + siblings (``<branch>/slice-N``) without git's ``directory file + conflict`` rejection — see #2399. A leaf ref at ``<branch>`` and a + child at ``<branch>/slice-N`` cannot coexist on origin, so the + pipeline tip is moved one level deeper into the namespace. + + Idempotent and bounded to ``egg/<id>``-shaped branches: + + * ``None`` → ``None`` (prompt-driven; the caller generates a + ``/work``-shaped branch later). + * ``egg/<id>`` → ``egg/<id>/work`` (issue submissions). + * ``egg/<id>/work`` → unchanged (resubmission, internal callers). + * non-``egg/`` (passed unchanged) — a pipeline pointed at a foreign + branch (e.g. ``feature/foo``). Slices on a non-``egg/`` branch are + not a guaranteed-safe shape and are intentionally not normalised + here — the conflict would resurface at the slice push and is + tracked separately. + + The trailing-``/work`` check is structural rather than a plain + suffix match (``branch.count("/") >= 2 and branch.rsplit("/", 1)[1] + == "work"``) so a degenerate input like ``egg/work`` — a single + segment that *happens* to end in ``/work`` — gets normalised to + ``egg/work/work`` (siblings ``egg/work/slice-N``) rather than + treated as already-normalised. Trailing slashes are stripped first + so ``egg/`` does not collapse to a double-slash ``egg//work``. + """ + if branch is None: + return None + branch = branch.rstrip("/") + if not branch.startswith("egg/"): + return branch + # Structural check: only treat ``egg/<id>/work`` (≥2 slashes, last + # segment is ``work``) as already-normalised. ``egg/work`` looks + # like a suffix match but is a single-segment id and still needs the + # ``/work`` namespace deepening. + if branch.count("/") >= 2 and branch.rsplit("/", 1)[1] == "work": + return branch + return f"{branch}/work" + + +def _slice_namespace_root(pipeline_branch: str) -> str: + """Return the slice-integration-branch namespace root for a pipeline branch. + + Slice integration branches live as siblings of the pipeline tip + under ``egg/<id>/`` (see :func:`_ensure_pipeline_work_ref`). The + namespace root is the pipeline branch with the trailing ``/work`` + stripped — that's the prefix slice paths (``<root>/slice-N``) are + built from. For legacy / non-normalised branches that do not end in + ``/work``, the branch itself is the root. + + The trailing-``/work`` check mirrors the structural check in + :func:`_ensure_pipeline_work_ref` (≥2 slashes, last segment is + ``work``) so a degenerate single-segment input like ``egg/work`` + is treated as the root itself rather than collapsing to ``egg``. + """ + if pipeline_branch.count("/") >= 2 and pipeline_branch.rsplit("/", 1)[1] == "work": + return pipeline_branch.rsplit("/", 1)[0] + return pipeline_branch + + +def _pipeline_identifier( + issue_number: int | None, + pipeline_id: str, +) -> int | str: + """Derive the pipeline identifier used for namespaced .egg-state filenames. + + Prefers ``issue_number`` when available, falling back to ``pipeline_id``. + + A pipeline whose id carries a qualifier beyond the bare ``issue-<N>`` + form (e.g. ``issue-1557-v2`` for a versioned re-run) keys by + ``pipeline_id`` instead, so concurrent pipelines on the same issue + don't collide on ``.egg-state/drafts/<N>-analysis.md``. + """ + if pipeline_id and issue_number is not None: + expected_issue_prefix = f"issue-{issue_number}" + if pipeline_id.startswith(expected_issue_prefix + "-"): + # A qualifier is present beyond the bare ``issue-<N>`` form; + # key by pipeline_id so concurrent runs on the same issue do + # not collide on draft files. + return pipeline_id + return issue_number if issue_number is not None else pipeline_id + + +def _brc_history_identifier(pipeline) -> int | str: + """Return the identifier used to namespace BRC-history artifacts. + + Mirrors :func:`_pipeline_identifier` (favouring the issue number). + """ + return _pkg._pipeline_identifier( + getattr(pipeline, "issue_number", None), + getattr(pipeline, "id", "") or "", + ) + + +def _emit_pipeline_event( + pipeline: _pkg.Pipeline, + event_type_str: str, +) -> None: + """Emit a pipeline event to the EventBus for SSE streaming.""" + if _pkg._emit_event is None: + return + mapped = _pkg._EVENT_TYPE_MAP.get(event_type_str) + if mapped is None: + return + _pkg._emit_event( + mapped, + pipeline.id, + data={ + "status": pipeline.status.value, + "phase": pipeline.current_phase.value, + }, + ) + + +def _resolve_pipeline( + pipeline_id: str, base_path: _pkg.Path +) -> tuple[_pkg.StateStore, _pkg.Pipeline]: + """Load a pipeline, resolving the correct repo subdirectory. + + Each repo has its own state store and worktree. This function + searches all repos under ``base_path`` to find the pipeline. + + Returns: + (store, pipeline) tuple + + Raises: + PipelineNotFoundError: if the pipeline cannot be found anywhere + InvalidPipelineIdError: if the ID format is invalid + GitOperationError: if the state-store worktree cannot be loaded + (e.g. ``git worktree add`` contention). Callers should + surface this as 500, not 404 — it is recoverable + infrastructure failure, not a missing pipeline. + """ + from state_store import discover_repo_paths + + for repo_path in discover_repo_paths(base_path): + try: + store = _pkg.get_state_store(repo_path) + pipeline = store.load_pipeline(pipeline_id) + return store, pipeline + except _pkg.PipelineNotFoundError: + continue + # NOTE: do NOT broaden this to ``StateStoreError``. Swallowing + # ``GitOperationError`` here re-raised every state-store wedge + # as ``Pipeline not found`` and surfaced to operators as 404, + # masking a recoverable git contention as a missing pipeline + # (#2167). Let infrastructure failures propagate so the route + # can return 500 with the actual error. + + raise _pkg.PipelineNotFoundError(f"Pipeline {pipeline_id} not found") from None + + +def _collect_all_pipelines(base_path: _pkg.Path) -> list: + """Collect pipelines from all git repos under base_path. + + Each repo has its own state store and worktree. Pipelines are + deduplicated by ID in case of overlapping stores. + """ + from state_store import discover_repo_paths + + seen: set[str] = set() + pipelines = [] + + def _add_from_store(store): + for pid in store.list_pipelines(): + if pid in seen: + continue + try: + pipelines.append(store.load_pipeline(pid)) + seen.add(pid) + except _pkg.StateStoreError: + continue + + for repo_path in discover_repo_paths(base_path): + try: + _add_from_store(_pkg.get_state_store(repo_path)) + except _pkg.StateStoreError: + continue + + return pipelines diff --git a/orchestrator/routes/pipelines/_reviews.py b/orchestrator/routes/pipelines/_reviews.py new file mode 100644 index 0000000000..3877869e3c --- /dev/null +++ b/orchestrator/routes/pipelines/_reviews.py @@ -0,0 +1,186 @@ +"""reviews helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim; patched/barrel-resident globals reached via _pkg so +patch("routes.pipelines.<name>") keeps intercepting. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import routes.pipelines as _pkg # noqa: E402,F401 +from models import AggregatedReviewResult, ReviewVerdict + +from ._drafts import _verdict_path_for_type + + +def _read_review_verdict( + repo_path: Path, + phase: str, + reviewer_type: str = "code", + pipeline_mode: str = "issue", + issue_number: int | None = None, + pipeline_id: str | None = None, +) -> ReviewVerdict | None: + """Read a typed review verdict JSON from the repo. + + Returns None if the file is missing or malformed (treated as approved + for graceful degradation). + """ + verdict_rel = _verdict_path_for_type( + phase, + reviewer_type, + issue_number=issue_number, + pipeline_id=pipeline_id, + ) + verdict_file = repo_path / verdict_rel + + if not verdict_file.exists(): + _pkg.logger.warning( + "Verdict file not found, treating as approved", + path=str(verdict_file), + reviewer_type=reviewer_type, + ) + return None + + try: + raw = verdict_file.read_text() + data = json.loads(raw) + return ReviewVerdict(**data) + except (json.JSONDecodeError, Exception) as e: + _pkg.logger.warning( + "Failed to parse verdict file, treating as approved", + path=str(verdict_file), + reviewer_type=reviewer_type, + error=str(e), + ) + return None + + +def _read_tester_gaps( + repo_path: Path, + identifier: int | str | None = None, +) -> str | None: + """Read tester output and extract gap findings for feedback to the coder. + + Reads `.egg-state/agent-outputs/{identifier}-tester-output.json` (with + fallback to `tester-output.json`) and formats any test failures and gaps + found into a summary string. + + Falls back to scanning the `summary` field for failure keywords when + `gaps_found` is not present (backwards compat with old tester outputs). + + Args: + repo_path: Path to the repository. + identifier: Pipeline/issue identifier for namespaced filenames. + + Returns: + Formatted gap summary string, or None if no gaps found. + """ + outputs_dir = repo_path / ".egg-state" / "agent-outputs" + + # Try prefixed filename first, fall back to old global filename + tester_output_file = None + if identifier is not None: + prefixed = outputs_dir / f"{identifier}-tester-output.json" + if prefixed.exists(): + tester_output_file = prefixed + if tester_output_file is None: + tester_output_file = outputs_dir / "tester-output.json" + + if not tester_output_file.exists(): + return None + + try: + raw = tester_output_file.read_text() + data = json.loads(raw) + except (json.JSONDecodeError, OSError) as e: + _pkg.logger.warning( + "Failed to parse tester output file", + path=str(tester_output_file), + error=str(e), + ) + return None + + if not isinstance(data, dict): + return None + + sections: list[str] = [] + + tests_failed = data.get("tests_failed", 0) + if tests_failed: + sections.append(f"- **{tests_failed}** test(s) failed") + + gaps_found = data.get("gaps_found") + if gaps_found and isinstance(gaps_found, list): + # Cap at 10 gaps to avoid prompt bloat + capped = gaps_found[:10] + for gap in capped: + gap_str = str(gap)[:200] + sections.append(f"- {gap_str}") + if len(gaps_found) > 10: + sections.append(f"- ... and {len(gaps_found) - 10} more gaps") + elif not tests_failed: + # Backwards compat: scan summary for failure keywords + summary = data.get("summary", "") + if isinstance(summary, str) and any( + kw in summary.lower() for kw in ("fail", "gap", "missing", "error", "deficien") + ): + sections.append(f"- Tester summary: {summary}") + + if not sections: + return None + + return f"{_pkg.TESTER_FINDINGS_HEADER}\n" + "\n".join(sections) + + +def _aggregate_review_verdicts( + verdicts: dict[str, ReviewVerdict | None], +) -> AggregatedReviewResult: + """Aggregate multiple typed review verdicts into an overall result. + + Returns: + AggregatedReviewResult with: + - verdict: "approved" or "needs_revision" (any needs_revision → overall needs_revision) + - blocking_feedback: combined feedback from needs_revision verdicts only + - advisory_content: analysis and suggestions from ALL verdicts (including approved) + + Missing/None verdicts are skipped. + """ + overall = "approved" + feedback_sections: list[str] = [] + advisory_sections: list[str] = [] + + for reviewer_type, verdict in verdicts.items(): + if verdict is None: + continue + + # Collect blocking feedback from needs_revision verdicts + if verdict.verdict == "needs_revision": + overall = "needs_revision" + section = f"### {reviewer_type} reviewer\n" + if verdict.feedback: + section += verdict.feedback + elif verdict.summary: + section += verdict.summary + feedback_sections.append(section) + + # Collect analysis and suggestions from ALL verdicts (including approved) + advisory_parts: list[str] = [] + if verdict.analysis: + advisory_parts.append(verdict.analysis) + if verdict.suggestions: + advisory_parts.append(f"**Suggestions:** {verdict.suggestions}") + if advisory_parts: + advisory_sections.append( + f"### {reviewer_type} reviewer\n" + "\n\n".join(advisory_parts) + ) + + blocking_feedback = "\n\n".join(feedback_sections) if feedback_sections else "" + advisory_content = "\n\n".join(advisory_sections) if advisory_sections else "" + return AggregatedReviewResult( + verdict=overall, + blocking_feedback=blocking_feedback, + advisory_content=advisory_content, + ) diff --git a/orchestrator/routes/pipelines/_routes_crud.py b/orchestrator/routes/pipelines/_routes_crud.py new file mode 100644 index 0000000000..c113698d1b --- /dev/null +++ b/orchestrator/routes/pipelines/_routes_crud.py @@ -0,0 +1,1047 @@ +"""CRUD-route bodies helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +from typing import Any + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _create_pipeline_body() -> tuple[_pkg.Response, int]: + """ + Create a new pipeline. + + Request body: + { + "issue_number": 123, + "repo": "owner/name", + "branch": "egg/issue-123", + "config": {...} // optional + } + + Response: + { + "success": true, + "message": "Pipeline created", + "data": { + "pipeline": {...} + } + } + """ + data = _pkg.request.get_json() + if data is None: + return _pkg.make_error_response("Missing request body") + if not isinstance(data, dict): + return _pkg.make_error_response("Request body must be a JSON object") + + network_mode = data.get("network_mode") + if network_mode is not None and network_mode not in ("public", "private"): + return _pkg.make_error_response( + f"Invalid network_mode: {network_mode!r} (must be 'public' or 'private')" + ) + + issue_number = data.get("issue_number") + repo = data.get("repo") + branch = data.get("branch") + base_branch = data.get("base_branch") + prompt = data.get("prompt") + + # #3393 (multi-repo): a submission may carry a ``repos`` list instead of + # (or in addition to) the single ``repo``. Normalize it up front and derive + # the primary onto the legacy ``repo``/``base_branch`` scalars so the + # single-repo plumbing below (naming, base-branch detection, branch checks) + # keeps working and a direct HTTP submission — one that bypasses the + # submit_task MCP tool that would otherwise mirror the primary — is + # supported. ``repos_entries`` is None for a single-repo submission. + repos_entries: list[dict[str, str | None]] | None = None + repos_arg = data.get("repos") + if repos_arg is not None: + _repos_err, repos_entries, _primary_repo, _primary_base = _pkg._normalize_submission_repos( + repos_arg + ) + if _repos_err: + return _pkg.make_error_response( + _repos_err, status_code=400, details={"reason": "invalid_repos"} + ) + if repo and _primary_repo and repo != _primary_repo: + return _pkg.make_error_response( + f"Conflicting repo {repo!r} and repos primary {_primary_repo!r}; " + "pass one or the other.", + status_code=400, + details={"reason": "repo_repos_conflict"}, + ) + if not repo: + repo = _primary_repo + if not base_branch: + base_branch = _primary_base + mode = data.get("mode", "issue") + analysis = data.get("analysis") + plan = data.get("plan") + source_branch = data.get("source_branch") + if source_branch is not None: + if not _pkg.re.match(r"^[a-zA-Z0-9_./-]+$", source_branch) or ".." in source_branch: + return _pkg.make_error_response( + f"Invalid source_branch: {source_branch!r}", + status_code=400, + ) + source_artifact_prefix = data.get("source_artifact_prefix") + if source_artifact_prefix is not None: + if not _pkg.re.match(r"^[a-zA-Z0-9_.-]+$", source_artifact_prefix): + return _pkg.make_error_response( + f"Invalid source_artifact_prefix: {source_artifact_prefix!r}", + status_code=400, + ) + + # Issue #1557: Jira-epic SDLC parameters. ``jira_ticket`` is the + # Atlassian key; ``epic_mode`` is the operator's override + # (``'auto' | 'fresh' | 'reassess'``). The MCP submit_task tool + # normalises ``jira_ticket`` to upper-case before forwarding. + jira_ticket_arg = data.get("jira_ticket") + epic_mode_arg = data.get("epic_mode") + if jira_ticket_arg is not None: + if not isinstance(jira_ticket_arg, str) or not _pkg.re.fullmatch( + r"[A-Z][A-Z0-9_]*-\d+", jira_ticket_arg + ): + return _pkg.make_error_response( + f"Invalid jira_ticket: {jira_ticket_arg!r} (expected <PROJECT>-<number>)", + status_code=400, + details={"reason": "invalid_jira_ticket"}, + ) + if epic_mode_arg is not None: + if epic_mode_arg not in ("auto", "fresh", "reassess"): + return _pkg.make_error_response( + f"Invalid epic_mode: {epic_mode_arg!r} (must be 'auto' / 'fresh' / 'reassess')", + status_code=400, + details={"reason": "invalid_epic_mode"}, + ) + if not jira_ticket_arg: + return _pkg.make_error_response( + "epic_mode requires jira_ticket", + status_code=400, + details={"reason": "epic_mode_without_ticket"}, + ) + + # Validate mode + valid_modes = {m.value for m in _pkg.PipelineMode} + if mode not in valid_modes: + return _pkg.make_error_response( + f"Invalid mode: {mode!r} (must be one of {sorted(valid_modes)})" + ) + + if not repo: + return _pkg.make_error_response("Missing repo") + + # Repo format sanity check — a lightweight shell-metacharacter guard. + # The repo_config allowlist (repositories.yaml) is enforced gateway-side. + if not _pkg.re.match(r"^[a-zA-Z0-9._-]+/[a-zA-Z0-9._-]+$", repo): + return _pkg.make_error_response( + f"Invalid repo format: {repo!r} (expected owner/name)", + status_code=400, + details={"reason": "repo_not_allowed"}, + ) + + # Validate branch and base_branch — reject values that could be + # interpreted as git flags (e.g. "--upload-pack=...") or contain + # path-traversal sequences. Same regex used for source_branch above. + for _ref_name, _ref_val in [("branch", branch), ("base_branch", base_branch)]: + if _ref_val is not None: + if not _pkg.re.match(r"^[a-zA-Z0-9_./-]+$", _ref_val) or ".." in _ref_val: + return _pkg.make_error_response( + f"Invalid {_ref_name}: {_ref_val!r}", + status_code=400, + ) + + # Issue-driven or explicitly-named pipelines require a branch; + # prompt-driven ones do not. + pipeline_id = data.get("pipeline_id") + + if (issue_number or pipeline_id) and not branch: + return _pkg.make_error_response("Missing branch") + + # #2399 — push the pipeline tip to ``<branch>/work`` so slice + # integration branches at ``<branch>/slice-N`` can coexist as + # siblings under the same namespace (git rejects a leaf ref and + # children of that ref's path with ``directory file conflict``). + branch = _pkg._ensure_pipeline_work_ref(branch) + + # Wait for the gateway to be ready before any gateway-dependent work. + # On fresh deploys / pod restarts the orchestrator can accept requests + # while the gateway HTTP listener is still coming up; without this gate + # the first submission proceeds, hits the gateway during pipeline-level + # worktree creation or per-agent fan-out, and surfaces as a cascade of + # generic per-agent ConnectionRefused / "Remote end closed connection" + # errors that operators have to reverse-engineer. See #1851. + try: + _ready_timeout = int(_pkg.os.environ.get("EGG_GATEWAY_READY_TIMEOUT_SECONDS", "60")) + except ValueError: + _ready_timeout = 60 + _ready_timeout = max(0, _ready_timeout) + if _ready_timeout > 0: + _gw_ready = _pkg.get_gateway_client() + if not _gw_ready.wait_for_healthy(timeout_seconds=_ready_timeout): + _last = _gw_ready.check_health() + _resp, _status = _pkg.make_error_response( + f"Gateway not ready after {_ready_timeout}s " + f"(status={_last.status}): {_last.error or 'unhealthy'}. " + "Retry once the gateway has finished starting up.", + status_code=503, + details={ + "reason": "gateway_not_ready", + "gateway_status": _last.status, + "gateway_error": _last.error, + "timeout_seconds": _ready_timeout, + }, + ) + _resp.headers["Retry-After"] = str(_ready_timeout) + return _resp, _status + + repo_path = _pkg.get_repo_path() + + # #3038: resolve the repo's default branch ONCE at submit time and + # persist it on the pipeline record, so every downstream consumer + # (the context-PR opener, the restart/spawn paths, the gateway + # ``register_session`` base, the spawner ``EGG_BASE_BRANCH`` export) + # reads a concrete base off the record instead of re-deriving it on + # every invocation. Re-deriving each time opened a narrow race the + # #3035 reviewer flagged: a single flaky ``git symbolic-ref + # origin/HEAD`` read drops the opener into the ``origin/main → + # origin/master → "main"`` fallback chain, which can pick the wrong + # default on a ``master`` repo and 422 a second ``create_pr``. + # Persisting closes the race because the consumers' ``base_branch or + # _detect_default_branch(...)`` short-circuits on the stored value and + # never reaches the subprocess. ``_detect_default_branch`` is the + # local/fast helper (``git symbolic-ref``) and is the same resolution + # the stale-branch reuse check below already performs. + # + # An explicit ``base_branch`` (validated above) is passed through + # untouched; ``repo`` is already guaranteed non-empty by the early + # ``Missing repo`` guard, so only the ``base_branch`` side needs a + # check here. + if not base_branch: + base_branch = _pkg._detect_default_branch(repo_path) + + # Check that the target branch does not already exist on the remote. + # This catches conflicts early (before spawning agents). However, + # allow branch reuse when the pipeline is in a terminal state + # (CANCELLED/FAILED/COMPLETE) or doesn't exist at all — this lets + # callers resubmit against the same branch after a prior run ended. + if branch: + try: + gw = _pkg.get_gateway_client() + if gw.ls_remote_branch( + pipeline_id=pipeline_id or f"branch-check-{_pkg.uuid4().hex[:8]}", + repo_path=str(repo_path), + ref=f"refs/heads/{branch}", + ): + # Branch exists — only block if there is an active pipeline + _branch_store = _pkg.get_state_store(repo_path) + _has_active_pipeline = False + # When pipeline_id is None (auto-generated later), we skip + # the existence check — we can't look up a pipeline that + # hasn't been assigned an ID yet. This is acceptable because + # auto-generated IDs are unique and won't collide. + if pipeline_id and _branch_store.pipeline_exists(pipeline_id): + try: + _existing = _branch_store.load_pipeline(pipeline_id) + _terminal = { + _pkg.PipelineStatus.CANCELLED, + _pkg.PipelineStatus.FAILED, + _pkg.PipelineStatus.COMPLETE, + } + _has_active_pipeline = _existing.status not in _terminal + except Exception: + # If we can't load the pipeline, treat as no active pipeline + pass + + if _has_active_pipeline: + hint = "" + if pipeline_id: + hint = ( + f" Use a qualifier to create a separate pipeline" + f" (e.g. '{pipeline_id}-<qualifier>')." + ) + return _pkg.make_error_response( + f"Branch '{branch}' already exists on remote.{hint}", + status_code=409, + details={"reason": "branch_exists", "branch": branch}, + ) + else: + # No active pipeline, but the branch may carry commits + # from a prior failed/cancelled run. Inheriting that + # state was the precondition for #2222 (stale + # pipeline-branch tip + advanced main → contaminated + # PR via the push-reconcile fallback). Compare the + # branch tip to the configured base; only a fresh + # branch (tip == base) is safe to silently reuse. + # + # Resolve the default branch via ``_detect_default_branch`` + # rather than hardcoding ``"main"`` so repos whose default + # is ``master`` / ``develop`` still get the stale-branch + # check (otherwise the ``origin/main`` lookup returns + # ``None``, the guard falls through, and the precondition + # check is silently disabled). + _resolved_base = base_branch or _pkg._detect_default_branch(repo_path) + _branch_sha = gw.get_remote_branch_sha( + pipeline_id=pipeline_id or f"branch-check-{_pkg.uuid4().hex[:8]}", + repo_path=str(repo_path), + ref=f"refs/heads/{branch}", + ) + _base_sha = gw.get_remote_branch_sha( + pipeline_id=pipeline_id or f"branch-check-{_pkg.uuid4().hex[:8]}", + repo_path=str(repo_path), + ref=f"refs/heads/{_resolved_base}", + ) + # When either lookup returns ``None`` the stale-branch + # check is bypassed. ``get_remote_branch_sha`` swallows + # transient gateway errors and returns ``None`` (same + # value it returns when the ref legitimately doesn't + # exist), so we surface a warning here to make the + # silent skip visible to operators investigating a + # post-merge contamination — rather than letting the + # precondition fix vanish behind a transient hiccup. + if _branch_sha is None or _base_sha is None: + _pkg.logger.warning( + "Stale-branch check skipped: SHA lookup returned None " + "(transient gateway error or ref missing — see #2222)", + branch=branch, + base_branch=_resolved_base, + branch_sha=_branch_sha, + base_sha=_base_sha, + ) + if _branch_sha and _base_sha and _branch_sha != _base_sha: + _pkg.logger.warning( + "Branch exists with prior-pipeline commits — refusing reuse (#2222)", + branch=branch, + base_branch=_resolved_base, + branch_sha=_branch_sha, + base_sha=_base_sha, + ) + cleanup_hint = ( + f" Run cancel_task(task_id='{pipeline_id}', cleanup=true) " + "to delete the stale branch and pipeline state, then " + "resubmit." + if pipeline_id + else ( + " Delete the stale branch and any associated " + "pipeline state, then resubmit." + ) + ) + return _pkg.make_error_response( + f"Branch '{branch}' exists with commits from a prior " + f"pipeline run (tip {_branch_sha[:8]} != " + f"origin/{_resolved_base} {_base_sha[:8]}). Starting a " + "new pipeline on top of it would inherit that history.", + status_code=409, + details={ + "reason": "stale_branch", + "branch": branch, + "branch_sha": _branch_sha, + "base_sha": _base_sha, + "hint": cleanup_hint.strip(), + }, + ) + _pkg.logger.info( + "Branch exists but no active pipeline — allowing reuse", + branch=branch, + pipeline_id=pipeline_id, + branch_sha=_branch_sha, + base_sha=_base_sha, + ) + except Exception as e: + # Non-fatal — if we can't reach the gateway, let creation proceed + # and fail later on push. + _pkg.logger.warning( + "Branch existence check failed, proceeding anyway", + branch=branch, + error=str(e), + ) + + # Validate config before creating the pipeline so invalid config + # returns a 400 instead of bubbling up as a 500. + config = data.get("config") + if config is not None: + if isinstance(config, str): + try: + config = _pkg.json.loads(config) + except _pkg.json.JSONDecodeError as e: + return _pkg.make_error_response(f"Invalid config JSON: {e}") + try: + from models import PipelineConfig + from pydantic import ValidationError + + PipelineConfig.model_validate(config) + except ValidationError as e: + errors = [ + {"field": ".".join(str(loc) for loc in err["loc"]), "message": err["msg"]} + for err in e.errors() + ] + return _pkg.make_error_response( + f"Invalid pipeline config: {errors}", + details={"validation_errors": errors}, + ) + + # Validate analysis/plan size before creating the pipeline. + _MAX_DRAFT_LEN = 200_000 + for field_name in ("analysis", "plan"): + value = data.get(field_name) + if isinstance(value, str) and len(value) > _MAX_DRAFT_LEN: + return _pkg.make_error_response( + f"{field_name} exceeds maximum length ({len(value)} > {_MAX_DRAFT_LEN})" + ) + + # Issue #1557: epic detection. Before persisting, resolve + # is_epic + pipeline_mode against the gateway when a jira_ticket + # was supplied. Failures are non-fatal (the helper fails open) — + # we surface them as warnings in the API response but always + # proceed with the pipeline creation. + epic_warnings: list[str] = [] + is_epic_resolved = False + pipeline_mode_resolved: str | None = None + if jira_ticket_arg: + try: + from jira_epic import resolve_epic_mode + except ImportError: # pragma: no cover - defensive + try: + from orchestrator.jira_epic import resolve_epic_mode # type: ignore[no-redef] + except ImportError: + resolve_epic_mode = None # type: ignore[assignment] + if resolve_epic_mode is not None: + try: + is_epic_resolved, pipeline_mode_resolved, epic_warnings = resolve_epic_mode( + ticket=jira_ticket_arg, + epic_mode_arg=epic_mode_arg, + ) + except Exception as exc: # pragma: no cover - defensive + _pkg.logger.warning( + "Epic detection raised; treating as non-epic", + pipeline_id=pipeline_id, + ticket=jira_ticket_arg, + error=str(exc), + ) + # Both explicit overrides (``reassess`` and ``fresh``) against + # a non-epic ticket are operator errors: the operator + # specifically asked for epic-mode treatment but the ticket + # doesn't qualify. Surface as HTTP 400 rather than the + # silent demotion ``resolve_epic_mode`` returns + # (is_epic=False with a warning). ``mode='auto'`` continues + # to demote silently to standard ticket mode — that's the + # whole point of auto. + if epic_mode_arg in {"reassess", "fresh"} and not is_epic_resolved: + return _pkg.make_error_response( + f"epic_mode={epic_mode_arg!r} but Jira ticket {jira_ticket_arg!r} is not an Epic", + status_code=400, + details={ + "reason": f"{epic_mode_arg}_not_epic", + "warnings": epic_warnings, + }, + ) + + # #3393 (multi-repo): enforce uniform visibility + auth across the run's + # repos before creating the pipeline. Single-repo submissions are trivially + # uniform and short-circuit without a gateway round-trip. Runs after the + # gateway-ready gate above so the visibility lookup can reach the gateway. + _uniform_repos = ( + [e["repo"] for e in repos_entries] if repos_entries else ([repo] if repo else []) + ) + _uniformity_err = _pkg._assert_repo_set_uniform([r for r in _uniform_repos if r]) + if _uniformity_err: + return _pkg.make_error_response( + _uniformity_err, + status_code=400, + details={"reason": "non_uniform_repo_set"}, + ) + + # Assemble the full list-shaped repo set persisted onto the Pipeline. The + # primary (entries[0]) carries the resolved ``base_branch`` (detected above + # when absent); secondary repos keep their submitted base_branch (None ⇒ + # auto-detected downstream). For a single-repo submission we leave + # ``repos_specs`` as None and let the Pipeline validator synthesize a + # one-element list from the legacy singleton (N=1 back-compat). + repos_specs: list[_pkg.RepoSpec] | None = None + if repos_entries is not None: + repos_specs = [ + _pkg.RepoSpec( + repo=entry["repo"], + base_branch=(base_branch if idx == 0 else entry["base_branch"]), + ) + for idx, entry in enumerate(repos_entries) + ] + + try: + store = _pkg.get_state_store(repo_path) + pipeline = store.create_pipeline( + issue_number=issue_number, + repo=repo, + branch=branch, + base_branch=base_branch, + repos=repos_specs, + config=config, + prompt=prompt, + network_mode=network_mode, + pipeline_id=pipeline_id, + analysis=analysis, + plan=plan, + source_branch=source_branch, + source_artifact_prefix=source_artifact_prefix, + has_contract=True, + jira_ticket=jira_ticket_arg, + is_epic=is_epic_resolved, + pipeline_mode=pipeline_mode_resolved, + ) + + # Contract creation is deferred to _run_pipeline so it writes + # into the per-pipeline worktree instead of the main repo. + + # When state_store replaces a terminal pipeline with the same id + # (state_store.create_pipeline:850), the in-memory consensus + # tracker / message-store entries for the prior run survive. Same + # for Redis-backed message-store entries across orchestrator + # restarts. Clear here so the new run starts with empty consensus + # state regardless of how the prior run ended (#2053). + # + # This is the *primary* eviction site for auto-FAILED prior runs, + # not just a defensive backstop: paths like restart_agent spawn + # failure call store.update_pipeline / store.save_pipeline directly + # (bypassing PATCH), so the PATCH-site clear never fires for them. + # Without this POST-site clear, those auto-FAILED pipelines would + # leak consensus + message-store state into the next run that + # reuses the id. + _pkg._clear_pipeline_runtime_state(pipeline.id, reason="pipeline_create") + + _pkg.logger.info( + "Pipeline created", + pipeline_id=pipeline.id, + issue_number=issue_number, + ) + + return _pkg.make_success_response( + "Pipeline created", + data={"pipeline": pipeline.model_dump(mode="json")}, + ) + + except _pkg.StateStoreError as e: + if "already exists" in str(e): + # Include existing pipeline details so callers can decide + # whether to cancel+resubmit or resume monitoring. + details: dict[str, _pkg.Any] = {} + try: + # Derive pipeline ID using the same logic as state_store + pid = pipeline_id or (f"issue-{issue_number}" if issue_number else None) + if pid: + existing = store.load_pipeline(pid) + details = { + "existing_pipeline_id": existing.id, + "existing_status": existing.status.value, + "existing_phase": existing.current_phase.value, + } + except Exception: + pass # Best-effort enrichment + return _pkg.make_error_response(str(e), status_code=409, details=details) + _pkg.logger.error("Failed to create pipeline", error=str(e)) + return _pkg.make_error_response(f"Failed to create pipeline: {e}", status_code=500) + except Exception as e: + # Catch non-StateStoreError exceptions (e.g., ValidationError, + # OSError) that would otherwise produce a generic 500 from the + # Flask error handler with no detail (#1396). + _pkg.logger.error( + "Unexpected error creating pipeline", + error=str(e), + error_type=type(e).__name__, + exc_info=True, + ) + msg = f"{type(e).__name__}: {e}" + return _pkg.make_error_response( + f"Failed to create pipeline: {msg[:500]}", + status_code=500, + ) + + +def _update_pipeline_body(pipeline_id: str) -> tuple[_pkg.Response, int]: + """ + Update a pipeline. + + URL params: + pipeline_id: Pipeline ID + + Request body: + { + "status": "running", + "current_phase": "plan", + ... + } + + Response: + { + "success": true, + "data": { + "pipeline": {...} + } + } + """ + data = _pkg.request.get_json() + if data is None: + return _pkg.make_error_response("Missing request body") + if not isinstance(data, dict): + return _pkg.make_error_response("Request body must be a JSON object") + + repo_path = _pkg.get_repo_path() + + try: + store, _pipeline = _pkg._resolve_pipeline(pipeline_id, repo_path) + prev_status = _pipeline.status + pipeline = store.update_pipeline(pipeline_id, data) + + # Emit the terminal event before kicking off cleanup so /status/wait + # long-pollers wake immediately on cancellation rather than waiting + # for the late-subscriber synth path on their next poll (#2663). The + # run loop emits pipeline.completed / pipeline.failed from its own + # terminal transitions; the PATCH path is the only place the + # CANCELLED transition originates, so we emit it here. Gate on the + # status *transition* (not equality) so idempotent retries against an + # already-cancelled pipeline don't re-wake long-pollers. + if ( + pipeline.status == _pkg.PipelineStatus.CANCELLED + and prev_status != _pkg.PipelineStatus.CANCELLED + ): + _pkg._emit_pipeline_event(pipeline, "pipeline.cancelled") + + # If pipeline is being cancelled or failed, clean up containers + # and cancel any pending decisions so wait_for_decision() unblocks. + if pipeline.status in (_pkg.PipelineStatus.CANCELLED, _pkg.PipelineStatus.FAILED): + try: + dq = _pkg.get_decision_queue(pipeline_id, repo_path) + pending = dq.get_pending_decisions() + for decision in pending: + dq.cancel_decision(decision.id) + if pending: + _pkg.logger.info( + "Cancelled pending decisions after pipeline status change", + pipeline_id=pipeline_id, + decisions_cancelled=len(pending), + ) + except Exception as e: + _pkg.logger.warning( + "Failed to cancel pending decisions", + pipeline_id=pipeline_id, + error=str(e), + ) + + # Sync pipeline state: reload latest state (agents may have + # written updates between status change and container cleanup), + # mark all running records as stopped, and re-save. + try: + pipeline = _pkg._mark_pipeline_records_terminated(store, pipeline_id) + except Exception as e: + _pkg.logger.warning( + "Failed to sync pipeline state after termination", + pipeline_id=pipeline_id, + error=str(e), + ) + # Reload pipeline so the response reflects current state + # rather than the stale pre-cleanup object. + try: + pipeline = store.load_pipeline(pipeline_id) + except Exception: + pass # Use stale pipeline if reload also fails + + # Move container/worktree cleanup to a background daemon thread + # so the PATCH response returns immediately. The DELETE handler + # already re-runs cleanup_pipeline() as a safety net, so it will + # catch anything the background thread hasn't finished. + # + # Compute the salvage mode + base branch up front (in the + # request thread, where ``pipeline`` is still in scope) so the + # background thread can pass them to ``cleanup_pipeline`` + # without re-loading state. Using the wrong mode here would + # mismatch the policy the rest of the pipeline ran under and + # the launcher-auth push could be rejected — see #2429 + # review. + _bg_salvage_mode, _ = _pkg._compute_gateway_mode(pipeline) + _bg_salvage_base_branch = pipeline.base_branch + + def _background_cleanup(pid: str, status_value: str) -> None: + try: + spawner = _pkg._get_spawner() + # Preserve worktrees for CANCELLED pipelines so that + # restart_phase/restart_agent can resume with local + # committed work intact (see #1725). + removed = spawner.cleanup_pipeline( + pid, + force=True, + preserve_worktrees=(status_value == "cancelled"), + salvage_mode=_bg_salvage_mode, + salvage_base_branch=_bg_salvage_base_branch, + ) + if removed > 0: + _pkg.logger.info( + "Cleaned up pipeline containers after status change", + pipeline_id=pid, + status=status_value, + containers_removed=removed, + ) + except ( + _pkg.DockerClientError, + _pkg.DockerException, + _pkg.KubernetesClientError, + ) as e: + _pkg.logger.warning( + "Failed to clean up pipeline containers", + pipeline_id=pid, + error=str(e), + ) + except Exception as e: + _pkg.logger.error( + "Unexpected error during pipeline container cleanup", + pipeline_id=pid, + error=str(e), + exc_info=True, + ) + + cleanup_thread = _pkg.threading.Thread( + target=_background_cleanup, + args=(pipeline_id, pipeline.status.value), + daemon=True, + name=f"cleanup-{pipeline_id}", + ) + cleanup_thread.start() + + # Evict per-pipeline runtime state (consensus tracker, legacy + # consensus evaluator, message store) so a future pipeline + # that reuses this id (same branch) does not inherit this + # run's CONFIRMED consensus or message history (#2053). + _pkg._clear_pipeline_runtime_state( + pipeline_id, reason=f"pipeline_{pipeline.status.value}" + ) + + _pkg.logger.info("Pipeline updated", pipeline_id=pipeline_id) + + response_data = {"pipeline": pipeline.model_dump(mode="json")} + if pipeline.status in (_pkg.PipelineStatus.CANCELLED, _pkg.PipelineStatus.FAILED): + response_data["cleanup_pending"] = True + + return _pkg.make_success_response( + "Pipeline updated", + data=response_data, + ) + + except _pkg.InvalidPipelineIdError: + return _pkg.make_error_response( + f"Invalid pipeline ID format: {pipeline_id}", + status_code=400, + ) + except _pkg.PipelineNotFoundError: + return _pkg.make_error_response( + f"Pipeline {pipeline_id} not found", + status_code=404, + ) + except _pkg.StateValidationError as e: + return _pkg.make_error_response( + f"Invalid update: {e}", + status_code=400, + ) + + +def _update_pipeline_config_body(pipeline_id: str) -> tuple[_pkg.Response, int]: + """Update the safely-mutable subset of a live pipeline's config (#3174). + + That subset is ``agent_models`` plus the ``consensus_timeout_minutes*`` + family (#3490). + + ``agent_models`` semantics are a per-role merge with the pipeline's + existing override map: roles absent from the request keep their + current value, a string value sets that role's override, and an + explicit ``null`` clears it (the role falls back to the repository + default / built-in tiers). The updated map takes effect at the next + agent spawn; currently running agents keep the model they were + started with. Pair with ``restart_phase`` / ``restart_agent`` to + apply the change to a running phase. Model *values* are not + validated against a registry here (any non-Claude string routes to + LiteLLM, mirroring submit-time behavior); a typo surfaces as a + model-not-found error at spawn. + + ``consensus_timeout_minutes`` / ``consensus_timeout_minutes_refine`` + / ``_plan`` / ``_implement`` set the corresponding override to an + integer number of minutes (>= 1); ``null`` clears the override so + the phase falls back to the resolution chain (per-phase override, + then legacy global, then the phase-aware default). The phase poll + loop re-resolves the budget from fresh config right before the + consensus wall fires, so a widened window takes effect on a running + slice without a restart (#3490). + + URL params: + pipeline_id: Pipeline ID + + Request body (any non-empty subset of the mutable keys): + { + "agent_models": { + "coder": "deepseek-v4-pro", + "tester": null + }, + "consensus_timeout_minutes_implement": 480 + } + + Response: + { + "success": true, + "data": { + "pipeline_id": "issue-123", + "agent_models": {...}, # effective map after the merge + "updated_roles": {...}, # roles set by this request + "cleared_roles": [...], # roles cleared by this request + "consensus_timeouts": {...},# effective timeout overrides + "updated_timeouts": {...} # timeout keys set/cleared here + } + } + """ + data = _pkg.request.get_json() + if data is None: + return _pkg.make_error_response("Missing request body") + if not isinstance(data, dict): + return _pkg.make_error_response("Request body must be a JSON object") + + unsupported = sorted(set(data) - _pkg._MUTABLE_CONFIG_KEYS) + if unsupported: + return _pkg.make_error_response( + f"Unsupported config keys: {unsupported}. This endpoint updates " + f"only the safely-mutable config subset: {sorted(_pkg._MUTABLE_CONFIG_KEYS)}", + status_code=400, + ) + if not data: + return _pkg.make_error_response( + f"Request body must set at least one mutable config key: " + f"{sorted(_pkg._MUTABLE_CONFIG_KEYS)}", + status_code=400, + ) + + agent_models = data.get("agent_models") + if "agent_models" in data: + if not isinstance(agent_models, dict) or not agent_models: + return _pkg.make_error_response( + "agent_models must be a non-empty object mapping role -> model " + "(use null as the model to clear a role's override)", + status_code=400, + ) + + # Pre-validate role keys against MODEL_OVERRIDE_ROLES so the operator + # gets the same actionable message as PipelineConfig's field validator + # instead of a wrapped pydantic StateValidationError. Lazy import + # mirrors models._validate_agent_models_roles. + from egg_contracts.agent_roles import MODEL_OVERRIDE_ROLES + + valid_roles = {role.value for role in MODEL_OVERRIDE_ROLES} + invalid_roles = sorted(role for role in agent_models if role not in valid_roles) + if invalid_roles: + return _pkg.make_error_response( + f"Invalid agent_models role keys: {invalid_roles}. agent_models " + f"is honored only for SDLC phase producer and reviewer roles: " + f"{sorted(valid_roles)}", + status_code=400, + ) + invalid_values = sorted( + role + for role, model in agent_models.items() + if model is not None and (not isinstance(model, str) or not model.strip()) + ) + if invalid_values: + return _pkg.make_error_response( + f"Invalid agent_models values for roles {invalid_values}: each " + f"value must be a non-empty model string, or null to clear the " + f"role's override", + status_code=400, + ) + + timeout_updates: dict[str, int | None] = {} + invalid_timeout_keys: list[str] = [] + for timeout_key in _pkg._CONSENSUS_TIMEOUT_CONFIG_KEYS: + if timeout_key not in data: + continue + timeout_value = data[timeout_key] + if timeout_value is None or ( + isinstance(timeout_value, int) + and not isinstance(timeout_value, bool) + and timeout_value >= 1 + ): + timeout_updates[timeout_key] = timeout_value + else: + invalid_timeout_keys.append(timeout_key) + if invalid_timeout_keys: + return _pkg.make_error_response( + f"Invalid values for {invalid_timeout_keys}: each consensus " + f"timeout must be an integer number of minutes >= 1, or null to " + f"clear the override (the phase falls back to the resolution " + f"chain: per-phase override, legacy global, phase-aware default)", + status_code=400, + ) + + repo_path = _pkg.get_repo_path() + + try: + store, _pipeline = _pkg._resolve_pipeline(pipeline_id, repo_path) + + # Merge under the pipeline state lock so a concurrent writer + # (another config update, the run loop persisting state) can't + # interleave between our load and the store's load-modify-save. + # The per-pipeline lock is an RLock, so update_pipeline's own + # acquisition nests cleanly. + with _pkg.get_pipeline_state_lock(pipeline_id): + current = store.load_pipeline(pipeline_id) + + # Reject mutations on terminal pipelines (#3174 review). Nothing + # consumes config once a pipeline is COMPLETE / FAILED / + # CANCELLED, so the merge would be a silent no-op; a 409 gives + # the operator a clear signal and matches restart_phase's + # terminal-state precondition style. Checked under the lock + # against freshly-loaded state so a concurrent terminal + # transition can't slip a mutation through. + if current.status in _pkg.PipelineStatus.terminal(): + return _pkg.make_error_response( + f"Pipeline {pipeline_id} is in terminal state " + f"{current.status.value}; config cannot be updated " + "(nothing would consume the change).", + status_code=409, + ) + + updates: dict[str, Any] = {} + updated_roles: dict[str, str] = {} + cleared_roles: list[str] = [] + if isinstance(agent_models, dict): + merged = dict(current.config.agent_models) + for role_key, model in agent_models.items(): + if model is None: + if merged.pop(role_key, None) is not None: + cleared_roles.append(role_key) + else: + merged[role_key] = model.strip() + updated_roles[role_key] = model.strip() + updates["config.agent_models"] = merged + for timeout_key, timeout_value in timeout_updates.items(): + updates[f"config.{timeout_key}"] = timeout_value + pipeline = store.update_pipeline(pipeline_id, updates) + + _pkg.logger.info( + "Pipeline config updated", + pipeline_id=pipeline_id, + updated_roles=updated_roles, + cleared_roles=cleared_roles, + updated_timeouts=timeout_updates, + ) + + return _pkg.make_success_response( + "Pipeline config updated", + data={ + "pipeline_id": pipeline.id, + "agent_models": pipeline.config.agent_models, + "updated_roles": updated_roles, + "cleared_roles": cleared_roles, + "consensus_timeouts": { + timeout_key: getattr(pipeline.config, timeout_key) + for timeout_key in _pkg._CONSENSUS_TIMEOUT_CONFIG_KEYS + }, + "updated_timeouts": timeout_updates, + }, + ) + + except _pkg.InvalidPipelineIdError: + return _pkg.make_error_response( + f"Invalid pipeline ID format: {pipeline_id}", + status_code=400, + ) + except _pkg.PipelineNotFoundError: + return _pkg.make_error_response( + f"Pipeline {pipeline_id} not found", + status_code=404, + ) + except _pkg.StateValidationError as e: + return _pkg.make_error_response( + f"Invalid update: {e}", + status_code=400, + ) + + +def _delete_pipeline_body(pipeline_id: str) -> tuple[_pkg.Response, int]: + """ + Delete a pipeline. + + URL params: + pipeline_id: Pipeline ID + + Response: + { + "success": true, + "message": "Pipeline deleted" + } + """ + repo_path = _pkg.get_repo_path() + + try: + store, _pipeline = _pkg._resolve_pipeline(pipeline_id, repo_path) + + # Clean up any running containers for this pipeline + try: + spawner = _pkg._get_spawner() + # Pass the running pipeline's gateway mode + base branch so the + # auto-salvage hook in cleanup_pipeline pushes recovery refs + # under the same policy the pipeline ran under (#2429 review). + _delete_salvage_mode, _ = _pkg._compute_gateway_mode(_pipeline) + removed = spawner.cleanup_pipeline( + pipeline_id, + force=True, + salvage_mode=_delete_salvage_mode, + salvage_base_branch=_pipeline.base_branch, + ) + if removed > 0: + _pkg.logger.info( + "Cleaned up pipeline containers", + pipeline_id=pipeline_id, + containers_removed=removed, + ) + except (_pkg.DockerClientError, _pkg.DockerException, _pkg.KubernetesClientError) as e: + _pkg.logger.warning( + "Failed to clean up pipeline containers", + pipeline_id=pipeline_id, + error=str(e), + ) + except Exception as e: + _pkg.logger.error( + "Unexpected error during pipeline container cleanup", + pipeline_id=pipeline_id, + error=str(e), + exc_info=True, + ) + + # Clean up remote branches (best-effort) + try: + _pkg._cleanup_remote_branches(pipeline_id, _pipeline, repo_path) + except Exception as e: + _pkg.logger.warning( + "Failed to clean up remote branches", + pipeline_id=pipeline_id, + error=str(e), + ) + + # Clean up the message store stream/counters AND the in-memory + # consensus tracker / legacy evaluator so a fresh pipeline that + # later reuses this id starts with empty consensus state (#2053). + _pkg._clear_pipeline_runtime_state(pipeline_id, reason="pipeline_delete") + + store.delete_pipeline(pipeline_id) + + _pkg.logger.info("Pipeline deleted", pipeline_id=pipeline_id) + + return _pkg.make_success_response("Pipeline deleted") + + except _pkg.InvalidPipelineIdError: + return _pkg.make_error_response( + f"Invalid pipeline ID format: {pipeline_id}", + status_code=400, + ) + except _pkg.PipelineNotFoundError: + return _pkg.make_error_response( + f"Pipeline {pipeline_id} not found", + status_code=404, + ) diff --git a/orchestrator/routes/pipelines/_routes_lifecycle.py b/orchestrator/routes/pipelines/_routes_lifecycle.py new file mode 100644 index 0000000000..e1359acd98 --- /dev/null +++ b/orchestrator/routes/pipelines/_routes_lifecycle.py @@ -0,0 +1,822 @@ +"""lifecycle-route bodies helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _list_pipeline_local_commits_body(pipeline_id: str) -> tuple[_pkg.Response, int]: + """List unpushed commits across this pipeline's per-agent worktrees. + + Inspects every per-agent worktree on disk + (``{pipeline_id}``, ``{pipeline_id}-{role}``, + ``{pipeline_id}-slice-{N}-{role}``) and reports the commits on its + local ``egg/{worktree_id}/work`` branch that are not reachable from + ``origin/<assigned_branch>`` (or ``origin/<base_branch>`` as a + fallback). Read-only — no fetch, no push. + + Query string (optional): + agent_role: Filter to a single agent role (e.g. ``coder``). + slice_id: Filter to a single slice scope (e.g. ``slice-2``). + + Response: + { + "success": true, + "data": { + "pipeline_id": "issue-2261-v9", + "worktrees": [ + { + "worktree_id": "issue-2261-v9-slice-2-coder", + "agent_role": "coder", + "slice_id": "slice-2", + "local_branch": "egg/issue-2261-v9-slice-2-coder/work", + "assigned_branch": "egg/issue-2261-v9/slice-2", + "anchor_ref": "refs/remotes/origin/egg/issue-2261-v9/slice-2", + "commits": [ + {"sha": "...", "summary": "...", "author": "...", + "authored_at": "...", "files_changed": 3} + ], + "error": null + } + ] + } + } + """ + repo_path = _pkg.get_repo_path() + + try: + _store, pipeline = _pkg._resolve_pipeline(pipeline_id, repo_path) + except _pkg.InvalidPipelineIdError: + return _pkg.make_error_response( + f"Invalid pipeline ID format: {pipeline_id}", status_code=400 + ) + except _pkg.PipelineNotFoundError: + return _pkg.make_error_response(f"Pipeline {pipeline_id} not found", status_code=404) + + agent_role = _pkg.request.args.get("agent_role") or None + if agent_role is not None: + try: + _pkg.AgentRole(agent_role) + except ValueError: + return _pkg.make_error_response(f"Invalid agent role: {agent_role}", status_code=400) + + raw_slice_id = _pkg.request.args.get("slice_id") + try: + slice_id = _pkg.extract_slice_id( + {"slice_id": raw_slice_id} if raw_slice_id is not None else {} + ) + except ValueError as e: + return _pkg.make_error_response(str(e), status_code=400) + + worktrees = _pkg._filter_salvage_worktrees( + _pkg.agent_salvage.enumerate_agent_worktrees(pipeline_id), + agent_role=agent_role, + slice_id=slice_id, + ) + reports = [ + _pkg.agent_salvage.list_unpushed_commits(wt, base_branch=pipeline.base_branch) + for wt in worktrees + ] + + return _pkg.make_success_response( + f"Listed local commits for pipeline {pipeline_id}", + data={ + "pipeline_id": pipeline_id, + "worktrees": [_pkg._serialize_commit_report(r) for r in reports], + }, + ) + + +def _salvage_pipeline_local_commits_body(pipeline_id: str) -> tuple[_pkg.Response, int]: + """Push unpushed agent commits to recovery refs (#2429). + + For every matching per-agent worktree, push its HEAD to + ``egg/recovered/<pipeline_id>/<scope>/<short_sha>`` via the gateway's + launcher-auth path. Launcher auth bypasses the agent-targeted + branch-allowlist check so this works even when the agent's own + pushes were rejected for the wrong-branch reason this verb exists + to recover from. + + Query string (optional): + agent_role: Salvage only this role's worktree. + slice_id: Salvage only this slice scope. + + Response (always ``success: true`` when the request was well-formed + — per-worktree failures are reported in ``data.results``): + { + "success": true, + "data": { + "pipeline_id": "issue-2261-v9", + "results": [ + {"worktree_id": "...", "agent_role": "coder", "slice_id": "slice-2", + "recovery_ref": "egg/recovered/issue-2261-v9/slice-2-coder/9665f37a6...", + "head_sha": "9665f37a6...", "n_commits": 14, "ok": true, "error": null} + ] + } + } + """ + repo_path = _pkg.get_repo_path() + + try: + _store, pipeline = _pkg._resolve_pipeline(pipeline_id, repo_path) + except _pkg.InvalidPipelineIdError: + return _pkg.make_error_response( + f"Invalid pipeline ID format: {pipeline_id}", status_code=400 + ) + except _pkg.PipelineNotFoundError: + return _pkg.make_error_response(f"Pipeline {pipeline_id} not found", status_code=404) + + agent_role = _pkg.request.args.get("agent_role") or None + if agent_role is not None: + try: + _pkg.AgentRole(agent_role) + except ValueError: + return _pkg.make_error_response(f"Invalid agent role: {agent_role}", status_code=400) + + raw_slice_id = _pkg.request.args.get("slice_id") + try: + slice_id = _pkg.extract_slice_id( + {"slice_id": raw_slice_id} if raw_slice_id is not None else {} + ) + except ValueError as e: + return _pkg.make_error_response(str(e), status_code=400) + + worktrees = _pkg._filter_salvage_worktrees( + _pkg.agent_salvage.enumerate_agent_worktrees(pipeline_id), + agent_role=agent_role, + slice_id=slice_id, + ) + + gateway_mode, _vis = _pkg._compute_gateway_mode(pipeline) + gateway = _pkg.get_gateway_client() + + results = [] + for wt in worktrees: + try: + result = _pkg.agent_salvage.salvage_worktree( + gateway, + wt, + base_branch=pipeline.base_branch, + mode=gateway_mode, + ) + except Exception as e: # noqa: BLE001 — must always return a result row + _pkg.logger.warning( + "Salvage raised unexpectedly", + pipeline_id=pipeline_id, + worktree_id=wt.worktree_id, + error=str(e), + ) + result = _pkg.agent_salvage.SalvageResult( + worktree_id=wt.worktree_id, + agent_role=wt.agent_role, + slice_id=wt.slice_id, + recovery_ref=None, + head_sha=None, + n_commits=0, + ok=False, + error=str(e), + ) + results.append(result) + + return _pkg.make_success_response( + f"Salvaged {sum(1 for r in results if r.ok and r.recovery_ref)} of " + f"{len(results)} per-agent worktrees for pipeline {pipeline_id}", + data={ + "pipeline_id": pipeline_id, + "results": [_pkg._serialize_salvage_result(r) for r in results], + }, + ) + + +def _start_pipeline_body(pipeline_id: str) -> tuple[_pkg.Response, int]: + """ + Start pipeline execution. + + Spawns containers for each phase in sequence, advancing through + the phase DAG until completion or failure. Runs in a background thread. + + URL params: + pipeline_id: Pipeline ID + + Response: + { + "success": true, + "message": "Pipeline started", + "data": { + "pipeline_id": "local-a1b2c3d4", + "status": "running" + } + } + """ + repo_path = _pkg.get_repo_path() + + # Parse force / force_reason from body. ``force=true`` skips the + # live-pod orphan guard before the phase reset (#2420). force_reason + # is recorded in the structured warning log, mirroring the + # complete_phase audit pattern. + body = _pkg.request.get_json(silent=True) or {} + # Strict boolean — `body.get("force") is True` rather than + # `bool(body.get("force"))` so non-boolean truthy values + # (`"false"`, `[]`, `{}`, `1`) don't silently flip the predicate. + force = body.get("force") is True + force_reason = body.get("force_reason") + if force_reason is not None and not isinstance(force_reason, str): + return _pkg.make_error_response( + "force_reason must be a string", + status_code=400, + reason="invalid_force_reason", + ) + if isinstance(force_reason, str) and not force_reason.strip(): + force_reason = None + + try: + store, pipeline = _pkg._resolve_pipeline(pipeline_id, repo_path) + # Use the store's repo_path so _run_pipeline operates on the correct directory + repo_path = store.repo_path + + # Compute gateway mode for session operations in the recovery path + _gw_mode, _gw_vis = _pkg._compute_gateway_mode(pipeline) + + if pipeline.status == _pkg.PipelineStatus.RUNNING: + return _pkg.make_error_response( + f"Pipeline {pipeline_id} is already running", + status_code=409, + ) + + if pipeline.status == _pkg.PipelineStatus.AWAITING_HUMAN: + # No pending decisions — the polling thread died (e.g. restart) + # but the human already resolved everything. Recover based on + # the latest phase_gate decision's resolution. + # + # #2593 review issue 1 — initialised before the lock so the + # post-lock deferred context-PR opener invocation has a + # stable name to read regardless of which branch inside the + # lock executes. + _hitl_open_context_pr_after_lock: bool = False + _hitl_pr_worktree_path: _pkg.Path | None = None + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + + # Re-validate status after acquiring the lock — another + # concurrent start_pipeline call may have already recovered + # this pipeline. + if pipeline.status != _pkg.PipelineStatus.AWAITING_HUMAN: + return _pkg.make_error_response( + f"Pipeline {pipeline_id} status changed to " + f"{pipeline.status.value} (concurrent recovery)", + status_code=409, + ) + + pending = pipeline.get_pending_decisions() + if len(pending) > 0: + return _pkg.make_error_response( + f"Pipeline {pipeline_id} is awaiting human approval " + f"({len(pending)} pending decision(s))", + status_code=409, + ) + + # Find the latest resolved phase_gate decision + phase_gate_decisions = [ + d + for d in reversed(pipeline.decisions) + if d.decision_type == "phase_gate" and d.status.value == "resolved" + ] + latest_resolution = ( + phase_gate_decisions[0].resolution if phase_gate_decisions else None + ) + + # Determine if approved or request_changes using the shared + # parser (handles approve, select, submit_feedback, + # request_changes, change_approach, and legacy bare strings). + is_approved, revision_feedback = _pkg._parse_resolution(latest_resolution) + + if is_approved: + # Mark current phase COMPLETE and advance + phase_execution = pipeline.get_phase_execution(pipeline.current_phase) + phase_execution.status = _pkg.PipelineStatus.COMPLETE + if phase_execution.completed_at is None: + phase_execution.completed_at = _pkg.datetime.now(_pkg.UTC) + + # Persist phase gate resolution so next-phase agents see it. #1295 + # + # The contract and phase draft both live under the + # per-pipeline worktree (``<worktree>/.egg-state/``), + # not the orchestrator's main repo. Resolve the + # worktree explicitly here — the inline path inside + # ``_run_pipeline`` already has ``worktree_repo_path`` + # in scope, but this recovery branch only has the + # main ``repo_path``. Passing ``repo_path`` would + # silently no-op the contract write and draft append + # (#2357, same shape as #2345). + if phase_gate_decisions: + worktree_repo_path = _pkg._resolve_pipeline_worktree_path( + pipeline, repo_path + ) + if worktree_repo_path == repo_path: + # No materialised worktree — recovery degrades to + # the pre-fix shape (contract write typically + # no-ops via ContractNotFoundError, draft append + # skipped). The contract write *may* succeed if + # the orchestrator's main repo happens to carry a + # contract for this pipeline, but it would land + # against the wrong tree. Surface this either way + # so operators can correlate missing next-phase + # context with worktree-cleanup races. + _pkg.logger.warning( + "No materialised worktree found for phase gate " + "persistence; falling back to main repo path. " + "Contract write may silently no-op.", + pipeline_id=pipeline_id, + phase=pipeline.current_phase.value, + ) + _pkg._persist_phase_gate_resolution( + worktree_repo_path, + pipeline_id, + phase_gate_decisions[0], + pipeline.current_phase.value, + pipeline.issue_number, + ) + + # Commit statefiles so worktrees created by _run_pipeline + # include the contract/draft changes. + try: + _pkg._commit_statefiles_to_worktree( + worktree_repo_path, + f"Persist HITL resolution after {pipeline.current_phase.value} phase gate", + pipeline_identifier=_pkg._pipeline_identifier( + pipeline.issue_number, pipeline_id + ), + pipeline_id=pipeline_id, + ) + except Exception as git_err: + # Catch broadly: see #2219. The helper raises + # ``TimeoutExpired`` and ``OSError`` paths that a + # ``CalledProcessError``-only handler did not catch. + _pkg.logger.warning( + "Failed to commit statefiles after phase gate resolution (continuing)", + pipeline_id=pipeline_id, + error=str(git_err), + ) + + # Push if this repo tracks a remote branch and a + # worktree was materialised. Mirrors the inline + # path's guard at pipelines.py:16044 — pushing from + # the orchestrator's main repo would target the + # wrong working tree. + if pipeline.branch and worktree_repo_path != repo_path: + try: + _spawner = _pkg._get_spawner() + _spawner.gateway.push_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(worktree_repo_path), + branch=pipeline.branch, + mode=_gw_mode, + base_branch=pipeline.base_branch, + ) + except Exception as push_err: + _pkg.logger.warning( + "Failed to push statefiles after phase gate resolution (continuing)", + pipeline_id=pipeline_id, + error=str(push_err), + ) + + from routes.phases import PHASE_TRANSITIONS + + transitions = PHASE_TRANSITIONS + current_phase = pipeline.current_phase + # Issue #1557 — route epic pipelines through APPLY + # between PLAN and IMPLEMENT. Non-epic pipelines + # see the default transition unchanged. + next_phases = _pkg._next_phases_for_epic( + pipeline, + current_phase, + transitions.get(current_phase, []), + ) + # #2593 — populate contract from the plan draft when + # the HITL recovery is advancing the pipeline out + # of the plan phase. Without this, contract.pr is + # empty (so the PR phase falls back to placeholder + # title/body and the context PR hook short-circuits + # on "contract has no pr block"), and the slice + # stack ends up rooted on ``/work`` with no PR to + # ``main`` — exactly the symptom reported on the + # in-flight #2474 pipeline. Mirrors the plan-exit + # logic in ``advance_phase`` (routes/phases.py) + # and the auto-advance path in ``_run_pipeline``. + # Best-effort: failures warn and continue so a + # transient infra problem cannot strand the HITL + # recovery. The actual context-PR open is + # deferred until after the lock is released + # (#2593 review issue 1) so the multi-second + # gateway sequence does not extend the + # per-pipeline state lock's hold time. + _next_phase_peek = next_phases[0] if next_phases else None + if ( + current_phase == _pkg.PipelinePhase.PLAN + and _next_phase_peek == _pkg.PipelinePhase.IMPLEMENT + ): + _hitl_worktree_path = _pkg._resolve_pipeline_worktree_path( + pipeline, repo_path + ) + try: + _pipeline_mode = pipeline.mode.value if pipeline.mode else "issue" + _hitl_populate_result = _pkg._populate_contract_from_plan_safe( + _hitl_worktree_path, + pipeline_id, + _pipeline_mode, + pipeline.issue_number, + source="hitl_plan_gate_approval", + ) + # #1941: HITL plan-gate approval is a recovery + # hammer like force-advance — blocking it on a + # populate failure defeats the purpose. We log + # the structured outcome but never raise. + if _hitl_populate_result.outcome != _pkg.PopulateOutcome.POPULATED: + _pkg.logger.warning( + "HITL plan-gate approval populate produced non-POPULATED outcome", + pipeline_id=pipeline_id, + outcome=_hitl_populate_result.outcome.value, + ) + try: + _pkg._commit_statefiles_to_worktree( + _hitl_worktree_path, + "Populate contract from plan on HITL plan-gate approval", + pipeline_identifier=_pkg._pipeline_identifier( + pipeline.issue_number, pipeline_id + ), + pipeline_id=pipeline_id, + ) + except Exception as _hitl_commit_err: # noqa: BLE001 + _pkg.logger.warning( + "Failed to commit populated contract on HITL plan-gate approval (continuing) (#2593)", + pipeline_id=pipeline_id, + error=str(_hitl_commit_err), + ) + + # #2593 review issue 5 — the earlier + # ``push_worktree_branch`` at line ~20598 + # ran *before* this populate commit, so + # the populated ``contract.pr`` only + # exists locally until the IMPLEMENT + # phase's next phase-boundary sync. Push + # again now so any slice-agent container + # that materialises a fresh worktree from + # origin before that sync still sees + # ``contract.pr``. Mirrors the + # auto-advance flow's pre-context-PR push + # in ``_run_pipeline``. + if pipeline.branch and _hitl_worktree_path != repo_path: + try: + _pkg._get_spawner().gateway.push_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(_hitl_worktree_path), + branch=pipeline.branch, + mode=_gw_mode, + base_branch=pipeline.base_branch, + ) + except Exception as _hitl_push_err: # noqa: BLE001 + _pkg.logger.warning( + "Failed to push populated contract on HITL plan-gate approval (continuing) (#2593)", + pipeline_id=pipeline_id, + error=str(_hitl_push_err), + ) + except Exception as _hitl_pop_err: # noqa: BLE001 + _pkg.logger.warning( + "Failed to run plan-exit populate on HITL recovery (continuing) (#2593)", + pipeline_id=pipeline_id, + error=str(_hitl_pop_err), + ) + + # Defer the context-PR open until after the + # per-pipeline state lock is released — see + # ``_open_context_pr_at_implement_start``'s + # idempotency docstring on why this multi- + # second network sequence (one ``gh pr list`` + # + maybe one ``gh pr create``) must not run + # under the lock (#2593 review issue 1). + _hitl_open_context_pr_after_lock = True + _hitl_pr_worktree_path = _hitl_worktree_path + + if not next_phases: + # Terminal phase — pipeline complete. + # Bump run_epoch so any lingering old _run_pipeline + # thread (e.g. stuck in its finally block) detects the + # recreation and exits without double-cleaning up. + pipeline.status = _pkg.PipelineStatus.COMPLETE + pipeline.run_epoch = _pkg.datetime.now(_pkg.UTC) + store.save_pipeline(pipeline) + return _pkg.make_success_response( + "Pipeline recovered and completed", + data={ + "pipeline_id": pipeline_id, + "status": "complete", + "current_phase": pipeline.current_phase.value, + }, + ) + + # Advance to next phase + next_phase = next_phases[0] + pipeline.current_phase = next_phase + + # Issue #1557: PLAN → APPLY transition on epic + # pipelines (mirrors auto-advance path). Write the + # applier handoff JSON before the next _run_pipeline + # thread is respawned so the APPLIER container's + # first read finds it on disk. + if ( + getattr(pipeline, "is_epic", False) + and current_phase == _pkg.PipelinePhase.PLAN + and next_phase == _pkg.PipelinePhase.APPLY + ): + _hitl_apply_worktree = _pkg._resolve_pipeline_worktree_path( + pipeline, repo_path + ) + _pkg._write_apply_phase_handoff( + pipeline, + _hitl_apply_worktree, + approved_phase="plan", + ) + + # Issue #1557 task-2-7: when the resolved phase was + # APPLY (BRC consensus confirmed via HITL recovery + # path), drain the Won't-Do handoff before advancing. + if current_phase == _pkg.PipelinePhase.APPLY: + _hitl_drain_worktree = _pkg._resolve_pipeline_worktree_path( + pipeline, repo_path + ) + _pkg._drain_wontdo_batch_after_apply(pipeline, _hitl_drain_worktree) + + # Update health monitor phase threshold before agents spawn + try: + from health_monitor import get_health_monitor + + _hm_instance = get_health_monitor() + if _hm_instance is not None: + _hm_instance.set_current_phase(next_phase.value) + except ImportError: + pass + + else: + # request_changes/change_approach — reset phase for re-run + phase_execution = pipeline.get_phase_execution(pipeline.current_phase) + # #2795: derive iteration_n monotonically. The + # ``max(len(iteration_history), max(directive_idx) + 1)`` + # form does not depend on ``hitl_review_cycles``, so + # this expression is safe to evaluate either before + # or after ``_clear_concurrent_state`` resets the + # per-phase counter. What *is* order-sensitive is + # the tracker snapshot a few lines below: the BRC + # tracker is in-memory only and gets wiped by + # ``_clear_concurrent_state``, so the snapshot MUST + # happen first. On a crash-recovery resolution the + # snapshot will typically have empty verdict detail, + # but the iteration index + artifacts are still + # useful context for iteration N+1's prompts. + # The ``max(...) + 1`` floor ensures a legacy- + # hitl_feedback migration (which synthesises a + # directive but leaves iteration_history empty) + # doesn't restart the index at 0. + _recovery_iteration_n = max( + len(phase_execution.iteration_history), + max( + (d.iteration_n for d in phase_execution.operator_directives), + default=-1, + ) + + 1, + ) + _recovery_tracker = None + try: + from peer_consensus import ( + get_peer_consensus_tracker as _gpct_recovery, + ) + + _recovery_tracker = _gpct_recovery(pipeline_id) + except Exception as tracker_err: # noqa: BLE001 + _pkg.logger.debug( + "Tracker lookup failed during recovery snapshot", + pipeline_id=pipeline_id, + error=str(tracker_err), + ) + _recovery_summary = _pkg._build_iteration_summary_from_tracker( + _recovery_tracker, + iteration_n=_recovery_iteration_n, + artifacts=phase_execution.artifacts, + ) + + if phase_execution.status in ( + _pkg.PipelineStatus.COMPLETE, + _pkg.PipelineStatus.FAILED, + _pkg.PipelineStatus.RUNNING, + _pkg.PipelineStatus.AWAITING_HUMAN, + ): + # Refuse to clear containers/agents/artifacts when + # pods labeled to this pipeline are still alive — + # the reset would orphan them (#2420). + guard = _pkg._guard_live_pods_or_force(pipeline_id, force, force_reason) + if guard is not None: + return guard + phase_execution.status = _pkg.PipelineStatus.PENDING + phase_execution.started_at = None + phase_execution.work_started_at = None + phase_execution.completed_at = None + phase_execution.error = None + phase_execution.review_cycles = 0 + phase_execution.hitl_review_cycles = 0 + phase_execution.containers = [] + phase_execution.agents = [] + phase_execution.artifacts = {} + + # Clear stale consensus state so re-run doesn't + # short-circuit (issue #1296). + from routes.phases import _clear_concurrent_state + + _clear_concurrent_state(pipeline_id) + + # #2795: append the operator directive + iteration + # summary so iteration N+1 prompts can render them + # with precedence prose. Both lists accumulate + # across kickbacks (no clear). + if revision_feedback: + phase_execution.operator_directives.append( + _pkg.OperatorDirective( + iteration_n=_recovery_iteration_n, + feedback_text=revision_feedback, + ) + ) + phase_execution.iteration_history.append(_recovery_summary) + + pipeline.error = None + pipeline.run_epoch = _pkg.datetime.now(_pkg.UTC) + pipeline.status = _pkg.PipelineStatus.RUNNING + store.save_pipeline(pipeline) + + # TEST_MARKER: recover_advance_clear (load-bearing: brackets + # the post-lock clear for TestRecoverPipelineClearsConcurrentState; + # do not remove without updating that test class). + # Drop the previous phase's in-memory consensus tracker on + # cross-phase advance (#2502). The request_changes / + # change_approach branch above already cleared inside the + # lock for same-phase re-runs (#1296); the advance branch + # needs its own post-lock clear so persisted state lands + # before the tracker is wiped, matching the persist-then- + # clear-then-spawn order used by ``advance_phase`` and the + # auto-advance block. + if is_approved: + from routes.phases import _clear_concurrent_state + + _clear_concurrent_state(pipeline_id) + + # #2593 review issue 1 — context-PR open moved out of the + # per-pipeline state lock so the multi-second gateway + # sequence does not hold the lock and block concurrent + # ``advance_phase`` / status reads. + # + # #2777 (cq-4, TASK-1-2) — HITL-recovery context-PR site + # calls the new idempotent + # ``_open_context_pr_at_implement_start`` opener directly. + # HITL recovery in ``start_pipeline`` does NOT route + # through ``advance_phase`` REST (the runner thread is + # spawned inline below), so without this call site an + # operator-resumed pipeline would silently strand its + # slice stack on ``egg/<id>/work``. The opener's + # ``gh pr list`` pre-flight makes a redundant call from a + # later ``advance_phase`` invocation a one-round-trip + # no-op (reviewer_code_holistic blocker 1 fix; v1 deleted + # this site under the incorrect "single canonical site" + # plan AC). + if _hitl_open_context_pr_after_lock and _hitl_pr_worktree_path is not None: + try: + _pkg._open_context_pr_at_implement_start(pipeline_id, repo_path=repo_path) + except _pkg.ContextPrCreationError as ctx_err: + _pkg.logger.warning( + "Context PR opener: HITL-resume failed " + "(continuing — hard-require enforced at " + "advance_phase and the implement-start plan " + "pre-flight gate) (#2777, #3100)", + pipeline_id=pipeline_id, + reason=ctx_err.reason, + error=str(ctx_err), + ) + except Exception as hitl_err: # noqa: BLE001 + _pkg.logger.warning( + "Context PR opener: HITL-resume outer wrapper raised (continuing) (#2777)", + pipeline_id=pipeline_id, + error=str(hitl_err), + ) + + # Launch runner thread + thread = _pkg.threading.Thread( + target=_pkg._run_pipeline, + args=(pipeline_id, repo_path), + daemon=True, + name=f"pipeline-{pipeline_id}", + ) + thread.start() + + _pkg.logger.info( + "Pipeline recovered from AWAITING_HUMAN", + pipeline_id=pipeline_id, + recovery_action="advance" if is_approved else "rerun", + ) + + return _pkg.make_success_response( + "Pipeline recovered and started", + data={ + "pipeline_id": pipeline_id, + "status": "running", + "current_phase": pipeline.current_phase.value, + }, + ) + + if pipeline.status == _pkg.PipelineStatus.COMPLETE: + return _pkg.make_error_response( + f"Pipeline {pipeline_id} is already complete", + status_code=409, + ) + + if pipeline.status == _pkg.PipelineStatus.CANCELLED: + return _pkg.make_error_response( + f"Pipeline {pipeline_id} is cancelled", + status_code=409, + ) + + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + + if pipeline.status == _pkg.PipelineStatus.FAILED: + # Reset the failed phase so it can be re-run. + # Also reset phases stuck in RUNNING — a pipeline-level exception + # sets the pipeline to FAILED without updating the phase status. + phase_execution = pipeline.get_phase_execution(pipeline.current_phase) + if phase_execution.status in ( + _pkg.PipelineStatus.FAILED, + _pkg.PipelineStatus.RUNNING, + ): + # Refuse to clear containers/agents/artifacts when pods + # labeled to this pipeline are still alive — the reset + # would orphan them (#2420). + guard = _pkg._guard_live_pods_or_force(pipeline_id, force, force_reason) + if guard is not None: + return guard + prev_status = phase_execution.status.value + phase_execution.status = _pkg.PipelineStatus.PENDING + phase_execution.started_at = None + phase_execution.work_started_at = None + phase_execution.completed_at = None + phase_execution.error = None + phase_execution.review_cycles = 0 + phase_execution.hitl_review_cycles = 0 + phase_execution.containers = [] + phase_execution.agents = [] + phase_execution.artifacts = {} + _pkg.logger.info( + "Resetting phase for restart", + pipeline_id=pipeline_id, + phase=pipeline.current_phase.value, + previous_phase_status=prev_status, + ) + pipeline.error = None + + # Bump run_epoch so the old _run_pipeline thread's finally block + # detects the restart and skips worktree cleanup. + pipeline.run_epoch = _pkg.datetime.now(_pkg.UTC) + + # Mark pipeline as running + pipeline.status = _pkg.PipelineStatus.RUNNING + store.save_pipeline(pipeline) + + # Run the pipeline in a background thread + thread = _pkg.threading.Thread( + target=_pkg._run_pipeline, + args=(pipeline_id, repo_path), + daemon=True, + name=f"pipeline-{pipeline_id}", + ) + thread.start() + + _pkg.logger.info("Pipeline started", pipeline_id=pipeline_id) + + return _pkg.make_success_response( + "Pipeline started", + data={ + "pipeline_id": pipeline_id, + "status": "running", + "current_phase": pipeline.current_phase.value, + }, + ) + + except _pkg.InvalidPipelineIdError: + return _pkg.make_error_response( + f"Invalid pipeline ID format: {pipeline_id}", + status_code=400, + ) + except _pkg.PipelineNotFoundError: + return _pkg.make_error_response( + f"Pipeline {pipeline_id} not found", + status_code=404, + ) diff --git a/orchestrator/routes/pipelines/_routes_read.py b/orchestrator/routes/pipelines/_routes_read.py new file mode 100644 index 0000000000..ed5ddb5dec --- /dev/null +++ b/orchestrator/routes/pipelines/_routes_read.py @@ -0,0 +1,120 @@ +"""read-route bodies helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _list_pipelines_body() -> tuple[_pkg.Response, int]: + """ + List all pipelines. + + Query params: + repo_path: Path to repository (optional) + active_only: Only return active pipelines (default: false) + + Response: + { + "success": true, + "data": { + "pipelines": [ + {"id": "issue-123", "status": "running", ...}, + ... + ] + } + } + """ + repo_path = _pkg.get_repo_path() + active_only = _pkg.request.args.get("active_only", "false").lower() == "true" + + try: + all_pipelines = _pkg._collect_all_pipelines(repo_path) + + if active_only: + pipelines = [ + p + for p in all_pipelines + if p.status + not in ( + _pkg.PipelineStatus.COMPLETE, + _pkg.PipelineStatus.FAILED, + _pkg.PipelineStatus.CANCELLED, + ) + ] + else: + pipelines = all_pipelines + + # Convert to response format + pipeline_data = [ + { + "id": p.id, + "issue_number": p.issue_number, + "repo": p.repo, + "branch": p.branch, + "status": p.status.value, + "current_phase": p.current_phase.value, + "created_at": p.created_at.isoformat(), + "updated_at": p.updated_at.isoformat(), + } + for p in pipelines + ] + + return _pkg.make_success_response( + f"Found {len(pipelines)} pipeline(s)", + data={"pipelines": pipeline_data}, + ) + + except _pkg.StateStoreError as e: + _pkg.logger.error("Failed to list pipelines", error=str(e)) + return _pkg.make_error_response(f"Failed to list pipelines: {e}", status_code=500) + + +def _get_pipeline_body(pipeline_id: str) -> tuple[_pkg.Response, int]: + """ + Get a pipeline by ID. + + URL params: + pipeline_id: Pipeline ID (e.g., "issue-123") + + Query params: + repo_path: Path to repository (optional) + + Response: + { + "success": true, + "data": { + "pipeline": {...} + } + } + """ + repo_path = _pkg.get_repo_path() + + try: + _store, pipeline = _pkg._resolve_pipeline(pipeline_id, repo_path) + + return _pkg.make_success_response( + "Pipeline retrieved", + data={"pipeline": pipeline.model_dump(mode="json")}, + ) + + except _pkg.InvalidPipelineIdError: + return _pkg.make_error_response( + f"Invalid pipeline ID format: {pipeline_id}", + status_code=400, + ) + except _pkg.PipelineNotFoundError: + return _pkg.make_error_response( + f"Pipeline {pipeline_id} not found", + status_code=404, + ) + except _pkg.StateValidationError as e: + _pkg.logger.error("Pipeline validation failed", pipeline_id=pipeline_id, error=str(e)) + return _pkg.make_error_response( + f"Pipeline state is invalid: {e}", + status_code=500, + ) diff --git a/orchestrator/routes/pipelines/_routes_restart.py b/orchestrator/routes/pipelines/_routes_restart.py new file mode 100644 index 0000000000..4d8d047889 --- /dev/null +++ b/orchestrator/routes/pipelines/_routes_restart.py @@ -0,0 +1,1070 @@ +"""restart-route bodies helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _restart_agent_body(pipeline_id: str, agent_role: str) -> tuple[_pkg.Response, int]: + """Restart a single agent in a pipeline (orchestrator-native). + + After #3164 the orchestrator unconditionally owns the BRC event + loop: agent work runs as one-shot Jobs spawned per actionable + event by the event loop, and the in-pod wait arm is gone. A + resident pod spawned here without ``EGG_EVENT_ACTION`` would + immediately log FATAL and ``exit 64``, so ``restart_agent`` no + longer spawns anything itself. Instead it: + + 0. Enforces the per-(pipeline, role, slice) restart budget + (``check_and_increment_restart_count``); a request over budget is + rejected with HTTP 429 before any state is mutated (#3244). + 1. Best-effort deletes the role's live one-shot Job(s) (to kill a + stuck pod). One-shot Jobs carry an event-discriminator suffix + in their name, so they are found by label + (``LABEL_PIPELINE_ID`` + ``LABEL_AGENT_ROLE`` [+ ``LABEL_SLICE_ID`` + when slice-scoped]), not by name. + 2. Resets the role's consensus state and health-monitor anchor. + 3. Marks the agent record RUNNING with ``container_id = None``. + + For a pipeline that is already RUNNING, the live event loop (polling + ~every 5s during the concurrent phase) spawns a fresh one-shot pod once + the role's consensus state is reset — that is the respawn. For a pipeline + that was FAILED/CANCELLED the event loop and its ``_run_pipeline`` driver + thread are already dead, so the route also relaunches a fresh driver + thread (mirroring ``restart_phase``) to restart the event loop; otherwise + the reset would leave the pipeline RUNNING-but-idle with nothing to + respawn it (#3244). The agent's per-agent worktree is preserved so + committed work is retained. + + URL params: + pipeline_id: Pipeline ID + agent_role: Agent role to restart (e.g. "coder", "tester") + + Query string (optional): + slice_id: Slice scope (``slice-<N>``). When supplied, the + slice-scoped Job and worktree are restarted, ``EGG_SLICE_ID`` + is propagated to the new Job, and consensus reset targets + the per-slice tracker. ``slice_id`` may also be supplied via + the JSON body. When omitted for a role that runs as a + per-slice agent, it is derived from the phase's agent records + (#2759): if exactly one slice has a non-complete record for + the role, that slice is used; otherwise the request is + rejected with the candidate list rather than spawning an + unscoped agent. The scan is scoped to ``pipeline.current_phase`` + only — if the pipeline has advanced past the slice's phase + (e.g. to ``pr`` or a later iteration) no current-phase records + will name the role, derivation falls through, and the operator + should supply ``slice_id`` explicitly. This is operator guidance, + not a code-enforced precondition: the fall-through branch + proceeds to a pipeline-level spawn rather than rejecting. + Genuinely pipeline-level agents (no per-slice records for the + role) omit ``slice_id``. + + Request body (optional): + { + "reason": "Human-readable reason for the restart", + "slice_id": "slice-2" + } + + Response: + { + "success": true, + "data": { + "agent_role": "coder", + "slice_id": "slice-2", + "respawn": "delegated to orchestrator event loop", + "restart_count": 1 + } + } + """ + repo_path = _pkg.get_repo_path() + + try: + store, pipeline = _pkg._resolve_pipeline(pipeline_id, repo_path) + except _pkg.InvalidPipelineIdError: + return _pkg.make_error_response( + f"Invalid pipeline ID format: {pipeline_id}", status_code=400 + ) + except _pkg.PipelineNotFoundError: + return _pkg.make_error_response(f"Pipeline {pipeline_id} not found", status_code=404) + + # Validate agent role + try: + role = _pkg.AgentRole(agent_role) + except ValueError: + return _pkg.make_error_response(f"Invalid agent role: {agent_role}", status_code=400) + + # Validate pipeline is in a restartable state. CANCELLED is included so + # that a cancel_task(cleanup=false) pipeline can be resumed without a + # full resubmission (see #1725). + if pipeline.status not in ( + _pkg.PipelineStatus.RUNNING, + _pkg.PipelineStatus.AWAITING_HUMAN, + _pkg.PipelineStatus.FAILED, + _pkg.PipelineStatus.CANCELLED, + ): + return _pkg.make_error_response( + f"Pipeline {pipeline_id} is not in a restartable state (status: {pipeline.status.value})", + status_code=409, + ) + + body = _pkg.request.get_json(silent=True) or {} + reason = body.get("reason", "Manual restart via API") + + # Slice scope (#2410): query param wins over body so the URL + # form is unambiguous; both forms validate against the canonical + # ``slice-<N>`` shape via ``extract_slice_id``. + raw_slice_id = _pkg.request.args.get("slice_id") + slice_payload = {"slice_id": raw_slice_id} if raw_slice_id is not None else body + try: + slice_id = _pkg.extract_slice_id(slice_payload) + except ValueError as e: + return _pkg.make_error_response(str(e), status_code=400) + + # Slice auto-derivation (#2759). A slice-mode restart that omits + # ``slice_id`` would otherwise spawn the agent pipeline-level: + # ``EGG_SLICE_ID`` is set by the spawner only when ``slice_id`` is + # non-None, so the respawned agent's BRC signals route to the bare + # pipeline tracker instead of the slice's tracker. The slice's own + # tracker keeps the dead agent registered while the live one ACKs + # into the wrong tracker — the slice's consensus then wedges with no + # message-bus recovery path. Since ``restart_agent`` is the + # operator's normal tool for recovering a failed container, the + # omission must not silently produce an unscoped agent. + # + # When the role runs as a per-slice agent (it has slice-scoped + # records in the current phase), derive the slice: the k8s monitor + # marks a cleanly-exited agent COMPLETE and a crashed one FAILED, so + # a single non-COMPLETE record isolates the slice that needs the + # restart. If the choice is ambiguous — multiple non-COMPLETE + # records, or none at all — reject with the candidate list so the + # operator re-issues with an explicit ``slice_id``. + if slice_id is None: + derive_phase_exec = pipeline.phases.get(pipeline.current_phase.value) + if derive_phase_exec is not None: + role_records = [ + a + for a in derive_phase_exec.agents + if hasattr(a, "role") + and (a.role == role or (hasattr(a.role, "value") and a.role.value == role.value)) + ] + sliced_records = [a for a in role_records if getattr(a, "slice_id", None)] + if sliced_records: + known_slices = sorted({a.slice_id for a in sliced_records}) + restart_candidates = sorted( + { + a.slice_id + for a in sliced_records + if a.status != _pkg.AgentExecutionStatus.COMPLETE + } + ) + if len(restart_candidates) == 1: + slice_id = restart_candidates[0] + _pkg.logger.info( + "restart_agent: derived slice_id from phase agent records", + pipeline_id=pipeline_id, + agent_role=agent_role, + slice_id=slice_id, + ) + else: + detail = ( + "no slice has a non-complete agent record for this role" + if not restart_candidates + else f"{len(restart_candidates)} slices have a non-complete record" + ) + return _pkg.make_error_response( + f"Agent role {agent_role!r} runs as a per-slice agent in " + f"pipeline {pipeline_id}; restart_agent could not derive " + f"slice_id ({detail}). Re-issue with an explicit slice_id.", + status_code=400, + details={ + "agent_role": agent_role, + "known_slices": known_slices, + "restart_candidates": restart_candidates, + }, + reason="slice_id_required", + ) + + # Slice-existence check (#2421): a well-formed but unknown + # ``slice_id`` would otherwise spawn an orphan Job + worktree + # the rest of the system has no record of. The shape regex in + # ``extract_slice_id`` only catches malformed values; only the + # contract knows which slices the pipeline actually has. + # + # Pipelines without a contract are not + # slice-aware, so any non-``None`` ``slice_id`` targeting them is + # by definition unknown — reject outright. For contracted + # pipelines, load the contract and check membership; fall through + # silently if the contract can't be loaded (worktree pruned, + # contract not yet populated, filesystem error) so we don't + # regress legitimate restarts on the existing pipeline-level path. + # + # After #3164 ``restart_agent`` no longer spawns a worktree itself, + # so the slice's parent-edge / base-branch resolution that used to + # feed the spawn is gone. Only the existence check below remains. + if slice_id is not None: + if not pipeline.has_contract: + return _pkg.make_error_response( + f"slice_id {slice_id!r} is invalid for pipeline " + f"{pipeline_id} (pipeline has no contract; not slice-aware)", + status_code=404, + details={ + "slice_id": slice_id, + "known_slices": [], + }, + ) + try: + from egg_contracts.loader import ( + ContractNotFoundError, + ContractValidationError, + load_contract, + ) + from routes import resolve_worktree_path + except ImportError: + _pkg.logger.warning( + "Required modules unavailable; skipping slice_id existence check", + pipeline_id=pipeline_id, + slice_id=slice_id, + ) + else: + contract = None + try: + worktree_path = resolve_worktree_path(pipeline_id, _pkg.Path(repo_path)) + contract_id = _pkg._pipeline_identifier(pipeline.issue_number, pipeline_id) + try: + contract = load_contract(contract_id, worktree_path) + except ContractNotFoundError: + # Contract not yet populated — fall through silently + # (``contract`` already initialised to ``None`` above). + pass + except (OSError, ValueError, ContractValidationError) as exc: + # Worktree pruned, filesystem failure, or corrupt/invalid + # contract JSON: log and fall through. The reviewer's #2421 + # ask was to catch the easy "wrong slice_id" case, not to + # gate restarts on contract reachability. Programmer errors + # (AttributeError, TypeError, NameError) are left to + # propagate so they surface during development. + _pkg.logger.warning( + "Could not load contract for slice_id existence check; allowing restart", + pipeline_id=pipeline_id, + slice_id=slice_id, + error=str(exc), + ) + if contract is not None: + slice_obj = next((s for s in contract.slices if s.id == slice_id), None) + if slice_obj is None: + return _pkg.make_error_response( + f"slice_id {slice_id!r} does not match any slice in " + f"pipeline {pipeline_id}'s contract", + status_code=404, + details={ + "slice_id": slice_id, + "known_slices": sorted(s.id for s in contract.slices), + }, + ) + + spawner = _pkg._get_spawner() + + current_phase = pipeline.current_phase.value + phase_exec = pipeline.phases.get(current_phase) + + # Enforce the per-(pipeline, role, slice) restart budget BEFORE any + # destructive action (#3244 review). Pre-#3164 this cap lived inside + # ``restart_agent_job``, which the route no longer calls — without + # re-enforcing it here an operator/overseer could call ``restart_agent`` + # without bound, each call resetting consensus and actively preventing a + # live phase from converging. ``check_and_increment_restart_count`` raises + # when the budget is exhausted; reject loudly (429) instead of flipping + # status / resetting consensus and returning a misleading success. The + # returned count is the source of truth for the ``restart_count`` + # telemetry below (the old read-only ``get_restart_count`` read always + # reported 0 on this path since nothing incremented it). + try: + new_restart_count = spawner.check_and_increment_restart_count( + pipeline_id, role, slice_id=slice_id + ) + except _pkg.KubernetesSpawnError as budget_err: + _pkg.logger.warning( + "restart_agent rejected: restart budget exhausted", + pipeline_id=pipeline_id, + agent_role=agent_role, + slice_id=slice_id, + error=str(budget_err), + ) + return _pkg.make_error_response(str(budget_err), status_code=429) + + # Early status update: transition FAILED/CANCELLED -> RUNNING so that + # get_status returns "running" immediately. Unlike a RUNNING pipeline — + # whose live event loop picks up the consensus reset below and respawns + # within one poll — a FAILED/CANCELLED pipeline has NO live event loop: + # ``_run_concurrent_phase`` already returned and ``stop_event_loop()`` + # tore the loop down on its way out, and the ``_run_pipeline`` driver + # thread has exited. Resetting consensus alone would leave the pipeline + # RUNNING-but-idle with nothing to respawn it (#3244 review). So when we + # make this transition we record it and relaunch a fresh ``_run_pipeline`` + # driver thread at the end of the route (mirroring ``restart_phase`` step + # 7) — that restarts the event loop, which then performs the respawn. + pipeline_was_inactive = pipeline.status in ( + _pkg.PipelineStatus.FAILED, + _pkg.PipelineStatus.CANCELLED, + ) + if pipeline_was_inactive: + early_lock = _pkg.get_pipeline_state_lock(pipeline_id) + with early_lock: + pipeline = store.load_pipeline(pipeline_id) + if pipeline.status in (_pkg.PipelineStatus.FAILED, _pkg.PipelineStatus.CANCELLED): + pipeline.status = _pkg.PipelineStatus.RUNNING + _phase_exec = pipeline.phases.get(current_phase) + if _phase_exec is not None: + _phase_exec.status = _pkg.PipelineStatus.RUNNING + # Bump run_epoch so the relaunched driver thread (below) owns a + # fresh epoch namespace and any stale thread that observes the + # transition detects itself as superseded (mirrors + # ``restart_phase`` / ``advance_phase``). + pipeline.run_epoch = _pkg.datetime.now(_pkg.UTC) + pipeline.updated_at = _pkg.datetime.now(_pkg.UTC) + store.update_pipeline(pipeline_id, pipeline.model_dump(mode="json")) + else: + # Lost the race — another writer already moved it off + # FAILED/CANCELLED, so its driver thread / event loop is + # live and will own the respawn. Don't relaunch a duplicate. + pipeline_was_inactive = False + + # #3164: ``restart_agent`` no longer spawns a resident pod. The + # orchestrator event loop owns the BRC respawn — once the role's + # consensus state is reset (below), it spawns a fresh one-shot pod + # within one ~5s poll. Here we only (1) kill any live one-shot Job + # for the role so a stuck pod is torn down, then (2) reset consensus + # + health so the event loop reschedules. + + # Delete the role's live one-shot Job(s), best-effort. One-shot + # event Jobs carry an event-discriminator SUFFIX in their name (one + # Job per actionable BRC event), so they can't be addressed by a + # deterministic name — find them by LABEL. Match on pipeline + + # role (and slice when scoped). Zero matches is fine (the role may + # have already exited cleanly); the event loop will respawn either + # way once consensus is reset. Wrap broadly so a k8s/list failure + # never fails the restart. + job_labels = { + _pkg.LABEL_PIPELINE_ID: pipeline_id, + # The role label value is the underscore form (e.g. + # ``reviewer_code``), which is exactly ``agent_role`` / ``role.value``. + _pkg.LABEL_AGENT_ROLE: role.value, + } + if slice_id is not None: + job_labels[_pkg.LABEL_SLICE_ID] = slice_id + try: + live_jobs = spawner.k8s.list_containers(labels=job_labels) + removed_jobs = 0 + for job in live_jobs: + try: + # Mirror the cleanup call sites: prefer the explicit + # ``job_name`` (already Job-prefixed), fall back to the + # container id which ``remove_agent_job`` -> ``remove_container`` + # resolves to a Job name. + spawner.remove_agent_job(job.job_name or job.container_id, force=True) + removed_jobs += 1 + except Exception as job_err: # noqa: BLE001 - best-effort teardown + _pkg.logger.warning( + "Failed to delete live one-shot Job during restart (best-effort)", + pipeline_id=pipeline_id, + agent_role=agent_role, + slice_id=slice_id, + job_name=getattr(job, "job_name", None), + error=str(job_err), + ) + _pkg.logger.info( + "restart_agent: deleted live one-shot Job(s) for role", + pipeline_id=pipeline_id, + agent_role=agent_role, + slice_id=slice_id, + removed=removed_jobs, + ) + except Exception as list_err: # noqa: BLE001 - best-effort teardown + _pkg.logger.warning( + "Failed to list live one-shot Jobs during restart (best-effort)", + pipeline_id=pipeline_id, + agent_role=agent_role, + slice_id=slice_id, + error=str(list_err), + ) + + # Reset consensus state for this agent so the event loop reschedules + # a fresh one-shot pod for it. If consensus reset fails, log a + # warning but don't fail the restart: the agent will re-enter + # consensus on its own. Slice-scoped restarts (#2410) target the + # per-slice tracker; the pipeline-level tracker has no record of the + # slice agent. + # Slice-scoped restarts (#2410) target the per-slice tracker; the + # pipeline-level tracker has no record of the slice agent. + # + # INVARIANT (#3200 task-7-1, mid-phase BRC record survival): this reset + # clears the *peer consensus tracker* (the ephemeral ACK/NACK/proposal + # bookkeeping the restarted agent rebuilds by re-proposing) but MUST NOT + # clear the *Redis message store* (``pipeline:{id}:messages``). That store + # is the durable BRC message record — CONSENSUS_PROPOSE/ACK/NACK and the + # conditional-ACK obligations — and a mid-phase restart deliberately + # preserves it so the reseeded/resumed session can re-pull it via + # ``GET /<pipeline_id>/brc-transcript`` + ``read_peer_artifact`` and + # re-derive the #3189 deterministic anchors. The store is cleared only at + # phase transitions (``_clear_concurrent_state``) and pipeline + # create/delete (``_clear_pipeline_runtime_state``), never here. Do NOT + # add ``get_message_store().clear()`` / ``_clear_concurrent_state`` to the + # restart path — that would lose the record across the restart boundary. + try: + try: + from peer_consensus import get_peer_consensus_tracker + except ImportError: + from ..peer_consensus import ( + get_peer_consensus_tracker, # type: ignore[import-not-found] + ) + + tracker = get_peer_consensus_tracker(pipeline_id, slice_id) + if tracker: + tracker.remove_agent(agent_role) + _pkg.logger.info( + "Reset consensus state for agent", + pipeline_id=pipeline_id, + agent_role=agent_role, + slice_id=slice_id, + ) + except ImportError: + pass + except Exception as e: + _pkg.logger.warning( + "Failed to reset consensus state (agent will re-enter consensus)", + pipeline_id=pipeline_id, + agent_role=agent_role, + slice_id=slice_id, + error=str(e), + ) + + # Reset health-monitor anchor so the pre-respawn _last_heartbeat does not + # generate a stale-elapsed heartbeat_timeout alert against the fresh + # container (issue #2084). + # + # #2270 slice-5 (restart hygiene): ``reset_agent`` also drops the agent's + # accumulated per-agent escalation state (escalation flags, error counts, + # active alerts). Clearing it on restart is what stops a freshly-restarted + # agent from inheriting a stale redirect/escalation history that would push + # it straight to HITL on its first post-restart stall. The Tier-2 overseer's + # own escalation-history clear + generation reset live on + # ``OverseerMonitor`` (overseer/monitor.py:reset_escalation_history / + # reset_generation), which the on-demand adjudicator constructs fresh. + try: + try: + from health_monitor import get_health_monitor + except ImportError: + from ..health_monitor import ( + get_health_monitor, # type: ignore[import-not-found] + ) + _hm = get_health_monitor() + if _hm is not None: + _hm.reset_agent(agent_role) + except Exception as e: + _pkg.logger.warning( + "Failed to reset health-monitor state for restarted agent", + pipeline_id=pipeline_id, + agent_role=agent_role, + error=str(e), + ) + + # Update pipeline state. No resident container is spawned (#3164) — + # the event loop will respawn a one-shot pod within one poll once the + # consensus reset above takes effect. We mark the agent RUNNING with + # ``container_id = None`` (the live pod is set by the event loop) and + # refresh ``started_at`` so the overseer's + # phase_minimum_working_window suppression on the + # ``agent-heartbeat-stall`` trigger anchors on the restart (#2084). + lock = _pkg.get_pipeline_state_lock(pipeline_id) + with lock: + pipeline = store.load_pipeline(pipeline_id) + if phase_exec is not None: + # Re-fetch from the freshly loaded pipeline (the outer check gates + # on "did the phase exist before the restart?"). + fresh_phase_exec = pipeline.phases.get(current_phase) + if fresh_phase_exec is not None: + from models import AgentExecution # type: ignore + + respawn_started_at = _pkg.datetime.now(_pkg.UTC) + # Match on ``(role, slice_id)`` — without the slice tiebreaker + # the first matching role wins, which on a multi-slice phase + # mutates the wrong slice's record (#2422). ``slice_id`` is + # the route-level scope already plumbed into the consensus + # tracker above. + found = False + for agent in fresh_phase_exec.agents: + if not hasattr(agent, "role"): + continue + role_match = agent.role == role or ( + hasattr(agent.role, "value") and agent.role.value == role.value + ) + if not role_match: + continue + if getattr(agent, "slice_id", None) != slice_id: + continue + agent.container_id = None + agent.status = _pkg.AgentExecutionStatus.RUNNING + agent.started_at = respawn_started_at + found = True + break + if not found: + fresh_phase_exec.agents.append( + AgentExecution( + role=role, + container_id=None, + status=_pkg.AgentExecutionStatus.RUNNING, + started_at=respawn_started_at, + slice_id=slice_id, + ) + ) + + pipeline.updated_at = _pkg.datetime.now(_pkg.UTC) + store.update_pipeline(pipeline_id, pipeline.model_dump(mode="json")) + + # ``restart_count`` is the value just incremented by + # ``check_and_increment_restart_count`` above (#3244). It is scoped to the + # same ``(pipeline_id, agent_role, slice_id)`` bucket the cap is enforced + # on, so it correctly reports the operator's "you've burned N of M + # restarts" telemetry — the pre-fix read-only ``get_restart_count`` read + # always reported 0 here because nothing on this path incremented it. + response_data: dict[str, object] = { + "agent_role": agent_role, + "slice_id": slice_id, + "respawn": "delegated to orchestrator event loop", + "restart_count": new_restart_count, + } + + # When the pipeline was FAILED/CANCELLED its event loop and driver thread + # are dead (see the early-status comment above), so the consensus reset + # alone has nothing to act on it. Relaunch a fresh ``_run_pipeline`` driver + # thread — exactly as ``restart_phase`` step 7 does — to restart the event + # loop, which then respawns the role's one-shot Job within one poll. For a + # pipeline that was already RUNNING we skip this: its live event loop owns + # the respawn and a second driver thread would race it (#3244 review). + if pipeline_was_inactive: + _pkg._spawn_pipeline_run_thread(pipeline_id, store.repo_path, pipeline.run_epoch) + _pkg.logger.info( + "restart_agent: relaunched driver thread for inactive pipeline", + pipeline_id=pipeline_id, + agent_role=agent_role, + slice_id=slice_id, + run_epoch=pipeline.run_epoch.isoformat() if pipeline.run_epoch else None, + ) + + _pkg.logger.info( + "Agent restart requested (respawn delegated to event loop)", + pipeline_id=pipeline_id, + agent_role=agent_role, + slice_id=slice_id, + restart_count=response_data.get("restart_count"), + reason=reason, + ) + + return _pkg.make_success_response( + f"Agent {agent_role} restarted", + data=response_data, + ) + + +def _restart_phase_body(pipeline_id: str, phase: str) -> tuple[_pkg.Response, int]: + """Restart all agents in a pipeline phase. + + Stops and removes all containers for the phase, resets consensus and + review cycle state, and respawns all agents. Prior phase artifacts + (from earlier phases) are preserved. + + Preservation semantics (#3080): per-agent worktrees AND their local + branches are deleted, so per-role branch tips do not survive a phase + restart. Unpushed commits are salvaged to ``egg/recovered/*`` refs + on a best-effort basis (#2429) — ``auto_salvage_pipeline`` + re-enumerates worktrees with ``validate_git=True``, so worktrees + with a corrupted ``.git`` marker (the #1723 failure class) may be + skipped without salvage. The respawned agents' fresh worktrees + re-fork from the shared work branch tip (``origin/<assigned_branch>``, + base-branch fallback when unpushed — see #3068). Anything that + lived only on a per-role branch (e.g. a reviewer's merge history) + is therefore discarded from agent trees; only state pushed to the + shared work branch is re-materialised on respawn. Operators needing + per-worktree retention should use ``restart_agent`` instead. + + URL params: + pipeline_id: Pipeline ID + phase: Phase name to restart (e.g. "implement") + + Request body (optional): + { + "reason": "Human-readable reason for the restart" + } + + Response: + { + "success": true, + "data": { + "phase": "implement", + "agents_to_restart": ["coder", "tester", "documenter", ...] + } + } + """ + repo_path = _pkg.get_repo_path() + + try: + store, pipeline = _pkg._resolve_pipeline(pipeline_id, repo_path) + except _pkg.InvalidPipelineIdError: + return _pkg.make_error_response( + f"Invalid pipeline ID format: {pipeline_id}", status_code=400 + ) + except _pkg.PipelineNotFoundError: + return _pkg.make_error_response(f"Pipeline {pipeline_id} not found", status_code=404) + + # Validate phase + try: + _pkg.PipelinePhase(phase) + except ValueError: + return _pkg.make_error_response(f"Invalid phase: {phase}", status_code=400) + + # Validate pipeline is in a restartable state. CANCELLED is included so + # that a cancel_task(cleanup=false) pipeline can be resumed without a + # full resubmission (see #1725). + if pipeline.status not in ( + _pkg.PipelineStatus.RUNNING, + _pkg.PipelineStatus.AWAITING_HUMAN, + _pkg.PipelineStatus.FAILED, + _pkg.PipelineStatus.CANCELLED, + ): + return _pkg.make_error_response( + f"Pipeline {pipeline_id} is not in a restartable state (status: {pipeline.status.value})", + status_code=409, + ) + + # Only the current phase can be restarted — restarting a completed or + # future phase would corrupt pipeline state. + if phase != pipeline.current_phase.value: + return _pkg.make_error_response( + f"Phase {phase} is not the current phase (current: {pipeline.current_phase.value})", + status_code=409, + ) + + phase_exec = pipeline.phases.get(phase) + if phase_exec is None: + return _pkg.make_error_response( + f"Phase {phase} not found in pipeline {pipeline_id}", status_code=404 + ) + + body = _pkg.request.get_json(silent=True) or {} + reason = body.get("reason", "Manual phase restart via API") + + # Compute gateway mode from pipeline config (not hardcoded "public") + gateway_mode, _ = _pkg._compute_gateway_mode(pipeline) + + spawner = _pkg._get_spawner() + + # Acquire the pipeline state lock to collect agent roles, snapshot + # container IDs, and update pipeline status to RUNNING *before* the + # slow container teardown. This ensures that ``get_status`` returns + # ``running`` immediately, even if the MCP call times out during + # container stop/remove (see #1594). + lock = _pkg.get_pipeline_state_lock(pipeline_id) + with lock: + # Re-load pipeline under the lock so agent_roles reflects the + # latest state (guards against concurrent modifications). + pipeline = store.load_pipeline(pipeline_id) + + # Re-check current phase under the lock to prevent TOCTOU race: + # the pipeline could have advanced between the earlier check and + # lock acquisition. + if phase != pipeline.current_phase.value: + return _pkg.make_error_response( + f"Phase {phase} is not the current phase (current: {pipeline.current_phase.value})", + status_code=409, + ) + + phase_exec = pipeline.phases.get(phase) + if phase_exec is None: + return _pkg.make_error_response( + f"Phase {phase} not found in pipeline {pipeline_id}", status_code=404 + ) + + # 1. Collect agent roles for respawning. Prefer the runtime cache + # on ``phase_exec.agents`` since it reflects the roster from + # the most recent spawn, but fall back to the deterministic + # source the executor itself consults — ``get_roles_for_phase``. + # Without this fallback a restart whose clear step ran + # (``phase_exec.agents = []`` below) but whose spawn step + # failed leaves the pipeline unrecoverable: every subsequent + # ``restart_phase`` 400s on the now-empty cache, and + # ``start_pipeline`` 409s on the CANCELLED state (#2515). + agent_roles: list[_pkg.AgentRole] = [] + for agent in phase_exec.agents: + if hasattr(agent, "role"): + role = ( + agent.role + if isinstance(agent.role, _pkg.AgentRole) + else _pkg.AgentRole(agent.role) + ) + agent_roles.append(role) + + if not agent_roles: + # Mirror ``_run_concurrent_phase`` exactly so the route's + # response (and the downstream worktree-delete / health- + # monitor reset) matches the roster the spawn will actually + # produce. + try: + from egg_contracts.agent_roles import ( + get_roles_for_phase as _get_roles_for_phase, + ) + + for r in _get_roles_for_phase( + phase, + include_reviewers=True, + repo=pipeline.repo, + has_contract=getattr(pipeline, "has_contract", True), + ): + try: + agent_roles.append(_pkg.AgentRole(r.value)) + except ValueError: + continue + except Exception as exc: # noqa: BLE001 + # Catch derivation failures so the route returns 400 + # rather than 500 — deliberate divergence from + # ``_run_concurrent_phase``, which lets the same failure + # propagate up the worker thread. In a synchronous HTTP + # context an honest 400 ("No agents found") is more + # useful to the operator than a 500. + _pkg.logger.warning( + "restart_phase: failed to derive default roster fallback", + pipeline_id=pipeline_id, + phase=phase, + error=str(exc), + ) + + if not agent_roles: + return _pkg.make_error_response( + f"No agents found in phase {phase} to restart", status_code=400 + ) + + _pkg.logger.info( + "restart_phase: phase_exec.agents empty, derived roster from pipeline config", + pipeline_id=pipeline_id, + phase=phase, + agent_roles=[r.value for r in agent_roles], + ) + + # 2. Snapshot container IDs for teardown outside the lock + old_container_ids = [c.container_id for c in phase_exec.containers] + + # 3. Fully reset phase execution state so the new _run_pipeline + # thread treats this as a fresh phase. Set pipeline status to + # RUNNING and bump run_epoch so any lingering old _run_pipeline + # thread detects the restart and exits (see #1638). + # NOTE: artifacts are intentionally preserved — they may contain + # outputs from partial work useful as context for the retry. + phase_exec.containers = [] + phase_exec.agents = [] + phase_exec.review_cycles = 0 + phase_exec.hitl_review_cycles = 0 + phase_exec.status = _pkg.PipelineStatus.PENDING + phase_exec.started_at = None + phase_exec.work_started_at = None + phase_exec.completed_at = None + phase_exec.error = None + phase_exec.cycle_timings = [] + pipeline.status = _pkg.PipelineStatus.RUNNING + pipeline.error = None + pipeline.run_epoch = _pkg.datetime.now(_pkg.UTC) + # ``updated_at`` is unconditionally set by ``StateStore.save_pipeline`` + # (which ``update_pipeline`` routes through). + store.update_pipeline(pipeline_id, pipeline.model_dump(mode="json")) + + # --- Outside the lock: slow, idempotent, best-effort operations --- + + # 3b. Persist the in-flight phase's BRC message record to disk BEFORE the + # destructive container/worktree teardown (#3200 task-7-1, mid-phase + # BRC record survival). Today ``_write_brc_history`` runs only at phase + # transitions (``_persist_phase_brc_history`` in complete/advance_phase, + # #1827); a mid-phase restart never wrote the durable on-disk + # transcript. + # + # PRIMARY mechanism is option (a), the live Redis stream: it survives a + # bare restart (the store is cleared only at phase transitions / + # pipeline create+delete, never here — see step 5), so a reseeded + # session re-pulls the in-flight record from Redis via + # ``/brc-transcript`` + ``read_peer_artifact``. The slice-scoped + # CONSENSUS_PROPOSE/ACK/NACK records of an in-flight implement slice + # rely on (a) for survival. + # + # This disk persist (option (b)) is a belt-and-suspenders ADD-ON with a + # deliberately NARROW durability scope — do not overstate it. It calls + # ``_persist_phase_brc_history`` -> ``_write_brc_history( + # write_per_slice=False)``. For a slice-aware implement phase that path + # writes ONLY the ``{id}-implement-unattributed.{md,json}`` sibling + # (non-CONSENSUS BRC types: HEARTBEAT/STATUS/HANDOFF/AGENT_FAILED/ + # NUDGE/OVERSEER_ALERT) and SKIPS the per-slice bucket loop; the + # slice's CONSENSUS_* proposals/verdicts/open-NACKs are NOT written to + # disk here (write_per_slice=False avoids the #2755 add/add conflict on + # ``work``; per-slice files are owned by the slice integration branch). + # So across a FULL Redis loss (orchestrator pod death, the cold-start + # case task-6-1 covers) the in-flight slice record does NOT survive on + # disk — only (a) preserves it. What (b) does buy: for non-slice phases + # (plan/refine/pr) and non-slice implement runs the aggregate + # ``{id}-{phase}.{md,json}`` transcript IS written, and for slice runs + # the unattributed audit sibling is captured — extending the #1827 + # persist-before-clear invariant to the restart path for everything + # except the per-slice CONSENSUS buckets. Best-effort and front-running + # teardown: a transcript-write hiccup must never block recovery of a + # wedged phase (mirrors the salvage step below). + try: + _pkg._persist_phase_brc_history(pipeline, store, phase) + except Exception as brc_persist_err: # noqa: BLE001 + _pkg.logger.warning( + "Failed to persist in-flight BRC history during phase restart (continuing)", + pipeline_id=pipeline_id, + phase=phase, + error=str(brc_persist_err), + ) + + # 4. Stop and remove old containers + for container_id in old_container_ids: + try: + spawner.stop_agent_container(container_id, cleanup_session=True) + except Exception as e: + _pkg.logger.warning( + "Failed to stop container during phase restart", + container_id=container_id[:12] if container_id else "?", + error=str(e), + ) + try: + spawner.remove_agent_container(container_id, force=True, cleanup_session=False) + except Exception as e: + _pkg.logger.warning( + "Failed to remove container during phase restart", + container_id=container_id[:12] if container_id else "?", + error=str(e), + ) + + # 4b. Delete per-agent worktrees so respawned containers get fresh mounts. + # Without this, stale worktree directories (e.g. broken btrfs mounts) + # survive container removal and cause create_worktree to skip creation + # or fail. Mirrors cleanup_pipeline's worktree deletion. (#1723) + # + # Enumerate from disk rather than guess names: slice-scoped worktrees + # are ``{pipeline_id}-slice-{N}-{role}``, not ``{pipeline_id}-{role}``, + # so a name-guess loop misses every per-slice worktree on a slice + # pipeline and leaves them behind. (#2522) + # + # ``validate_git=False`` so that broken/corrupted worktrees (missing + # or unreadable ``.git`` marker — exactly the #1723 btrfs failure + # class) still reach ``delete_worktrees``. The default + # ``validate_git=True`` is salvage-correct (you can't salvage a + # broken worktree) but cleanup-incorrect (you must still delete it). + restart_role_values = {role.value for role in agent_roles} + try: + all_worktrees = _pkg.agent_salvage.enumerate_agent_worktrees( + pipeline_id, validate_git=False + ) + except (OSError, ImportError, RuntimeError) as e: + _pkg.logger.warning( + "Failed to enumerate per-agent worktrees during phase restart", + pipeline_id=pipeline_id, + error=str(e), + ) + all_worktrees = [] + worktrees_to_delete = [wt for wt in all_worktrees if wt.agent_role in restart_role_values] + + # Salvage unpushed agent commits before deleting worktrees (#2429). + # Restart is *the* scenario where unpushed commits accumulate: an + # operator hits this endpoint precisely because agents are wedged or + # timed out — the same conditions that prevent pushes from landing on + # ``origin/<assigned_branch>``. Without this hook, restart would be + # the one orchestrator-side worktree-delete code path that bypasses + # salvage and silently destroys recoverable work. Best-effort: any + # failure logs and continues so cleanup cannot be blocked by salvage. + if worktrees_to_delete: + try: + _pkg.agent_salvage.auto_salvage_pipeline( + spawner.gateway, + pipeline_id, + worktree_filter={wt.worktree_id for wt in worktrees_to_delete}, + mode=gateway_mode, + base_branch=pipeline.base_branch, + ) + except Exception as e: + _pkg.logger.warning( + "Auto-salvage failed during phase restart; proceeding with worktree deletion", + pipeline_id=pipeline_id, + error=str(e), + ) + + for wt in worktrees_to_delete: + log_extras: dict[str, str] = {} + if wt.slice_id is not None: + log_extras["slice_id"] = wt.slice_id + try: + spawner.gateway.delete_worktrees(container_id=wt.worktree_id, force=True) + _pkg.logger.info( + "Deleted per-agent worktree during phase restart", + agent_worktree_id=wt.worktree_id, + pipeline_id=pipeline_id, + **log_extras, + ) + except Exception as e: + _pkg.logger.warning( + "Failed to delete per-agent worktree during phase restart", + agent_worktree_id=wt.worktree_id, + pipeline_id=pipeline_id, + error=str(e), + **log_extras, + ) + + # 5. Reset consensus state. + # Slice-4 TASK-4-1: mirror the slice-aware semantics of + # ``restart_agent`` (line ~2859) — clear BOTH the pipeline-level + # tracker AND every per-slice tracker keyed + # ``f"{pipeline_id}/{slice_id}"`` (see + # ``peer_consensus._tracker_key``). Phase-level restart wipes + # the entire phase, so any per-slice consensus state that + # survived the restart is stale and would deadlock the new run + # if left in place. + # + # INVARIANT (#3200 task-7-1, mid-phase BRC record survival): like + # ``restart_agent`` above, this clears the *peer consensus tracker* + # (ephemeral ACK/NACK state) but MUST NOT clear the *Redis message + # store* (``pipeline:{id}:messages``). That store is the durable BRC + # message record; a mid-phase phase restart preserves it so the + # reseeded session can re-pull it (``/brc-transcript`` + + # ``read_peer_artifact``) and re-derive the #3189 anchors. The store is + # cleared only at phase transitions / pipeline create+delete, never on + # restart. Do NOT add ``get_message_store().clear()`` here. + try: + try: + from peer_consensus import get_peer_consensus_tracker + except ImportError: + from ..peer_consensus import ( + get_peer_consensus_tracker, # type: ignore[import-not-found] + ) + + tracker = get_peer_consensus_tracker(pipeline_id) + if tracker: + tracker.clear() + _pkg.logger.info("Cleared peer consensus tracker", pipeline_id=pipeline_id) + + # Per-slice trackers. Best-effort contract load: if the + # contract cannot be read (corrupt on disk, etc.), the + # pipeline-level clear above still ran, and the slice + # trackers will be reconstructed lazily on next consensus + # activity — preserving the historical pipeline-level-only + # behaviour as a fallback rather than blocking the restart. + # **Worktree-path resolution (reviewer_code v1 blocker 2)**: + # active pipelines' contracts live in the per-pipeline + # worktree at ``/home/egg/.egg-worktrees/<pipeline_id>/<repo>/`` + # — NOT under ``store.repo_path`` (the main orchestrator repo). + # Without ``resolve_worktree_path`` the ``load_contract`` call + # below silently fails with ``ContractNotFoundError`` for every + # active pipeline, the per-slice loop never iterates, and the + # whole per-slice clear becomes a no-op. Pattern mirrors + # ``routes/signals.py:709`` and ``routes/pipelines.py:10017``. + try: + from egg_contracts.loader import load_contract + except ImportError: + load_contract = None # type: ignore[assignment] + if load_contract is not None: + try: + from routes import resolve_worktree_path + except ImportError: + try: + from .. import ( + resolve_worktree_path, # type: ignore[no-redef] + ) + except ImportError: + resolve_worktree_path = None # type: ignore[assignment] + try: + if resolve_worktree_path is not None: + _contract_repo_path = resolve_worktree_path( + pipeline_id, _pkg.Path(store.repo_path) + ) + else: + _contract_repo_path = _pkg.Path(store.repo_path) + _contract = load_contract(pipeline_id, _contract_repo_path) + except Exception as load_err: # noqa: BLE001 — best-effort + _pkg.logger.warning( + "Could not load contract to enumerate slice trackers " + "during phase restart; per-slice consensus state may " + "be left stale until lazy reconstruction", + pipeline_id=pipeline_id, + error=str(load_err), + ) + _contract = None + if _contract is not None and getattr(_contract, "slices", None): + for _s in _contract.slices: + _slice_tracker = get_peer_consensus_tracker(pipeline_id, slice_id=_s.id) + if _slice_tracker: + _slice_tracker.clear() + _pkg.logger.info( + "Cleared per-slice peer consensus tracker", + pipeline_id=pipeline_id, + slice_id=_s.id, + ) + except ImportError: + pass + except Exception as e: + _pkg.logger.warning( + "Failed to clear peer consensus", + pipeline_id=pipeline_id, + error=str(e), + ) + + # 6. Reset restart counts for this pipeline + spawner.reset_restart_counts(pipeline_id) + + # 6b. Drop health-monitor anchors for every respawned role so the Tier-1 + # heartbeat clock does not survive the restart and fire stale-elapsed + # alerts that the overseer would faithfully escalate (issue #2084). + try: + try: + from health_monitor import get_health_monitor + except ImportError: + from ..health_monitor import ( + get_health_monitor, # type: ignore[import-not-found] + ) + _hm = get_health_monitor() + if _hm is not None: + for role in agent_roles: + _hm.reset_agent(role.value) + except Exception as e: + _pkg.logger.warning( + "Failed to reset health-monitor state during phase restart", + pipeline_id=pipeline_id, + phase=phase, + error=str(e), + ) + + # 7. Launch a new _run_pipeline thread to monitor the restarted phase. + # Container spawning is handled by _run_concurrent_phase within the + # thread, matching the recovery pattern used by start_pipeline. + # See #1638: the original polling thread died when the pipeline + # failed; without this, consensus completion is never detected. + agents_to_restart = [role.value for role in agent_roles] + repo_path_for_thread = store.repo_path + + _pkg._spawn_pipeline_run_thread(pipeline_id, repo_path_for_thread, pipeline.run_epoch) + + _pkg.logger.info( + "Phase restarted", + pipeline_id=pipeline_id, + phase=phase, + agents_to_restart=agents_to_restart, + reason=reason, + ) + + return _pkg.make_success_response( + f"Phase {phase} restarted with {len(agents_to_restart)} agent(s)", + data={ + "phase": phase, + "agents_to_restart": agents_to_restart, + }, + ) diff --git a/orchestrator/routes/pipelines/_routes_status.py b/orchestrator/routes/pipelines/_routes_status.py new file mode 100644 index 0000000000..4d3787f322 --- /dev/null +++ b/orchestrator/routes/pipelines/_routes_status.py @@ -0,0 +1,572 @@ +"""status-route bodies helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _get_pipeline_status_body(pipeline_id: str) -> tuple[_pkg.Response, int]: + """ + Get pipeline status summary. + + URL params: + pipeline_id: Pipeline ID + + Response: + { + "success": true, + "data": { + "id": "issue-123", + "status": "running", + "current_phase": "implement", + "pending_decisions": 0 + } + } + """ + repo_path = _pkg.get_repo_path() + + # Validate ``slice_id`` BEFORE the StateStore disk read in + # ``_resolve_pipeline`` — a malformed value is going to 400 anyway, + # and the read is wasted (#2764 review). ``InvalidPipelineIdError`` + # / ``PipelineNotFoundError`` from ``_resolve_pipeline`` still + # naturally take precedence on the happy path: this validator only + # fires when a slice scope is supplied at all. + raw_slice_id = _pkg.request.args.get("slice_id") + try: + status_slice_id = _pkg.extract_slice_id( + {"slice_id": raw_slice_id} if raw_slice_id is not None else {} + ) + except ValueError as e: + return _pkg.make_error_response(str(e), status_code=400) + + try: + _store, pipeline = _pkg._resolve_pipeline(pipeline_id, repo_path) + + pending = pipeline.get_pending_decisions() + + data = { + "id": pipeline.id, + "status": pipeline.status.value, + "current_phase": pipeline.current_phase.value, + "pending_decisions": len(pending), + "updated_at": pipeline.updated_at.isoformat(), + } + + # Include first pending decision details so the collaborator + # doesn't need a second round-trip to fetch it + if pending: + d = pending[0] + data["pending_decision"] = { + "id": d.id, + "question": d.question, + "context": d.context, + "options": d.options, + "created_at": d.created_at.isoformat(), + } + + # Include PR info once the PR phase has created a PR (#1625) so + # monitoring clients don't need to scrape `gh pr list` by title. + pr_url, pr_number = _pkg._get_pr_info(pipeline) + if pr_url: + data["pr_url"] = pr_url + if pr_number is not None: + data["pr_number"] = pr_number + + # Include concurrent execution monitoring when enabled. The + # ``?slice_id=`` query param (validated above before the + # StateStore read) scopes the consensus block to one slice's + # BRC tracker in a slice-DAG implement phase (#2761); without + # it, only pipeline-level consensus is reported. + concurrent_data = _pkg._get_concurrent_status(pipeline, slice_id=status_slice_id) + if concurrent_data: + data["concurrent"] = concurrent_data + + # Surface the orchestrator-process-wide slice-admission state + # (#2241 gap 1) so operators can see when slices are queued + # behind the global cap rather than wedged. The shape is + # {cap, admitted, admitted_keys}; ``admitted_keys`` lists + # ``"<pipeline_id>/<slice_id>"`` so the operator can tell + # which slices currently hold the budget. + try: + try: + from orchestrator import global_slice_admit + except ImportError: + import global_slice_admit # type: ignore[no-redef] + + data["slice_admit"] = global_slice_admit.snapshot() + except Exception: # noqa: BLE001 + # Defensive: never let admit-state collection crash the + # status endpoint — the cap is advisory, not load-bearing + # for the pipeline's own progress. + pass + + # Issue #1962 TASK-1-2: include the overseer-relevant config + # subset in the status payload so the sandbox-side overseer + # monitor can read PipelineConfig values (advisor model, + # threshold knobs, host-detection flag) without a separate + # endpoint. Only the new + load-bearing knobs are exposed + # here to keep the response compact; full config is available + # via the dedicated config endpoint. + try: + cfg = getattr(pipeline, "config", None) + if cfg is not None: + data["config"] = { + "overseer_advisor_model": getattr(cfg, "overseer_advisor_model", None), + "overseer_advisor_recent_log_bytes_cap": getattr( + cfg, "overseer_advisor_recent_log_bytes_cap", None + ), + "overseer_auto_file_issues_mode": getattr( + cfg, "overseer_auto_file_issues_mode", None + ), + "overseer_owns_host_detection": getattr( + cfg, "overseer_owns_host_detection", False + ), + "overseer_stuck_phase_transition_seconds": getattr( + cfg, "overseer_stuck_phase_transition_seconds", 180 + ), + "overseer_agent_stall_seconds": getattr( + cfg, "overseer_agent_stall_seconds", 180 + ), + "overseer_silent_agent_threshold_seconds": getattr( + cfg, "overseer_silent_agent_threshold_seconds", 600 + ), + "overseer_long_running_phase_seconds": getattr( + cfg, "overseer_long_running_phase_seconds", 3600 + ), + "overseer_nack_unresolved_seconds": getattr( + cfg, "overseer_nack_unresolved_seconds", 180 + ), + } + except AttributeError, TypeError: + # Defensive: never let a config-shape change crash the + # status endpoint. + pass + + return _pkg.make_success_response("Status retrieved", data=data) + + except _pkg.InvalidPipelineIdError: + return _pkg.make_error_response( + f"Invalid pipeline ID format: {pipeline_id}", + status_code=400, + ) + except _pkg.PipelineNotFoundError: + return _pkg.make_error_response( + f"Pipeline {pipeline_id} not found", + status_code=404, + ) + + +def _wait_pipeline_status_body(pipeline_id: str) -> tuple[_pkg.Response, int]: + """Block up to ``wait`` seconds on the next pipeline-relevant event. + + Query params: + wait: seconds to block, default 25, clamped to + ``GET_STATUS_MAX_WAIT`` (25) so the caller stays + safely inside the Claude Code MCP tool-call timeout. + since: opaque cursor ``msg:<id>|evt:<seq>`` from a prior + response. An empty / missing cursor snaps to the tip + on both sources (first-call semantics). Returns 400 + if the cursor is syntactically malformed. + + Responses: + 200 — either a ``changed=true`` envelope (event or message + fired before the timeout) or a ``changed=false, + no_change=true`` envelope (timeout elapsed with no + pipeline-relevant event). Always carries ``cursor`` + so the caller can seed the next request. + 400 — malformed cursor or malformed ``wait``. + 404 — pipeline does not exist. + + Implementation: + * ``queue.Queue(maxsize=16)`` coordinates the two sources: + a wildcard EventBus handler (synchronous) and a daemon + thread running ``message_store.get_messages(wait=...)``. + * First source wins. On return the EventBus handler is + unsubscribed in ``finally``; the daemon thread is left + lame-duck for up to ``wait`` seconds (accepted per plan + risk R14 — bounded, non-blocking on shutdown). + * ``egg_inflight_host_waits`` gauge is incremented at entry + and decremented on return. + + Args: + pipeline_id: Pipeline ID from the URL. + """ + # Validate pipeline exists before doing any expensive setup. + repo_path = _pkg.get_repo_path() + try: + _store, pipeline = _pkg._resolve_pipeline(pipeline_id, repo_path) + except _pkg.InvalidPipelineIdError: + return _pkg.make_error_response( + f"Invalid pipeline ID format: {pipeline_id}", + status_code=400, + ) + except _pkg.PipelineNotFoundError: + return _pkg.make_error_response( + f"Pipeline {pipeline_id} not found", + status_code=404, + ) + + # Parse + clamp ``wait``. ``GET_STATUS_MAX_WAIT`` lives in + # ``mcp_server`` — importing it here keeps the cap in one place. + try: + from mcp_server import GET_STATUS_MAX_WAIT + except ImportError: + try: + from ..mcp_server import GET_STATUS_MAX_WAIT # type: ignore[no-redef] + except ImportError: + GET_STATUS_MAX_WAIT = 25 # conservative fallback + try: + requested_wait = int(_pkg.request.args.get("wait", str(GET_STATUS_MAX_WAIT))) + except ValueError, TypeError: + return _pkg.make_error_response( + "Invalid 'wait' query parameter: must be an integer", + status_code=400, + ) + timeout = min(max(requested_wait, 1), GET_STATUS_MAX_WAIT) + + # Parse the opaque compound cursor. ``ok=False`` is the only + # 400 path here — unknown cursors on either source are tolerated + # and degrade to "snap to tip". + ok, msg_since_id, event_since_seq = _pkg._parse_status_wait_cursor( + _pkg.request.args.get("since") + ) + if not ok: + return _pkg.make_error_response( + "Invalid 'since' cursor — expected 'msg:<id>|evt:<seq>' (either half may be empty).", + status_code=400, + ) + + # Lazy imports keep the route cheap to load at module import and + # match the pattern used elsewhere in this file. We compare events + # against ``_STATUS_WAIT_EVENT_TYPES`` by the string value of + # ``event.event_type`` — the ``EventType`` class itself is not + # needed here. + try: + from events import get_event_bus + except ImportError: # pragma: no cover + try: + from ..events import get_event_bus # type: ignore[no-redef] + except ImportError: + return _pkg.make_error_response("Event bus not available", status_code=500) + + try: + from routes.messages import _apply_delphi_filter as _delphi + except ImportError: # pragma: no cover + try: + from ..messages import _apply_delphi_filter as _delphi # type: ignore[no-redef] + except ImportError: + _delphi = None # type: ignore[assignment] + + import queue as _queue + + event_bus = get_event_bus() + + # Synchronous up-front cursor-staleness probe (issue #2464). The route + # used to silently keep re-emitting ``msg_since_id`` whenever the store + # tip was empty (post-phase-clear), so a polling client kept feeding + # the dead cursor back forever. Probe once at entry with ``wait=0`` so + # we can both stop re-emitting it and surface ``since_id_stale: True`` + # in the envelope, letting consumers (sandbox CLI cursor file, agent + # wait_loop) drop the stale cursor and re-snap to tip. Done before + # the terminal short-circuit below so a request that arrives after + # both a phase clear and pipeline completion still sees the flag. + since_id_stale = False + if msg_since_id is not None: + try: + store_fn = _pkg._get_message_store() + store = store_fn() + _msgs, meta = store.get_messages_with_meta( + pipeline_id, + since_id=msg_since_id, + limit=1, + wait=0, + # Suppress the "since_id not found in store" warning on + # this probe so a single ``/status/wait`` request that + # hits a stale cursor doesn't double-log: the long-poll + # daemon below makes its own ``get_messages`` call with + # the same cursor and emits the warning once. Pre-PR + # cadence was one warning per request; we preserve that. + _suppress_stale_warning=True, + ) + since_id_stale = meta.since_id_stale + except Exception as exc: # pragma: no cover + _pkg.logger.debug( + "status_wait staleness probe error", + pipeline_id=pipeline_id, + error=str(exc), + ) + + # Late-subscriber short-circuit (issue #2378): if the pipeline is + # already terminal at request time, the relevant ``pipeline.*`` + # event was emitted before this call could subscribe — and the + # snap-to-tip below would cement that miss. Synthesize a Path-A + # envelope so callers don't loop until the 1-hour cap. This covers + # the common path where ``mark FAILED`` succeeds; the synthetic + # emit at ``_run_pipeline``'s mark-FAILED-failed branch covers the + # rarer case where the FAILED-mark itself raises. + _TERMINAL_EVENT_TYPES = { + _pkg.PipelineStatus.COMPLETE: "pipeline.completed", + _pkg.PipelineStatus.FAILED: "pipeline.failed", + _pkg.PipelineStatus.CANCELLED: "pipeline.cancelled", + } + if pipeline.status in _TERMINAL_EVENT_TYPES: + # Issue #2464: don't fall back to ``msg_since_id`` when the tip + # is empty — that's exactly the post-clear state that perpetuates + # the dead cursor. + terminal_cursor = _pkg._build_status_wait_cursor( + _pkg._message_store_tip_id(pipeline_id), + event_bus.current_sequence(), + ) + terminal_envelope = _pkg._build_minimal_status_envelope(pipeline, terminal_cursor) + terminal_envelope.update( + { + "changed": True, + "trigger": "event", + "event_type": _TERMINAL_EVENT_TYPES[pipeline.status], + } + ) + if since_id_stale: + terminal_envelope["since_id_stale"] = True + return _pkg.make_success_response("Pipeline already terminal", data=terminal_envelope) + + # Snap event_since_seq to the current tip on first call. This + # preserves the "events before the call are already seen" + # semantic and matches the message-bus ``from_tip`` behaviour + # used by ``/messages/wait`` (issue #1925). + if event_since_seq is None: + event_since_seq = event_bus.current_sequence() + + wake_q: _queue.Queue[tuple[str, _pkg.Any]] = _queue.Queue(maxsize=16) + + def _on_event(event) -> None: # pragma: no cover - exercised via tests + if event.pipeline_id != pipeline_id: + return + if event.event_type.value not in _pkg._STATUS_WAIT_EVENT_TYPES: + return + if event.sequence <= event_since_seq: + return + try: + wake_q.put_nowait(("event", event)) + except _queue.Full: + _pkg.logger.warning( + "status_wait event queue full; dropping event", + pipeline_id=pipeline_id, + event_type=event.event_type.value, + ) + + def _on_message_store_wake() -> None: # pragma: no cover - exercised via tests + try: + store_fn = _pkg._get_message_store() + store = store_fn() + messages = store.get_messages( + pipeline_id, + since_id=msg_since_id, + limit=100, + wait=timeout, + wait_for_types=list(_pkg._STATUS_WAIT_MESSAGE_TYPES), + from_tip=msg_since_id is None, + ) + except Exception as exc: # pragma: no cover + _pkg.logger.debug( + "status_wait daemon error", + pipeline_id=pipeline_id, + error=str(exc), + ) + return + if not messages: + return + try: + wake_q.put_nowait(("message", messages)) + except _queue.Full: + _pkg.logger.warning( + "status_wait message queue full; dropping message", + pipeline_id=pipeline_id, + ) + + _pkg._track_host_wait_start() + event_bus.subscribe(None, _on_event) + daemon: _pkg.threading.Thread | None = None + try: + daemon = _pkg.threading.Thread( + target=_on_message_store_wake, + name=f"status-wait-msg-{pipeline_id}", + daemon=True, + ) + daemon.start() + + try: + source, payload = wake_q.get(timeout=timeout) + except _queue.Empty: + source = None + payload = None + + # Re-load the pipeline once here so both paths share a + # consistent snapshot for the minimal envelope. + try: + _store2, fresh_pipeline = _pkg._resolve_pipeline(pipeline_id, repo_path) + except _pkg.InvalidPipelineIdError, _pkg.PipelineNotFoundError: + fresh_pipeline = pipeline + + if source == "event": + event = payload + # Issue #2464: never fall back to ``msg_since_id`` when the + # tip is empty. After a phase-boundary clear the caller's + # cursor is dead; re-emitting it here is what kept the + # ``since_id not found in store`` warning firing on every + # subsequent poll. ``since_id_stale: True`` in the envelope + # tells the consumer to drop its cached cursor. + tip_msg_id = _pkg._message_store_tip_id(pipeline_id) + cursor = _pkg._build_status_wait_cursor(tip_msg_id, event.sequence) + envelope = _pkg._build_minimal_status_envelope(fresh_pipeline, cursor) + envelope.update( + { + "changed": True, + "trigger": "event", + "event_type": event.event_type.value, + } + ) + if since_id_stale: + envelope["since_id_stale"] = True + return _pkg.make_success_response("Event wake", data=envelope) + + if source == "message": + messages = payload + # Issue #2464: same as the event path — fall back to None + # when the message half is unavailable rather than re-emitting + # the stale ``msg_since_id``. + last_id = messages[-1].id if messages else None + # Delphi filter pass — currently a no-op for the host caller + # (role=None returns messages unchanged) but plumbed here so a + # future role parameter can enable reviewer-redaction (R13). + if _delphi is not None: + try: + messages = _delphi(pipeline_id, None, messages) + except Exception: # pragma: no cover + pass + tip_evt_seq = event_bus.current_sequence() + cursor = _pkg._build_status_wait_cursor(last_id, tip_evt_seq) + envelope = _pkg._build_minimal_status_envelope(fresh_pipeline, cursor) + envelope.update( + { + "changed": True, + "trigger": "message", + "messages": [m.to_dict() for m in messages], + } + ) + if since_id_stale: + envelope["since_id_stale"] = True + return _pkg.make_success_response("Message wake", data=envelope) + + # Timeout path — minimal envelope only. + tip_msg_id = _pkg._message_store_tip_id(pipeline_id) + tip_evt_seq = event_bus.current_sequence() + cursor = _pkg._build_status_wait_cursor(tip_msg_id, tip_evt_seq) + envelope = _pkg._build_minimal_status_envelope(fresh_pipeline, cursor) + envelope.update({"changed": False, "no_change": True}) + if since_id_stale: + envelope["since_id_stale"] = True + return _pkg.make_success_response("No change within wait window", data=envelope) + finally: + try: + event_bus.unsubscribe(None, _on_event) + except Exception: # pragma: no cover — unsubscribe is best-effort + pass + _pkg._track_host_wait_end() + + +def _get_pipeline_visualization_body(pipeline_id: str) -> tuple[_pkg.Response, int]: + """ + Get pipeline DAG visualization. + + URL params: + pipeline_id: Pipeline ID + + Query params: + format: Output format - "full" (default), "compact", "text", "json" + ascii: Use ASCII-only characters (default: false) + + Response: + { + "success": true, + "data": { + "pipeline_id": "issue-123", + "visualization": { + "dag": "...", // Full DAG visualization + "compact": "...", // Single-line status + "progress": "..." // Progress bar + }, + "phases": {...}, // Phase status summary + "status": "running", + "current_phase": "implement" + } + } + """ + # Check if visualization module is available (imported at module level) + if not _pkg._DAG_VISUALIZER_AVAILABLE: + return _pkg.make_error_response( + "Visualization module not available", + status_code=500, + ) + + repo_path = _pkg.get_repo_path() + output_format = _pkg.request.args.get("format", "full") + use_ascii = _pkg.request.args.get("ascii", "false").lower() == "true" + + try: + _store, pipeline = _pkg._resolve_pipeline(pipeline_id, repo_path) + + if output_format == "json": + # Return structured JSON report + report = _pkg.generate_status_report(pipeline, use_ascii=use_ascii) + return _pkg.make_success_response( + "Visualization generated", + data=report, + ) + + elif output_format == "text": + # Return plain text DAG + dag_text = _pkg.render_pipeline_dag(pipeline, use_ascii=use_ascii) + return _pkg.Response( + dag_text, + mimetype="text/plain", + status=200, + ) + + elif output_format == "compact": + # Return compact single-line status + compact = _pkg.render_compact_status(pipeline, use_ascii=use_ascii) + progress = _pkg.render_progress_bar(pipeline, use_ascii=use_ascii) + return _pkg.make_success_response( + "Visualization generated", + data={ + "pipeline_id": pipeline.id, + "compact": compact, + "progress": progress, + "status": pipeline.status.value, + "current_phase": pipeline.current_phase.value, + }, + ) + + else: + # Full format with all visualizations + report = _pkg.generate_status_report(pipeline, use_ascii=use_ascii) + return _pkg.make_success_response( + "Visualization generated", + data=report, + ) + + except _pkg.InvalidPipelineIdError: + return _pkg.make_error_response( + f"Invalid pipeline ID format: {pipeline_id}", + status_code=400, + ) + except _pkg.PipelineNotFoundError: + return _pkg.make_error_response( + f"Pipeline {pipeline_id} not found", + status_code=404, + ) diff --git a/orchestrator/routes/pipelines/_routes_stream.py b/orchestrator/routes/pipelines/_routes_stream.py new file mode 100644 index 0000000000..63e929123d --- /dev/null +++ b/orchestrator/routes/pipelines/_routes_stream.py @@ -0,0 +1,123 @@ +"""stream-route bodies helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _stream_all_pipelines_body() -> _pkg.Response: + """ + Stream unified events for all pipelines via Server-Sent Events (SSE). + + Provides real-time updates for ALL pipeline state changes in a single + SSE connection. Unlike the per-pipeline stream, terminal events for + individual pipelines do not end the stream. + + Query params: + ascii: Use ASCII-only characters (default: false) + active_only: Only include active pipelines (default: true) + full_dag: Include full DAG visualization (default: false) + + Response: + text/event-stream with the following event types: + - snapshot: Initial state of all active pipelines + - pipeline.*: Pipeline lifecycle events + - phase.*: Phase transition events + - agent.*: Agent lifecycle events + - decision.*: HITL decision events + - done: Stream is ending (timeout) + """ + if not _pkg._UNIFIED_SSE_AVAILABLE: + return _pkg.make_error_response( + "Unified SSE streaming module not available", + status_code=500, + ) + + use_ascii = _pkg.request.args.get("ascii", "false").lower() == "true" + active_only = _pkg.request.args.get("active_only", "true").lower() == "true" + full_dag = _pkg.request.args.get("full_dag", "false").lower() == "true" + + repo_path = _pkg.get_repo_path() + + return _pkg.Response( + _pkg.stream_with_context( + _pkg.create_unified_sse_stream( + repo_path=repo_path, + use_ascii=use_ascii, + active_only=active_only, + full_dag=full_dag, + ) + ), + mimetype="text/event-stream", + headers={ + "Cache-Control": "no-cache", + "X-Accel-Buffering": "no", + }, + ) + + +def _stream_pipeline_body(pipeline_id: str) -> _pkg.Response: + """ + Stream pipeline events via Server-Sent Events (SSE). + + Provides real-time updates for pipeline state changes including + phase transitions, agent lifecycle, and DAG visualization. + + URL params: + pipeline_id: Pipeline ID + + Query params: + ascii: Use ASCII-only characters (default: false) + + Response: + text/event-stream with the following event types: + - snapshot: Initial pipeline state + - pipeline.*: Pipeline lifecycle events + - phase.*: Phase transition events + - agent.*: Agent lifecycle events + - decision.*: HITL decision events + - done: Stream is ending (terminal state or timeout) + - error: An error occurred + + The stream automatically closes when the pipeline reaches a + terminal state (completed, failed, cancelled) or after the + maximum connection time (1 hour). + """ + if not _pkg._SSE_AVAILABLE: + return _pkg.make_error_response( + "SSE streaming module not available", + status_code=500, + ) + + use_ascii = _pkg.request.args.get("ascii", "false").lower() == "true" + + # Validate pipeline exists before starting stream + repo_path = _pkg.get_repo_path() + try: + _pkg._resolve_pipeline(pipeline_id, repo_path) + except _pkg.InvalidPipelineIdError: + return _pkg.make_error_response( + f"Invalid pipeline ID format: {pipeline_id}", + status_code=400, + ) + except _pkg.PipelineNotFoundError: + return _pkg.make_error_response( + f"Pipeline {pipeline_id} not found", + status_code=404, + ) + + return _pkg.Response( + _pkg.stream_with_context( + _pkg.create_sse_stream(pipeline_id, repo_path=repo_path, use_ascii=use_ascii) + ), + mimetype="text/event-stream", + headers={ + "Cache-Control": "no-cache", + "X-Accel-Buffering": "no", + }, + ) diff --git a/orchestrator/routes/pipelines/_run_concurrent.py b/orchestrator/routes/pipelines/_run_concurrent.py new file mode 100644 index 0000000000..2c4efe47bf --- /dev/null +++ b/orchestrator/routes/pipelines/_run_concurrent.py @@ -0,0 +1,1476 @@ +"""concurrent-phase runner + impasse-retry wrapper helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _run_concurrent_phase( + pipeline_id: str, + pipeline: _pkg.Pipeline, + phase: str, + spawner, + repo_volumes: dict[str, str], + gateway_mode: str, + repos: list[str], + sandbox_env: dict[str, str], + store, + certs_volume: str | None, + worktree_repo_path: _pkg.Path, + review_feedback: str | None = None, + slice_id: str | None = None, + operator_directives: list[_pkg.OperatorDirective] | None = None, + iteration_history: list[_pkg.IterationSummary] | None = None, + run_epoch: _pkg.datetime | None = None, +) -> tuple[int, str]: + """Run a phase using concurrent all-agents-at-once execution. + + Creates a ConcurrentPhaseExecutor that spawns all agents simultaneously, + all sharing the pipeline branch. Each container receives a role-specific + prompt built via ``_build_agent_prompt``. After spawning, waits for all + containers to exit and records their state in the pipeline store. + + Returns: + (exit_code, logs) — 0 on success. + + Raises: + SpawnFailureError: If any agent fails to spawn. Survivors are stopped + and their pipeline-state records are marked FAILED before the + exception propagates. Distinguishes spawn failures from container + exits so the outer caller's ``pipeline.error`` is accurate. + """ + from models import ( + AgentExecutionStatus as StateAgentStatus, + ) + from models import ( + ContainerInfo, + ContainerStatus, + PipelinePhase, + resolve_consensus_timeout_minutes, + ) + + try: + from concurrent_executor import ConcurrentPhaseExecutor + except ImportError: + from ..concurrent_executor import ConcurrentPhaseExecutor # type: ignore + + phase_str = phase if isinstance(phase, str) else phase.value + pipeline_mode = "issue" if pipeline.issue_number is not None else "prompt" + + # Slice-aware sandbox env (#2137 TASK-4-3 / #2403): when running a + # per-slice team, the spawner exposes the slice id via + # ``EGG_SLICE_ID`` and leaves ``EGG_PIPELINE_ID`` as the bare + # pipeline id. An earlier shape encoded the slice into + # ``EGG_PIPELINE_ID`` itself (``{pipeline_id}/{slice_id}``) so the + # orchestrator's ``_tracker_key`` would route CONSENSUS_* to the + # slice tracker without an extra signal-level field. That broke + # every agent → orchestrator round-trip: + # + # * the orchestrator-side ``PIPELINE_ID_PATTERN`` and the agent + # handler validator (``[a-zA-Z0-9_-]+``) both reject the slash, + # * Flask's default URL converter doesn't allow ``/``, so every + # ``POST /api/v1/pipelines/{pid}/...`` route 404s — i.e. all + # of progress, BRC, heartbeat, message, phase, decision, etc. + # + # Slice routing is plumbed explicitly instead: the BRC handlers + # pull ``EGG_SLICE_ID`` and forward it on the signal payload, and + # the orchestrator's signal handlers feed it into + # ``get_peer_consensus_tracker(pipeline_id, slice_id)``. CONSENSUS_* + # isolation is preserved; HEARTBEAT and OVERSEER_ALERT are not + # tracker-scoped at all — ``handle_heartbeat_signal`` is a no-op + # ACK with no tracker lookup, and OVERSEER_ALERT flows through the + # message bus (``MessageType.OVERSEER_ALERT``) rather than the + # consensus tracker. So per-slice scoping doesn't apply to either, + # and operator telemetry stays pipeline-wide as before. The + # pipeline-level fan-out for OVERSEER_ALERT mentioned in earlier + # comments here is tracked alongside the per-slice MCP control + # verbs in #2199. + # + # Single source of truth (#2410 v2 review): ``EGG_SLICE_ID`` is + # injected by ``KubernetesSpawner.spawn_agent_job`` from the same + # ``slice_id`` parameter that drives Job naming and worktree id, so + # there is no need to also stuff it into ``sandbox_env`` here. The + # key is in ``_PROTECTED_ENV_KEYS`` so any future caller that does + # supply a value via ``extra_env`` is logged and overridden. + + # Build per-role prompts for concurrent phase execution. + from egg_contracts.agent_roles import get_roles_for_phase as _get_roles_for_phase + + roles: list[_pkg.AgentRole] = [] + for r in _get_roles_for_phase( + phase_str, + include_reviewers=True, + repo=pipeline.repo, + has_contract=getattr(pipeline, "has_contract", True), + ): + try: + roles.append(_pkg.AgentRole(r.value)) + except ValueError: + # New roles not yet in orchestrator AgentRole — skip + continue + + # Build a review graph filtered to only active roles so consensus + # tracking doesn't wait for unspawned agents. + from review_graph import ReviewGraph + from review_graph import get_review_graph_for_phase as _get_graph + + full_graph = _get_graph(phase_str, repo=pipeline.repo) + active_role_names = {r.value for r in roles} + filtered_edges = [ + e + for e in full_graph.edges + if e.reviewer_role in active_role_names and e.producer_role in active_role_names + ] + filtered_graph = ReviewGraph(filtered_edges) + + # Scope the per-slice team to the slice's repo (#3393 task-6-1). + # + # Every slice maps to exactly one repo (slice ↔ repo, 1:1). For a + # multi-repo pipeline the slice's work, worktree, test gate, reviewer + # diff and PR all live in ITS repo — not necessarily the pipeline + # primary. We resolve the slice's repo via ``resolve_slice_repo`` and + # thread the slice-scoped repo / worktree / base-branch into the agent + # prompts (which drive ``get_repo_checks`` for the tester's configured + # checks, the file-boundary patterns, and the reviewer's + # ``git diff origin/<base>...HEAD``) and the spawn (via ``base_branch`` + # → ``EGG_BASE_BRANCH`` and a slice-primary-first ``repos`` ordering so + # the spawner sets the agent cwd / ``EGG_REPO_PATH`` to the slice's + # repo worktree). + # + # N=1 stays byte-identical: a single-repo pipeline has one RepoSpec, so + # the block below is skipped entirely (``len(pipeline.repos) <= 1``), + # leaving ``slice_repo == pipeline.repo``, ``worktree_repo_path``, and + # the pipeline base branch exactly as before — no extra contract read. + slice_repo = pipeline.repo + slice_repo_path = worktree_repo_path + slice_repos = repos + slice_base_branch: str | None = None + if slice_id and len(getattr(pipeline, "repos", None) or []) > 1: + from egg_contracts.loader import load_contract + + slice_obj = None + try: + _contract = load_contract(pipeline_id, worktree_repo_path) + slice_obj = next((s for s in _contract.slices if s.id == slice_id), None) + except Exception as contract_err: # noqa: BLE001 + # Best-effort: a contract load/parse failure degrades to the + # pipeline-primary repo (today's behaviour), it does not block + # the spawn. The slice still runs, just against the primary. + _pkg.logger.warning( + "Slice-repo scoping: contract load failed; using pipeline primary repo (#3393)", + pipeline_id=pipeline_id, + slice_id=slice_id, + error=str(contract_err), + ) + + # Single gate-repo accessor (shared with the tester's task-6-2 + # TestSliceGateRepoAccessor): the repo the whole slice team scopes to. + resolved = _pkg._resolve_slice_gate_repo(slice_obj, pipeline) if slice_obj else None + if resolved and resolved != pipeline.repo: + slice_repo = resolved + slice_repo_path = _pkg._resolve_slice_worktree_path( + pipeline, resolved, worktree_repo_path + ) + # Per-repo base branch from the pipeline's RepoSpec list. + for spec in pipeline.repos or []: + if getattr(spec, "repo", None) == resolved: + slice_base_branch = getattr(spec, "base_branch", None) + break + # Order the slice's repo first so the spawner treats it as the + # effective repo for this per-slice team (cwd / EGG_REPO_PATH). + # ``repo_volumes`` already carries every repo owner/repo-keyed + # (slice-3), so only the ordering changes here. + slice_repos = [resolved, *[r for r in repos if r != resolved]] + _pkg.logger.info( + "Slice scoped to secondary repo (#3393 task-6-1)", + pipeline_id=pipeline_id, + slice_id=slice_id, + slice_repo=slice_repo, + slice_worktree=str(slice_repo_path), + ) + + # Resolve base branch for diff commands in agent prompts. Prefer the + # slice repo's own base (its RepoSpec.base_branch) over the pipeline + # singleton, then fall back to auto-detecting the default branch in the + # slice's worktree (#3393 task-6-1). For N=1 this is the pipeline base / + # pipeline worktree exactly as before. + _resolved_base_branch = slice_base_branch or pipeline.base_branch + if not _resolved_base_branch: + try: + _resolved_base_branch = _pkg.get_default_branch(slice_repo_path) + except Exception: + _resolved_base_branch = None + + # A producer with no work in this slice is no longer pre-seeded (#3027 + # retired the #2581 pre-seed). It stays spawned and, if it finds it has + # nothing to contribute, submits a generic no-op propose + # (``no_changes_needed=true``) — the prompts below tell every producer + # about that path. The consensus protocol accepts the no-op durably, so + # no orchestrator-side roster pre-classification is needed. + agent_prompts: dict[_pkg.AgentRole, str] = {} + for role in roles: + prompt = _pkg._build_agent_prompt( + role_value=role.value, + phase=phase_str, + pipeline_id=pipeline_id, + pipeline_mode=pipeline_mode, + prompt=pipeline.prompt, + issue_number=pipeline.issue_number, + # Slice-scoped repo / worktree (#3393 task-6-1): drives the + # tester's ``get_repo_checks`` (per-repo configured checks), + # the role file-boundary patterns, and the reviewer diff base — + # all resolve from the slice's repo, not the pipeline primary. + # N=1 ⇒ these equal ``pipeline.repo`` / ``worktree_repo_path``. + repo=slice_repo, + branch=pipeline.branch, + base_branch=_resolved_base_branch, + repo_path=str(slice_repo_path), + concurrent=True, + review_feedback=review_feedback, + network_mode=gateway_mode, + operator_directives=operator_directives, + iteration_history=iteration_history, + ) + agent_prompts[role] = prompt + + # Create spawn function and executor. + spawn_fn = spawner.create_concurrent_spawn_fn( + pipeline_id=pipeline_id, + issue_number=pipeline.issue_number, + repo_volumes=repo_volumes, + mode=gateway_mode, + # Slice's repo first (#3393 task-6-1): the spawner derives the agent + # cwd / EGG_REPO_PATH from the primary (first) repo, so ordering the + # slice's repo first sets the working directory to that repo's + # worktree. N=1 / primary-repo slices leave ``repos`` unchanged. + repos=slice_repos, + phase=phase_str, + sandbox_env=sandbox_env, + certs_volume=certs_volume, + # Pass the *resolved* base branch (above) rather than the raw + # ``pipeline.base_branch`` so a ``None`` (auto-detect) base still + # reaches the spawner as a concrete branch name. The spawner exports + # it as ``EGG_BASE_BRANCH`` for the BRC event-pump's per-producer + # ``git log --not origin/<base>`` delta (#2967); without a concrete + # value the wrapper + composer fall back to ``origin/main`` and the + # delta errors out on every non-``main`` repo. Worktree creation is + # unaffected: the gateway resolves the same default branch when handed + # ``None``, so resolving one layer up here is equivalent. + base_branch=_resolved_base_branch, + spawn_max_retries=pipeline.config.spawn_max_retries, + spawn_retry_initial_backoff_seconds=pipeline.config.spawn_retry_initial_backoff_seconds, + slice_id=slice_id, + ) + + max_concurrent = getattr(pipeline.config, "max_concurrent_agents", 6) + # #3064 slice-3: in orchestrator-ownership mode the event loop watches + # one-shot Job termination to drive failure supervision (backoff / + # respawn / OVERSEER_ALERT). Hand it a Job-status observer when the + # spawner can provide one (the kubernetes spawner); spawners without it + # leave supervision observation dormant (pod mode is unaffected either way). + event_status_view = None + _make_status_view = getattr(spawner, "create_event_job_status_view", None) + if callable(_make_status_view): + event_status_view = _make_status_view() + executor = ConcurrentPhaseExecutor( + pipeline=pipeline, + spawn_fn=spawn_fn, + max_concurrent=max_concurrent, + review_graph=filtered_graph, + roles=roles, + slice_id=slice_id, + event_status_view=event_status_view, + ) + + # Spawn all agents with their prompts. + executions = executor.spawn_all(agent_prompts=agent_prompts) + + # Phase-level retry for transient spawn failures (#1879). + executions = _pkg._retry_transient_spawn_failures_impl( + executions, + pipeline=pipeline, + executor=executor, + agent_prompts=agent_prompts, + spawner=spawner, + pipeline_id=pipeline_id, + phase_str=phase_str, + ) + # Record spawned containers/agents in pipeline state. + _pkg._record_spawned_agents_impl( + executions, + store=store, + pipeline_id=pipeline_id, + phase_str=phase_str, + slice_id=slice_id, + ) + + # Check for spawn failures before waiting. Stop successfully-spawned + # containers so they don't continue running after the phase is aborted, + # then write their terminal status back to pipeline state so get_status + # agrees with list_containers (kubernetes_monitor won't reconcile a + # non-RUNNING pipeline, so we must finalize here). + spawn_failures = [e for e in executions if e.status.value == "failed"] + if spawn_failures: + survivor_container_ids: set[str] = set() + for e in executions: + if e.container_id and e.status.value != "failed": + survivor_container_ids.add(e.container_id) + try: + spawner.backend.stop_container(e.container_id, timeout=10) + except Exception: + pass + + if store is not None: + try: + with _pkg.get_pipeline_state_lock(pipeline_id): + pip = store.load_pipeline(pipeline_id) + phase_execution = pip.get_phase_execution(PipelinePhase(phase_str)) + abort_error = "Aborted during spawn-failure cleanup" + now = _pkg.datetime.now(_pkg.UTC) + for agent_state in phase_execution.agents: + if ( + agent_state.container_id in survivor_container_ids + and agent_state.status == StateAgentStatus.RUNNING + ): + agent_state.status = StateAgentStatus.FAILED + agent_state.error = abort_error + agent_state.completed_at = now + for container_info in phase_execution.containers: + if ( + container_info.container_id in survivor_container_ids + and container_info.status == ContainerStatus.RUNNING + ): + container_info.status = ContainerStatus.FAILED + container_info.exited_at = now + store.save_pipeline(pip) + except Exception as cleanup_err: + _pkg.logger.warning( + "Failed to record spawn-failure cleanup in pipeline state", + pipeline_id=pipeline_id, + error=str(cleanup_err), + ) + + raise _pkg.SpawnFailureError([(e.role.value, e.error) for e in spawn_failures]) + + # Consensus-driven polling loop with container-exit fallback. + # + # The loop periodically checks consensus via executor.check_consensus(). + # When all agents signal READY, the phase completes immediately without + # waiting for containers to exit. If consensus is never reached (timeout + # or all containers exit first), fall back to exit-code-based completion. + active_executions = [e for e in executions if e.container_id] + docker_client = spawner.backend + all_logs: list[str] = [] + has_failures = [False] # Mutable container for closure access + # Lock kept for forward-compat; the polling loop is single-threaded + # after the #1921 refactor but _record_container_exit uses the lock + # and is called from multiple code paths. + _logs_lock = _pkg.threading.Lock() + + poll_interval = 5 # seconds + raw_timeout = resolve_consensus_timeout_minutes(pipeline.config, phase_str) + consensus_timeout = max(raw_timeout, 1) * 60 # minimum 1 minute + start_time = _pkg.time.monotonic() + objection_decision_created = False + + # ``run_epoch`` is the authoritative epoch the owning ``_run_pipeline`` + # thread captured at start (#1638). The poll loop uses it to detect a + # ``restart_phase`` (or any restart that bumps ``run_epoch``) that + # superseded this thread (#3315). ``start_time`` is a fresh monotonic + # clock per call, but a parked-then-restarted phase leaves the *old* + # ``_run_concurrent_phase`` thread alive in its poll loop with a + # ``start_time`` from the original phase start; once its ``elapsed`` + # crosses ``consensus_timeout`` it would fire a spurious consensus-timeout + # OVERSEER_ALERT + HITL decision against the freshly-restarted phase. The + # new ``_run_pipeline`` thread owns the pipeline now, so this stale thread + # must bail before escalating. When ``run_epoch`` is not supplied (legacy + # / direct-call callers) the guard is dormant — behaviour is unchanged. + + _superseded_by_restart = _pkg.functools.partial( + _pkg._superseded_by_restart_impl, + store=store, + pipeline_id=pipeline_id, + run_epoch=run_epoch, + ) + + # Track which containers have exited and their results. + exited_containers: dict[str, ContainerInfo] = {} + + _record_container_exit = _pkg.functools.partial( + _pkg._record_container_exit_impl, + docker_client=docker_client, + _logs_lock=_logs_lock, + has_failures=has_failures, + all_logs=all_logs, + store=store, + phase_str=phase_str, + pipeline_id=pipeline_id, + ) + + _stop_running_containers = _pkg.functools.partial( + _pkg._stop_running_containers_impl, + active_executions=active_executions, + exited_containers=exited_containers, + docker_client=docker_client, + ) + + _latest_proposal_ts = _pkg._latest_proposal_ts_impl + + _update_agents_complete = _pkg.functools.partial( + _pkg._update_agents_complete_impl, + store=store, + phase_str=phase_str, + pipeline_id=pipeline_id, + slice_id=slice_id, + ) + + _demoted_agents: set[str] = set() + + # #2243 progress-gate state: log on first defer + first un-defer only + # so the polling loop doesn't spam at every iteration once we cross + # ``consensus_timeout``. + _progress_gate_deferring = False + + # #3426 HITL-gate state: same log-once discipline for the + # operator-gated suspension of the consensus timeout. + _hitl_gate_deferring = False + + while True: + elapsed = _pkg.time.monotonic() - start_time + + # 0. Bail if a restart superseded this thread (#3315). A parked phase + # that is restarted after the consensus-timeout budget elapsed + # leaves this old thread polling with a stale ``start_time``; the + # new ``_run_pipeline`` thread already owns the pipeline. Exit + # cleanly — stop this executor's event loop so it stops requesting + # one-shot spawns — WITHOUT firing the timeout escalation. Return a + # NON-zero exit so the caller never mistakes this for success and + # advances the phase; the post-return epoch check (#1638) at the + # call site re-confirms the restart and exits the old thread without + # marking the phase FAILED. + if _superseded_by_restart(): + _pkg.logger.info( + "Phase superseded by restart (run_epoch changed) — exiting stale " + "_run_concurrent_phase thread without escalation", + pipeline_id=pipeline_id, + phase=phase, + slice_id=slice_id, + ) + executor.stop_event_loop() + return 1, "Phase superseded by restart; stale monitor thread exited." + + # 1. Check consensus + try: + consensus = executor.check_consensus() + except Exception as e: + _pkg.logger.warning( + "Consensus check failed, continuing poll", + pipeline_id=pipeline_id, + error=str(e), + ) + consensus = {"is_complete": False, "has_objections": False, "blocking_agents": []} + + # 2. Consensus reached — stop containers and return + if consensus.get("is_complete"): + # Recover pipeline if externally marked FAILED (issue #1273). + # The container_monitor reconciliation thread may have marked the + # pipeline FAILED while we were polling. Now that consensus is + # confirmed complete, restore the pipeline to RUNNING so stored + # state matches the successful outcome. + # + # NOTE: consensus staleness is acceptable here. The `consensus` + # dict was fetched earlier in this loop iteration and is not + # re-evaluated under the lock. If consensus regressed between + # the outer check and lock acquisition (extremely unlikely), the + # next iteration of this monitoring loop will re-evaluate and + # self-correct. + if store is not None: + try: + _current_pip = store.load_pipeline(pipeline_id) + if _current_pip.status == _pkg.PipelineStatus.FAILED: + _pkg.logger.warning( + "Pipeline externally marked FAILED but consensus is complete — recovering", + pipeline_id=pipeline_id, + ) + with _pkg.get_pipeline_state_lock(pipeline_id): + _current_pip = store.load_pipeline(pipeline_id) + if _current_pip.status == _pkg.PipelineStatus.FAILED: + _current_pip.status = _pkg.PipelineStatus.RUNNING + _current_pip.error = None + store.save_pipeline(_current_pip) + except Exception as recovery_err: + _pkg.logger.warning( + "External FAILED recovery check failed", + pipeline_id=pipeline_id, + error=str(recovery_err), + ) + + if _pkg._emit_event is not None: + _pkg._emit_event( + _pkg.EventType.CONSENSUS_REACHED, + pipeline_id, + data={"elapsed_seconds": elapsed}, + ) + _pkg.logger.info( + "Consensus reached, stopping containers", + pipeline_id=pipeline_id, + elapsed_seconds=round(elapsed, 1), + has_failures=has_failures[0], + ) + _update_agents_complete() + _stop_running_containers() + combined_logs = ( + "\n".join(all_logs) if all_logs else "Consensus reached; phase complete." + ) + # Consensus is the authoritative success signal. When all agents + # have confirmed (is_complete=True), container-level failures + # (e.g. OOM kills that happened *before* the surviving agents + # reached agreement) should not override the consensus result. + # Any pending HITL decisions from handle_agent_failure remain + # active for human review, but the pipeline itself succeeds. + if has_failures[0]: + _pkg.logger.warning( + "Container failures detected but consensus is complete — treating as success", + pipeline_id=pipeline_id, + has_failures=has_failures[0], + ) + # Orchestrator mode (#3064): tear down the BRC event loop now that + # the slice has converged so it stops requesting one-shot spawns. + # No-op in pod mode. + executor.stop_event_loop() + return 0, combined_logs + + # 3. Handle objections (create HITL decision once). + # The decision is fire-and-forget: resolution is processed by the + # orchestrator's decision queue (outside this function). If the + # human selects "Override objections", the orchestrator updates + # agent readiness, which is picked up by check_consensus() on + # the next poll iteration. "Abort phase" triggers pipeline + # cancellation via a separate control path. + if consensus.get("has_objections") and not objection_decision_created: + decision = _pkg._persist_hitl_decision( + pipeline_id, + pipeline, + store, + question="Agent(s) objecting to phase completion. How to proceed?", + options=["Override objections", "Wait for resolution", "Abort phase"], + phase=pipeline.current_phase, + ) + if decision is not None: + objection_decision_created = True + _pkg.logger.info( + "Objection detected, HITL decision created", + pipeline_id=pipeline_id, + blocking_agents=consensus.get("blocking_agents", []), + ) + + # 3b. RC3: Stall demotion for dual-role agents. + # If a dual-role agent has missed heartbeats for 5+ minutes, + # demote its reviewer edges to ADVISORY so other agents can proceed. + try: + from health_monitor import get_health_monitor + + _hm = get_health_monitor() + if _hm is not None: + try: + from peer_consensus import get_peer_consensus_tracker + except ImportError: + from ..peer_consensus import ( + get_peer_consensus_tracker, # type: ignore[no-redef] + ) + + # Slice-aware tracker lookup (#2137): per-slice trackers + # are namespaced ``{pipeline_id}/{slice_id}`` so the + # stall-demotion check fires against the correct scope. + try: + _brc_tracker = get_peer_consensus_tracker(pipeline_id, slice_id) + except TypeError: + _brc_tracker = get_peer_consensus_tracker(pipeline_id) + if _brc_tracker is not None: + heartbeat_actions = _hm.check_heartbeats() + for hb_action in heartbeat_actions: + stalled_agent = hb_action.get("agent_id", "") + stall_elapsed = hb_action.get("elapsed_seconds", 0) + if ( + stall_elapsed >= 300 + and stalled_agent not in _demoted_agents + and _brc_tracker.graph.is_dual_role(stalled_agent) + ): + try: + _brc_tracker.handle_stall_demotion( + stalled_agent, + reason=f"Missed heartbeats for {stall_elapsed}s", + ) + _demoted_agents.add(stalled_agent) + except Exception as demote_err: + _pkg.logger.debug( + "Stall demotion skipped", + agent=stalled_agent, + error=str(demote_err), + ) + except Exception as stall_err: + _pkg.logger.debug( + "Stall demotion check failed", + pipeline_id=pipeline_id, + error=str(stall_err), + ) + + # 4. Non-blocking check for exited containers + for exec_info in active_executions: + if exec_info.container_id in exited_containers: + continue + try: + info = docker_client.get_container_info(exec_info.container_id) + except ( + _pkg.ContainerNotFoundError, + _pkg.ContainerOperationError, + _pkg.PodNotFoundError, + _pkg.JobOperationError, + ) as e: + _pkg.logger.warning( + "Container lost during poll", + container_id=exec_info.container_id, + role=exec_info.role.value, + error=str(e), + ) + info = ContainerInfo( + container_id=exec_info.container_id, + container_name=f"{pipeline_id}-{exec_info.role.value}", + status=ContainerStatus.FAILED, + exit_code=-1, + exited_at=_pkg.datetime.now(_pkg.UTC), + ) + + if info.status in ( + ContainerStatus.EXITED, + ContainerStatus.FAILED, + ContainerStatus.REMOVED, + ): + exited_containers[exec_info.container_id] = info + _record_container_exit(exec_info, info) + + # Handle non-clean exit as agent failure. 0 = normal, + # 143 = orchestrator-initiated SIGTERM (#2210) — both + # are classified as clean here to match the K8s monitor's + # _classify_exit, so the two layers can't race to write + # contradictory agent.status values. + if info.exit_code not in (0, 143): + # Issue #2806 (Option A): a producer's consensus-wrapper + # exhausting its retry budget is unrecoverable — the + # slice state machine cannot replace a permanently dead + # producer, and the surviving reviewers will heartbeat + # forever waiting on a proposal that will never come. + # Detect this case and short-circuit the polling loop + # with a non-zero return so the caller transitions the + # pipeline (or slice) to FAILED. Reviewer-only deaths + # still flow through ``handle_agent_failure`` because + # peer-review redistribution can recover them. + role_value = exec_info.role.value + if filtered_graph.is_producer(role_value): + # Race window guard: a producer can legitimately + # exit non-zero after CONFIRMED (wrapper cleanup + # crash) — between step 1 (consensus check) and + # step 4 (exit detection) the producer could have + # written CONFIRMED and then died. Re-query + # consensus before hard-failing; if it has + # completed, fall through and let the next + # iteration's step 1/2 return success. + try: + recheck = executor.check_consensus() + except Exception as recheck_err: + _pkg.logger.warning( + "Producer-death consensus recheck failed", + pipeline_id=pipeline_id, + role=role_value, + error=str(recheck_err), + ) + recheck = {"is_complete": False} + if recheck.get("is_complete"): + _pkg.logger.info( + "Producer container exited non-zero but consensus already complete — skipping hard-fail", + pipeline_id=pipeline_id, + role=role_value, + exit_code=info.exit_code, + ) + # Consensus completed in the race window before + # the producer's wrapper-cleanup crash. Step 5 + # (or the next iteration's step 1/2) will return + # success; skip handle_agent_failure (reviewer + # recovery path, not applicable to producers). + continue + _pkg._emit_producer_death_alert( + pipeline_id=pipeline_id, + role=role_value, + phase=phase_str, + slice_id=slice_id, + exit_code=info.exit_code, + ) + _pkg.logger.error( + "Producer agent died permanently — failing phase", + pipeline_id=pipeline_id, + phase=phase_str, + slice_id=slice_id, + role=role_value, + exit_code=info.exit_code, + ) + _stop_running_containers() + combined_logs = "\n".join( + all_logs + + [ + "--- PRODUCER PERMANENT DEATH ---", + ( + f"Producer '{role_value}' container exited with code " + f"{info.exit_code} after the consensus-wrapper exhausted " + f"its retry budget. Pipeline failing (issue #2806)." + ), + ] + ) + return 1, combined_logs + try: + executor.handle_agent_failure( + role=role_value, + error=f"Container exited with code {info.exit_code}", + ) + except Exception as e: + _pkg.logger.warning( + "handle_agent_failure error", + role=role_value, + error=str(e), + ) + else: + # Clean exit (0 or 143): the consensus wrapper inside + # the container handles restarts if the agent didn't + # signal READY. We do NOT auto-register READY here — + # agents must explicitly participate in consensus. + _pkg.logger.info( + "Container exited cleanly, wrapper handles consensus", + pipeline_id=pipeline_id, + role=exec_info.role.value, + exit_code=info.exit_code, + ) + + # 5. All containers exited — fall back to exit-code-based result. + # + # Guarded on a non-empty ``active_executions`` so an empty set is + # never misread as "everything exited" (``0 >= 0``). In orchestrator + # mode (#3064) ``spawn_all`` returns ``[]`` by design — the + # orchestrator owns the BRC loop and spawns one-shot pods per event, + # so there are no up-front containers to track. Completion is driven + # purely off ``check_consensus()`` (step 2) and the consensus timeout + # (step 6); a zero-container fallback here would otherwise fail the + # phase on the first poll, before any event-driven pod ran. + if active_executions and len(exited_containers) >= len(active_executions): + combined_logs = "\n".join(all_logs) + if has_failures[0]: + # Final consensus recheck: consensus may have completed between + # the step-2 check and now (race window while containers were + # shutting down). Re-query before giving up. + try: + final_consensus = executor.check_consensus() + except Exception as e: + _pkg.logger.warning( + "Final consensus recheck failed, treating as incomplete", + pipeline_id=pipeline_id, + error=str(e), + ) + final_consensus = {"is_complete": False} + + if final_consensus.get("is_complete"): + # Guard: consensus may be "complete" by quorum but still + # have unresolved NACKs — mirror the step 5 no-failure + # NACK check and the timeout path NACK check. + if final_consensus.get("has_unresolved_nacks"): + nack_details = final_consensus.get("unresolved_nacks", []) + nack_summary = _pkg._format_nack_summary(nack_details) + _pkg.logger.warning( + "Consensus complete on final recheck but unresolved NACKs remain (has_failures path)", + pipeline_id=pipeline_id, + nack_count=len(nack_details), + nack_summary=nack_summary, + ) + # Tag with the consensus-timeout context so "Retry + # phase" dispatches through restart_phase on resolve + # (#3421), for symmetry with the incomplete-consensus + # sites below. This question is hand-built and does not + # promise restart copy, but restart_phase is the correct + # "Retry phase" action regardless. Like its siblings + # this pod-mode path is unreachable today (spawn_all + # returns [] post-#3164, so active_executions is always + # empty); tagging keeps the dispatch honest if pod mode + # is ever revived. + _pkg._persist_hitl_decision( + pipeline_id, + pipeline, + store, + question=( + f"Consensus reached but {len(nack_details)} NACK(s) " + f"remain unresolved: {nack_summary}. How to proceed?" + ), + options=["Retry phase", "Accept current state", "Abort phase"], + phase=pipeline.current_phase, + context=_pkg._CONSENSUS_TIMEOUT_HITL_CONTEXT, + ) + combined_logs += ( + f"\n--- UNRESOLVED NACKs ({len(nack_details)}) ---\n{nack_summary}" + ) + return 1, combined_logs + + # Consensus reached after all — recover pipeline if needed + if store is not None: + try: + _current_pip = store.load_pipeline(pipeline_id) + if _current_pip.status == _pkg.PipelineStatus.FAILED: + _pkg.logger.warning( + "Pipeline externally marked FAILED but consensus is complete — recovering", + pipeline_id=pipeline_id, + ) + with _pkg.get_pipeline_state_lock(pipeline_id): + _current_pip = store.load_pipeline(pipeline_id) + if _current_pip.status == _pkg.PipelineStatus.FAILED: + _current_pip.status = _pkg.PipelineStatus.RUNNING + _current_pip.error = None + store.save_pipeline(_current_pip) + except Exception as recovery_err: + _pkg.logger.warning( + "External FAILED recovery check failed", + pipeline_id=pipeline_id, + error=str(recovery_err), + ) + + _elapsed_final = _pkg.time.monotonic() - start_time + if _pkg._emit_event is not None: + _pkg._emit_event( + _pkg.EventType.CONSENSUS_REACHED, + pipeline_id, + data={"elapsed_seconds": _elapsed_final}, + ) + _pkg.logger.info( + "Consensus reached on final recheck, stopping containers", + pipeline_id=pipeline_id, + elapsed_seconds=round(_elapsed_final, 1), + has_failures=has_failures[0], + ) + _update_agents_complete() + _stop_running_containers() + return 0, combined_logs + + # Incomplete consensus + container failures: surface an HITL + # decision so the operator can drive recovery (issue #2203). + # Without this, the phase fails terminally with no signal — + # the agent's committed work is still on the per-role branch + # and `restart_phase` would recover, but the operator has no + # way to know that without out-of-band investigation. + # + # If an objection HITL was created earlier in the polling loop + # this is intentionally a *second* pending decision: it + # carries different options ("Retry phase" / "Accept current + # state" / "Abort phase" vs the objection set) and conveys a + # different operator action. The test + # `test_objection_dedup_distinct_from_incomplete_consensus_hitl` + # locks in the two-decision UX. + failure_count = sum(1 for info in exited_containers.values() if info.exit_code != 0) + question, log_suffix = _pkg._incomplete_consensus_decision_text( + final_consensus, container_failure_count=failure_count + ) + _pkg.logger.warning( + "Incomplete consensus with container failures — escalating to HITL", + pipeline_id=pipeline_id, + failure_count=failure_count, + blocking_agents=final_consensus.get("blocking_agents", []), + nack_count=len(final_consensus.get("unresolved_nacks", []) or []), + ) + # Tag with the consensus-timeout context so "Retry phase" + # dispatches through restart_phase on resolve (#3421), matching + # the restart semantics `_incomplete_consensus_decision_text` + # promises. This pod-mode container-exit path is unreachable + # today (spawn_all returns [] post-#3164, so active_executions + # is always empty), but tagging keeps the copy honest if pod + # mode is ever revived. + _pkg._persist_hitl_decision( + pipeline_id, + pipeline, + store, + question=question, + options=["Retry phase", "Accept current state", "Abort phase"], + phase=pipeline.current_phase, + context=_pkg._CONSENSUS_TIMEOUT_HITL_CONTEXT, + ) + combined_logs += log_suffix + return 1, combined_logs + + # Before returning success, check the BRC approval matrix for + # unresolved NACKs. If reviewers NACKed but producers exited + # without iterating, we must NOT report success — escalate to + # HITL so a human can decide how to proceed. + if consensus.get("has_unresolved_nacks"): + nack_details = consensus.get("unresolved_nacks", []) + nack_summary = _pkg._format_nack_summary(nack_details) + _pkg.logger.warning( + "All containers exited with unresolved NACKs", + pipeline_id=pipeline_id, + nack_count=len(nack_details), + nack_summary=nack_summary, + ) + # Same as the unresolved-NACK site above: tag with the + # consensus-timeout context so "Retry phase" dispatches through + # restart_phase (#3421) for symmetry. Hand-built question, no + # restart copy, but restart_phase is the right action here too. + # Dead pod-mode path today; tagging is cheap insurance. + _pkg._persist_hitl_decision( + pipeline_id, + pipeline, + store, + question=( + f"All agents exited but {len(nack_details)} NACK(s) remain " + f"unresolved: {nack_summary}. How to proceed?" + ), + options=["Retry phase", "Accept current state", "Abort phase"], + phase=pipeline.current_phase, + context=_pkg._CONSENSUS_TIMEOUT_HITL_CONTEXT, + ) + combined_logs += f"\n--- UNRESOLVED NACKs ({len(nack_details)}) ---\n{nack_summary}" + return 1, combined_logs + + # Final consensus completeness check: all containers exited + # cleanly (no failures, no NACKs) but consensus may not have + # been reached. Mirror the has_failures branch pattern to + # prevent advancing without confirmed BRC consensus. + try: + final_consensus = executor.check_consensus() + except Exception as e: + _pkg.logger.warning( + "Final consensus recheck failed on clean exit, treating as incomplete", + pipeline_id=pipeline_id, + error=str(e), + ) + final_consensus = {"is_complete": False} + + if not final_consensus.get("is_complete"): + # Symmetric to the has_failures path: clean exits with no + # consensus also need an HITL decision so the operator can + # drive recovery (issue #2203). + question, log_suffix = _pkg._incomplete_consensus_decision_text( + final_consensus, container_failure_count=0 + ) + _pkg.logger.warning( + "All containers exited cleanly but consensus not reached — escalating to HITL", + pipeline_id=pipeline_id, + elapsed_seconds=round(elapsed, 1), + blocking_agents=final_consensus.get("blocking_agents", []), + ) + # Same as the container-failure path above: tag with the + # consensus-timeout context so "Retry phase" dispatches through + # restart_phase (#3421) and honors the restart copy. Also a + # dead pod-mode path today; tagging is cheap insurance. + _pkg._persist_hitl_decision( + pipeline_id, + pipeline, + store, + question=question, + options=["Retry phase", "Accept current state", "Abort phase"], + phase=pipeline.current_phase, + context=_pkg._CONSENSUS_TIMEOUT_HITL_CONTEXT, + ) + combined_logs += log_suffix + return 1, combined_logs + + # Consensus confirmed on clean exit — mirror the has_failures + # success path: emit event, update agent state, stop containers. + if _pkg._emit_event is not None: + _pkg._emit_event( + _pkg.EventType.CONSENSUS_REACHED, + pipeline_id, + data={"elapsed_seconds": elapsed}, + ) + _pkg.logger.info( + "Consensus reached on final recheck, stopping containers", + pipeline_id=pipeline_id, + elapsed_seconds=round(elapsed, 1), + has_failures=has_failures[0], + ) + _update_agents_complete() + _stop_running_containers() + return 0, combined_logs + + # 6. Consensus timeout + if elapsed >= consensus_timeout: + # #3426 HITL gate: while an unresolved operator HITL decision + # (contract ``cq-N``) gates the running phase, the slice is + # provably operator-gated — a reviewer withholding its ACK + # pending a human ruling is the system working as designed, not + # a convergence failure. Suspend the timeout (keep polling, no + # alert, no failure) until the operator answers. On release, + # reset the convergence clock so the agents folding in the + # resolution get a full fresh window instead of a clock that + # already expired while the human was thinking. + _hitl_ids = _pkg._unresolved_contract_hitl_ids(pipeline_id, pipeline, phase_str) + if _hitl_ids: + if not _hitl_gate_deferring: + _pkg.logger.info( + "Consensus timeout suspended — phase is operator-gated " + "on unresolved HITL decision(s)", + pipeline_id=pipeline_id, + slice_id=slice_id, + elapsed_seconds=round(elapsed, 1), + decision_ids=_hitl_ids, + ) + _hitl_gate_deferring = True + _pkg.time.sleep(poll_interval) + continue + if _hitl_gate_deferring: + _hitl_gate_deferring = False + start_time = _pkg.time.monotonic() + _pkg.logger.info( + "Consensus timeout clock reset — operator HITL decision(s) resolved", + pipeline_id=pipeline_id, + slice_id=slice_id, + suspended_after_seconds=round(elapsed, 1), + ) + continue + + # #2243 progress gate: keep polling instead of publishing + # the consensus-timeout alert while producer/reviewer + # activity is still live on the BRC bus or in container + # heartbeats. Without this gate, the historical decision-15 + # / decision-17 misfires on ``issue-1557-v2`` (now + # ``OVERSEER_ALERT`` post-#2264) fired minutes before the + # next commit landed. + _gate_seconds = max( + 0, + int(getattr(pipeline.config, "brc_consensus_progress_gate_seconds", 300)), + ) + _gate_defer, _gate_reason = _pkg._check_brc_progress_gate( + pipeline_id, + slice_id, + [e.role.value for e in active_executions], + _gate_seconds, + ) + if _gate_defer: + if not _progress_gate_deferring: + _pkg.logger.info( + "Consensus timeout deferred by progress gate", + pipeline_id=pipeline_id, + elapsed_seconds=round(elapsed, 1), + gate_seconds=_gate_seconds, + reason=_gate_reason, + ) + _progress_gate_deferring = True + _pkg.time.sleep(poll_interval) + continue + if _progress_gate_deferring: + _pkg.logger.info( + "Consensus timeout proceeding — progress gate window elapsed", + pipeline_id=pipeline_id, + elapsed_seconds=round(elapsed, 1), + gate_seconds=_gate_seconds, + ) + _progress_gate_deferring = False + + # #3490 live-widening gate: re-resolve the budget from freshly + # loaded config so a PATCH /config update of + # ``consensus_timeout_minutes*`` takes effect any time before the + # wall fires; an operator watching a giant slice can widen the + # window without letting the slice fail and restarting. Checked + # here, after the HITL and progress gates, so the load only + # happens once per firing rather than on every deferred poll. A + # load failure keeps the current budget: a transient store hiccup + # must never widen or shrink the window on its own. + if store is not None: + _fresh_minutes: int | None = None + try: + _fresh_config = store.load_pipeline(pipeline_id).config + _fresh_minutes = resolve_consensus_timeout_minutes(_fresh_config, phase_str) + except Exception as _reresolve_err: + _pkg.logger.warning( + "Consensus-timeout config re-resolve failed; keeping current budget", + pipeline_id=pipeline_id, + error=str(_reresolve_err), + ) + # The isinstance guard keeps a malformed store payload (or a + # test double) from replacing the numeric budget. + if isinstance(_fresh_minutes, int) and not isinstance(_fresh_minutes, bool): + _fresh_timeout = max(_fresh_minutes, 1) * 60 + if _fresh_timeout != consensus_timeout: + _pkg.logger.info( + "Consensus timeout budget updated from live config", + pipeline_id=pipeline_id, + slice_id=slice_id, + old_timeout_minutes=consensus_timeout / 60, + new_timeout_minutes=_fresh_timeout / 60, + ) + consensus_timeout = _fresh_timeout + if elapsed < consensus_timeout: + _pkg.time.sleep(poll_interval) + continue + + _pkg.logger.warning( + "Consensus timeout reached, falling back to container exit", + pipeline_id=pipeline_id, + timeout_minutes=consensus_timeout / 60, + ) + # Orchestrator mode (#3064): we are giving up on convergence, so + # stop the BRC event loop before the fallback wait so it does not + # keep spawning one-shot pods past the deadline. No-op in pod + # mode. (The progress-gate ``continue`` above is taken before + # this point, so a deferral never reaches here and the loop keeps + # running across the deferral window.) + executor.stop_event_loop() + _pkg._handle_brc_consensus_timeout( + pipeline, + pipeline_id, + consensus_timeout, + consensus.get("blocking_agents", []), + store, + slice_id=slice_id, + active_role_names=[e.role.value for e in active_executions], + ) + + # Fall back: event-driven wait for remaining containers. + # + # Issue #1921: the previous implementation used a + # ThreadPoolExecutor with a blocking + # wait_for_container(timeout=3600) per container. During + # that hour the polling loop was blind to BRC progress — + # a NACK → re-propose → ACK cycle completing in the final + # minute could still be force-killed. Now we poll + # container status in short steps and re-check consensus + # between steps, early-returning on completion before + # force-killing anything. + # + # Issue #2245: the per-iteration budget rebaselines on + # producer progress. Each new CONSENSUS_PROPOSE (initial + # or NACK→re-propose) resets ``last_progress_at`` so the + # producer's next iteration gets a clean clock instead of + # inheriting the prior iterations' wall-clock spend. An + # absolute cap (``post_consensus_max_total_seconds``) + # bounds the total wait so an unbounded propose churn + # can't stall the pipeline indefinitely. + remaining = [e for e in active_executions if e.container_id not in exited_containers] + if remaining: + post_timeout_iteration_budget = ( + pipeline.config.post_consensus_iteration_budget_seconds + ) + post_timeout_max_total = pipeline.config.post_consensus_max_total_seconds + post_timeout_poll_interval = 30 # seconds between checks + post_timeout_start = _pkg.time.monotonic() + last_progress_at = post_timeout_start + + # Snapshot the latest proposal timestamp at entry so we + # only count *new* proposals as progress signals. ``None`` + # is fine: the rebaseline check at the bottom of the loop + # short-circuits on ``last_seen_proposal_ts is None`` + # before any datetime comparison runs. + last_seen_proposal_ts = _latest_proposal_ts(pipeline_id, slice_id) + + while remaining: + now_monotonic = _pkg.time.monotonic() + total_elapsed = now_monotonic - post_timeout_start + iteration_elapsed = now_monotonic - last_progress_at + if total_elapsed >= post_timeout_max_total: + _pkg.logger.warning( + "Post-consensus-timeout absolute cap reached", + pipeline_id=pipeline_id, + total_elapsed_seconds=round(total_elapsed, 1), + max_total_seconds=post_timeout_max_total, + ) + break + if iteration_elapsed >= post_timeout_iteration_budget: + _pkg.logger.warning( + "Post-consensus-timeout iteration budget exhausted", + pipeline_id=pipeline_id, + iteration_elapsed_seconds=round(iteration_elapsed, 1), + iteration_budget_seconds=post_timeout_iteration_budget, + total_elapsed_seconds=round(total_elapsed, 1), + ) + break + + # A. Re-check consensus; if agents converged during + # the wait, stop containers and return success + # before force-killing them. + try: + _wait_consensus = executor.check_consensus() + except Exception as _wait_consensus_err: + _pkg.logger.warning( + "Consensus recheck during post-timeout wait failed", + pipeline_id=pipeline_id, + error=str(_wait_consensus_err), + ) + _wait_consensus = None + + if ( + _wait_consensus + and _wait_consensus.get("is_complete") + and not _wait_consensus.get("has_unresolved_nacks") + ): + combined_logs = "\n".join(all_logs) + _total_elapsed = _pkg.time.monotonic() - start_time + if _pkg._emit_event is not None: + _pkg._emit_event( + _pkg.EventType.CONSENSUS_REACHED, + pipeline_id, + data={"elapsed_seconds": _total_elapsed}, + ) + _pkg.logger.info( + "Consensus reached during post-timeout wait", + pipeline_id=pipeline_id, + elapsed_post_timeout_seconds=round(total_elapsed, 1), + total_elapsed_seconds=round(_total_elapsed, 1), + ) + _update_agents_complete() + _stop_running_containers() + return 0, combined_logs + + # A'. Rebaseline the iteration clock on producer + # progress (#2245). A fresh CONSENSUS_PROPOSE + # timestamp means a producer just landed work + # (initial propose or NACK→re-propose) — the next + # round of reviews deserves its own iteration + # budget, not whatever's left of the prior round's. + current_proposal_ts = _latest_proposal_ts(pipeline_id, slice_id) + if current_proposal_ts is not None and ( + last_seen_proposal_ts is None or current_proposal_ts > last_seen_proposal_ts + ): + _pkg.logger.info( + "Post-consensus-timeout clock rebaselined on producer progress", + pipeline_id=pipeline_id, + iteration_elapsed_seconds=round(iteration_elapsed, 1), + total_elapsed_seconds=round(total_elapsed, 1), + proposal_timestamp=current_proposal_ts.isoformat(), + ) + last_seen_proposal_ts = current_proposal_ts + last_progress_at = _pkg.time.monotonic() + + # B. Non-blocking container status check; record + # any that have exited naturally. + still_running = [] + for exec_info in remaining: + try: + info = docker_client.get_container_info(exec_info.container_id) + except ( + _pkg.ContainerNotFoundError, + _pkg.ContainerOperationError, + _pkg.PodNotFoundError, + _pkg.JobOperationError, + ) as _wait_status_err: + _pkg.logger.warning( + "Container lost during post-timeout wait", + container_id=exec_info.container_id, + role=exec_info.role.value, + error=str(_wait_status_err), + ) + info = ContainerInfo( + container_id=exec_info.container_id, + container_name=f"{pipeline_id}-{exec_info.role.value}", + status=ContainerStatus.FAILED, + exit_code=-1, + exited_at=_pkg.datetime.now(_pkg.UTC), + ) + + if info.status in ( + ContainerStatus.EXITED, + ContainerStatus.FAILED, + ContainerStatus.REMOVED, + ): + exited_containers[exec_info.container_id] = info + _record_container_exit(exec_info, info) + else: + still_running.append(exec_info) + + remaining = still_running + if not remaining: + break + + _pkg.time.sleep(post_timeout_poll_interval) + + # Budget exhausted with containers still running — + # force-kill so they don't orphan (issue #1691). + for exec_info in remaining: + try: + docker_client.stop_container(exec_info.container_id, timeout=30) + except Exception: + pass + final_info = ContainerInfo( + container_id=exec_info.container_id, + container_name=f"{pipeline_id}-{exec_info.role.value}", + status=ContainerStatus.FAILED, + exit_code=-1, + exited_at=_pkg.datetime.now(_pkg.UTC), + ) + exited_containers[exec_info.container_id] = final_info + _record_container_exit(exec_info, final_info) + + combined_logs = "\n".join(all_logs) + if has_failures[0]: + # Consensus recheck: consensus may have completed right as the + # post-timeout budget elapsed and containers were force-killed + # (issue #1691). The in-loop consensus check covers the common + # case; this recheck catches the narrow race where consensus + # completed between the last in-loop check and force-kill. + try: + _timeout_consensus = executor.check_consensus() + except Exception as e: + _pkg.logger.warning( + "Consensus recheck after timeout failed, treating as incomplete", + pipeline_id=pipeline_id, + error=str(e), + ) + _timeout_consensus = {"is_complete": False} + + if _timeout_consensus.get("is_complete"): + # Guard: consensus may be "complete" by quorum but still + # have unresolved NACKs — mirror the step 5 NACK check. + if _timeout_consensus.get("has_unresolved_nacks"): + nack_details = _timeout_consensus.get("unresolved_nacks", []) + nack_summary = _pkg._format_nack_summary(nack_details) + _pkg.logger.warning( + "Consensus complete on timeout recheck but unresolved NACKs remain", + pipeline_id=pipeline_id, + nack_count=len(nack_details), + nack_summary=nack_summary, + ) + # Tag with the consensus-timeout context so "Retry + # phase" dispatches through restart_phase on resolve + # (#3421), for symmetry with the incomplete-consensus + # sites above. This question is hand-built and does not + # promise restart copy, but restart_phase is the correct + # "Retry phase" action regardless. Like its siblings + # this pod-mode path is unreachable today: has_failures[0] + # is only set in _record_container_exit, called solely for + # active_executions / remaining members, which are always + # empty in orchestrator mode (spawn_all returns [] + # post-#3164). Tagging keeps the dispatch honest if pod + # mode is ever revived. + _pkg._persist_hitl_decision( + pipeline_id, + pipeline, + store, + question=( + f"Consensus reached after timeout but {len(nack_details)} NACK(s) " + f"remain unresolved: {nack_summary}. How to proceed?" + ), + options=["Retry phase", "Accept current state", "Abort phase"], + phase=pipeline.current_phase, + context=_pkg._CONSENSUS_TIMEOUT_HITL_CONTEXT, + ) + combined_logs += ( + f"\n--- UNRESOLVED NACKs ({len(nack_details)}) ---\n{nack_summary}" + ) + return 1, combined_logs + + # Consensus reached during the wait — recover pipeline + if store is not None: + try: + _current_pip = store.load_pipeline(pipeline_id) + if _current_pip.status == _pkg.PipelineStatus.FAILED: + _pkg.logger.warning( + "Pipeline externally marked FAILED but consensus is complete — recovering (timeout path)", + pipeline_id=pipeline_id, + ) + with _pkg.get_pipeline_state_lock(pipeline_id): + _current_pip = store.load_pipeline(pipeline_id) + if _current_pip.status == _pkg.PipelineStatus.FAILED: + _current_pip.status = _pkg.PipelineStatus.RUNNING + _current_pip.error = None + store.save_pipeline(_current_pip) + except Exception as recovery_err: + _pkg.logger.warning( + "External FAILED recovery check failed (timeout path)", + pipeline_id=pipeline_id, + error=str(recovery_err), + ) + + _elapsed_timeout = _pkg.time.monotonic() - start_time + if _pkg._emit_event is not None: + _pkg._emit_event( + _pkg.EventType.CONSENSUS_REACHED, + pipeline_id, + data={"elapsed_seconds": _elapsed_timeout}, + ) + _pkg.logger.info( + "Consensus reached on recheck after timeout, treating as success", + pipeline_id=pipeline_id, + elapsed_seconds=round(_elapsed_timeout, 1), + has_failures=has_failures[0], + ) + _update_agents_complete() + _stop_running_containers() + return 0, combined_logs + + # Consensus not complete on recheck. Mirror the non-failure + # branch's NACK summary so operators see which reviewer edges + # are still blocking, even when containers had non-zero exits. + if _timeout_consensus.get("has_unresolved_nacks"): + nack_details = _timeout_consensus.get("unresolved_nacks", []) + nack_summary = _pkg._format_nack_summary(nack_details) + _pkg.logger.warning( + "Timeout with unresolved NACKs (has_failures path)", + pipeline_id=pipeline_id, + nack_count=len(nack_details), + ) + combined_logs += ( + f"\n--- UNRESOLVED NACKs ({len(nack_details)}) ---\n{nack_summary}" + ) + return 1, combined_logs + + # After timeout, check the BRC approval matrix for unresolved + # NACKs before declaring success. Producers that exited without + # addressing reviewer feedback should not be treated as passing. + try: + _final_consensus = executor.check_consensus() + except Exception: + _pkg.logger.warning("Failed to check consensus at timeout", exc_info=True) + _final_consensus = {} + if _final_consensus.get("has_unresolved_nacks"): + nack_details = _final_consensus.get("unresolved_nacks", []) + nack_summary = _pkg._format_nack_summary(nack_details) + _pkg.logger.warning( + "Timeout with unresolved NACKs — returning failure", + pipeline_id=pipeline_id, + nack_count=len(nack_details), + ) + combined_logs += f"\n--- UNRESOLVED NACKs ({len(nack_details)}) ---\n{nack_summary}" + return 1, combined_logs + + # Orchestrator-owned event loop: this timeout fallthrough is the + # dominant non-convergence terminal. spawn_all returns [] by + # design, so step 5's "all containers exited" path is guarded off + # (it requires a non-empty active set) and a slice that never + # converged — producer never proposed, a reviewer pod failed to + # ACK, reviews pending with no NACK — lands here with no NACKs. + # Unlike pod mode, where a clean all-exited phase already routed + # through step 5's is_complete check, nothing upstream has verified + # consensus completeness on this path. Mirror step 5: when the + # orchestrator owns the loop and consensus is incomplete, escalate + # an HITL and fail rather than reporting a non-converged slice as + # success (a bare `return 0` here would advance the phase toward PR + # creation past the BRC consensus gate). + if executor.owns_event_loop() and not _final_consensus.get("is_complete"): + question, log_suffix = _pkg._incomplete_consensus_decision_text( + _final_consensus, container_failure_count=0, orchestrator_mode=True + ) + _pkg.logger.warning( + "Consensus timed out and is incomplete (orchestrator-owned loop) — escalating to HITL", + pipeline_id=pipeline_id, + blocking_agents=_final_consensus.get("blocking_agents", []), + ) + _pkg._persist_hitl_decision( + pipeline_id, + pipeline, + store, + question=question, + options=["Retry phase", "Accept current state", "Abort phase"], + phase=pipeline.current_phase, + context=_pkg._CONSENSUS_TIMEOUT_HITL_CONTEXT, + ) + combined_logs += log_suffix + return 1, combined_logs + + return 0, combined_logs + + # 7. Sleep before next poll + _pkg.time.sleep(poll_interval) diff --git a/orchestrator/routes/pipelines/_run_concurrent_retry.py b/orchestrator/routes/pipelines/_run_concurrent_retry.py new file mode 100644 index 0000000000..3262d3c3b1 --- /dev/null +++ b/orchestrator/routes/pipelines/_run_concurrent_retry.py @@ -0,0 +1,215 @@ +"""impasse-retry wrapper for the concurrent-phase runner (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _run_concurrent_phase_with_impasse_retry( + pipeline_id: str, + pipeline: _pkg.Pipeline, + phase: str, + spawner, + repo_volumes: dict[str, str], + gateway_mode: str, + repos: list[str], + sandbox_env: dict[str, str], + store, + certs_volume: str | None, + worktree_repo_path: _pkg.Path, + review_feedback: str | None = None, + slice_id: str | None = None, + operator_directives: list[_pkg.OperatorDirective] | None = None, + iteration_history: list[_pkg.IterationSummary] | None = None, + run_epoch: _pkg.datetime | None = None, +) -> tuple[int, str]: + """Run a concurrent phase, auto-delegating impasses once before HITL. + + Wraps :func:`_run_concurrent_phase` with the runtime escape-hatch + introduced in #2529: + + 1. Run the BRC cycle as usual. + 2. After it exits, scan each producer's ``AgentOutput`` for a typed + :class:`egg_contracts.Impasse`. + 3. For ``WRONG_ROLE`` impasses with a single eligible alternative + producer role and ``task.delegation_attempts == 0``, mutate + ``task.role`` to the suggested role and re-run the BRC cycle + once. The new spawn picks up the role flip when + ``_build_agent_prompt`` re-reads the contract. + 4. For everything else (second impasse, non-WRONG_ROLE category, + no eligible alternative role, unresolvable task_id) the helper + creates a HITL decision on the contract and the slice exits + so the operator can choose between cancel / re-plan / manual + resolution. ``feedback_no_auto_hitl.md``: the orchestrator + creates the decision; surfacing to the user is the operator + layer's job. + + Pipeline-level (non-sliced) callers can pass ``slice_id=None``; + the routing helper falls back to a contract-wide search for the + impassed task. + """ + try: + from orchestrator.impasse_routing import ( + ImpasseAction, + collect_impasses, + route_impasses, + ) + except ImportError: + from impasse_routing import ( # type: ignore[no-redef] + ImpasseAction, + collect_impasses, + route_impasses, + ) + + try: + from egg_contracts.agent_roles import AgentRole as ContractAgentRoleEnum + except ImportError: # pragma: no cover - import seam parity + from shared.egg_contracts.agent_roles import ( # type: ignore[no-redef] + AgentRole as ContractAgentRoleEnum, + ) + # Two attempts max: original + at most one delegated retry. The + # ``delegation_attempts`` counter on the contract task enforces the + # same bound when the slice is restarted out-of-band by an + # operator, so a long-lived pipeline can never escape this gate. + MAX_IMPASSE_ATTEMPTS = 2 + + # Producer roles only — impasses are a producer concept; reviewers + # don't author tasks. Mirrors the producer trio in + # ``shared/egg_restrictions/patterns.py``. + producer_roles = [ + ContractAgentRoleEnum.CODER, + ContractAgentRoleEnum.TESTER, + ContractAgentRoleEnum.DOCUMENTER, + ] + + last_exit = 0 + last_logs = "" + for attempt in range(MAX_IMPASSE_ATTEMPTS): + is_terminal = attempt + 1 == MAX_IMPASSE_ATTEMPTS + + last_exit, last_logs = _pkg._run_concurrent_phase( + pipeline_id=pipeline_id, + pipeline=pipeline, + phase=phase, + spawner=spawner, + repo_volumes=repo_volumes, + gateway_mode=gateway_mode, + repos=repos, + sandbox_env=sandbox_env, + store=store, + certs_volume=certs_volume, + worktree_repo_path=worktree_repo_path, + review_feedback=review_feedback, + slice_id=slice_id, + operator_directives=operator_directives, + iteration_history=iteration_history, + run_epoch=run_epoch, + ) + + try: + impasses = collect_impasses( + _pkg.Path(worktree_repo_path), + pipeline_id, + producer_roles, + ) + except Exception as scan_err: # noqa: BLE001 + _pkg.logger.warning( + "Impasse scan raised; continuing without delegation", + pipeline_id=pipeline_id, + slice_id=slice_id, + error=str(scan_err), + ) + return last_exit, last_logs + + if not impasses: + return last_exit, last_logs + + # Defense-in-depth (#3315 facet a, slice path): if a restart bumped + # ``run_epoch`` while this thread was running, a stale producer-written + # impasse file could otherwise drive ``route_impasses`` into a HITL + # against the freshly-restarted phase. The poll loop in + # ``_run_concurrent_phase`` already bails on supersession before any + # escalation; mirror that here so the "no escalation when superseded" + # property holds on the slice path too — return the (superseded) result + # without routing. + if _pkg._pipeline_superseded_by_restart(store, pipeline_id, run_epoch): + _pkg.logger.info( + "Restart superseded this thread before impasse routing; " + "skipping route_impasses to avoid escalating against a " + "freshly-restarted phase", + pipeline_id=pipeline_id, + slice_id=slice_id, + ) + return last_exit, last_logs + + try: + # On the terminal iteration we have no remaining BRC cycle + # to respawn with a new role, so a delegation made here + # would silently dangle (review feedback #2 on PR #2553). + # Force every impasse onto the escalate path instead. + decisions = route_impasses( + repo_path=_pkg.Path(worktree_repo_path), + pipeline_id=pipeline_id, + contract_identifier=pipeline_id, + impasses=impasses, + slice_id=slice_id, + force_escalate=is_terminal, + ) + except Exception as route_err: # noqa: BLE001 + _pkg.logger.error( + "Impasse routing raised; surfacing slice failure", + pipeline_id=pipeline_id, + slice_id=slice_id, + error=str(route_err), + ) + return last_exit, last_logs + + all_delegated = decisions and all(d.action == ImpasseAction.DELEGATE for d in decisions) + if not all_delegated: + # Any escalation, or an empty decision list, means the + # operator gates the next move. Don't auto-retry. + for d in decisions: + _pkg.logger.info( + "Impasse decision", + pipeline_id=pipeline_id, + slice_id=slice_id, + action=d.action.value, + role=d.role, + task_id=d.task_id, + new_role=d.new_role, + reason=d.reason, + hitl_decision_id=d.hitl_decision_id, + ) + return last_exit, last_logs + + # All impasses delegated cleanly — the contract has been + # mutated, log the swap and let the loop respawn with the new + # roles. Last attempt falls through and returns whatever the + # second BRC cycle produced. + for d in decisions: + _pkg.logger.info( + "Impasse delegated; retrying slice with new role", + pipeline_id=pipeline_id, + slice_id=slice_id, + attempt=attempt + 1, + from_role=d.role, + to_role=d.new_role, + task_id=d.task_id, + ) + + # Drop the now-routed impasse signals before the next BRC + # cycle, so a producer that crashes pre-handoff in iter-N+1 + # cannot resurrect this iteration's impasse via a stale file. + _pkg._clear_stale_impasses_for_producers( + _pkg.Path(worktree_repo_path), + pipeline_id, + producer_roles, + cleanup_reason="post-delegation cleanup", + ) + + return last_exit, last_logs diff --git a/orchestrator/routes/pipelines/_run_concurrent_support.py b/orchestrator/routes/pipelines/_run_concurrent_support.py new file mode 100644 index 0000000000..1e3956310c --- /dev/null +++ b/orchestrator/routes/pipelines/_run_concurrent_support.py @@ -0,0 +1,422 @@ +"""concurrent-phase lifted helpers helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _superseded_by_restart_impl(*, store, pipeline_id, run_epoch) -> bool: + """True if a newer run_epoch means another thread owns this pipeline. + + Reloads pipeline state and compares its ``run_epoch`` against the + epoch this thread runs under. Mirrors the post-return epoch check + (#1638) but runs *inside* the poll loop so a superseded thread stops + polling before it can fire stale escalations. Best-effort: a load + failure returns ``False`` so a transient store hiccup never tears + down a legitimately-running phase. + """ + return _pkg._pipeline_superseded_by_restart(store, pipeline_id, run_epoch) + + +def _record_container_exit_impl( + exec_info, + final_info, + *, + docker_client, + _logs_lock, + has_failures, + all_logs, + store, + phase_str, + pipeline_id, +) -> None: + from models import AgentExecutionStatus as StateAgentStatus + + """Capture logs and update pipeline state for an exited container.""" + container_logs = "" + if final_info.exit_code != 0: + try: + container_logs = docker_client.get_container_logs( + exec_info.container_id, + tail=200, + ) + except Exception: + pass + + with _logs_lock: + # 143 (SIGTERM) is orchestrator-initiated teardown, not a + # failure — match the K8s monitor's classifier (#2210) so + # the two layers don't disagree about what 143 means. + if final_info.exit_code not in (0, 143): + has_failures[0] = True + all_logs.append( + f"--- {exec_info.role.value} (exit={final_info.exit_code}) ---\n{container_logs}" + ) + + if store is not None: + try: + with _pkg.get_pipeline_state_lock(pipeline_id): + pip = store.load_pipeline(pipeline_id) + pe = pip.get_phase_execution(_pkg.PipelinePhase(phase_str)) + + for ci in pe.containers: + if ci.container_id == exec_info.container_id: + ci.status = final_info.status + ci.exited_at = final_info.exited_at + ci.exit_code = final_info.exit_code + break + + for agent in pe.agents: + if agent.container_id == exec_info.container_id: + agent.completed_at = _pkg.datetime.now(_pkg.UTC) + if final_info.exit_code in (0, 143): + agent.status = StateAgentStatus.COMPLETE + else: + agent.status = StateAgentStatus.FAILED + agent.error = f"Container exited with code {final_info.exit_code}" + break + + # Cap each tail line at 4096 chars: containers that print + # large JSON blobs on one line could otherwise persist + # multi-MB lines into pipeline state on every chatty exit. + last_lines = ( + [ln[:4096] for ln in container_logs.splitlines()[-200:]] + if container_logs + else [] + ) + pe.agent_exits.append( + _pkg.AgentExitInfo( + role=exec_info.role, + exit_code=final_info.exit_code, + last_lines=last_lines, + terminated_at=_pkg.datetime.now(_pkg.UTC), + container_id=exec_info.container_id, + ) + ) + + store.save_pipeline(pip) + except Exception as track_err: + _pkg.logger.warning( + "Failed to update concurrent agent state", + container_id=exec_info.container_id, + error=str(track_err), + ) + + +def _stop_running_containers_impl(*, active_executions, exited_containers, docker_client) -> None: + """Gracefully stop all containers that haven't exited yet.""" + for e in active_executions: + if e.container_id not in exited_containers: + try: + docker_client.stop_container(e.container_id, timeout=30) + except Exception: + pass + + +def _latest_proposal_ts_impl(_pid, _sid): + _get_brc_tracker = None + try: + from peer_consensus import get_peer_consensus_tracker as _get_brc_tracker + except ImportError: + from ..peer_consensus import ( + get_peer_consensus_tracker as _get_brc_tracker, # type: ignore[no-redef] + ) + + """Return the latest CONSENSUS_PROPOSE timestamp from the BRC tracker. + + Used by the post-consensus-timeout poll loop (#2245) to rebaseline + the per-iteration budget on producer progress. Returns ``None`` if + the tracker is unavailable, has no proposals, or any lookup raises — + callers treat ``None`` as "no progress signal yet" and proceed + without a rebaseline. + """ + if _get_brc_tracker is None: + return None + try: + _t = _get_brc_tracker(_pid, _sid) + except Exception: + return None + if _t is None: + return None + try: + return _t.get_latest_proposal_timestamp() + except Exception: + return None + + +def _update_agents_complete_impl(*, store, phase_str, pipeline_id, slice_id) -> None: + from models import AgentExecutionStatus as StateAgentStatus + + _get_brc_tracker = None + try: + from peer_consensus import get_peer_consensus_tracker as _get_brc_tracker + except ImportError: + from ..peer_consensus import ( + get_peer_consensus_tracker as _get_brc_tracker, # type: ignore[no-redef] + ) + + """Mark all running agents as COMPLETE in pipeline state (consensus path).""" + if store is None: + return + try: + with _pkg.get_pipeline_state_lock(pipeline_id): + pip = store.load_pipeline(pipeline_id) + pe = pip.get_phase_execution(_pkg.PipelinePhase(phase_str)) + completed_container_ids: set[str] = set() + + # Look up proposal commit SHAs from the BRC tracker so we can + # populate agent.commit (issue #1691). The lookup is slice- + # aware (#2137) — when ``slice_id`` is set the tracker key + # is the nested ``{pipeline_id}/{slice_id}`` form. + _brc = None + if _get_brc_tracker is not None: + try: + _brc = _get_brc_tracker(pipeline_id, slice_id) + except TypeError: + # Older tracker import-shim without slice_id support. + try: + _brc = _get_brc_tracker(pipeline_id) + except Exception: + pass + except Exception: + pass + + # Filter to this slice's agents — without the filter, slice-2 + # BRC completing flips slice-3's still-running agents to + # COMPLETE because they share ``pe.agents`` (#2422). For + # pipeline-level (non-sliced) phases ``slice_id`` is ``None`` + # and we still match all agents whose ``slice_id`` is ``None``. + for agent in pe.agents: + if getattr(agent, "slice_id", None) != slice_id: + continue + if agent.status in (StateAgentStatus.RUNNING, StateAgentStatus.FAILED): + agent.status = StateAgentStatus.COMPLETE + agent.completed_at = _pkg.datetime.now(_pkg.UTC) + if agent.container_id: + completed_container_ids.add(agent.container_id) + # Populate commit SHA from the consensus tracker's proposal + # records. Only producers have SHAs; reviewers get "". + if _brc is not None and not agent.commit: + sha = _brc.get_proposal_commit_sha(agent.role.value) + if sha and sha != "RECONSTRUCTED_NO_SHA": + agent.commit = sha + elif sha is None or sha == "RECONSTRUCTED_NO_SHA": + # Diagnostic only (#1911): log when the BRC + # tracker returns null or the + # RECONSTRUCTED_NO_SHA sentinel for a role + # so we can see on real runs whether the + # three-role implement phase + # (coder/tester/documenter) wiring misses + # SHAs. Deliberately no auto-fallback — + # that would mask the real bug. Empty + # string is the expected reviewer default + # (reviewers never propose) — do NOT warn + # for that case or the signal drowns in + # noise. + _pkg.logger.warning( + "BRC tracker returned no commit sha for completed agent", + pipeline_id=pipeline_id, + phase=phase_str, + role=agent.role.value, + brc_value=sha, + ) + + # Also mark containers as exited so the container monitor + # doesn't find stale RUNNING entries and mark pipeline FAILED. + # See issue #1294. + for ci in pe.containers: + if ( + ci.container_id in completed_container_ids + and ci.status == _pkg.ContainerStatus.RUNNING + ): + ci.status = _pkg.ContainerStatus.EXITED + # Synthetic: container will be stopped next, but 0 + # reflects successful consensus completion. + ci.exit_code = 0 + ci.exited_at = _pkg.datetime.now(_pkg.UTC) + + # Auto-withdraw any stale consensus-timeout HITL a superseded + # thread opened before this phase converged (#3315 facet c). + # Folded into this already-locked load→save so it costs no + # extra lock and rides every consensus-success path. + _withdrawn = _pkg._cancel_consensus_timeout_decisions(pip) + if _withdrawn: + _pkg.logger.info( + "Auto-withdrew stale consensus-timeout HITL decision(s) on convergence", + pipeline_id=pipeline_id, + phase=phase_str, + withdrawn=_withdrawn, + ) + + store.save_pipeline(pip) + except Exception as track_err: + _pkg.logger.warning( + "Failed to update agents to COMPLETE after consensus", + pipeline_id=pipeline_id, + error=str(track_err), + ) + + +def _record_spawned_agents_impl(executions, *, store, pipeline_id, phase_str, slice_id) -> None: + from models import ( + AgentExecution as StateAgentExecution, + ) + from models import ( + AgentExecutionStatus as StateAgentStatus, + ) + + if store is None: + return + try: + with _pkg.get_pipeline_state_lock(pipeline_id): + pip = store.load_pipeline(pipeline_id) + phase_execution = pip.get_phase_execution(_pkg.PipelinePhase(phase_str)) + for exec_info in executions: + if exec_info.container_id: + spawn_info = exec_info.container_info + if spawn_info is not None: + # Preserve backend-specific fields (pod_name, + # namespace, job_name on k8s) from the spawner + # while overriding the live bookkeeping fields. + container_info = spawn_info.model_copy( + update={ + "status": _pkg.ContainerStatus.RUNNING, + "started_at": _pkg.datetime.now(_pkg.UTC), + "agent_role": exec_info.role, + } + ) + else: + container_info = _pkg.ContainerInfo( + container_id=exec_info.container_id, + container_name=f"{pipeline_id}-{exec_info.role.value}", + status=_pkg.ContainerStatus.RUNNING, + started_at=_pkg.datetime.now(_pkg.UTC), + agent_role=exec_info.role, + ) + phase_execution.containers.append(container_info) + + agent_state = StateAgentExecution( + role=exec_info.role, + status=( + StateAgentStatus.RUNNING + if exec_info.status == StateAgentStatus.RUNNING + else StateAgentStatus.FAILED + ), + container_id=exec_info.container_id, + started_at=_pkg.datetime.now(_pkg.UTC), + slice_id=slice_id, + # Carry the per-agent resolved model through the + # reconstruction (#3174). ``_spawn_agent`` stamps this on + # the in-memory execution, but the persisted record is + # rebuilt from scratch here — without this copy the field + # dead-ends at None and both operator confirmation + # channels (get_status, list_containers), which read from + # persisted state, surface ``resolved_model: null`` for + # every concurrent-phase agent (initial spawn and + # restart_phase respawn alike). + resolved_model=exec_info.resolved_model, + ) + phase_execution.agents.append(agent_state) + store.save_pipeline(pip) + except Exception as track_err: + _pkg.logger.warning( + "Failed to record concurrent agents in pipeline state", + pipeline_id=pipeline_id, + error=str(track_err), + ) + + +def _retry_transient_spawn_failures_impl( + executions, + *, + pipeline, + executor, + agent_prompts, + spawner, + pipeline_id, + phase_str, +): + """Phase-level respawn of transiently-failed roles; returns the + updated executions list (verbatim extraction from _run_concurrent_phase).""" + try: + from concurrent_executor import _is_transient_agent_error + except ImportError: + from ..concurrent_executor import _is_transient_agent_error # type: ignore + + # Phase-level retry for transient spawn failures (#1879). Per-role + # retries in kubernetes_spawner handle short blips (~7s budget); this + # outer budget bridges longer outages like a gateway cold start by + # respawning only the failed roles while survivors wait idle. BRC can + # not start without the full cohort anyway, so leaving survivors alone + # during the retry window does not risk correctness. + phase_max_retries = getattr(pipeline.config, "phase_spawn_max_retries", 2) + phase_initial_backoff = getattr( + pipeline.config, "phase_spawn_retry_initial_backoff_seconds", 30.0 + ) + _PHASE_RETRY_BACKOFF_MULTIPLIER = 3.0 + for attempt in range(phase_max_retries): + failed = [e for e in executions if e.status.value == "failed"] + if not failed: + break + transient_failed = [e for e in failed if _is_transient_agent_error(e.error)] + if not transient_failed: + # All remaining failures are permanent — retrying would just + # burn the budget for no benefit. + break + + delay = phase_initial_backoff * (_PHASE_RETRY_BACKOFF_MULTIPLIER**attempt) + failed_roles = [e.role for e in failed] + _pkg.logger.warning( + "Phase-level spawn retry scheduled", + pipeline_id=pipeline_id, + phase=phase_str, + attempt=attempt + 1, + max_attempts=phase_max_retries, + delay_seconds=delay, + failed_roles=[r.value for r in failed_roles], + transient_roles=[e.role.value for e in transient_failed], + ) + _pkg.time.sleep(delay) + + # Clear any half-created gateway worktree state for failed roles + # so the retry sees a clean slate. Survivors' worktrees use + # different container_ids and are untouched. + for role in failed_roles: + agent_worktree_id = f"{pipeline_id}-{role.value}" + try: + spawner.gateway.delete_worktrees( + container_id=agent_worktree_id, + force=True, + ) + except Exception as clear_err: + _pkg.logger.warning( + "Failed to clear partial worktree before retry", + pipeline_id=pipeline_id, + agent_worktree_id=agent_worktree_id, + error=str(clear_err), + ) + + retry_executions = executor.spawn_specific_roles(failed_roles, agent_prompts=agent_prompts) + by_role = {e.role: e for e in retry_executions} + executions = [ + by_role.get(e.role, e) if e.status.value == "failed" else e for e in executions + ] + + still_failed = [e for e in executions if e.status.value == "failed"] + _pkg.logger.info( + "Phase-level spawn retry outcome", + pipeline_id=pipeline_id, + phase=phase_str, + attempt=attempt + 1, + recovered_roles=[ + r.value for r in failed_roles if r not in {e.role for e in still_failed} + ], + still_failed_roles=[e.role.value for e in still_failed], + ) + + return executions diff --git a/orchestrator/routes/pipelines/_run_hitl_gate.py b/orchestrator/routes/pipelines/_run_hitl_gate.py new file mode 100644 index 0000000000..e0c7e077b1 --- /dev/null +++ b/orchestrator/routes/pipelines/_run_hitl_gate.py @@ -0,0 +1,722 @@ +"""run_pipeline HITL-gate converge-before-advance loop block helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _run_hitl_gate_converge( + pipeline, + *, + current_phase, + gateway_mode, + pipeline_id, + repo_path, + spawner, + store, + worktree_repo_path, +): + """Refine/plan HITL-gate converge-before-advance loop-body block + (extracted verbatim from _run_pipeline). Returns (pipeline, action); + action=="continue" -> caller re-enters the outer while-loop.""" + if current_phase.value in _pkg._HITL_GATE_PHASES and not pipeline.config.hitl_gates: + _pkg.report_pipeline_status( + pipeline, + event_type="phase.gate_skipped", + message=( + f"{current_phase.value} phase gate skipped " + f"(hitl_gates=False) — advancing autonomously" + ), + ) + _pkg.logger.warning( + "HITL gate: refine/plan gate on an autonomous pipeline " + "(hitl_gates=False); surfacing but not blocking — advancing " + "without human approval (the converge-before-advance loop " + "requires a human, so it cannot run unattended)", + pipeline_id=pipeline_id, + phase=current_phase.value, + ) + # Decision-ledger visibility on the autonomous path (#3390): + # no human is present to resolve a backstop HITL, so mirror + # the gate-skip posture — surface a missing ledger loudly + # (event + warning) but never block. + try: + _ledger_note, _ledger_missing, _ledger_explicit_none = ( + _pkg._collect_decision_ledger_status( + worktree_repo_path, + pipeline_id, + _pkg._pipeline_identifier(pipeline.issue_number, pipeline_id), + current_phase, + ) + ) + if _ledger_missing: + _pkg.logger.warning( + "Decision ledger missing at autonomous gate skip (#3390)", + pipeline_id=pipeline_id, + phase=current_phase.value, + ) + _pkg.report_pipeline_status( + pipeline, + event_type="phase.decision_ledger_missing", + message=( + f"{current_phase.value} phase advanced autonomously " + f"with no decision ledger — {_ledger_note}" + ), + ) + elif _ledger_explicit_none is not None: + # No human is present to confirm the attestation + # (#3462) — mirror the gate-skip posture: surface + # loudly, never block. + _pkg.report_pipeline_status( + pipeline, + event_type="phase.decision_ledger_explicit_none", + message=( + f"{current_phase.value} phase advanced autonomously " + f"on an unconfirmed no-decisions attestation — " + f"{_ledger_note}" + ), + ) + except Exception as ledger_err: # noqa: BLE001 + _pkg.logger.warning( + "Decision-ledger check raised on autonomous path (continuing)", + pipeline_id=pipeline_id, + phase=current_phase.value, + error=str(ledger_err), + ) + elif current_phase.value in _pkg._HITL_GATE_PHASES: + # --- Decision-ledger backstop (#3390) --- + # Propose-time validation guarantees every refine/plan + # producer attested its ledger, so reaching this gate with + # zero registered decisions AND no explicit-none attestation + # means a path bypassed consensus (force-advance, resume) or + # the claim was lost. Never silently advance past that: + # surface a dedicated HITL whose default remedy is a phase + # re-run (the converge loop's standard corrective), with an + # explicit operator override to proceed. + _ledger_note = "" + _ledger_missing = False + _ledger_explicit_none: tuple[str, str] | None = None + try: + _ledger_note, _ledger_missing, _ledger_explicit_none = ( + _pkg._collect_decision_ledger_status( + worktree_repo_path, + pipeline_id, + _pkg._pipeline_identifier(pipeline.issue_number, pipeline_id), + current_phase, + ) + ) + except Exception as ledger_err: # noqa: BLE001 + # Never let a helper bug strand the pipeline — the + # propose-time hard gate remains the primary enforcement. + _pkg.logger.warning( + "Decision-ledger status check raised (continuing)", + pipeline_id=pipeline_id, + phase=current_phase.value, + error=str(ledger_err), + ) + + if _ledger_missing: + dq = _pkg.get_decision_queue(pipeline_id, repo_path) + _backstop = dq.queue_decision( + question=( + f"The {current_phase.value} phase reached its gate " + f"without a decision ledger (#3390). {_ledger_note}\n\n" + f"Re-running the phase lets its agents register the " + f"decisions the drafts should have surfaced (or attest " + f"an explicit empty ledger); proceeding accepts the " + f"unverified ledger and presents the normal phase gate." + ), + context=_ledger_note, + options=[ + _pkg._LEDGER_BACKSTOP_RERUN_OPTION, + _pkg._LEDGER_BACKSTOP_PROCEED_OPTION, + ], + decision_type="choice", + phase=current_phase, + ) + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + pipeline.status = _pkg.PipelineStatus.AWAITING_HUMAN + phase_execution = pipeline.get_phase_execution(current_phase) + phase_execution.status = _pkg.PipelineStatus.AWAITING_HUMAN + store.save_pipeline(pipeline) + _pkg.report_pipeline_status( + pipeline, + event_type="decision.created", + message=( + f"Decision ledger missing for {current_phase.value} " + f"phase — awaiting operator direction" + ), + ) + _pkg._emit_pipeline_event(pipeline, "decision.created") + + _backstop_resolved = dq.wait_for_decision(_backstop.id) + _backstop_resolution = str( + getattr(_backstop_resolved, "resolution", None) or "" + ).strip() + _proceed = ( + _backstop_resolved.status != _pkg.DecisionStatus.RESOLVED + or "proceed" in _backstop_resolution.lower() + ) + if not _proceed: + # Default remedy: re-run the phase so producers can + # register (or explicitly attest) the ledger. Any + # free-text resolution rides along as the directive. + _rerun_directive = ( + f"The {current_phase.value} phase reached its gate " + f"without a decision ledger: no HITL decisions were " + f"registered and no producer attested an explicit " + f"empty ledger (#3390). Review your draft for " + f"operator-grade choices; register each via " + f"`egg-contract add-decision` and cite its cq-N in " + f"the draft, or attest `no_decisions_rationale` when " + f"proposing if the phase genuinely raises none." + ) + if _backstop_resolution.lower() != (_pkg._LEDGER_BACKSTOP_RERUN_OPTION.lower()): + _rerun_directive += f"\n\nOperator note: {_backstop_resolution}" + _pkg.logger.info( + "Decision-ledger backstop: re-running phase (#3390)", + pipeline_id=pipeline_id, + phase=current_phase.value, + ) + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + pipeline.status = _pkg.PipelineStatus.RUNNING + phase_execution = pipeline.get_phase_execution(current_phase) + phase_execution.status = _pkg.PipelineStatus.RUNNING + phase_execution.completed_at = None + phase_execution.hitl_review_cycles += 1 + _alert_threshold = pipeline.config.max_hitl_review_cycles + if phase_execution.hitl_review_cycles >= _alert_threshold: + _pkg._broadcast_hitl_nonconvergence_alert( + pipeline_id, + pipeline, + current_phase, + phase_execution.hitl_review_cycles, + _alert_threshold, + ) + _pkg._perform_hitl_phase_rerun( + store=store, + spawner=spawner, + pipeline=pipeline, + phase_execution=phase_execution, + pipeline_id=pipeline_id, + current_phase=current_phase, + feedback_text=_rerun_directive, + event_message=( + f"Re-running {current_phase.value}: decision ledger missing (#3390)" + ), + ) + return pipeline, "continue" # Re-enter outer loop → re-run phase + _pkg.logger.warning( + "Decision-ledger backstop: operator chose to proceed without a ledger (#3390)", + pipeline_id=pipeline_id, + phase=current_phase.value, + resolution=_backstop_resolution[:200], + ) + elif _ledger_explicit_none is not None: + # --- Explicit-none attestation confirmation (#3462) --- + # The producer's claim that this phase raises no operator + # decisions bypasses the entire register → bridge → + # resolve chain, and is itself a judgment call the HITL + # contract assigns to the operator. Surface it as its own + # confirmable decision (see the helper): confirming records + # the operator's endorsement on the ledger note; rejecting + # re-runs the phase to register cq-N entries. + _rerun_requested, _ledger_note, pipeline = _pkg._handle_explicit_none_attestation_gate( + pipeline=pipeline, + pipeline_id=pipeline_id, + repo_path=repo_path, + current_phase=current_phase, + ledger_note=_ledger_note, + explicit_none=_ledger_explicit_none, + store=store, + spawner=spawner, + ) + if _rerun_requested: + return pipeline, "continue" # Re-enter outer loop → re-run phase + + # Check for an existing pending phase_gate decision for this + # phase. A prior agent-exit event may + # have already created one — creating a duplicate confuses the + # human reviewer. See #1152. + existing_pending_gate = any( + d.decision_type == "phase_gate" + and d.phase == current_phase + and d.status == _pkg.DecisionStatus.PENDING + for d in pipeline.decisions + ) + + if existing_pending_gate: + _pkg.logger.info( + "HITL gate: reusing existing pending phase_gate decision", + pipeline_id=pipeline_id, + phase=current_phase.value, + ) + # Find the existing decision to wait on + dq = _pkg.get_decision_queue(pipeline_id, repo_path) + decision = next( + d + for d in reversed(pipeline.decisions) + if d.decision_type == "phase_gate" + and d.phase == current_phase + and d.status == _pkg.DecisionStatus.PENDING + ) + else: + draft_content = _pkg._read_phase_draft( + worktree_repo_path, + current_phase.value, + issue_number=pipeline.issue_number, + pipeline_id=pipeline_id, + branch=pipeline.branch, + ) + phase_label = "analysis" if current_phase.value == "refine" else current_phase.value + + # Warn if draft is missing — the agent may not have written + # it to the expected path. See #1016. + if draft_content is None: + _pkg.logger.warning( + "HITL gate: draft not found on work branch", + pipeline_id=pipeline_id, + phase=current_phase.value, + worktree_path=str(worktree_repo_path), + ) + draft_content = ( + f"**Warning**: No {phase_label} draft was found on the " + f"work branch. The agent may not have written the output " + f"to the expected path." + ) + + question = ( + f"The {current_phase.value} phase has completed. " + f"Please review the {phase_label} and approve to continue, " + f"or provide feedback to request changes." + ) + # Auditability (#3390): make "N registered" vs "explicitly + # none" vs "MISSING (operator overrode)" readable at the + # gate without a get_contract round-trip. + if _ledger_note: + question += f"\n\n{_ledger_note}" + + # Lead the gate comment with the simplifier's human-focused + # companion (simplified, jargon-free) when present, and link + # the full agent draft for depth. Falls back to the full + # draft inline when no companion exists (older pipelines, + # or the companion failed to land). + human_content = _pkg._read_human_phase_draft( + worktree_repo_path, + current_phase.value, + issue_number=pipeline.issue_number, + pipeline_id=pipeline_id, + branch=pipeline.branch, + ) + gate_context = draft_content + if human_content: + full_draft_link = "" + draft_rel = _pkg._get_draft_path( + current_phase.value, + issue_number=pipeline.issue_number, + pipeline_id=pipeline_id, + ) + if pipeline.repo and pipeline.branch and draft_rel: + blob = f"https://github.com/{pipeline.repo}/blob/{pipeline.branch}" + full_draft_link = ( + f"\n\n[View the full detailed {phase_label} draft]({blob}/{draft_rel})" + ) + gate_context = f"{human_content}{full_draft_link}" + + # Detect whether the gate content changed compared to the + # previous phase_gate decision for this phase (if any). + # + # NB: this compares ``gate_context``, which leads with the + # simplifier's human-focused summary when a companion + # exists. That summary is intentionally high-level and + # lossy, so a re-refinement that materially changes the + # detailed agent draft *without* altering the summary will + # report ``content_changed=False``. The flag only feeds the + # overseer's no-op-rerun health heuristic + # (``overseer/monitor.py`` ``_check_rerun_anomaly``) — it + # never gates re-prompting — so a missed change here is at + # worst a suppressed advisory alert, not a correctness + # issue. We compare the gate content (not the full draft) + # deliberately so the heuristic tracks what the operator + # actually sees at the gate. + _content_changed: bool | None = None + _prev_gate = next( + ( + d + for d in reversed(pipeline.decisions) + if d.decision_type == "phase_gate" + and d.phase == current_phase + and d.status == _pkg.DecisionStatus.RESOLVED + ), + None, + ) + if _prev_gate is not None: + _content_changed = gate_context != _prev_gate.context + + dq = _pkg.get_decision_queue(pipeline_id, repo_path) + decision = dq.queue_decision( + question=question, + context=gate_context, + options=["approve", "request changes"], + decision_type="phase_gate", + phase=current_phase, + content_changed=_content_changed, + ) + + # Reload pipeline to pick up the decision persisted by queue_decision(), + # otherwise the stale local object overwrites it with an empty decisions list. + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + pipeline.status = _pkg.PipelineStatus.AWAITING_HUMAN + # Also mark the phase as awaiting human so the DAG visualization + # shows the HITL gate on the correct phase box. + phase_execution = pipeline.get_phase_execution(current_phase) + phase_execution.status = _pkg.PipelineStatus.AWAITING_HUMAN + store.save_pipeline(pipeline) + + # Report HITL gate to collaborator + _pkg.report_pipeline_status( + pipeline, + event_type="decision.created", + message=f"Awaiting human approval for {current_phase.value} phase", + ) + _pkg._emit_pipeline_event(pipeline, "decision.created") + + dq.wait_for_decision(decision.id) + + # Check resolution — did the human approve or request changes? + resolved_decision = dq.get_decision(decision.id) + resolution = (resolved_decision.resolution or "").strip() + + # JSON-first resolution parsing: try structured payload before + # falling back to keyword matching for legacy bare-string resolutions. + _is_approved = False + _needs_revision = False + _revision_feedback: str | None = None + + try: + payload = _pkg.json.loads(resolution) + if isinstance(payload, dict) and "action" in payload: + action = payload["action"] + feedback_text = payload.get("feedback", "") + + if action == "approve": + _is_approved = True + elif action == "select": + # Selection from a choice menu — treat as approval + _is_approved = True + elif action == "submit_feedback": + # Feedback submission — treat as approval (info collected) + _is_approved = True + elif action in ("request_changes", "change_approach"): + if feedback_text: + # R-1: Extract readable feedback, not raw JSON + _needs_revision = True + _revision_feedback = feedback_text + else: + # JSON request_changes without feedback — same as bare label + _needs_revision = True + _revision_feedback = None + else: + # Unknown action — fall through to legacy matching + raise _pkg.json.JSONDecodeError("unknown action", resolution, 0) + else: + # Valid JSON but no action field — fall through to legacy + raise _pkg.json.JSONDecodeError("no action field", resolution, 0) + except _pkg.json.JSONDecodeError, TypeError, AttributeError: + # Legacy bare-string resolution — existing keyword matching + if resolution.lower() in _pkg._APPROVE_KEYWORDS: + _is_approved = True + elif resolution.lower() in _pkg._BARE_OPTION_LABELS: + # Bare "request changes" without feedback + _needs_revision = True + _revision_feedback = None + elif resolution: + # Free-text feedback + _needs_revision = True + _revision_feedback = resolution + + # Holds the operator's resolution from the "bare request → + # asked for specifics → approve-with-context" follow-up path, + # if that path is taken. When set, it (not the original + # ``resolution``) carries any context attached to the final + # gate approval, so the convergence re-run below must thread it + # rather than the stale original resolution (#3392 review). + followup_resolution: str | None = None + + if _needs_revision and _revision_feedback is None: + # Bare request without actionable feedback — ask for specifics. + # This handles both legacy "request changes" and JSON + # {"action":"request_changes"} without feedback text. + _pkg.logger.info( + "HITL gate: bare option label without feedback, requesting specifics", + pipeline_id=pipeline_id, + phase=current_phase, + resolution=resolution, + ) + # Extract a human-friendly label from the resolution for the + # follow-up prompt (avoid displaying raw JSON to the user). + try: + _parsed = _pkg.json.loads(resolution) + display_resolution = ( + _parsed.get("action", resolution).replace("_", " ") + if isinstance(_parsed, dict) + else resolution + ) + except _pkg.json.JSONDecodeError, TypeError, AttributeError: + display_resolution = resolution + followup = dq.queue_decision( + question=( + f'You selected "{display_resolution}" but didn\'t provide specific feedback. ' + f"Please describe what changes you'd like to see in the {phase_label}, " + f"or approve to continue." + ), + context=draft_content, + options=["approve"], + decision_type="phase_gate", + phase=current_phase, + ) + dq.wait_for_decision(followup.id) + resolved_followup = dq.get_decision(followup.id) + followup_resolution = (resolved_followup.resolution or "").strip() + + # Parse follow-up resolution (also JSON-first) + try: + fp = _pkg.json.loads(followup_resolution) + if isinstance(fp, dict) and "action" in fp: + fa = fp["action"] + if fa == "approve": + _is_approved = True + _needs_revision = False + elif fa in ("request_changes", "change_approach"): + ft = fp.get("feedback", "") + if ft: + _revision_feedback = ft + else: + _is_approved = True + _needs_revision = False + else: + raise _pkg.json.JSONDecodeError("unknown", followup_resolution, 0) + else: + raise _pkg.json.JSONDecodeError("no action", followup_resolution, 0) + except _pkg.json.JSONDecodeError, TypeError, AttributeError: + if ( + followup_resolution.lower() in _pkg._APPROVE_KEYWORDS + or followup_resolution.lower() in _pkg._BARE_OPTION_LABELS + ): + _pkg.logger.info( + "HITL follow-up: no actionable feedback, treating as approval", + pipeline_id=pipeline_id, + phase=current_phase, + ) + _is_approved = True + _needs_revision = False + elif followup_resolution: + _revision_feedback = followup_resolution + + if _needs_revision and _revision_feedback: + # Human provided feedback — re-run the phase with corrections + _pkg.logger.info( + "HITL gate: changes requested, re-running phase", + pipeline_id=pipeline_id, + phase=current_phase, + feedback_preview=_revision_feedback[:200], + ) + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + pipeline.status = _pkg.PipelineStatus.RUNNING + phase_execution = pipeline.get_phase_execution(current_phase) + phase_execution.status = _pkg.PipelineStatus.RUNNING + phase_execution.completed_at = None # Reset — phase is re-running + phase_execution.hitl_review_cycles += 1 + + # No force-advance (#3392). The converge-before-advance + # loop is human-gated every round, so an unbounded loop + # cannot burn compute silently and we must never advance + # with the operator's feedback unaddressed. After the + # configured number of rounds, emit a non-fatal overseer + # alert for visibility, then always re-run. The + # ``max_hitl_review_cycles`` config is now this alert + # threshold, not a force-advance budget. + _alert_threshold = pipeline.config.max_hitl_review_cycles + if phase_execution.hitl_review_cycles >= _alert_threshold: + _pkg._broadcast_hitl_nonconvergence_alert( + pipeline_id, + pipeline, + current_phase, + phase_execution.hitl_review_cycles, + _alert_threshold, + ) + # #2795: the directive + frozen iteration summary + # accumulate across kickbacks so iteration N+1's prompts + # render them with explicit precedence prose. + _pkg._perform_hitl_phase_rerun( + store=store, + spawner=spawner, + pipeline=pipeline, + phase_execution=phase_execution, + pipeline_id=pipeline_id, + current_phase=current_phase, + feedback_text=_revision_feedback, + event_message=f"Human requested changes to {current_phase.value}", + ) + return pipeline, "continue" # Re-enter outer loop → re-run phase with feedback + + # Before advancing, surface any contract-scoped decisions / + # feedback the phase's agents registered via ``egg-contract``. + # Without this bridge, approving the phase_gate silently + # discards them (#1889). Wrapped in try/except so a bug + # here can never strand the pipeline. + _decisions_resolved_this_round = 0 + try: + _decisions_resolved_this_round = _pkg._queue_and_await_contract_decisions( + dq, + worktree_repo_path, + pipeline_id, + _pkg._pipeline_identifier(pipeline.issue_number, pipeline_id), + current_phase, + ) + except Exception as bridge_err: + _pkg.logger.warning( + "Contract decision bridge failed (continuing)", + pipeline_id=pipeline_id, + phase=current_phase.value, + error=str(bridge_err), + ) + + # Converge-before-advance (#3392): if the operator just + # resolved one or more decisions, re-run the phase so the + # documents reflect those resolutions and any decision the + # resolutions induce is surfaced in the next round. Re-asks of + # already-answered questions are suppressed by carry-forward + # (find_resolved_question), so the open-decision set shrinks + # toward a fixpoint; we advance only on a round that resolved + # nothing new. The phase gate is re-presented after the re-run. + if _decisions_resolved_this_round and current_phase.value in _pkg._HITL_GATE_PHASES: + # Preserve any operator context attached to the approve so + # the re-run's agents see it (the bridge already persisted + # the decision answers themselves; this carries the gate + # prose that would otherwise be dropped on a re-run round). + # When the operator went through the "bare request → asked + # for specifics → approve-with-context" follow-up path, the + # context lives in ``followup_resolution`` (the final + # answer), not the stale original ``resolution`` — prefer + # it so that context is not silently dropped (#3392 review). + _context_source = followup_resolution if followup_resolution is not None else resolution + _approve_context = "" + try: + _ap = _pkg.json.loads(_context_source) + if isinstance(_ap, dict): + _approve_context = (_ap.get("context") or _ap.get("feedback") or "").strip() + except _pkg.json.JSONDecodeError, TypeError, AttributeError: + _approve_context = "" + + _rerun_feedback = ( + f"The operator resolved {_decisions_resolved_this_round} HITL " + f"decision(s) for the {current_phase.value} phase. Update the " + f"{current_phase.value} document(s) to reflect the resolved " + f"decisions (read them from the contract's `decisions`), and " + f"register any new decisions the resolutions induce." + ) + if _approve_context: + _rerun_feedback += f"\n\nOperator note at the gate: {_approve_context}" + + _pkg.logger.info( + "HITL gate: decisions resolved, re-running phase to fold them in", + pipeline_id=pipeline_id, + phase=current_phase.value, + resolved_count=_decisions_resolved_this_round, + ) + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + pipeline.status = _pkg.PipelineStatus.RUNNING + phase_execution = pipeline.get_phase_execution(current_phase) + phase_execution.status = _pkg.PipelineStatus.RUNNING + phase_execution.completed_at = None + phase_execution.hitl_review_cycles += 1 + _alert_threshold = pipeline.config.max_hitl_review_cycles + if phase_execution.hitl_review_cycles >= _alert_threshold: + _pkg._broadcast_hitl_nonconvergence_alert( + pipeline_id, + pipeline, + current_phase, + phase_execution.hitl_review_cycles, + _alert_threshold, + ) + _pkg._perform_hitl_phase_rerun( + store=store, + spawner=spawner, + pipeline=pipeline, + phase_execution=phase_execution, + pipeline_id=pipeline_id, + current_phase=current_phase, + feedback_text=_rerun_feedback, + event_message=( + f"Folding {_decisions_resolved_this_round} resolved " + f"decision(s) into {current_phase.value}" + ), + ) + return pipeline, "continue" # Re-enter outer loop → re-run phase, re-surface gate + + # Approved — resume and advance + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + pipeline.status = _pkg.PipelineStatus.RUNNING + # Restore phase status to COMPLETE now that the HITL gate is cleared + phase_execution = pipeline.get_phase_execution(current_phase) + phase_execution.status = _pkg.PipelineStatus.COMPLETE + if phase_execution.completed_at is None: + phase_execution.completed_at = _pkg.datetime.now(_pkg.UTC) + store.save_pipeline(pipeline) + + # Persist phase gate resolution to contract and draft so + # next-phase agents can see the human's decisions. #1295 + _pkg._persist_phase_gate_resolution( + worktree_repo_path, + pipeline_id, + resolved_decision, + current_phase.value, + pipeline.issue_number, + ) + + # Commit and push updated statefiles (contract + draft with resolution) + try: + _pkg._commit_statefiles_to_worktree( + worktree_repo_path, + f"Persist HITL resolution after {current_phase.value} phase gate", + pipeline_identifier=_pkg._pipeline_identifier(pipeline.issue_number, pipeline_id), + pipeline_id=pipeline_id, + ) + except Exception as git_err: + # Catch broadly: see #2219. The helper raises + # ``TimeoutExpired`` and ``OSError`` paths that a + # ``CalledProcessError``-only handler did not catch. + _pkg.logger.warning( + "Failed to commit statefiles after phase gate resolution (continuing)", + pipeline_id=pipeline_id, + error=str(git_err), + ) + + if pipeline.branch and worktree_repo_path != repo_path: + try: + spawner.gateway.push_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(worktree_repo_path), + branch=pipeline.branch, + mode=gateway_mode, + base_branch=pipeline.base_branch, + ) + except Exception as push_err: + _pkg.logger.warning( + "Failed to push statefiles after phase gate resolution (continuing)", + pipeline_id=pipeline_id, + error=str(push_err), + ) + return pipeline, None diff --git a/orchestrator/routes/pipelines/_run_implement.py b/orchestrator/routes/pipelines/_run_implement.py new file mode 100644 index 0000000000..e3a9bf6254 --- /dev/null +++ b/orchestrator/routes/pipelines/_run_implement.py @@ -0,0 +1,1496 @@ +"""implement-phase slice loop, extracted verbatim from the pipelines barrel; barrel-resident/test-patched globals via ``_pkg`` (#3312 slice-4).""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _run_implement_phase_slices( + pipeline_id: str, + pipeline: _pkg.Pipeline, + spawner, + repo_volumes: dict[str, str], + gateway_mode: str, + repos: list[str], + sandbox_env: dict[str, str], + store, + certs_volume: str | None, + worktree_repo_path: _pkg.Path, + run_epoch: _pkg.datetime | None = None, +) -> tuple[int, str]: + """Drive the implement phase as a DAG of independent slices (#2137). + + For each wave produced by :class:`SliceScheduler`, spawns a fresh + BRC team per slice and waits for that slice's consensus before + advancing the scheduler. Each slice runs through the existing + :func:`_run_concurrent_phase` machinery with a slice-scoped tracker + namespace (``{pipeline_id}/{slice_id}``) and slice-scoped per-role + branches (``egg/issue-N/{slice_id}/{role}/work``). + + Per-slice PRs are opened via ``GatewayClient.create_slice_pr`` after + each slice reaches CONSENSUS_CONFIRMED — root slices target the + pipeline branch; child slices target their parent slice's + integration branch. The stacked-PR reconciler runs in parallel as a + daemon thread for the lifetime of this call. + + Returns ``(exit_code, logs)`` where ``exit_code == 0`` means every + slice reached CONFIRMED; non-zero means at least one slice failed. + """ + try: + from orchestrator.slice_scheduler import SliceScheduler + except ImportError: + from slice_scheduler import SliceScheduler + + try: + from egg_contracts.loader import load_contract, save_contract + except ImportError as exc: + _pkg.logger.error( + "Slice loop: egg_contracts.loader unavailable — falling back", + pipeline_id=pipeline_id, + error=str(exc), + ) + return 1, "slice loop bootstrap failed" + + contract = load_contract(pipeline_id, worktree_repo_path) + slices = list(getattr(contract, "slices", []) or []) + if not slices: + _pkg.logger.warning( + "Slice loop: contract has no slices, falling back to monolithic implement", + pipeline_id=pipeline_id, + ) + return 1, "no slices in contract" + + pipeline_branch = pipeline.branch or ( + f"egg/issue-{pipeline.issue_number}/work" + if pipeline.issue_number is not None + else f"egg/{pipeline_id}/work" + ) + issue_number = pipeline.issue_number + # Slice integration branches stack as siblings of the pipeline tip + # under ``egg/<id>/`` (see :func:`_ensure_pipeline_work_ref` for the + # ``/work`` namespace decision in #2399). The namespace root drops the + # trailing ``/work`` so slice paths build to ``<root>/slice-M`` rather + # than ``<root>/work/slice-M``. The qualifier suffix (``-v3``, + # ``-backend``) is preserved through ``pipeline.branch`` so two + # qualified pipelines for the same issue do not collide on + # ``egg/issue-N/slice-M`` (#2368). + issue_branch = _pkg._slice_namespace_root(pipeline_branch) + + # Wrap scheduler construction so the run loop doesn't crash if the + # contract bypassed plan-ingestion validation and reaches the + # scheduler with a multi-parent / cyclic forest. ``SliceScheduler`` + # raises ``ValueError`` with the structured forest errors; surface + # them to the operator via the existing return path so the run + # loop can route to HITL escalation rather than wedge the pipeline. + try: + scheduler = SliceScheduler( + contract, + max_parallel_slices=pipeline.config.max_parallel_slices, + ) + except ValueError as exc: + _pkg.logger.error( + "Slice loop: scheduler refused to start (forest validation failed)", + pipeline_id=pipeline_id, + error=str(exc), + ) + return 1, f"slice scheduler validation failed: {exc}" + + # Defensive idempotent context-PR opener (#2777 cq-4). The + # canonical advance_phase REST path enforces hard-required, but + # the runner-driven entries (auto-advance, implement-entry, + # HITL-resume, this slice-loop entry) must also fire it to avoid + # silent strands on ``egg/<id>/work``. Soft-fail on transient + # gateway errors here — the canonical site already enforces the + # 422 contract. + try: + # Pass the main repo path (``store.repo_path``) — not + # ``worktree_repo_path`` — so all four opener call sites of + # ``_open_context_pr_at_implement_start`` read identically. + # The opener rederives its own per-pipeline worktree internally + # via ``resolve_worktree_path(pipeline_id, store.repo_path)``. + _pkg._open_context_pr_at_implement_start(pipeline_id, repo_path=_pkg.Path(store.repo_path)) + except _pkg.ContextPrCreationError as ctx_err: + _pkg.logger.warning( + "Context PR opener: slice-loop entry safety net failed " + "(continuing — hard-require enforced at advance_phase and " + "the implement-start plan pre-flight gate) (#2777, #3100)", + pipeline_id=pipeline_id, + reason=ctx_err.reason, + error=str(ctx_err), + ) + except Exception as safety_err: # noqa: BLE001 + # Defence in depth: import / lookup failures must not strand + # the slice loop. + _pkg.logger.warning( + "Context PR opener: slice-loop entry safety net outer " + "wrapper raised (continuing) (#2777)", + pipeline_id=pipeline_id, + error=str(safety_err), + ) + + _contract_loader = _pkg.functools.partial( + _pkg._contract_loader_impl, + pipeline_id=pipeline_id, + worktree_repo_path=worktree_repo_path, + ) + + # Stacked-PR reconciler starts after the bootstrap pass below so + # an unhandled bootstrap exception cannot leak its daemon thread + # (the ``finally`` at the bottom of the run loop owns teardown). + aggregate_logs: list[str] = [] + overall_exit = 0 + poll_interval = 5.0 + + from egg_contracts.models import SliceStatus + + try: + from orchestrator import global_slice_admit + except ImportError: + import global_slice_admit # type: ignore[no-redef] + try: + from orchestrator.peer_consensus import remove_peer_consensus_tracker + except ImportError: + from peer_consensus import remove_peer_consensus_tracker # type: ignore[no-redef] + try: + from orchestrator.state_store import get_pipeline_state_lock + except ImportError: + from state_store import get_pipeline_state_lock # type: ignore[no-redef] + + _commit_and_push_slice_statefiles = _pkg.functools.partial( + _pkg._commit_and_push_slice_statefiles_impl, + pipeline_id=pipeline_id, + worktree_repo_path=worktree_repo_path, + pipeline=pipeline, + store=store, + spawner=spawner, + gateway_mode=gateway_mode, + issue_number=issue_number, + ) + _persist_slice_status_complete = _pkg.functools.partial( + _pkg._persist_slice_status_complete_impl, + pipeline_id=pipeline_id, + worktree_repo_path=worktree_repo_path, + commit_and_push=_commit_and_push_slice_statefiles, + ) + + # Bootstrap reconciliation pass (#2549). Before the run loop ticks, + # fold in two sources of "this slice is already done" state that + # the scheduler (a pure rebuild from ``contract.slices``) cannot + # see on its own: + # + # (A) Slices the contract already records as + # ``SliceStatus.COMPLETE`` — trusted directly, no I/O. + # (B) Slices whose integration branch on origin is reachable from + # their parent's tip (PR merged). On a hit, also persist (A) + # so subsequent restarts skip the GitHub round-trip. + # + # Without this pass the scheduler would re-yield merged slices as + # READY and ``create_slice_integration_branch`` would + # non-fast-forward-reject. Best-effort: failure falls through to + # the run loop. + bootstrap_complete: list[str] = [] + bootstrap_merged: list[str] = [] + + # Layer (A): cheap, no I/O. Trust contract-recorded COMPLETE status — + # but verify the recorded COMPLETE is not itself a #3214 false-complete + # (an interior forest node persisted COMPLETE with pending tasks, no + # PR, no merge). Blindly trusting a corrupt contract here is how the + # false-complete propagated into the scheduler and wedged the chain. + # On an invalid record, alert and decline to trust it — route the + # slice through Layer-B/C so it is re-evaluated and (re-)run rather + # than silently skipped. + # + # Note: a COMPLETE slice that recorded *no* durable evidence (no + # pr_number, no integration_base_sha — e.g. a legacy pre-#2871 + # contract from before integration_base_sha existed) is distrusted + # here on every restart, even when it was genuinely merged. That is + # intentional, not a bug: such a slice falls through to Layer-B, + # where origin-side merge detection re-confirms it and re-marks it + # COMPLETE. The outcome stays correct; the only cost is one extra + # GitHub round-trip per restart. A slice that forked under current + # code *usually* records integration_base_sha and is trusted here + # directly — but that write is best-effort (the get_remote_branch_sha + # call at slice spawn swallows failures and degrades to ancestor-only + # detection), so a current-code slice whose base-SHA write failed also + # falls through to Layer-B and self-corrects identically to the legacy + # case above. + # + # Known limitation (#3253): a slice that pre-fix code *already* persisted + # COMPLETE basis="merged" with a stale ``integration_base_sha`` and no + # produced commits / PR is still trusted here — Layer-A validates with no + # ``basis`` (the #3253 merged-empty guard keys on ``basis == "merged"``, + # which Layer-A never supplies), so the ``forked`` free-pass below accepts + # the stale fork base. This is deliberately *not* fixed by broadening the + # guard to the basis-less path: a legitimate ``basis="consensus_complete"`` + # slice can also have no PR and no recorded task commit (best-effort agent + # recording + ``pr_number`` None on an unparseable PR URL, #3122), so + # re-running on "no commit + no PR" alone here would re-run genuinely + # completed work. The #3253 fix prevents the corrupt write going forward; + # a pipeline already wedged by this exact bug *before* the upgrade needs a + # manual contract touch-up (clear the slice's COMPLETE status) rather than + # self-healing on restart. + layer_b_candidates = [] + for s in slices: + if s.status == SliceStatus.COMPLETE: + invalid = _pkg._validate_slice_completion_basis(s, pr_number=s.pr_number) + if invalid is not None: + _pkg.logger.error( + "Contract records slice COMPLETE but the completion basis is " + "invalid — NOT trusting it; re-evaluating the slice (#3214)", + pipeline_id=pipeline_id, + slice_id=s.id, + reason=invalid, + ) + layer_b_candidates.append(s) + continue + scheduler.record_complete(s.id) + bootstrap_complete.append(s.id) + continue + layer_b_candidates.append(s) + + # Layer (B): origin-side detection for slices not yet recorded as + # COMPLETE on the contract. Each helper call uses its own synthetic + # gateway session, so we parallelise across slices to keep startup + # latency bounded as forests grow. Cap workers so a large forest + # doesn't burst against the gateway. + if pipeline.repo and layer_b_candidates: + + def _bootstrap_check_one(slice_obj: _pkg.Any) -> tuple[str, bool]: + # Prefer the parent branch the slice was actually forked + # off of (recorded by ``_run_one_slice_inner``). Falls back + # to the dependency-derived parent for slices that never + # made it through ``_run_one_slice_inner`` (e.g. fresh + # contract on first run). Both should agree today, but a + # future re-plan that mutates ``dependencies`` post-creation + # would diverge — preferring the recorded value future- + # proofs the check. + if slice_obj.parent_branch_at_creation: + parent_branch_for_check = slice_obj.parent_branch_at_creation + elif slice_obj.dependencies: + parent_branch_for_check = f"{issue_branch}/{slice_obj.dependencies[0]}" + else: + parent_branch_for_check = pipeline_branch + integration_branch_for_check = f"{issue_branch}/{slice_obj.id}" + try: + merged = spawner.gateway.is_slice_branch_merged_into_parent( + pipeline_id, + str(worktree_repo_path), + integration_branch=integration_branch_for_check, + parent_branch=parent_branch_for_check, + # #2871 — pass the recorded fork base so an empty + # (un-started) slice branch whose tip is still at its + # creation base is not mistaken for merged work. + integration_base_sha=slice_obj.integration_base_sha, + # Read-only ancestry check run by the orchestrator's + # slice-loop scheduler; attribute to the orchestrator + # in the audit log, not a phantom coder (#2919). + agent_role="orchestrator", + mode=gateway_mode, # type: ignore[arg-type] + ) + except Exception as detect_err: # noqa: BLE001 + # Gateway `is_slice_branch_merged_into_parent` call. + # Catches gateway HTTP/timeout errors (GatewayError), + # low-level socket / DNS errors (OSError), and any + # rare argument-shape errors. Default to "not merged" + # so the slice can still spawn fresh. + _pkg.logger.warning( + "Bootstrap merged-detection raised; treating slice as not-merged", + pipeline_id=pipeline_id, + slice_id=slice_obj.id, + error=str(detect_err), + ) + return slice_obj.id, False + # #3253 — guard against a false-positive merged result. A slice + # whose producers never committed (no produced task commit) and + # that has no slice PR has an empty integration branch: its tip + # is still the fork base, so it is trivially an ancestor of an + # advanced parent and the origin ancestry check reports it + # merged. Marking it COMPLETE basis=merged silently drops the + # slice and lets the pipeline complete with its work missing — + # the restart-to-retry failure mode (#3138 producer exhaustion → + # operator restart → false-complete). Override to not-merged so + # the slice falls through to Layer-C and re-runs. A genuine merge + # has produced commits or a recorded PR, so this never overrides + # a real merge. + if ( + merged + and not _pkg._slice_produced_commits(slice_obj) + and getattr(slice_obj, "pr_number", None) is None + ): + _pkg.logger.warning( + "Bootstrap merged-detection overridden: origin ancestry " + "reports merged but the slice has no produced task commit " + "and no PR — empty/un-started branch, re-running rather than " + "false-completing as merged (#3253)", + pipeline_id=pipeline_id, + slice_id=slice_obj.id, + ) + return slice_obj.id, False + return slice_obj.id, bool(merged) + + max_workers = min(len(layer_b_candidates), 8) + with _pkg.concurrent.futures.ThreadPoolExecutor( + max_workers=max_workers, + thread_name_prefix=f"slice-bootstrap-{pipeline_id}", + ) as bootstrap_pool: + results = list(bootstrap_pool.map(_bootstrap_check_one, layer_b_candidates)) + + for slice_id, already_merged in results: + if already_merged: + scheduler.record_complete(slice_id) + _persist_slice_status_complete(slice_id, basis="merged", commit_to_branch=False) + bootstrap_merged.append(slice_id) + + # Layer (C): non-COMPLETE slice classification (slice-4 TASK-4-4). + # After layers A (contract-recorded COMPLETE) and B (merged on + # origin), classify the remaining slices per the 5-way matrix + # so crash recovery does not respawn agents for a slice that is + # already running, silently advance a slice whose HITL is still + # pending, or treat a corrupt status enum as a benign default: + # + # (1) IN_PROGRESS, no commits on integration branch → no Layer-C + # action; the scheduler will re-yield the slice as READY and + # the run loop spawns fresh agents. + # (2) IN_PROGRESS, commits on integration branch, consensus + # NOT reached → call ``scheduler.mark_spawned`` so the run + # loop does NOT respawn. Per-slice tracker reconstruction + # is handled at orchestrator boot by + # startup_reconciliation.py (slice-4 TASK-4-5); the + # producer pods (if alive) or the lazy spawn-on-need path + # carry the slice forward. + # (3) IN_PROGRESS, commits on integration branch, consensus + # REACHED, slice PR NOT opened → mark COMPLETE so the + # slice-PR opener path (with TASK-3-2 idempotency + # pre-flight) fires on the next loop iteration; do not + # respawn agents. + # (4) BLOCKED (HITL pending) → preserve the BLOCKED status. + # Verify the HITL decision is still on the contract; if + # not, surface an OVERSEER_ALERT so a human investigates. + # (5) Unknown / corrupt state (impossible status enum value) + # → surface an OVERSEER_ALERT instead of silently + # re-yielding as READY. + bootstrap_resumed: list[str] = [] + bootstrap_consensus_complete: list[str] = [] + bootstrap_blocked: list[str] = [] + bootstrap_corrupt: list[str] = [] + bootstrap_reclassified_fresh: list[str] = [] # resume-but-dead → fresh (#2914) + layer_b_marked_complete = set(bootstrap_merged) + for s in layer_b_candidates: + if s.id in layer_b_marked_complete: + continue + classification = _pkg._classify_non_complete_slice( + pipeline_id=pipeline_id, + slice_obj=s, + issue_branch=issue_branch, + pipeline_repo=pipeline.repo, + worktree_repo_path=worktree_repo_path, + gateway=spawner.gateway, + gateway_mode=gateway_mode, + consensus_tracker_lookup=_pkg._lookup_peer_consensus_tracker_or_none, + ) + if classification == "consensus_complete": + # Case 3 — louder than fresh-spawn but quieter than + # case-4/5 HITL. A warning here makes the non-trivial + # recovery (consensus reached pre-crash, PR not opened) + # auditable in operator logs without paging anyone + # (reviewer_code v1 non-blocking). + _pkg.logger.warning( + "Layer-C case 3 — slice consensus reached pre-restart but " + "slice PR was never opened; marking COMPLETE so the next " + "loop iteration runs the slice-PR opener (slice-4 TASK-4-4)", + pipeline_id=pipeline_id, + slice_id=s.id, + ) + scheduler.record_complete(s.id) + _persist_slice_status_complete(s.id, basis="consensus_complete", commit_to_branch=False) + bootstrap_consensus_complete.append(s.id) + continue + if classification == "resume": + # Verify agents are actually live before marking as spawned (#2914). + # On restart_phase, agents were torn down but contract still shows + # IN_PROGRESS with commits — we must not mark_spawned when cohort + # is absent, or the pipeline wedges with no agents running. + if _pkg._slice_agents_alive(spawner, pipeline_id, s.id): + scheduler.mark_spawned(s.id) + bootstrap_resumed.append(s.id) + else: + _pkg.logger.warning( + "Layer-C resume classification but no live agents; " + "treating as fresh to force re-spawn (#2914)", + pipeline_id=pipeline_id, + slice_id=s.id, + ) + bootstrap_reclassified_fresh.append(s.id) + continue + if classification == "blocked": + bootstrap_blocked.append(s.id) + continue + if classification == "corrupt": + bootstrap_corrupt.append(s.id) + continue + # "fresh" → no Layer-C action, scheduler re-yields READY. + + # The bootstrap passes above persist with ``commit_to_branch=False`` + # — one batched commit+push here covers every reconciled slice + # (Layer B merged-detection + Layer-C case 3) instead of a commit + # per slice (#3117). + if bootstrap_merged or bootstrap_consensus_complete: + _commit_and_push_slice_statefiles( + "Persist slice completion statuses after bootstrap reconciliation (#3117)" + ) + + if bootstrap_complete or bootstrap_merged: + _pkg.logger.info( + "Slice bootstrap reconciliation marked slices complete", + pipeline_id=pipeline_id, + already_complete_on_contract=bootstrap_complete, + detected_merged_on_origin=bootstrap_merged, + ) + if ( + bootstrap_resumed + or bootstrap_consensus_complete + or bootstrap_blocked + or bootstrap_corrupt + or bootstrap_reclassified_fresh + ): + # NOTE: include ``bootstrap_blocked`` in the gate (reviewer_code + # v3 NACK fix) — a bootstrap pass whose only Layer-C activity is + # BLOCKED slices was previously suppressing the audit-trail line + # entirely. Case-4 escalation still fires, but operators need + # the structured "we saw a blocked slice" log to spot + # pending-HITL backlogs without grepping for the side-effect. + # + # Also include ``bootstrap_reclassified_fresh`` (#2914) — resume- + # classified slices that were re-verified against k8s and found + # to have no live agents. Surfacing the reclassification here + # gives operators a structured audit trail for the + # ``restart_phase``-recovery path. + _pkg.logger.info( + "Slice bootstrap reconciliation classified non-COMPLETE slices (slice-4 TASK-4-4)", + pipeline_id=pipeline_id, + resumed=bootstrap_resumed, + consensus_complete_unrecorded=bootstrap_consensus_complete, + blocked=bootstrap_blocked, + corrupt=bootstrap_corrupt, + reclassified_fresh=bootstrap_reclassified_fresh, + ) + # Case 5 — escalate via HITL so the pipeline pauses until the + # operator picks an option (reviewer_contract / reviewer_code v1 + # blocker). OVERSEER_ALERT alone is too weak — it surfaces but + # does not gate progress. The Decision lands on the contract via + # ``_escalate_corrupt_slice_to_hitl`` so ``/sdlc`` reads it on + # the next poll. + _current_phase = getattr(pipeline, "current_phase", None) + for _corrupt_slice_id in bootstrap_corrupt: + try: + _pkg._escalate_corrupt_slice_to_hitl( + pipeline_id=pipeline_id, + slice_id=_corrupt_slice_id, + worktree_repo_path=worktree_repo_path, + current_phase=_current_phase, + ) + except Exception as escalate_err: # noqa: BLE001 + _pkg.logger.warning( + "Failed to escalate corrupt-state slice to HITL during " + "bootstrap (slice-4 TASK-4-4 case 5)", + pipeline_id=pipeline_id, + slice_id=_corrupt_slice_id, + error=str(escalate_err), + ) + # Case 4 — symmetric HITL escalation for BLOCKED-without-HITL. + for _blocked_slice_id, _escalate_reason in [ + (sid, "no pending HITL decision found on contract") + for sid in bootstrap_blocked + if not _pkg._slice_has_pending_decision(sid, getattr(contract, "decisions", None) or []) + ]: + try: + _pkg._escalate_blocked_slice_to_hitl( + pipeline_id=pipeline_id, + slice_id=_blocked_slice_id, + reason=_escalate_reason, + worktree_repo_path=worktree_repo_path, + current_phase=_current_phase, + ) + except Exception as escalate_err: # noqa: BLE001 + _pkg.logger.warning( + "Failed to escalate blocked-without-HITL slice to HITL " + "during bootstrap (slice-4 TASK-4-4 case 4)", + pipeline_id=pipeline_id, + slice_id=_blocked_slice_id, + error=str(escalate_err), + ) + + reconciler_thread, reconciler_stop = _pkg._start_stacked_pr_reconciler( + pipeline_id, + _contract_loader, + spawner.gateway, + pipeline, + worktree_repo_path=worktree_repo_path, + repo=getattr(pipeline, "repo", None), + ) + + try: + while not scheduler.all_done(): + # 1. Snapshot ready slices for this tick. + ready_batch = list(scheduler.iter_ready()) + if not ready_batch: + # 2. Drain cascades whose grace window expired so the + # descendants are visibly BLOCKED in the runtime view + # and we don't busy-spin. + events = scheduler.poll_cascades() + for event in events: + _pkg.logger.warning( + "Slice cascade fired", + pipeline_id=pipeline_id, + failed_slice=event.failed_slice_id, + blocked=event.blocked_subtree, + ) + try: + from orchestrator.gateway_client import ( + get_gateway_client as _get_gateway_client, + ) + + _ = _get_gateway_client # noqa: F841 — kept for symmetry + except ImportError: + # Symmetry-only import; the module not being + # available means the cascade alert path can't + # call the gateway, but the warning above is + # the always-on fallback. + pass + if scheduler.all_done(): + break + _pkg.time.sleep(poll_interval) + continue + + # Run every ready slice in this wave in parallel + # (#2137 TASK-4-4 + decision-5: unbounded). The + # ``max_parallel_slices`` cap from ``iter_ready`` already + # bounds ``ready_batch`` so the executor's worker pool + # mirrors that cap. Each slice runs through the existing + # ``_run_concurrent_phase`` machinery in its own thread. + # Per-slice failure / completion events are recorded back + # on the scheduler from inside ``_run_one_slice`` so the + # cascade machinery sees the same wall-clock as the run + # loop. + + def _run_one_slice(slice_id: str, parent_slice_id: str | None) -> tuple[int, str]: + # Release the global-admission slot when the slice + # exits, regardless of how (consensus, failure, raised + # exception). Idempotent — safe even if a future + # codepath calls release() somewhere else (#2241 gap 1). + try: + return _run_one_slice_inner(slice_id, parent_slice_id) + finally: + global_slice_admit.release(pipeline_id, slice_id) + + def _run_one_slice_inner( + slice_id: str, + parent_slice_id: str | None, # noqa: ARG001 — kept for caller compat; resolver reads contract + ) -> tuple[int, str]: + # Resolve parent branch for stacking via + # :func:`_resolve_slice_base_branch` (#2777, cq-2 / cq-4 / + # cq-9 / cq-10). The helper handles both: + # + # * eager-persisted ``parent_branch_at_creation`` (the + # primary path post-slice-4 TASK-4-2), and + # * fresh-pipeline derivation from + # ``slice.dependencies[0]`` (the path #2777's slice-2 + # takes before slice-4 lands). + # + # The legacy ``egg/<id>/context`` branch was removed in + # cq-4 so slice-1 (the root) now stacks on + # ``pipeline_branch`` like every other root slice — the + # work-branch context PR's diff already encompasses the + # slice-1 integration branch via ancestry. + # #2928: wire a parent-branch-existence probe so the + # resolver can tell a FRESH non-root slice (whose + # dependency parent branch is still on origin → stack + # on it) apart from an orphaned one (parent merged + # into ``work`` and cascade-deleted → base on + # ``pipeline_branch``). This replaces the pre-#2928 + # merge-base probe, which probed the slice's OWN + # integration branch — non-existent on a first run — + # and so mis-routed every fresh non-root slice onto + # ``work`` whenever ``work`` had advanced ahead of the + # parent. Repoless test scaffolds short-circuit to + # ``True`` (no origin to check; the derived parent is + # the correct DAG target), mirroring the resolver's + # conservative "assume parent exists" default. + # + # IMPORTANT: this wrapper calls the STRICT ls-remote + # variant (``ls_remote_branch_strict``) so a gateway / + # network / policy failure RAISES into the resolver's + # ``try/except`` instead of being collapsed to + # ``False``. The lenient ``ls_remote_branch`` / + # ``get_remote_branch_sha`` helpers swallow all + # exceptions and return ``False`` / ``None`` for both + # "branch absent" AND "gateway error" — using either + # here would silently route a real slice onto + # ``pipeline_branch`` on a flaky gateway, re-creating + # the #2928 wedge that this PR claims to fix. + def _probe_parent_branch_exists(parent_branch: str) -> bool: + if not pipeline.repo: + return True + return spawner.gateway.ls_remote_branch_strict( + pipeline_id, + str(worktree_repo_path), + f"refs/heads/{parent_branch}", + mode=gateway_mode, # type: ignore[arg-type] + ) + + parent_branch = _pkg._resolve_slice_base_branch( + contract, + slice_id, + pipeline_id=pipeline_id, + pipeline_branch=pipeline_branch, + parent_branch_exists=_probe_parent_branch_exists, + ) + integration_branch = f"{issue_branch}/{slice_id}" + + # Persist the parent-branch reference on the contract + # under the per-pipeline state lock so a concurrent + # tester / documenter contract write doesn't race with + # ours (reviewer_code v4 #5). While we hold the contract, + # also read back any integration_base_sha recorded on a + # prior run (#2871) — on a restart this lets the race + # check below tell an empty branch apart from a merged + # one. It is ``None`` on a slice's first run (recorded + # only after the branch is created, just below). + recorded_base_sha: str | None = None + # #3253 — capture whether the slice has any produced task + # commit / PR while we hold the contract, so the race-merged + # skip below cannot mistake an empty / un-started branch for + # a merged one (see the merged-acceptance guard below). + slice_produced_work = False + try: + with get_pipeline_state_lock(pipeline_id): + contract_local = load_contract(pipeline_id, worktree_repo_path) + for s in contract_local.slices: + if s.id == slice_id: + s.parent_branch_at_creation = parent_branch + # Slice-4 TASK-4-2: flip PENDING → + # IN_PROGRESS in the SAME contract write + # that persists parent_branch_at_creation + # (cq-9). Crash recovery (TASK-4-4 + # Layer C) now has a single signal to + # distinguish a fresh slice from one + # whose run was interrupted between + # status flip and branch creation. + # Idempotent on re-entry (e.g. orphan + # reconciler): only PENDING is flipped; + # COMPLETE / BLOCKED / IN_PROGRESS are + # left untouched. + if s.status == SliceStatus.PENDING: + s.status = SliceStatus.IN_PROGRESS + recorded_base_sha = s.integration_base_sha + slice_produced_work = ( + _pkg._slice_produced_commits(s) or s.pr_number is not None + ) + break + save_contract(contract_local, worktree_repo_path) + except Exception as save_err: # noqa: BLE001 + # Contract load/save under per-pipeline state lock. + # Same exception surface as the COMPLETE-persist + # site above (loader validation, atomic-rename + # I/O, pydantic re-serialisation). Best-effort. + _pkg.logger.warning( + "Failed to persist parent_branch_at_creation", + pipeline_id=pipeline_id, + slice_id=slice_id, + error=str(save_err), + ) + + # Race protection: a slice's PR can be merged between + # bootstrap reconciliation and this spawn. Detect and + # skip to COMPLETE so the create-branch push below + # doesn't non-fast-forward (#2549). + if pipeline.repo: + try: + already_merged = spawner.gateway.is_slice_branch_merged_into_parent( + pipeline_id, + str(worktree_repo_path), + integration_branch=integration_branch, + parent_branch=parent_branch, + integration_base_sha=recorded_base_sha, + # Read-only ancestry check run by the + # orchestrator's slice-loop scheduler; attribute + # to the orchestrator, not a phantom coder (#2919). + agent_role="orchestrator", + mode=gateway_mode, # type: ignore[arg-type] + ) + except Exception as detect_err: # noqa: BLE001 + # Same `is_slice_branch_merged_into_parent` + # surface as the bootstrap pass above + # (GatewayError + OSError). Default to "not + # merged" so the slice can still spawn. + _pkg.logger.warning( + "Slice merged-detection raised; treating as not-merged", + pipeline_id=pipeline_id, + slice_id=slice_id, + error=str(detect_err), + ) + already_merged = False + # #3253 — a slice with no produced task commit and no PR + # has an empty integration branch (tip still at the fork + # base); origin ancestry reports it merged because that + # base is trivially an ancestor of an advanced parent. + # Don't skip it as merged — spawn so it actually runs. + if already_merged and not slice_produced_work: + _pkg.logger.info( + "Slice merged-detection ignored: no produced task commit " + "and no PR — empty/un-started branch, spawning instead of " + "skipping as merged (#3253)", + pipeline_id=pipeline_id, + slice_id=slice_id, + integration_branch=integration_branch, + parent_branch=parent_branch, + ) + already_merged = False + if already_merged: + _pkg.logger.info( + "Slice already merged into parent on origin — skipping spawn (#2549)", + pipeline_id=pipeline_id, + slice_id=slice_id, + integration_branch=integration_branch, + parent_branch=parent_branch, + ) + scheduler.record_complete(slice_id) + _persist_slice_status_complete(slice_id, basis="merged") + try: + remove_peer_consensus_tracker(pipeline_id, slice_id) + except Exception: # noqa: BLE001 + # In-memory dict pop under a lock; only + # programming errors (KeyError, AttributeError) + # could fire. Bare-except keeps the slice + # COMPLETE/return path crash-proof. + pass + return 0, ( + f"slice {slice_id}: already merged into " + f"{parent_branch} on origin — skipped" + ) + + # #2137 TASK-4-2: create the slice integration branch + # on origin BEFORE spawning containers. Push + # ``parent_branch:refs/heads/integration_branch`` + # through the existing per-agent push allowlist. Agents + # then push their commits directly to the slice's + # integration branch (``egg/issue-N/slice-M``) so the + # slice PR's diff is non-empty when ``gh pr create`` + # runs. On failure, mark the slice failed so the + # cascade machinery can surface the missing-parent + # error to the operator instead of silently spawning + # agents that would push to a missing parent. + if pipeline.repo: + try: + # #3185 — the helper now returns the fork-base + # SHA it pushed the integration branch at (the + # parent tip resolved inside the call), or None + # on failure. Recording that SHA directly here + # replaces a prior best-effort + # ``get_remote_branch_sha`` re-fetch that could + # silently fail (no ``retry_transient``) and + # leave ``integration_base_sha`` unset — arming + # the empty-pre-created-branch trap on the next + # restart. + created_base_sha = spawner.gateway.create_slice_integration_branch( + pipeline_id, + str(worktree_repo_path), + integration_branch=integration_branch, + parent_branch=parent_branch, + # #2947 — hand the slice's recorded fork + # base to the gateway so a crash/restart + # over a branch that already carries this + # slice's commits (with an additively + # advanced parent) resumes in place + # instead of non-fast-forward-failing. + integration_base_sha=recorded_base_sha, + # Orchestrator pre-creates the slice + # integration branch on a synthetic session + # before agents spawn; attribute to the + # orchestrator, not a phantom coder (#2919). + # The push rides the slice-integration + # exemption (synthetic + branch shape), not a + # role gate. + agent_role="orchestrator", + mode=gateway_mode, # type: ignore[arg-type] + ) + except Exception as branch_err: # noqa: BLE001 + # Gateway `create_slice_integration_branch` + # call. Catches GatewayError (HTTP/timeout) + # and OSError (DNS / socket). Treat as failure + # so the cascade machinery surfaces a + # missing-parent error. + _pkg.logger.error( + "Slice integration branch creation raised", + pipeline_id=pipeline_id, + slice_id=slice_id, + error=str(branch_err), + ) + created_base_sha = None + if created_base_sha is None: + _pkg.logger.error( + "Slice integration branch creation failed; " + "marking slice failed (agents not spawned)", + pipeline_id=pipeline_id, + slice_id=slice_id, + parent_branch=parent_branch, + integration_branch=integration_branch, + ) + scheduler.record_failure(slice_id) + return 1, ( + f"slice {slice_id}: integration branch " + f"{integration_branch} could not be created from " + f"{parent_branch}" + ) + + # #2871 / #3185 — record the integration branch's fork + # base exactly once, on first creation. The branch was + # just pushed at the parent's tip and no agent has been + # spawned yet, so its origin tip still equals its base. + # Persisting it now lets a later restart's bootstrap + # reconciliation (and the race check above) tell an + # *empty* slice branch — tip still at this base, hence + # a trivial ancestor of an advanced parent — apart from + # a genuinely *merged* one whose tip moved past it. We + # only write it when unset so a restart over a branch + # that already carries slice commits (#2512 recovery) + # keeps its original base rather than the advanced tip. + # ``created_base_sha`` is the SHA the create call + # returned (no extra round-trip); it is an empty string + # on the unreachable no-op path + # (``integration_branch == parent_branch``), which we + # skip here. + if recorded_base_sha is None and created_base_sha: + try: + with get_pipeline_state_lock(pipeline_id): + contract_local = load_contract(pipeline_id, worktree_repo_path) + for s in contract_local.slices: + if s.id == slice_id: + s.integration_base_sha = created_base_sha + break + save_contract(contract_local, worktree_repo_path) + recorded_base_sha = created_base_sha + except Exception as base_err: # noqa: BLE001 + # Contract load/save under per-pipeline state + # lock. Catches loader validation, atomic- + # rename I/O, and pydantic re-serialisation + # errors. Best-effort: the fork base is no + # longer a round-trip failure (the SHA came + # from the create call itself), so this now + # only fires on a contract-write failure — a + # transient the next run repairs on the same + # create path. + _pkg.logger.warning( + "Failed to persist slice integration_base_sha " + "(#2871); a future restart re-records it on the " + "create path", + pipeline_id=pipeline_id, + slice_id=slice_id, + integration_branch=integration_branch, + error=str(base_err), + ) + + _pkg.logger.info( + "Slice spawn", + pipeline_id=pipeline_id, + slice_id=slice_id, + parent_branch=parent_branch, + integration_branch=integration_branch, + ) + + exit_code_inner, logs_inner = _pkg._run_concurrent_phase_with_impasse_retry( + pipeline_id=pipeline_id, + pipeline=pipeline, + phase="implement", + spawner=spawner, + repo_volumes=repo_volumes, + gateway_mode=gateway_mode, + repos=repos, + sandbox_env=sandbox_env, + store=store, + certs_volume=certs_volume, + worktree_repo_path=worktree_repo_path, + slice_id=slice_id, + run_epoch=run_epoch, + ) + + if exit_code_inner != 0: + scheduler.record_failure(slice_id) + _pkg.logger.warning( + "Slice failed", + pipeline_id=pipeline_id, + slice_id=slice_id, + exit_code=exit_code_inner, + ) + return exit_code_inner, logs_inner + + # Slice consensus reached — load the contract ONCE + # under the per-pipeline state lock and reuse the same + # snapshot for the #3125 evidence-reachability gate + # AND the slice's PR data snapshot below. Both readers + # previously took the lock independently; collapsing + # them eliminates one file read + lock acquire per + # slice close (#3125 review). + # + # The slice_pr_data block below originally documented + # the lock as covering only the contract read so the + # gateway HTTP round-trip wouldn't serialise other + # writers — the same posture applies here: we release + # the lock before the gateway call inside the gate. + contract_post: _pkg.Any | None = None + try: + with get_pipeline_state_lock(pipeline_id): + contract_post = load_contract(pipeline_id, worktree_repo_path) + except Exception as load_err: # noqa: BLE001 + _pkg.logger.warning( + "Slice close: contract load failed (continuing) (#3125)", + pipeline_id=pipeline_id, + slice_id=slice_id, + error=str(load_err), + ) + + # #3125 — evidence-reachability gate: every commit SHA + # cited by this slice's contract task records must be + # an ancestor of the integration branch tip, or the + # slice PR would ship without a deliverable the task + # record claims is done (the post-confirmation + # ``complete-task --commit`` unblock flow, #3124). + # Fails the slice BEFORE any close side effect so the + # cascade + HITL machinery surfaces the gap loudly. + # ``contract_post`` may be None if the load above + # raised — the gate falls back to its own load in that + # case (and skips gracefully if that fails too). + if pipeline.repo: + evidence_failure = _pkg._check_slice_evidence_reachability( + pipeline_id, + spawner, + worktree_repo_path, + slice_id, + integration_branch, + gateway_mode=gateway_mode, # type: ignore[arg-type] + contract=contract_post, + ) + if evidence_failure is not None: + scheduler.record_failure(slice_id) + return 1, evidence_failure + + # #3398 — per-slice green gate: execute the repo's + # configured checks (repositories.yaml, via + # get_repo_checks) against the integration-branch tip + # in a sandboxed one-shot runner, and refuse to open + # the slice PR while any check is red. Closes the + # trust-vs-verify gap in the propose-time + # checks_passed self-report. Same posture as the + # evidence gate above: fail-open on infra errors, + # fail-closed only on a definitive red verdict; + # EGG_SLICE_GREEN_GATE is the operator switch + # (off during rollout / log / on). + if pipeline.repo: + try: + import slice_green_gate as _green_gate + except ImportError: + from .. import slice_green_gate as _green_gate # type: ignore[no-redef] + + green_gate_failure = _green_gate.run_slice_green_gate( + pipeline_id, + spawner, + slice_id, + integration_branch, + pipeline.repo, + gateway_mode=gateway_mode, # type: ignore[arg-type] + ) + if green_gate_failure is not None: + scheduler.record_failure(slice_id) + return 1, green_gate_failure + + # Snapshot the slice's PR data from the same loaded + # contract — no second lock acquire, no second file + # read. + slice_pr_data: dict[str, _pkg.Any] | None = None + try: + if contract_post is not None: + slice_obj = next( + (s for s in contract_post.slices if s.id == slice_id), + None, + ) + if slice_obj is not None and pipeline.repo: + # #2538: every slice carries the + # planner-authored narrative on its PR so + # reviewers see context on whichever slice + # they open first. Pre-#2777 cq-6 the + # terminal slice additionally carried a + # program-level rollup (test plan + manual + # steps + pre-merge obligations) and a + # ``[merge-gate]`` title marker. Under cq-4 + # the merge gate is the up-front context + # PR (``egg/<id>/work → main``) opened by + # ``_open_context_pr_at_implement_start``, + # so every slice PR — terminal or not — + # now uses the same lean shape and the + # terminal-slice computation is gone. + program_pr = contract_post.pr + # #2745: derive 1-based slice position + + # total slice count from declared contract + # order so the slice PR title can carry + # ``[slice-N/M]``. + slice_count = len(contract_post.slices) + slice_index_lookup = next( + ( + i + 1 + for i, s in enumerate(contract_post.slices) + if s.id == slice_id + ), + None, + ) + # Union of ``task.files_affected`` across the + # slice's tasks; rendered under + # ``## This slice`` so reviewers see what + # this slice actually touches without + # opening the diff (#2745). + slice_files_affected_list: list[str] = [] + seen_paths: set[str] = set() + for t in slice_obj.tasks or []: + for path in t.files_affected or []: + if path and path not in seen_paths: + seen_paths.add(path) + slice_files_affected_list.append(path) + # #3393 slice-4 / task-4-1: route this slice's + # PR to its OWN repo (``resolve_slice_repo`` → + # ``slice.repo`` else the pipeline primary) and + # gather CROSS-repo coordination references for + # the PR body. Same-repo relationships are left + # to ``## Stack``, so for an N=1 pipeline + # ``slice_repo`` is the single repo and both + # ref sets are empty — behaviour is unchanged. + try: + from models import ( # type: ignore[no-redef] + resolve_slice_repo, + ) + except ImportError: + from ..models import ( # type: ignore[no-redef] + resolve_slice_repo, + ) + slice_repo = resolve_slice_repo(slice_obj, pipeline) or pipeline.repo + sibling_pr_refs: list[dict[str, _pkg.Any]] = [] + for other in contract_post.slices: + if other.id == slice_id: + continue + other_repo = resolve_slice_repo(other, pipeline) or pipeline.repo + if other_repo and other_repo != slice_repo and other.pr_number: + sibling_pr_refs.append( + {"repo": other_repo, "number": other.pr_number} + ) + # Dependent-slice upstream PR — surfaced only + # when the upstream slice is in a DIFFERENT repo + # (a same-repo parent is the stack base already + # rendered by ``## Stack``). + upstream_pr_ref: dict[str, _pkg.Any] | None = None + upstream_ids = slice_obj.dependencies or [] + if upstream_ids: + upstream = next( + (s for s in contract_post.slices if s.id == upstream_ids[0]), + None, + ) + if upstream is not None and upstream.pr_number: + upstream_repo = ( + resolve_slice_repo(upstream, pipeline) or pipeline.repo + ) + if upstream_repo and upstream_repo != slice_repo: + upstream_pr_ref = { + "repo": upstream_repo, + "number": upstream.pr_number, + } + # #3393 slice-5 / task-5-1: a slice with a + # CROSS-repo dependency opens its PR as a DRAFT + # — cross-repo edges can't stack, so the + # dependent slice is developed in parallel and + # only its PR *ready* transition waits on the + # merge gate (auto draft→ready when the upstream + # merges, else a HITL hold). A dep is cross-repo + # iff the upstream slice resolves to a DIFFERENT + # repo; same-repo-only deps and N=1 pipelines + # stay non-draft (behaviour unchanged). Checks + # ALL deps so any cross-repo upstream holds it. + cross_repo_draft = False + for _dep_id in slice_obj.dependencies or []: + _dep = next( + (s for s in contract_post.slices if s.id == _dep_id), + None, + ) + if _dep is None: + continue + _dep_repo = resolve_slice_repo(_dep, pipeline) or pipeline.repo + if _dep_repo and _dep_repo != slice_repo: + cross_repo_draft = True + break + slice_pr_data = { + # #3393 slice-4: the repo this slice's PR is + # opened in + its cross-repo coordination + # references (empty for N=1). + "slice_repo": slice_repo, + # #3393 slice-5: open draft when this slice + # has a cross-repo dependency (see above). + "cross_repo_draft": cross_repo_draft, + "sibling_pr_refs": sibling_pr_refs, + "upstream_pr_ref": upstream_pr_ref, + "slice_name": slice_obj.name or slice_id, + # Planner's reviewer-facing summary — + # rendered as the slice PR body's lead + # paragraph (#3115). Empty for + # pre-#3115 contracts. + "slice_goal": getattr(slice_obj, "goal", "") or None, + "slice_tasks": [ + { + "id": t.id, + "description": t.description, + "acceptance_criteria": t.acceptance_criteria, + } + for t in (slice_obj.tasks or []) + ], + "slice_index": slice_index_lookup, + "slice_count": slice_count, + "slice_files_affected": slice_files_affected_list or None, + # ``context_pr_number`` is populated by + # ``_open_context_pr_at_implement_start`` + # at the plan→implement boundary (#2777 + # cq-4). When the contract linkage is + # missing (e.g. ``contract.pr`` is None + # on an implement-start resume, #3100), + # fall back to ``pipeline.pr_number`` — + # the pipeline-level mirror written by + # ``_persist_context_pr_number`` whose + # sole post-#2777 writer is the same + # opener — so the slice PR still links + # its base PR (#3115). When both are + # None — should be unreachable under + # the hard-required opener but kept as + # defence-in-depth — ``create_slice_pr`` + # falls back to the pre-#2745 inline- + # narrative body so the slice PR stays + # reviewable as a standalone diff + # against ``/work``. + "context_pr_number": ( + (program_pr.context_pr_number if program_pr else None) + or pipeline.pr_number + ), + "program_title": (program_pr.title if program_pr else None), + "program_description": ( + program_pr.description if program_pr else None + ), + "program_test_plan": (program_pr.test_plan if program_pr else None), + "program_manual_steps": ( + program_pr.manual_steps if program_pr else None + ), + } + except Exception as attr_err: # noqa: BLE001 + # Nested attribute traversal on slice/program PR + # objects (the contract load was lifted out to the + # block above). Surface is AttributeError / + # KeyError on partially-populated PR rollup + # fields. Continue without slice_pr_data (the + # gateway PR creation just below is gated on it + # being non-None). + _pkg.logger.warning( + "Slice PR pre-load failed (continuing)", + pipeline_id=pipeline_id, + slice_id=slice_id, + error=str(attr_err), + ) + + # Persist this slice's per-slice BRC consensus history + # onto its integration branch as the final + # orchestrator-authored commit before the slice PR is + # opened, so reviewers see the consensus transcript in + # the PR diff (#2548). Best-effort + idempotent on + # retry; per-slice files live ONLY on the integration + # branch. + if pipeline.repo: + try: + _pkg._commit_slice_brc_history_to_integration_branch( + pipeline, + spawner, + worktree_repo_path, + slice_id, + integration_branch, + gateway_mode=gateway_mode, # type: ignore[arg-type] + ) + except Exception as brc_commit_err: # noqa: BLE001 + # Per-slice BRC commit helper calls into the + # full git/gateway/message-store machinery + # — the exception surface is unbounded + # (gateway push failures, git plumbing + # errors, message-store reads, file I/O). + # Best-effort: the BRC transcript commit is + # non-essential to slice consensus. + _pkg.logger.warning( + "Per-slice BRC commit raised (continuing) (#2548)", + pipeline_id=pipeline_id, + slice_id=slice_id, + error=str(brc_commit_err), + ) + + pr_created = True + slice_pr_url: str | None = None + slice_pr_number: int | None = None + if slice_pr_data is not None and pipeline.repo: + # Best-effort real-diff summary for the PR body + # (#3115) — commit subjects + diffstat from the + # pushed integration branch. (None, None) on any + # failure; the PR opens without the section. + commit_subjects, diffstat = _pkg._build_slice_diff_summary( + pipeline, + spawner, + worktree_repo_path, + integration_branch, + parent_branch, + gateway_mode=gateway_mode, # type: ignore[arg-type] + ) + try: + slice_pr_url = spawner.gateway.create_slice_pr( + pipeline_id=pipeline_id, + # #3393 slice-4 / task-4-1: route to the slice's + # own repo (falls back to the pipeline primary + # when ``slice.repo`` is absent — the N=1 case). + repo=slice_pr_data["slice_repo"] or pipeline.repo, + slice_id=slice_id, + slice_name=slice_pr_data["slice_name"], + slice_tasks=slice_pr_data["slice_tasks"], + head=integration_branch, + base=parent_branch, + issue_number=issue_number, + agent_role="orchestrator", + mode=gateway_mode, # type: ignore[arg-type] + # #3393 slice-5 / task-5-1: draft when this + # slice has a cross-repo dependency; the merge + # gate marks it ready on upstream merge (or a + # HITL hold releases it). False for N=1. + draft=slice_pr_data["cross_repo_draft"], + program_title=slice_pr_data["program_title"], + program_description=slice_pr_data["program_description"], + program_test_plan=slice_pr_data["program_test_plan"], + program_manual_steps=slice_pr_data["program_manual_steps"], + slice_index=slice_pr_data["slice_index"], + slice_count=slice_pr_data["slice_count"], + slice_files_affected=slice_pr_data["slice_files_affected"], + context_pr_number=slice_pr_data["context_pr_number"], + slice_goal=slice_pr_data["slice_goal"], + diffstat=diffstat, + commit_subjects=commit_subjects, + sibling_pr_refs=slice_pr_data["sibling_pr_refs"], + upstream_pr_ref=slice_pr_data["upstream_pr_ref"], + ) + except Exception as pr_err: # noqa: BLE001 + # Single `gateway.create_slice_pr` HTTP call. + # Catches GatewayError (HTTP) and OSError + # (DNS / socket). Mark pr_created=False so + # the cascade machinery fires. + _pkg.logger.error( + "Slice PR creation failed", + pipeline_id=pipeline_id, + slice_id=slice_id, + error=str(pr_err), + ) + pr_created = False + + if not pr_created: + scheduler.record_failure(slice_id) + return 1, ( + f"slice {slice_id}: PR creation failed (head={integration_branch}, " + f"base={parent_branch})" + ) + + # Parse the slice PR number from the returned URL + # (#3122) — same trailing-boundary pattern the context- + # PR opener uses, narrowed to ``[1-9]\d*`` so a + # malformed ``/pull/0/...`` URL doesn't make it as far + # as ``Slice.pr_number``'s ``ge=1`` validator (which + # would silently downgrade to a warning log via the + # save try/except in ``_persist_slice_status_complete``). + # Best-effort: an unparseable URL just means the + # linkage isn't recorded this pass; the idempotent + # ``create_slice_pr`` re-yields it on a resume. + if slice_pr_url: + pr_match = _pkg.re.search(r"/pull/([1-9]\d*)(?:[/?#]|$)", slice_pr_url) + if pr_match: + slice_pr_number = int(pr_match.group(1)) + + # Hold the per-pipeline state lock across both the + # contract-write (``_persist_slice_status_complete`` + # itself reacquires this RLock) and the context-PR + # body refresh (load + compose + push). Without the + # outer lock, two slices in the same wave could + # interleave between persist and push so the slice + # whose refresh starts earlier but lands later + # clobbers the body that already included both links + # — and because no later slice fires a refresh, the + # final slice's ``— #N`` link would stay missing + # forever. Serializing here bounds the per-slice tail + # latency by one gateway PATCH per concurrent slice + # rather than racing them. + with get_pipeline_state_lock(pipeline_id): + scheduler.record_complete(slice_id) + # Reaching here means ``_run_concurrent_phase`` returned + # success (BRC consensus) AND ``pr_created`` gated above — + # a verified completion independent of whether the PR URL + # parsed to a number (#3122 stub URLs leave + # ``slice_pr_number`` None). Declare the consensus basis so + # the #3214 invariant accepts it; ``pr_number`` is still + # passed for the slice-table linkage. + _persist_slice_status_complete( + slice_id, + pr_number=slice_pr_number, + pr_url=slice_pr_url if slice_pr_number else None, + basis="consensus_complete", + ) + + # Refresh the context PR body so its slice table + # links the PR that just opened (#3122). Strictly + # cosmetic and best-effort: every failure path + # inside logs + returns False without raising, and + # the slice outcome below never depends on it. + if slice_pr_number: + _pkg._refresh_context_pr_body( + pipeline_id, + pipeline=pipeline, + spawner=spawner, + worktree_repo_path=worktree_repo_path, + identifier=_pkg._pipeline_identifier( + pipeline.issue_number, pipeline_id + ), + gateway_mode=gateway_mode, + ) + + try: + remove_peer_consensus_tracker(pipeline_id, slice_id) + except Exception: # noqa: BLE001 + # In-memory dict pop under a lock; same crash-proof + # defence-in-depth as the merged-skip branch above. + pass + return exit_code_inner, logs_inner + + # Gate every ready slice through the orchestrator-process-wide + # admission counter (#2241 gap 1). Slices the global cap + # rejects stay in READY and re-yield next tick — the per- + # pipeline ``iter_ready`` accounting is unaffected because + # we admit BEFORE ``mark_spawned``. If the entire batch is + # rejected, sleep one poll interval before re-checking so + # we don't burn CPU spinning on iter_ready. + admitted_batch: list[tuple[str, str | None]] = [ + (slice_id, parent_slice_id) + for slice_id, parent_slice_id in ready_batch + if global_slice_admit.try_admit(pipeline_id, slice_id) + ] + if not admitted_batch: + _pkg.logger.info( + "Slice wave deferred behind global cap", + pipeline_id=pipeline_id, + ready=[s for s, _ in ready_batch], + admit=global_slice_admit.snapshot(), + ) + _pkg.time.sleep(poll_interval) + continue + + # Mark admitted slices as spawned BEFORE submitting them to + # the executor so a subsequent ``iter_ready`` from any other + # thread sees the in-flight count correctly. + for slice_id, _parent in admitted_batch: + scheduler.mark_spawned(slice_id) + + max_workers = max(1, len(admitted_batch)) + with _pkg.concurrent.futures.ThreadPoolExecutor( + max_workers=max_workers, + thread_name_prefix=f"slice-wave-{pipeline_id}", + ) as wave_pool: + futures: dict[_pkg.concurrent.futures.Future, str] = {} + for slice_id, parent_slice_id in admitted_batch: + fut = wave_pool.submit(_run_one_slice, slice_id, parent_slice_id) + futures[fut] = slice_id + + for fut in _pkg.concurrent.futures.as_completed(futures): + slice_id_done = futures[fut] + try: + exit_code, logs = fut.result() + except Exception as exc: # noqa: BLE001 + # fut.result() re-raises whatever the slice + # worker raised. Workers call into the full + # implement-phase machinery (gateway, contract, + # spawner, message store, docker) so the + # exception surface is unbounded; mark the + # slice failed and continue rather than tearing + # down the whole wave. + scheduler.record_failure(slice_id_done) + exit_code = 1 + logs = f"slice {slice_id_done} raised: {exc!r}" + _pkg.logger.error( + "Slice worker raised", + pipeline_id=pipeline_id, + slice_id=slice_id_done, + error=str(exc), + ) + aggregate_logs.append(f"--- slice {slice_id_done} ---\n{logs}") + if exit_code != 0: + overall_exit = exit_code + + # Drain cascades after each wave so descendants of a + # failed slice are visibly BLOCKED before the next + # iteration computes ready slices. Emit an + # OVERSEER_ALERT per cascade so the human operator sees + # the blocked subtree (#2137 TASK-3-4 emission path). + events = scheduler.poll_cascades() + for event in events: + _pkg.logger.warning( + "Slice cascade fired", + pipeline_id=pipeline_id, + failed_slice=event.failed_slice_id, + blocked=event.blocked_subtree, + ) + # Emit OVERSEER_ALERT directly through the in-process + # message store so the human operator's overseer + # surface picks up the cascade-block event (TASK-3-4 + # emission path). + try: + try: + from message_store import Message, get_message_store + except ImportError: + from ..message_store import ( # type: ignore[no-redef] + Message, + get_message_store, + ) + + msg = Message( + pipeline_id=pipeline_id, + from_role="orchestrator", + to_role="all", + message_type="OVERSEER_ALERT", + subject=f"slice-cascade-block: {event.failed_slice_id}", + body=( + f"Slice {event.failed_slice_id} failed; " + f"downstream subtree {event.blocked_subtree} marked " + "BLOCKED_ON_FAILED_DEPENDENCY (60 s grace expired). " + "HITL resolution required to restart the failed slice." + ), + metadata={ + "anomaly": "slice-cascade-block", + "priority": "high", + "failed_slice_id": event.failed_slice_id, + "blocked_subtree": list(event.blocked_subtree), + }, + phase="implement", + ) + get_message_store().add_message(msg) + except Exception: # noqa: BLE001 + # Best-effort: the log line above is the + # always-on fallback so the operator still sees + # the cascade in the orchestrator log. + pass + finally: + reconciler_stop.set() + try: + reconciler_thread.join(timeout=5.0) + except RuntimeError: + # Thread.join only raises RuntimeError (e.g. joining the + # current thread). Other failures are silent timeouts. + pass + + aggregated = "\n".join(aggregate_logs) if aggregate_logs else "Slice loop completed." + return overall_exit, aggregated diff --git a/orchestrator/routes/pipelines/_run_implement_support.py b/orchestrator/routes/pipelines/_run_implement_support.py new file mode 100644 index 0000000000..1045cd5a88 --- /dev/null +++ b/orchestrator/routes/pipelines/_run_implement_support.py @@ -0,0 +1,242 @@ +"""implement-phase support (lifted closures) helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _commit_and_push_slice_statefiles_impl( + message: str, + *, + pipeline_id, + worktree_repo_path, + pipeline, + store, + spawner, + gateway_mode, + issue_number, +) -> None: + """Commit + push pipeline-scoped ``.egg-state/`` writes to the work branch. + + Contract mutations — agent task-record updates via + ``mutate_contract`` and the ``slice.status`` flips below — land + on the shared pipeline worktree's disk copy only. Without a + slice-boundary commit, the work branch's contract file stays + frozen at the init-time "Initialize SDLC contract" commit for + the entire implement phase, and a mid-phase orchestrator crash + or worktree prune loses every accumulated task record (#3117). + The phase-boundary commit at the end of the run loop is too + coarse for multi-slice phases. + + Scope (per #3117): this closes durability for the post-prune + audit record, operator/PR-side review of mid-phase contract + state, and orchestrator-restart resume at slice granularity. + It is deliberately NOT the read path for live agents — agents + read the contract via ``mcp__sdlc__show_contract`` against the + orchestrator's in-memory state, never from their checkout's + ``.egg-state/contracts/`` file (#3077). + + Best-effort: slice completion must not block on statefile + durability; failures are logged and the next boundary (later + slice close or phase completion) carries the writes. The commit + runs under the per-pipeline state lock to serialise concurrent + slice-close threads against the shared worktree's git index; + the push runs outside the lock. The expected case is a linear + fast-forward (lock-serialised commits stack), and a no-op FF + of the same SHA from two threads is harmless. The residual + hazard is ``_reconcile_and_retry_push`` on a non-FF rejection + (``gateway_client.py:1361``): two threads both fetching+rebasing + in the shared worktree can interleave ``.git/index.lock``. + Within the implement phase no other writer pushes to + ``pipeline.branch`` so non-FF shouldn't fire in normal + operation; an external push (operator hand-fix, stale + concurrent orchestrator) is the only known trigger. + """ + try: + with _pkg.get_pipeline_state_lock(pipeline_id): + committed = _pkg._commit_statefiles_to_worktree( + worktree_repo_path, + message, + _pkg._pipeline_identifier(issue_number, pipeline_id), + pipeline_id=pipeline_id, + ) + except Exception as commit_err: # noqa: BLE001 + # The helper raises CalledProcessError / TimeoutExpired + # from subprocess.run and OSError from glob (#2219 family). + _pkg.logger.warning( + "Failed to commit slice statefiles to work branch (continuing) (#3117)", + pipeline_id=pipeline_id, + commit_message=message, + error=str(commit_err), + ) + return + if not committed or not pipeline.branch or worktree_repo_path == store.repo_path: + return + try: + spawner.gateway.push_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(worktree_repo_path), + branch=pipeline.branch, + mode=gateway_mode, # type: ignore[arg-type] + base_branch=pipeline.base_branch, + ) + except Exception as push_err: # noqa: BLE001 + # Gateway HTTP push (GatewayError / OSError). The commit is + # already on the local work branch; the next successful + # push carries it. + _pkg.logger.warning( + "Failed to push slice statefiles to work branch (continuing) (#3117)", + pipeline_id=pipeline_id, + commit_message=message, + error=str(push_err), + ) + + +def _persist_slice_status_complete_impl( + slice_id: str, + *, + pipeline_id, + worktree_repo_path, + commit_and_push, + pr_number: int | None = None, + pr_url: str | None = None, + basis: str | None = None, + commit_to_branch: bool = True, +) -> None: + """Mark ``slice_id`` as ``SliceStatus.COMPLETE`` on the contract. + + Durable signal so the bootstrap reconciliation pass below and + the ``restart_agent`` parent-complete fallback can skip the + slice without a GitHub round-trip (#2549, #2470). Best-effort: + on save failure the in-memory scheduler state still reflects + completion for this pass and the next ``start_pipeline`` + re-detects via the merged-detection helper. + + With *commit_to_branch* (the default), the saved contract — + along with any other uncommitted pipeline statefiles, e.g. + agent task-record mutations made during the slice — is + committed and pushed to the pipeline work branch so the durable + copy tracks the live one (#3117). The bootstrap reconciliation + passes set it to ``False`` and batch a single commit after the + loop instead of one per reconciled slice. + + Called only after a slice successfully closes (BRC consensus + reached + PR opened, or merged-skip / bootstrap-COMPLETE + reconciliation). Failed slices — ``exit_code_inner != 0`` + (#16410) or ``pr_created == False`` (#16588) — return early + without calling this helper, so their accumulated task-record + mutations remain uncommitted in the worktree until the next + successful slice's commit (the pipeline-scoped glob picks them + up) or the phase-boundary commit, whichever fires first. + + ``basis`` lets a caller declare *why* the slice is complete when + not every task is marked COMPLETE on the contract: ``"merged"`` + (integration branch ancestry-verified merged into its parent) or + ``"consensus_complete"`` (BRC consensus reached pre-restart, PR + not yet opened). The PR-open caller passes ``pr_number`` instead. + Absent any of these — and with tasks still pending — the write is + a #3214 false-complete and :func:`_validate_slice_completion_basis` + raises :class:`SliceCompletionInvariantError` rather than persist + a slice as done that never ran. + + When the caller just opened the slice's PR it passes + ``pr_number`` / ``pr_url`` so the linkage lands in the same + contract write (#3122) — the context-PR body refresh and any + later stack consumer read them from ``Slice.pr_number``. + ``None`` (the merged-skip and bootstrap callers) leaves any + previously recorded linkage untouched. + + TODO(#3122): the three ``None`` callers — bootstrap layer-A + (contract-recorded COMPLETE), bootstrap layer-B (merged on + origin), and the run-loop merged-skip — do not recover the + slice PR number from GitHub (`gh pr list --head … --state + merged`), so on a resume past those points the slice-table + entries for merged slices stay unlinked. Acceptable for v1 + because the per-slice ``— #N`` link is most useful while the + stack is live, but worth backfilling if reviewers ask for + complete cross-linkage on archived stacks. + """ + from egg_contracts.loader import load_contract, save_contract + from egg_contracts.models import SliceStatus + + try: + with _pkg.get_pipeline_state_lock(pipeline_id): + contract_local = load_contract(pipeline_id, worktree_repo_path) + for s in contract_local.slices: + if s.id == slice_id: + # #3214 — refuse to persist a contradictory COMPLETE. + # An interior forest node marked COMPLETE without a + # valid basis (tasks pending, no PR, no verified + # merge/consensus) skips a slice that never ran and + # wedges the chain a phase later. Fail loud here, at + # the source of the bad write, instead. + invalid = _pkg._validate_slice_completion_basis( + s, pr_number=pr_number, basis=basis + ) + if invalid is not None: + _pkg.logger.error( + "Refusing to persist slice.status=COMPLETE — " + "invalid completion basis (#3214)", + pipeline_id=pipeline_id, + slice_id=slice_id, + reason=invalid, + ) + raise _pkg.SliceCompletionInvariantError(invalid) + s.status = SliceStatus.COMPLETE + if pr_number is not None: + s.pr_number = pr_number + if pr_url is not None: + s.pr_url = pr_url + _pkg.logger.info( + "Slice marked COMPLETE", + pipeline_id=pipeline_id, + slice_id=slice_id, + basis=( + basis + or ( + "pr" + if (pr_number is not None or s.pr_number is not None) + else "tasks_complete" + ) + ), + pr_number=pr_number if pr_number is not None else s.pr_number, + ) + break + save_contract(contract_local, worktree_repo_path) + except _pkg.SliceCompletionInvariantError: + # Fail loud — never swallow the completion invariant into the + # best-effort save handler below (#3214). + raise + except Exception as save_err: # noqa: BLE001 + # Contract load/save under per-pipeline state lock. + # Catches loader validation errors, atomic-rename / fdopen + # I/O failures, and pydantic re-serialisation errors. + # Best-effort: the in-memory scheduler still reflects + # COMPLETE for this pass; next start_pipeline re-detects. + _pkg.logger.warning( + "Failed to persist slice.status=COMPLETE", + pipeline_id=pipeline_id, + slice_id=slice_id, + error=str(save_err), + ) + return + if commit_to_branch: + commit_and_push(f"Persist contract after slice {slice_id} completion (#3117)") + + +def _contract_loader_impl(*, pipeline_id, worktree_repo_path) -> _pkg.Any: + from egg_contracts.loader import load_contract + + try: + return load_contract(pipeline_id, worktree_repo_path) + except Exception: # noqa: BLE001 + # Best-effort loader for callers that just need "current + # contract or None". Catches loader validation errors, + # OSError on the contract file read, and any pydantic + # re-serialisation failure. + return None diff --git a/orchestrator/routes/pipelines/_run_phase.py b/orchestrator/routes/pipelines/_run_phase.py new file mode 100644 index 0000000000..d8b92cf844 --- /dev/null +++ b/orchestrator/routes/pipelines/_run_phase.py @@ -0,0 +1,331 @@ +"""run_pipeline per-phase execution loop block helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _run_phase_execution( + pipeline, + phase_execution, + phase_failed, + *, + certs_volume, + current_phase, + gateway_mode, + pipeline_id, + pipeline_mode, + repo_volumes, + repos, + run_epoch, + sandbox_env, + spawner, + store, + worktree_repo_path, +): + """Run one phase (spawn agents / BRC) — the while-loop's phase-execution + block, extracted verbatim from _run_pipeline. Returns (pipeline, + phase_execution, phase_failed, action); action in {None, 'return', + 'break'} tells the thin loop whether to return / break / fall through.""" + if True: + while True: + # Reset tester gaps each cycle so stale findings don't accumulate + tester_gap_summary = None + + # Reload to get latest review_cycles count + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + phase_execution = pipeline.get_phase_execution(current_phase) + review_cycle = phase_execution.review_cycles + + # Reset status to RUNNING at cycle start so that a + # previous cycle's FAILED status doesn't persist and + # cause _derive_subphase_status() to misreport (see + # issue #1178). + phase_execution.status = _pkg.PipelineStatus.RUNNING + pipeline.status = _pkg.PipelineStatus.RUNNING + + # Record when actual agent work begins (excludes sandbox setup + # and HITL waiting time from the phase duration). + phase_execution.work_started_at = _pkg.datetime.now(_pkg.UTC) + + # Capture HEAD commit for delta reviews: reviewers in + # subsequent cycles can diff against this to see only + # the changes made since the last review. + cycle_commit_sha: str | None = None + try: + _git_result = _pkg.subprocess.run( + ["git", "rev-parse", "HEAD"], + capture_output=True, + text=True, + cwd=str(worktree_repo_path), + timeout=10, + ) + if _git_result.returncode == 0: + cycle_commit_sha = _git_result.stdout.strip() + except Exception: + pass # Non-fatal — delta review is best-effort + + phase_execution.cycle_timings.append( + _pkg.CycleTiming( + cycle=review_cycle, + started_at=phase_execution.work_started_at, + commit_sha=cycle_commit_sha, + ) + ) + store.save_pipeline(pipeline) + + # 1. Spawn workers — always use concurrent BRC execution. + _pkg.logger.info( + "Spawning concurrent phase execution", + pipeline_id=pipeline_id, + phase=current_phase, + review_cycle=review_cycle, + mode=gateway_mode, + ) + + # Read structured operator directives + prior iteration + # history off the phase so iteration N+1 prompts can render + # them with precedence prose (#2795). These lists accumulate + # across kickbacks and are never cleared, so no read-and- + # clear stash is needed. + _phase_operator_directives: list[_pkg.OperatorDirective] = [] + _phase_iteration_history: list[_pkg.IterationSummary] = [] + try: + with _pkg.get_pipeline_state_lock(pipeline_id): + _fb_pipeline = store.load_pipeline(pipeline_id) + _fb_phase = _fb_pipeline.get_phase_execution(current_phase) + _phase_operator_directives = list(_fb_phase.operator_directives) + _phase_iteration_history = list(_fb_phase.iteration_history) + except Exception as e: + _pkg.logger.debug("Failed to read operator directives for phase", error=str(e)) + + # #2137: route the implement phase through the slice + # DAG iterator when the contract has more than one + # slice. Single-slice and no-slice contracts continue + # to use the legacy monolithic path so existing + # pipelines are unaffected. + _use_slice_loop = False + _slice_gate_failure: _pkg.SliceGateMonolithicBlock | None = None + if current_phase.value == "implement": + try: + from egg_contracts.loader import ( + load_contract as _load_contract_for_slice_check, + ) + + _check_contract = _load_contract_for_slice_check( + pipeline_id, worktree_repo_path + ) + _slice_count = len(getattr(_check_contract, "slices", []) or []) + # #2777 cq-10 — route through ``_is_slice_dag_mode`` + # so the "what counts as slice-DAG" definition has + # a single source of truth. Local ``_slice_count`` + # is still used by the defensive recheck below for + # the structured log when the populator dropped + # slices (#2337). + _use_slice_loop = _pkg._is_slice_dag_mode(_check_contract) + + # #2915: Auto-populate contract if empty at implement start + # This fills the gap where start_phase=implement doesn't trigger + # the plan-completion populate path, leaving agents with nothing to do. + if _slice_count == 0: + _slice_count = _pkg._auto_populate_contract_at_implement_start( + worktree_repo_path, + pipeline_id, + pipeline_mode, + pipeline.issue_number, + pipeline.current_phase, + pipeline.branch, + gateway=spawner.gateway, + gateway_mode=gateway_mode, + base_branch=pipeline.base_branch, + ) + if _slice_count > 0: + # Reload contract after successful populate + _check_contract = _load_contract_for_slice_check( + pipeline_id, worktree_repo_path + ) + _use_slice_loop = _pkg._is_slice_dag_mode(_check_contract) + + # #2337 defensive recheck: if the contract has no + # slices but the on-disk plan draft parses to N>1 + # slices, the populator silently failed earlier. + # Refuse to demote to monolithic. + if _slice_count == 0: + _slice_gate_failure = _pkg._slice_gate_block_monolithic_demotion( + worktree_repo_path, + pipeline_id, + pipeline.issue_number, + ) + except Exception as _slice_check_err: # noqa: BLE001 + _pkg.logger.debug( + "Slice-loop gate: contract load failed, falling back to monolithic", + pipeline_id=pipeline_id, + error=str(_slice_check_err), + ) + + if _slice_gate_failure is not None: + _slice_gate_msg = _slice_gate_failure.message + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + phase_execution = pipeline.get_phase_execution(current_phase) + if phase_execution.cycle_timings: + phase_execution.cycle_timings[-1].completed_at = _pkg.datetime.now(_pkg.UTC) + phase_execution.status = _pkg.PipelineStatus.FAILED + phase_execution.error = _slice_gate_msg + phase_execution.completed_at = _pkg.datetime.now(_pkg.UTC) + pipeline.status = _pkg.PipelineStatus.FAILED + pipeline.error = _slice_gate_msg + store.save_pipeline(pipeline) + # #2627 follow-up: emit a dedicated HITL naming the + # empty-contract root cause inline. The generic + # post-failure Retry/Accept/Abort decision respawns + # implement into the same empty-contract state; this + # HITL's options map to repopulate / restart-plan / + # abort so the operator has a recovery path that + # actually changes state. + _pkg._emit_empty_contract_hitl( + pipeline_id, + pipeline, + store, + reason="slice_gate_blocked_monolithic_demotion", + draft_slice_count=_slice_gate_failure.draft_slice_count, + gate="slice_gate", + phase=current_phase, + ) + _pkg.logger.error( + "OVERSEER_ALERT slice_gate_blocked_monolithic_demotion", + pipeline_id=pipeline_id, + error=_slice_gate_msg, + draft_slice_count=_slice_gate_failure.draft_slice_count, + ) + phase_failed = True + break + + try: + if _use_slice_loop: + exit_code, container_logs = _pkg._run_implement_phase_slices( + pipeline_id=pipeline_id, + pipeline=pipeline, + spawner=spawner, + repo_volumes=repo_volumes, + gateway_mode=gateway_mode, + repos=repos, + sandbox_env=sandbox_env, + store=store, + certs_volume=certs_volume, + worktree_repo_path=worktree_repo_path, + run_epoch=run_epoch, + ) + else: + # Pre-#2137 monolithic-implement fallback. The + # impasse-retry wrapper deliberately wraps only + # the slice-loop call site (#2529): impasse + # delegation rewires a *task* between producer + # roles, which only makes sense per-slice. + # Pipelines that don't use the slice loop are + # legacy / single-PR-shape, so an impasse here + # surfaces as a normal slice failure and the + # operator handles it via the existing + # phase-failure HITL path. + exit_code, container_logs = _pkg._run_concurrent_phase( + pipeline_id=pipeline_id, + pipeline=pipeline, + phase=current_phase, + spawner=spawner, + repo_volumes=repo_volumes, + gateway_mode=gateway_mode, + repos=repos, + sandbox_env=sandbox_env, + store=store, + certs_volume=certs_volume, + worktree_repo_path=worktree_repo_path, + operator_directives=_phase_operator_directives, + iteration_history=_phase_iteration_history, + run_epoch=run_epoch, + ) + except (_pkg.ContainerSpawnError, _pkg.KubernetesSpawnError) as e: + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + phase_execution = pipeline.get_phase_execution(current_phase) + if phase_execution.cycle_timings: + phase_execution.cycle_timings[-1].completed_at = _pkg.datetime.now(_pkg.UTC) + phase_execution.status = _pkg.PipelineStatus.FAILED + phase_execution.error = str(e) + phase_execution.completed_at = _pkg.datetime.now(_pkg.UTC) + pipeline.status = _pkg.PipelineStatus.FAILED + pipeline.error = str(e) + store.save_pipeline(pipeline) + _pkg.logger.error( + "Failed to spawn concurrent containers", + pipeline_id=pipeline_id, + error=str(e), + ) + phase_failed = True + break + + if exit_code != 0: + # Check if pipeline was restarted while this thread + # was running (e.g. restart_phase bumped run_epoch). + # If so, a new _run_pipeline thread owns this pipeline + # — exit without marking the phase FAILED. See #1638. + _check_pip = store.load_pipeline(pipeline_id) + _check_epoch = _check_pip.run_epoch or _check_pip.created_at + if _check_epoch != run_epoch: + _pkg.logger.info( + "Pipeline was restarted during phase execution, exiting old thread", + pipeline_id=pipeline_id, + ) + return pipeline, phase_execution, phase_failed, "return" + + error_msg = f"Container exited with code {exit_code}" + if container_logs: + log_lines = container_logs.strip().splitlines() + tail = "\n".join(log_lines[-10:]) + error_msg += f"\n--- container logs (last 10 lines) ---\n{tail}" + + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + phase_execution = pipeline.get_phase_execution(current_phase) + if phase_execution.cycle_timings: + phase_execution.cycle_timings[-1].completed_at = _pkg.datetime.now(_pkg.UTC) + phase_execution.status = _pkg.PipelineStatus.FAILED + phase_execution.error = error_msg + phase_execution.completed_at = _pkg.datetime.now(_pkg.UTC) + pipeline.status = _pkg.PipelineStatus.FAILED + pipeline.error = error_msg + store.save_pipeline(pipeline) + _pkg.logger.error( + "Phase failed", + pipeline_id=pipeline_id, + phase=current_phase, + exit_code=exit_code, + container_logs=container_logs[-2000:] if container_logs else "", + ) + phase_failed = True + break + + # 2. Read tester gap findings (concurrent phases include a tester). + # Only read when the phase succeeded — a failed phase may + # have left stale output from a previous cycle on disk. + if not phase_failed: + tester_gap_summary = _pkg._read_tester_gaps( + worktree_repo_path, + identifier=_pkg._pipeline_identifier(pipeline.issue_number, pipeline_id), + ) + if tester_gap_summary: + _pkg.logger.info( + "Tester found gaps", + pipeline_id=pipeline_id, + phase=current_phase, + ) + + # Reviewers are handled within the BRC consensus protocol + # (see issue #1178) — advance to next phase. + break + return pipeline, phase_execution, phase_failed, None diff --git a/orchestrator/routes/pipelines/_run_phase_blocks.py b/orchestrator/routes/pipelines/_run_phase_blocks.py new file mode 100644 index 0000000000..1e0b3e4e70 --- /dev/null +++ b/orchestrator/routes/pipelines/_run_phase_blocks.py @@ -0,0 +1,338 @@ +"""run_pipeline per-phase advance loop blocks helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _run_implement_advance( + pipeline, + *, + current_phase, + gateway_mode, + pipeline_id, + repo_path, + spawner, + store, + worktree_repo_path, +): + """IMPLEMENT-phase advance loop block (extracted verbatim; pure fall-through).""" + if current_phase == _pkg.PipelinePhase.IMPLEMENT: + try: + gap_gated = _pkg._await_unresolved_gap_gate( + store, + pipeline_id, + repo_path, + worktree_repo_path, + _pkg._pipeline_identifier(pipeline.issue_number, pipeline_id), + current_phase, + pipeline.config.hitl_gates, + ) + pipeline = store.load_pipeline(pipeline_id) + # The gate ran after the statefile commit+push above, so + # when it changed the contract (operator resolved a gap, + # or the override audit landed) the resolution is still + # uncommitted in the worktree. Re-commit + push so the + # work branch tree CI sees reflects the post-gate + # contract, not the open-gap snapshot pushed earlier. + if gap_gated: + gate_committed = False + try: + gate_committed = _pkg._commit_statefiles_to_worktree( + worktree_repo_path, + f"Persist contract after {current_phase.value} gap gate", + pipeline_identifier=_pkg._pipeline_identifier( + pipeline.issue_number, pipeline_id + ), + pipeline_id=pipeline_id, + ) + except Exception as git_err: + _pkg.logger.warning( + "Failed to commit statefiles after gap gate (continuing)", + pipeline_id=pipeline_id, + phase=current_phase.value, + error=str(git_err), + ) + # Skip the follow-up push when nothing was committed + # (e.g. the override path leaves the contract + # unchanged) — it would be a no-op fast-forward + # (#2548). + if gate_committed and pipeline.branch and worktree_repo_path != repo_path: + try: + spawner.gateway.push_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(worktree_repo_path), + branch=pipeline.branch, + mode=gateway_mode, + base_branch=pipeline.base_branch, + ) + except Exception as push_err: + _pkg.logger.warning( + "Failed to push statefiles after gap gate (continuing)", + pipeline_id=pipeline_id, + phase=current_phase.value, + error=str(push_err), + ) + except Exception as gap_gate_err: # noqa: BLE001 + # Never let a gate bug strand the pipeline — the + # reactive test_models_gaps.py CI check remains the + # backstop if this fails open. + _pkg.logger.warning( + "Unresolved-gap gate raised (continuing)", + pipeline_id=pipeline_id, + phase=current_phase.value, + error=str(gap_gate_err), + ) + return pipeline + + +def _run_plan_advance( + pipeline, + phase_overseer_active, + *, + current_phase, + gateway_mode, + overseer_container_id, + overseer_lock, + pipeline_id, + pipeline_mode, + repo_path, + spawner, + store, + worktree_repo_path, +): + """Plan-phase populate/advance loop block (extracted verbatim).""" + if current_phase.value == "plan": + try: + _plan_complete_populate_result = _pkg._populate_contract_from_plan_safe( + worktree_repo_path, + pipeline_id, + pipeline_mode, + pipeline.issue_number, + source="plan_complete", + branch=pipeline.branch, + ) + # #2627 follow-up: populate-succeeded-but-empty is the + # orthogonal failure mode flagged in the issue. The + # draft existed (so neither PlanDraftMissing variant + # fired) but the populator did not produce a contract + # with tasks the implement-phase agents can act on. + # Synthesize a raise so the same FAILED-cleanup + # handler below runs. + # + # Routes through + # :func:`_populate_result_is_empty_contract` so the two + # empty-contract call sites (this handler and the + # ``start_phase=implement`` safety net) can't drift out + # of agreement. This widens the original + # ``EMPTY_RESULT`` / ``PARSE_FAILED`` check to cover + # every non-success outcome plus the POPULATED-with-no- + # slices case (#2627 review). + if _pkg._populate_result_is_empty_contract(_plan_complete_populate_result): + # Pre-raise OVERSEER_ALERT mirroring the two + # ``PlanDraftMissing*`` wrapper-side emits at + # :func:`_populate_contract_from_plan_safe` so the + # discriminator the FAILED-cleanup logger uses + # (``OVERSEER_ALERT plan_populate_produced_empty_contract``) + # is also emitted before the raise. Without this + # the third fail-loud branch had no pre-raise log + # while the two draft-missing branches did, + # asymmetric audit (#2627 review). + _pkg.logger.error( + "OVERSEER_ALERT plan_populate_produced_empty_contract", + pipeline_id=pipeline_id, + branch=pipeline.branch, + outcome=_plan_complete_populate_result.outcome.value, + slice_count=_plan_complete_populate_result.slice_count, + note=( + "plan populate did not produce a contract with " + "tasks the implement-phase agents can act on; " + "blocking phase advance (#2627)" + ), + ) + raise _pkg.PopulateProducedEmptyContractError( + _plan_complete_populate_result.outcome, + slice_count=_plan_complete_populate_result.slice_count, + ) + except ( + _pkg.PlanDraftMissingOnLocalError, + _pkg.PlanDraftMissingOnLocalAndOriginError, + _pkg.PopulateProducedEmptyContractError, + ) as missing_err: + # Mirror the slice-gate failure handler at the + # implement-phase entry: mark FAILED in state, + # then run the same cleanup sequence as the + # ``if phase_failed:`` block above (teardown phase + # overseer, report pipeline status, best-effort push + # for backup) so both load-bearing failure paths + # have a uniform cleanup story. Re #2337 / #2627 + # reviews. + teardown_reason, log_event = _pkg._empty_contract_failure_metadata(missing_err) + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + phase_execution = pipeline.get_phase_execution(current_phase) + phase_execution.status = _pkg.PipelineStatus.FAILED + phase_execution.error = str(missing_err) + phase_execution.completed_at = _pkg.datetime.now(_pkg.UTC) + pipeline.status = _pkg.PipelineStatus.FAILED + pipeline.error = str(missing_err) + store.save_pipeline(pipeline) + # #2627 follow-up: emit the dedicated empty-contract + # HITL so the operator sees an actionable decision + # (repopulate / restart-plan / abort) inline with the + # FAILED status, instead of having to dig through + # pipeline.error and the generic consensus-timeout + # decision. + _hitl_reason = _pkg._empty_contract_hitl_reason(missing_err) + _pkg._emit_empty_contract_hitl( + pipeline_id, + pipeline, + store, + reason=_hitl_reason, + draft_slice_count=None, + gate="plan_complete", + phase=current_phase, + ) + _pkg.logger.error( + log_event, + pipeline_id=pipeline_id, + error=str(missing_err), + ) + # Stop the phase-scoped overseer on failure. + # Hold the lock to prevent the poll thread from seeing + # the container as EXITED and respawning it. + with overseer_lock: + if overseer_container_id and phase_overseer_active: + phase_overseer_active = False + _pkg._teardown_phase_overseer( + spawner, + overseer_container_id, + pipeline_id, + phase_label=str(current_phase), + reason=teardown_reason, + ) + _pkg.report_pipeline_status( + pipeline, + event_type="pipeline.failed", + message=f"Pipeline failed: {(pipeline.error or 'unknown')[:100]}", + ) + _pkg._emit_pipeline_event(pipeline, "pipeline.failed") + # Best-effort: push worktree branch to remote so work + # is backed up before the pipeline exits. + if pipeline.branch and worktree_repo_path != repo_path: + try: + spawner.gateway.push_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(worktree_repo_path), + branch=pipeline.branch, + mode=gateway_mode, + base_branch=pipeline.base_branch, + ) + except Exception as push_err: + _pkg.logger.warning( + "Best-effort push on failure failed", + pipeline_id=pipeline_id, + error=str(push_err), + ) + return pipeline, phase_overseer_active, "break" + return pipeline, phase_overseer_active, None + + +def _run_pending_phase_init( + pipeline, + phase_execution, + *, + current_phase, + pipeline_id, + repo_path, + store, + worktree_repo_path, +): + """PENDING-phase init loop block (extracted verbatim; pure fall-through).""" + if phase_execution.status == _pkg.PipelineStatus.PENDING: + # Record branch tip SHA for completion signal verification. + # This allows the completion handler to detect if a commit + # was pushed to a different branch than expected. + # NOTE: Intentional TOCTOU — the SHA is captured before + # acquiring the state lock, so a push between rev-parse and + # lock acquisition could make it stale. Acceptable because + # phase_start_sha is only used for advisory "no new commits" + # logging, not for correctness decisions. + phase_start_sha: str | None = None + try: + _sha_result = _pkg.subprocess.run( + ["git", "rev-parse", f"origin/{pipeline.branch}"], + capture_output=True, + text=True, + cwd=str(worktree_repo_path), + timeout=10, + check=False, + ) + if _sha_result.returncode == 0: + phase_start_sha = _sha_result.stdout.strip() + except Exception: + pass # Non-fatal — verification is best-effort + + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + phase_execution = pipeline.get_phase_execution(current_phase) + phase_execution.status = _pkg.PipelineStatus.RUNNING + phase_execution.started_at = _pkg.datetime.now(_pkg.UTC) + phase_execution.phase_start_sha = phase_start_sha + pipeline.status = _pkg.PipelineStatus.RUNNING + store.save_pipeline(pipeline) + + # Report phase start to collaborator + _pkg.report_pipeline_status( + pipeline, + event_type="phase.started", + message=f"Phase {current_phase.value} started", + ) + _pkg._emit_pipeline_event(pipeline, "phase.started") + + # #2777 (cq-4, TASK-1-2) — implement-phase entry + # backstop. Calls the new + # ``_open_context_pr_at_implement_start`` opener for + # the runner-driven paths that bypass + # ``advance_phase`` REST (inline ``_run_pipeline`` + # auto-advance and the HITL-approval recovery in + # ``start_pipeline`` both leave + # ``phase_execution.status`` as PENDING and spawn the + # runner directly; the backstop catches both per + # #2593). The opener is idempotent so re-firing here + # after a successful advance_phase call is a one- + # round-trip ``gh pr list`` no-op. + # + # reviewer_code_holistic blocker 1 fix: v1 deleted + # this site under the (incorrect) "single canonical + # site" plan AC; the four soft-fail call sites are in + # fact the only context-PR opener calls on the + # runner-driven paths, so the deletion silently + # stranded slice stacks on ``egg/<id>/work``. + # Restored under the new idempotent opener. + if current_phase == _pkg.PipelinePhase.IMPLEMENT: + try: + _pkg._open_context_pr_at_implement_start(pipeline_id, repo_path=repo_path) + except _pkg.ContextPrCreationError as ctx_err: + _pkg.logger.warning( + "Context PR opener: implement-entry backstop " + "failed (continuing — hard-require enforced at " + "advance_phase and the implement-start plan " + "pre-flight gate) (#2777, #3100)", + pipeline_id=pipeline_id, + reason=ctx_err.reason, + error=str(ctx_err), + ) + except Exception as backstop_err: # noqa: BLE001 + _pkg.logger.warning( + "Context PR opener: implement-entry backstop " + "outer wrapper raised (continuing) (#2777)", + pipeline_id=pipeline_id, + error=str(backstop_err), + ) + return pipeline, phase_execution diff --git a/orchestrator/routes/pipelines/_run_pipeline.py b/orchestrator/routes/pipelines/_run_pipeline.py new file mode 100644 index 0000000000..f03d353d2f --- /dev/null +++ b/orchestrator/routes/pipelines/_run_pipeline.py @@ -0,0 +1,1483 @@ +"""pipeline per-phase driver loop helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _run_pipeline( + pipeline_id: str, + repo_path: _pkg.Path, + _respawn_attempt: int = 0, +) -> None: + """Run a pipeline by spawning containers for each phase. + + This runs in a background thread. For each phase it: + 1. Spawns agent containers via concurrent BRC execution + (_run_concurrent_phase) for all phases. + 2. For reviewed phases (refine, implement, plan): reviewers participate + in the BRC consensus protocol alongside workers, then the phase + loops back with feedback if revision is needed. + 3. Advances to the next phase once approved. + + Args: + pipeline_id: Pipeline ID + repo_path: Path to repository + _respawn_attempt: Internal — counts how many times this thread + has been respawned by the spurious-PNFE recovery path. + Bounded by ``_PNFE_RESPAWN_MAX_ATTEMPTS`` to prevent a + persistent transient from cascading into an unbounded + thread/overseer/commit storm. + """ + from routes.phases import PHASE_TRANSITIONS + + # Track which run of the pipeline this thread owns. If the pipeline + # is deleted and recreated with the same ID while we're still running, + # the new run creates its own worktrees under the same path. Without + # this guard, our finally block would delete the *new* run's worktrees. + run_epoch: _pkg.datetime | None = None + overseer_container_id: str | None = None + phase_overseer_active: bool = False + overseer_lock = _pkg.threading.Lock() + health_monitor_instance = None + health_monitor_timer: _pkg.threading.Event | None = None + poll_thread: _pkg.threading.Thread | None = None + + try: + store = _pkg.get_state_store(repo_path) + spawner = _pkg._get_spawner() + pipeline = store.load_pipeline(pipeline_id) + run_epoch = pipeline.run_epoch or pipeline.created_at + pipeline_mode = "issue" if pipeline.issue_number is not None else "prompt" + transitions = PHASE_TRANSITIONS + + def _make_overseer_teardown_hook( + *, + reason: str, + container_id: str | None, + phase: _pkg.PipelinePhase, + ) -> _pkg.Callable[[], None]: + """Build a pre_event_hook that tears down the per-phase overseer. + + ``container_id`` and ``phase`` are snapshotted as function + parameters (frozen per-call), so the returned closure binds + the loop-iteration values that were current when the + post-phase cleanup branch fired — late binding would race a + subsequent loop iteration. ``reason`` differs between the + doubly-failed and hard-reset-recovered call sites and is + forwarded to :func:`_teardown_phase_overseer`. + + #2797 follow-up: collapses the two duplicated closure + definitions at the two post-phase hard-reset emission sites + into one shared factory. The closure remains inside + ``_run_pipeline`` because the ``phase_overseer_active`` + bool is a local nonlocal of this function. + """ + + def _hook() -> None: + nonlocal phase_overseer_active + with overseer_lock: + if container_id and phase_overseer_active: + phase_overseer_active = False + _pkg._teardown_phase_overseer( + spawner, + container_id, + pipeline_id, + phase_label=str(phase), + reason=reason, + ) + + return _hook + + # Map pipeline to gateway session mode. + gateway_mode, detected_visibility = _pkg._compute_gateway_mode(pipeline) + if not pipeline.network_mode and pipeline.repo: + if detected_visibility is not None: + _pkg.logger.info( + "Auto-detected network mode from repo visibility", + repo=pipeline.repo, + visibility=detected_visibility, + gateway_mode=gateway_mode, + ) + else: + _pkg.logger.warning( + "Could not detect repo visibility, defaulting to public mode", + repo=pipeline.repo, + ) + + # Parse host repo map for volume mounts. When the orchestrator + # runs inside Docker, EGG_REPO_PATH is the *container* path but + # volume mounts need *host* paths (since the Docker socket + # operates on the host daemon). EGG_HOST_REPO_MAP provides a + # JSON mapping of repo_name -> host_path, auto-generated from + # repositories.yaml by the egg launcher. + host_repo_map_raw = _pkg.os.environ.get("EGG_HOST_REPO_MAP", "{}") + try: + host_repo_map: dict[str, str] = _pkg.json.loads(host_repo_map_raw) + except _pkg.json.JSONDecodeError as exc: + _pkg.logger.error( + "Failed to parse EGG_HOST_REPO_MAP — no repos will be mounted in sandbox containers", + raw_value=host_repo_map_raw, + ) + raise ValueError( + f"EGG_HOST_REPO_MAP contains invalid JSON: {host_repo_map_raw!r}" + ) from exc + + # Create a pipeline-level worktree via the gateway. This worktree + # is used by the orchestrator for reading/writing contracts, drafts, + # and state files. Individual agents get their own per-agent + # worktrees at spawn time (created in container_spawner.py) so + # concurrent agents cannot stomp on each other's uncommitted work. + # See #1481 for the per-agent worktree isolation design. + # + # We use the pipeline_id as the worktree container_id for the + # orchestrator-side worktree. Agent worktrees use + # "{pipeline_id}-{role}" as their container_id. + worktree_id = pipeline_id + repo_volumes: dict[str, str] = {} + worktree_repo_path = repo_path # default; overridden when worktrees exist + host_uid = int(_pkg.os.environ.get("HOST_UID", 1000)) + host_gid = int(_pkg.os.environ.get("HOST_GID", 1000)) + pipeline_repos = [pipeline.repo] if pipeline.repo else [] + + repo_volumes, worktree_repo_path = _pkg._map_host_repos( + pipeline, + host_gid=host_gid, + host_repo_map=host_repo_map, + host_uid=host_uid, + pipeline_id=pipeline_id, + pipeline_repos=pipeline_repos, + spawner=spawner, + worktree_id=worktree_id, + repo_volumes=repo_volumes, + worktree_repo_path=worktree_repo_path, + ) + + if not repo_volumes: + raise RuntimeError( + f"No repo volumes available for pipeline {pipeline_id} — " + f"worktree creation is required" + ) + + # Sync worktree with remote before starting pipeline phases. After an + # orchestrator restart, the local worktree branch may be behind origin: + # commits pushed by agents in previous phases (contracts, drafts, + # statefiles) exist on the remote but not in the local checkout. + # Fetching and resetting ensures downstream code (contract loading, + # draft reading) sees the full pipeline state from prior phases. + pipeline, _worktree_done = _pkg._resolve_worktree_repo( + pipeline, + gateway_mode=gateway_mode, + pipeline_id=pipeline_id, + repo_path=repo_path, + spawner=spawner, + store=store, + worktree_repo_path=worktree_repo_path, + ) + if _worktree_done: + return + + # Resolve the certs named volume for gateway CA trust. + # The docker-compose stack creates ${COMPOSE_PROJECT_NAME:-egg}-certs. + certs_volume_raw = _pkg.os.environ.get( + "EGG_CERTS_VOLUME", + _pkg.os.environ.get("COMPOSE_PROJECT_NAME", "egg") + "-certs", + ) + # Validate volume name: Docker allows [a-zA-Z0-9][a-zA-Z0-9_.-]* + # We use a permissive check that rejects obvious shell metacharacters. + if not _pkg.re.match(r"^[a-zA-Z0-9][a-zA-Z0-9_.-]*$", certs_volume_raw): + _pkg.logger.warning( + "Invalid certs volume name, using default", + raw_name=certs_volume_raw, + ) + certs_volume = "egg-certs" + else: + certs_volume = certs_volume_raw + + # Capture source_branch before _read_source_branch_artifacts clears + # it on success — the contract-pull path below (#2035) runs inside + # the contract_synced block and otherwise wouldn't see the value. + source_branch_for_contract_pull = pipeline.source_branch + + # Read artifacts from source branch if specified and inline values + # were not provided. This populates pipeline.plan and + # pipeline.analysis so the contract creation block below can use them. + _pkg._sync_source_branch_drafts( + gateway_mode=gateway_mode, + pipeline=pipeline, + pipeline_id=pipeline_id, + spawner=spawner, + store=store, + worktree_repo_path=worktree_repo_path, + ) + + # Create companion contract in the worktree (deferred from pipeline + # creation so it doesn't pollute the main repo working directory). + pipeline, _contract_setup_done = _pkg._sync_contract_setup( + pipeline, + gateway_mode=gateway_mode, + pipeline_id=pipeline_id, + pipeline_mode=pipeline_mode, + repo_path=repo_path, + source_branch_for_contract_pull=source_branch_for_contract_pull, + spawner=spawner, + store=store, + worktree_repo_path=worktree_repo_path, + ) + if _contract_setup_done: + return + + # Safety net: when start_phase=implement, the plan phase is + # skipped so the plan-completion hook at the end of the phase loop + # never fires. The inline-plan path above calls + # _populate_contract_from_plan inside the contract_synced block, + # but that block is skipped on pipeline restarts (contract already + # synced) and when _read_source_branch_artifacts writes the draft + # file to the worktree without going through the inline-plan + # branch. This catch-all ensures the contract has phases and + # tasks before agents spawn when the plan phase was skipped. + # When start_phase=plan, the plan phase runs normally and the + # plan-completion hook populates the contract, so no safety net + # is needed. + pipeline, _start_phase_done = _pkg._start_phase_setup( + pipeline, + pipeline_id=pipeline_id, + pipeline_mode=pipeline_mode, + store=store, + worktree_repo_path=worktree_repo_path, + ) + if _start_phase_done: + return + + # Operator directives + prior iteration history are persisted on + # ``PhaseExecution`` and accumulate across HITL kickbacks (#2795). + # They are read directly off the phase below each loop iteration — + # no separate "read once and clear" stash is needed. + + # Initialize the Tier 1 health monitor so deterministic tripwires + # (heartbeat timeout, container exit, repeated errors, message rate, + # progress stall) fire during pipeline execution. The monitor + # subscribes to EventBus events reactively, but check_heartbeats() + # and check_progress() need periodic polling. + try: + from events import get_event_bus + from health_monitor import init_health_monitor + + health_monitor_instance = init_health_monitor( + get_event_bus(), pipeline_id, pipeline.config + ) + # Sync the phase-aware threshold with the current pipeline phase + health_monitor_instance.set_current_phase(pipeline.current_phase.value) + + # Wake stuck producers directly when check_brc_progress fires + # so the deterministic detector actually drives remediation + # instead of relying on the overseer agent's discretion (#2079). + # The closure reads the monitor's current phase at fire time so + # the message records the phase the producer is actually in. + _on_health_escalation = _pkg.functools.partial( + _pkg._on_health_escalation_impl, + health_monitor_instance=health_monitor_instance, + pipeline_id=pipeline_id, + ) + + health_monitor_instance.on_escalation(_on_health_escalation) + + # Start a background polling thread for time-based tripwires + health_monitor_timer = _pkg.threading.Event() + + # SHAs we've already raised a branch-divergence alert for + # (#2224 PR 3). Per-pipeline dedupe so we fire once per + # offending commit, not once per 30s tick. + divergence_alerted_shas: set[str] = set() + + _health_monitor_poll = _pkg.functools.partial( + _pkg._health_monitor_poll_impl, + pipeline_id=pipeline_id, + worktree_repo_path=worktree_repo_path, + store=store, + divergence_alerted_shas=divergence_alerted_shas, + ) + + poll_thread = _pkg.threading.Thread( + target=_health_monitor_poll, + args=(health_monitor_instance, health_monitor_timer), + daemon=True, + name=f"health-monitor-{pipeline_id[:8]}", + ) + poll_thread.start() + _pkg.logger.info( + "Health monitor initialized", + pipeline_id=pipeline_id, + ) + except Exception as hm_err: + # Non-fatal: pipeline can run without Tier 1 monitoring + _pkg.logger.warning( + "Failed to initialize health monitor (continuing without Tier 1 monitoring)", + pipeline_id=pipeline_id, + error=str(hm_err), + ) + + while True: + try: + pipeline = store.load_pipeline(pipeline_id) + except Exception: + # Pipeline was deleted — exit quietly + _pkg.logger.info( + "Pipeline no longer exists, exiting thread", + pipeline_id=pipeline_id, + ) + return + + # Detect recreation/restart: another run now owns this pipeline ID + _current_epoch = pipeline.run_epoch or pipeline.created_at + if _current_epoch != run_epoch: + _pkg.logger.info( + "Pipeline was recreated, exiting old thread", + pipeline_id=pipeline_id, + ) + return + + if pipeline.status in (_pkg.PipelineStatus.FAILED, _pkg.PipelineStatus.CANCELLED): + _pkg.logger.info( + "Pipeline stopped", pipeline_id=pipeline_id, status=pipeline.status.value + ) + break + + current_phase = pipeline.current_phase + + # Start the current phase + phase_execution = pipeline.get_phase_execution(current_phase) + pipeline, phase_execution = _pkg._run_pending_phase_init( + pipeline, + phase_execution, + current_phase=current_phase, + pipeline_id=pipeline_id, + repo_path=repo_path, + store=store, + worktree_repo_path=worktree_repo_path, + ) + + # Spawn overseer container for this phase's health monitoring. + # The overseer is phase-scoped: spawned at phase start and torn + # down at phase completion/advance/failure. Each phase gets a + # fresh overseer instance with no accumulated state. + # + # #2270 slice-5: gate overseer presence on "agents actually + # running". During a zero-agent HITL park the pipeline has no phase + # agents in flight, so spawning an overseer there is pure churn + # (§3). The respawn loop that used to keep it alive across such + # parks was removed; this gate stops the phase-start spawn from + # doing the same thing. The agent count is the deterministic phase + # roster the concurrent executor itself consults — the cohort this + # phase is about to run. + _phase_agent_count = _pkg._count_phase_agents(pipeline, current_phase) + if pipeline.config.overseer_enabled and _pkg._overseer_should_be_present( + running_agent_count=_phase_agent_count, + pipeline_status=pipeline.status, + ): + try: + overseer_result = _pkg._spawn_overseer_agent( + spawner=spawner, + pipeline_id=pipeline_id, + issue_number=pipeline.issue_number, + gateway_mode=gateway_mode, + pipeline_repos=pipeline_repos if pipeline_repos else None, + max_turns=pipeline.config.overseer_max_turns, + decision_model=pipeline.config.overseer_decision_maker_model, + ) + with overseer_lock: + overseer_container_id = overseer_result.container_info.container_id + phase_overseer_active = True + _pkg.logger.info( + "Overseer container spawned for phase", + pipeline_id=pipeline_id, + phase=current_phase.value, + container_id=overseer_container_id[:12], + ) + except (_pkg.ContainerSpawnError, _pkg.KubernetesSpawnError) as e: + # Non-fatal: pipeline can run without overseer monitoring + _pkg.logger.warning( + "Failed to spawn overseer container (continuing without monitoring)", + pipeline_id=pipeline_id, + phase=current_phase.value, + error=str(e), + ) + + # Common sandbox environment for all containers in this phase. + # GATEWAY_URL, RUNTIME_UID/GID, proxy vars, DNS lockdown, and + # extra_hosts are now handled by the shared build_sandbox_config() + # inside spawn_agent_container(). Only pipeline-specific vars go here. + if gateway_mode == "private": + orchestrator_ip = _pkg.ORCHESTRATOR_ISOLATED_IP + else: + orchestrator_ip = _pkg.ORCHESTRATOR_EXTERNAL_IP + orchestrator_url = f"http://{orchestrator_ip}:{_pkg.ORCHESTRATOR_PORT}" + sandbox_env: dict[str, str] = { + "EGG_PIPELINE_ID": pipeline_id, + "EGG_PIPELINE_PHASE": current_phase.value, + "EGG_PIPELINE_MODE": pipeline_mode, + "EGG_ORCHESTRATOR_URL": orchestrator_url, + "EGG_ORCHESTRATOR_MODE": "distributed", + } + # ``EGG_BRANCH`` is intentionally NOT set here. The spawner + # is the single source of truth for the agent's assigned + # branch (#2428): ``KubernetesSpawner.spawn_agent_job`` + # derives ``EGG_BRANCH`` from its ``branch`` parameter, + # which the slice scheduler populates with the slice + # integration branch via + # ``ConcurrentPhaseExecutor.get_worktree_branch``. Stuffing + # ``pipeline.branch`` into ``sandbox_env`` here used to be + # threaded through ``extra_env``, where the spawner's + # override loop runs after the default-from-``branch`` + # assignment — deterministic precedence, not a race — so + # the pipeline-level value silently won and slice agents + # were downgraded to the pipeline tip, breaking every + # slice-coder push. The branch persistence below is the + # only side-effect the run loop still needs. + if not pipeline.branch: + generated_branch = f"egg/{pipeline_id}/work" + # Persist the generated branch so the PR phase can use it + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + if not pipeline.branch: + pipeline.branch = generated_branch + store.save_pipeline(pipeline) + _pkg.logger.info( + "Recorded generated branch on pipeline", + pipeline_id=pipeline_id, + branch=generated_branch, + ) + if pipeline.prompt: + sandbox_env["EGG_PIPELINE_PROMPT"] = pipeline.prompt + + if pipeline.repo: + repos = [pipeline.repo] + sandbox_env["EGG_REPO"] = pipeline.repo + else: + repos = [] + + # Jira ticket advisory env vars (issue #1556). These give sandbox + # agents a stable handle for the ticket the pipeline is working + # against (``jira ticket get "$EGG_JIRA_TICKET"``) without + # hard-coding the key. They are ADVISORY — the gateway's project + # allowlist is the only hard boundary, and we never export + # Atlassian credentials (JIRA_BASE_URL / JIRA_USERNAME / + # JIRA_API_TOKEN) to the sandbox. An empty string is exported + # when no ticket is configured so agent wrappers can rely on + # variable presence. + jira_ticket_value = getattr(pipeline, "jira_ticket", None) or "" + sandbox_env["EGG_JIRA_TICKET"] = jira_ticket_value + if jira_ticket_value and "-" in jira_ticket_value: + sandbox_env["EGG_JIRA_PROJECT"] = jira_ticket_value.split("-", 1)[0] + else: + sandbox_env["EGG_JIRA_PROJECT"] = "" + + # Jira-epic SDLC support (issue #1557). Export ``EGG_IS_EPIC`` + # (bool-string) and ``EGG_EPIC_MODE`` (one of + # 'epic-fresh', 'epic-reassess', 'ticket', 'github_issue') + # so the refiner / task-planner / applier prompts can select + # the right mode block. Mapping is derived via + # ``prompt_loader.derive_pipeline_mode`` so the orchestrator + # and any auxiliary callers agree on the canonical rule. + # + # Note: ``EGG_PIPELINE_MODE`` is already taken (PipelineMode: + # 'issue' — set above at L19349). + # ``EGG_EPIC_MODE`` is the orthogonal Jira-epic dimension. + try: + from prompt_loader import derive_pipeline_mode + except ImportError: # pragma: no cover - defensive + derive_pipeline_mode = None # type: ignore[assignment] + _is_epic_flag = bool(getattr(pipeline, "is_epic", False)) + _pipeline_mode_attr = getattr(pipeline, "pipeline_mode", None) + sandbox_env["EGG_IS_EPIC"] = "true" if _is_epic_flag else "false" + if derive_pipeline_mode is not None: + sandbox_env["EGG_EPIC_MODE"] = derive_pipeline_mode( + is_epic=_is_epic_flag, + pipeline_mode=_pipeline_mode_attr, + jira_ticket=jira_ticket_value or None, + ) + else: + sandbox_env["EGG_EPIC_MODE"] = "github_issue" if not jira_ticket_value else "ticket" + + # Issue #1557 reviewer_code v1 finding #4: run the reassess + # sweep before the planner / applier spawn on reassess-mode + # epic pipelines so the task-planner prompt's ``[mode: epic- + # reassess]`` branch and the applier's in-flight refusal + # have the children classification on disk. The sweep + # writes two JSON files under ``.egg-state/agent-outputs/``; + # we export both paths into the sandbox env so the prompts + # read them by env var rather than re-querying the gateway. + # Fail-open: a sweep failure logs a warning but never aborts + # the phase — the planner falls back to fresh-mode treatment + # of the children (which is safe because every action carries + # an explicit ``jira_action`` and the applier's in-flight + # refusal hinges on the sweep file's presence). + if ( + _is_epic_flag + and _pipeline_mode_attr == "reassess" + and current_phase.value in ("plan", "apply") + and jira_ticket_value + ): + try: + from jira_reassess import ( + run_reassess_sweep, + serialise_sweep_to_disk, + ) + except ImportError: # pragma: no cover - defensive + run_reassess_sweep = None # type: ignore[assignment] + serialise_sweep_to_disk = None # type: ignore[assignment] + if run_reassess_sweep is not None and serialise_sweep_to_disk is not None: + try: + sweep_result = run_reassess_sweep( + epic_key=jira_ticket_value, + state_store=store, + ) + agent_outputs_dir = ( + _pkg.Path(worktree_repo_path) / ".egg-state" / "agent-outputs" + ) + sweep_path, done_path = serialise_sweep_to_disk( + result=sweep_result, + agent_outputs_dir=agent_outputs_dir, + pipeline_id=pipeline_id, + ) + sandbox_env["EGG_REASSESS_SWEEP_PATH"] = str(sweep_path) + sandbox_env["EGG_DONE_CHILDREN_PATH"] = str(done_path) + _pkg.logger.info( + "Reassess sweep complete", + pipeline_id=pipeline_id, + epic_key=jira_ticket_value, + child_count=len(sweep_result.children), + done_count=len(sweep_result.done), + warnings=sweep_result.warnings, + ) + except Exception as sweep_err: # noqa: BLE001 — fail-open + _pkg.logger.warning( + "Reassess sweep failed (continuing without sweep handoff)", + pipeline_id=pipeline_id, + epic_key=jira_ticket_value, + error=str(sweep_err), + ) + + phase_failed = False + + # --- Inner review cycle --- + # NOTE: the legacy PR phase (and its auto-PR / slice-DAG-skip + # branches) was deleted in #2777 (cq-4 / TASK-2-2). The context + # PR now opens up-front via ``_open_context_pr_at_implement_start`` + # at the plan→implement boundary, slice PRs stack on it, and + # IMPLEMENT is the terminal phase — no per-phase auto-PR creation + # logic is reachable here for ``current_phase.value == "pr"``. + pipeline, phase_execution, phase_failed, _phase_exec_action = _pkg._run_phase_execution( + pipeline, + phase_execution, + phase_failed, + certs_volume=certs_volume, + current_phase=current_phase, + gateway_mode=gateway_mode, + pipeline_id=pipeline_id, + pipeline_mode=pipeline_mode, + repo_volumes=repo_volumes, + repos=repos, + run_epoch=run_epoch, + sandbox_env=sandbox_env, + spawner=spawner, + store=store, + worktree_repo_path=worktree_repo_path, + ) + if _phase_exec_action == "return": + return + if _phase_exec_action == "break": + break + + # If the phase failed, emit the failure event so the SSE stream + # terminates, then break out of the outer loop. + if phase_failed: + # Stop the phase-scoped overseer on failure. + # Hold the lock to prevent the poll thread from seeing the + # container as EXITED and respawning it. + with overseer_lock: + if overseer_container_id and phase_overseer_active: + phase_overseer_active = False + _pkg._teardown_phase_overseer( + spawner, + overseer_container_id, + pipeline_id, + phase_label=str(current_phase), + reason="phase failed", + ) + + # report_pipeline_status is a stub (no-op) unless status_reporter + # is installed. The actual SSE emission is _emit_pipeline_event + # below. Kept for consistency with the except block at the + # bottom of this function. + _pkg.report_pipeline_status( + pipeline, + event_type="pipeline.failed", + message=f"Pipeline failed: {(pipeline.error or 'unknown')[:100]}", + ) + _pkg._emit_pipeline_event(pipeline, "pipeline.failed") + + # Best-effort: push worktree branch to remote so work is backed up + if pipeline.branch and worktree_repo_path != repo_path: + try: + spawner.gateway.push_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(worktree_repo_path), + branch=pipeline.branch, + mode=gateway_mode, + base_branch=pipeline.base_branch, + ) + except Exception as push_err: + _pkg.logger.warning( + "Best-effort push on failure failed", + pipeline_id=pipeline_id, + error=str(push_err), + ) + + break + + # Phase succeeded — mark complete and advance + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + phase_execution = pipeline.get_phase_execution(current_phase) + phase_execution.status = _pkg.PipelineStatus.COMPLETE + phase_execution.completed_at = _pkg.datetime.now(_pkg.UTC) + + store.save_pipeline(pipeline) # Persist phase completion before HITL gate + + # Report phase completion to collaborator + _pkg.report_pipeline_status( + pipeline, + event_type="phase.completed", + message=f"Phase {current_phase.value} completed", + ) + _pkg._emit_pipeline_event(pipeline, "phase.completed") + + # Commit any uncommitted ``.egg-state/`` writes the agents + # made during the phase BEFORE the worktree sync runs. + # ``register_open_question`` / ``request_feedback`` mutate the + # contract live in the shared pipeline worktree (see + # ``orchestrator/contract_store.py``); those writes are + # uncommitted on disk. The ``git reset --hard`` step inside + # ``_sync_worktree_with_remote`` discards them, leaving the + # bridge below with an empty ``contract.decisions`` and + # silently dropping the operator-bound questions (#2488). + # Committing first lets the sync's rebase reconcile them + # against agent-pushed drafts cleanly. + try: + _pkg._commit_statefiles_to_worktree( + worktree_repo_path, + f"Persist agent statefile writes before {current_phase.value} sync", + pipeline_identifier=_pkg._pipeline_identifier( + pipeline.issue_number, pipeline_id + ), + pipeline_id=pipeline_id, + ) + except Exception as git_err: + _pkg.logger.warning( + "Failed to commit pre-sync agent statefiles (continuing)", + pipeline_id=pipeline_id, + phase=current_phase.value, + error=str(git_err), + ) + + # Sync worktree with remote before post-phase modifications + # so that agent-pushed commits (including plan drafts) are + # incorporated. This must run BEFORE _populate_contract_from_plan + # and _sync_pipeline_decisions_to_contract so the autoresolve + # rebase inside _sync_worktree_with_remote lands the remote + # state before the populate step reads ``.egg-state/`` — + # otherwise populate would read a stale local view and either + # produce an empty contract or overwrite agent-pushed drafts + # that only exist on origin. (Before #2979 the helper also + # issued ``git reset --hard`` on a doubly-failed divergence, + # which would have reverted local on-disk modifications; that + # destructive path is gone, so the modern rationale is purely + # about the autoresolve rebase, not a hard reset.) + post_phase_sync_outcome: _pkg.WorktreeSyncOutcome | None = None + post_phase_sync_aborted = False + if pipeline.branch and worktree_repo_path != repo_path: + # Best-effort for transient failures: a sync failure must + # not strand the auto-advance. Without this guard, a + # gateway HTTP error or git subprocess failure inside the + # helper propagates to the outer Exception handler and (if + # marking FAILED also fails) leaves the pipeline wedged with + # phase COMPLETE but no successor (#2219). + # + # #2979: on an unreconciled divergence the helper pauses + # (AWAITING_HUMAN) on a reconcile HITL and blocks until the + # operator acks, then re-runs the sync — nothing is + # discarded and the pipeline is NOT failed for a recoverable + # post-consensus sync. Only an operator abort (or an + # exhausted reconcile budget) returns aborted=True. + try: + post_phase_sync_outcome, post_phase_sync_aborted = ( + _pkg._sync_worktree_reconciling_divergence( + spawner, + pipeline_id, + store, + repo_path, + worktree_repo_path=worktree_repo_path, + phase=current_phase, + gateway_mode=gateway_mode, + base_branch=pipeline.base_branch, + pipeline_branch=pipeline.branch, + ) + ) + except Exception as sync_err: + _pkg.logger.warning( + "Failed to sync worktree with remote after phase (continuing)", + pipeline_id=pipeline_id, + phase=current_phase.value, + error=str(sync_err), + ) + + # #2979: operator aborted the manual reconcile (or the pause + # budget was exhausted). Fail the pipeline; nothing was + # discarded — the local commits remain pinned under the backup + # ref for offline recovery. ``pre_event_hook`` tears down the + # per-phase overseer under its own lock before the public + # ``pipeline.failed`` event, matching the prior ordering. + if post_phase_sync_aborted and post_phase_sync_outcome is not None: + _pkg._fail_pipeline_after_divergence_abort( + pipeline_id, + store, + phase=current_phase, + backup_ref=post_phase_sync_outcome.backup_ref, + local_only_commit_shas=post_phase_sync_outcome.local_only_commit_shas, + pre_event_hook=_make_overseer_teardown_hook( + reason="worktree divergence reconcile aborted", + container_id=overseer_container_id, + phase=current_phase, + ), + ) + break + + # After plan phase: populate contract with task structure. + # NOTE: worktree_repo_path is used for both draft reads and + # contract load/save inside _populate_contract_from_plan. + # The contract was created at worktree_repo_path above, so + # both operations must use the same path. + # Called on every successful plan completion (including after + # HITL revision) so the contract reflects the latest approved + # plan, not a previously rejected draft. + # + # Routed through _populate_contract_from_plan_safe so a raised + # exception here cannot skip the HITL gate below (#1890). The + # same helper is invoked from advance_phase so force-advances + # out of plan see the same populate step (#1941). + # + # ``source="plan_complete"`` makes the wrapper raise: + # * PlanDraftMissingOnLocalError — draft missing on local + # but present on origin (#2337 silent demotion). + # * PlanDraftMissingOnLocalAndOriginError — draft missing on BOTH local + # and origin (#2627 silent advance to empty contract). + # We catch either below and mark the pipeline FAILED so the + # operator can intervene rather than implement silently + # shipping slice-1 alone (#2337) or strand 8 agents on an + # empty contract (#2627). + pipeline, phase_overseer_active, _loopblk_action = _pkg._run_plan_advance( + pipeline, + phase_overseer_active, + current_phase=current_phase, + gateway_mode=gateway_mode, + overseer_container_id=overseer_container_id, + overseer_lock=overseer_lock, + pipeline_id=pipeline_id, + pipeline_mode=pipeline_mode, + repo_path=repo_path, + spawner=spawner, + store=store, + worktree_repo_path=worktree_repo_path, + ) + if _loopblk_action == "break": + break + + # After refine and plan phases: sync substantive HITL decisions + # (non-phase-gate) to the contract so implement-phase agents + # can see what was decided. Called for both refine and plan + # phases — refine decisions inform the plan, plan decisions + # inform the implementation. + if current_phase.value in _pkg._HITL_GATE_PHASES: + try: + _pkg._sync_pipeline_decisions_to_contract( + repo_path, + worktree_repo_path, + pipeline_id, + ) + except Exception as sync_err: + _pkg.logger.warning( + "Failed to sync pipeline decisions to contract (continuing)", + pipeline_id=pipeline_id, + phase=current_phase.value, + error=str(sync_err), + ) + + # Write BRC consensus history for this phase before committing + # statefiles so the history file is included in the commit. + try: + _pkg._write_brc_history( + worktree_repo_path, + pipeline_id, + current_phase.value, + _pkg._brc_history_identifier(pipeline), + # Per-slice implement-phase files are owned by each + # slice's integration branch; committing them onto + # ``work`` here would conflict with the slice + # branches' add of the same paths and break slice + # PR merges (#2755). The parameter is a no-op for + # non-implement phases. + write_per_slice=False, + ) + except Exception as brc_err: + _pkg.logger.debug( + "Failed to write BRC history (continuing)", + pipeline_id=pipeline_id, + phase=current_phase, + error=str(brc_err), + ) + + # Commit any .egg-state/ files produced during this phase + # (drafts, reviews, check results, contract updates). Mirrors + # the GHA workflow's `git add .egg-state/` at phase boundaries. + try: + _pkg._commit_statefiles_to_worktree( + worktree_repo_path, + f"Persist statefiles after {current_phase.value} phase", + pipeline_identifier=_pkg._pipeline_identifier( + pipeline.issue_number, pipeline_id + ), + pipeline_id=pipeline_id, + ) + except Exception as git_err: + # Catch broadly: the helper does ``subprocess.run(check=True, + # timeout=30)`` which can raise ``TimeoutExpired`` (not a + # CalledProcessError) and ``glob.glob`` which can raise + # ``OSError``. A narrow ``except`` here let either escape + # to the outer handler and stranded the pipeline (#2219). + _pkg.logger.warning( + "Failed to commit statefiles after phase (continuing)", + pipeline_id=pipeline_id, + phase=current_phase, + error=str(git_err), + ) + + # Push statefiles to remote so the next phase's agents + # don't have unpushed .egg-state/ files in their diff. + if pipeline.branch and worktree_repo_path != repo_path: + try: + spawner.gateway.push_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(worktree_repo_path), + branch=pipeline.branch, + mode=gateway_mode, + base_branch=pipeline.base_branch, + ) + except Exception as push_err: + _pkg.logger.warning( + "Failed to push statefiles after phase (continuing)", + pipeline_id=pipeline_id, + phase=current_phase, + error=str(push_err), + ) + + # --- Unresolved-gap gate (#3300) --- + # Block finalize while the contract carries an unresolved + # tester→coder TaskGap. Runs after the worktree sync above so + # the contract reflects the agents' final writes, and BEFORE + # the phase_gate / advance / finalize below so the gap can't + # ship into the committed contract (which would fail + # test_models_gaps.py red in CI on the already-open PR — + # #3298 class 4). Scoped to IMPLEMENT, where gaps are written; + # no-ops on a clean contract. On a fully-autonomous pipeline + # (hitl_gates=False) the gate surfaces the escalation but does + # not block — both options need a human, so blocking would + # stall the pipeline indefinitely; the reactive CI check stays + # the backstop there. + pipeline = _pkg._run_implement_advance( + pipeline, + current_phase=current_phase, + gateway_mode=gateway_mode, + pipeline_id=pipeline_id, + repo_path=repo_path, + spawner=spawner, + store=store, + worktree_repo_path=worktree_repo_path, + ) + + # --- HITL gate: pause for human approval --- + # Refine/plan are gated by the converge-before-advance loop + # (#3392): it resolves decisions with a human each round, which is + # what lets us drop the force-advance backstop — a human is present + # to resolve and approve. + # + # But a fully-autonomous pipeline (``hitl_gates is False``) has no + # human to resolve or approve, and ``wait_for_decision`` polls + # indefinitely — so unconditionally gating here would convert that + # explicitly-chosen, first-class config into an indefinite hang + # with no operator-facing signal that the flag was ignored. Mirror + # the unresolved-gap gate's autonomous escape (#3300): when + # ``hitl_gates is False`` we *surface* the gate (event + loud + # warning) but do not block, advancing autonomously instead. + # ``hitl_gates`` therefore still governs refine/plan, but only by + # toggling between the human-gated converge loop and an autonomous + # advance — never an indefinite stall. + pipeline, _hitl_gate_action = _pkg._run_hitl_gate_converge( + pipeline, + current_phase=current_phase, + gateway_mode=gateway_mode, + pipeline_id=pipeline_id, + repo_path=repo_path, + spawner=spawner, + store=store, + worktree_repo_path=worktree_repo_path, + ) + if _hitl_gate_action == "continue": + continue + + # ---------------------------------------------------------- + # #2777 (cq-4, TASK-1-2) — inline ``_run_pipeline`` + # auto-advance plan→implement transition. Calls the new + # idempotent ``_open_context_pr_at_implement_start`` + # opener directly; auto-advance does NOT route through + # ``routes/phases.py:advance_phase``, so without this call + # site a natural plan-exit (no operator REST call) would + # never get a context PR opened, leaving the slice stack + # stranded on ``egg/<id>/work`` (the #2593 / #2769 + # symptom). reviewer_code_holistic blocker 1 fix: + # restored after v1's incorrect "single canonical site" + # deletion. The opener's ``gh pr list`` pre-flight makes + # a redundant call from any other transition path a one- + # round-trip no-op. + # ---------------------------------------------------------- + if current_phase.value == "plan": + try: + _pkg._open_context_pr_at_implement_start(pipeline_id, repo_path=repo_path) + except _pkg.ContextPrCreationError as ctx_err: + _pkg.logger.warning( + "Context PR opener: _run_pipeline auto-advance " + "failed (continuing — hard-require enforced at " + "advance_phase and the implement-start plan " + "pre-flight gate) (#2777, #3100)", + pipeline_id=pipeline_id, + reason=ctx_err.reason, + error=str(ctx_err), + ) + except Exception as autoadvance_err: # noqa: BLE001 + _pkg.logger.warning( + "Context PR opener: _run_pipeline auto-advance " + "outer wrapper raised (continuing) (#2777)", + pipeline_id=pipeline_id, + error=str(autoadvance_err), + ) + + # Tear down the phase-scoped overseer before advancing. + # Each phase gets a fresh overseer instance — no state carries + # over between phases. + # Hold the lock to prevent the poll thread from seeing the + # container as EXITED and respawning it. + with overseer_lock: + if overseer_container_id and phase_overseer_active: + phase_overseer_active = False + _pkg._teardown_phase_overseer( + spawner, + overseer_container_id, + pipeline_id, + phase_label=current_phase.value, + reason="phase ended", + ) + + # Determine next phase. Issue #1557: epic-mode pipelines + # route through the new APPLY phase between PLAN and + # IMPLEMENT so the APPLIER role can drive Jira mutations on + # HITL approval. ``_next_phases_for_epic`` returns + # ``transitions.get(current_phase, [])`` unchanged for + # non-epic pipelines so the pre-#1557 scheduling is + # preserved bit-for-bit. + next_phases = _pkg._next_phases_for_epic( + pipeline, + current_phase, + transitions.get(current_phase, []), + ) + + if not next_phases: + # Terminal phase — pipeline complete + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + pipeline.status = _pkg.PipelineStatus.COMPLETE + store.save_pipeline(pipeline) + + # Report pipeline completion to collaborator + _pkg.report_pipeline_status( + pipeline, + event_type="pipeline.completed", + message="Pipeline completed successfully", + ) + _pkg._emit_pipeline_event(pipeline, "pipeline.completed") + _pkg.logger.info( + "Pipeline complete", + pipeline_id=pipeline_id, + ) + break + + # TEST_MARKER: auto_advance_block (load-bearing: brackets the + # block for TestAutoAdvanceRespawnsThread; do not remove without + # updating that test class). + # Advance to next phase by respawning a fresh _run_pipeline + # thread, mirroring advance_phase (#2165). Bumping run_epoch + # makes this thread's finally cleanup detect itself as superseded + # and skip worktree teardown; the new thread drives the next + # phase from clean local state. Without this, any exception in + # the new phase's first iteration takes the whole pipeline down. + next_phase = next_phases[0] + + # Issue #1557: when the just-completed phase is PLAN and the + # pipeline is_epic, we are advancing into APPLY. Write the + # applier handoff JSON now (before respawning the driver + # thread) so the APPLIER container can read it on its + # first wakeup. ``approved_phase='plan'`` so the applier + # drives plan-apply (Task.jira_action walk → child create / + # edit / link, Won't-Do handoff for the orchestrator drain). + if ( + getattr(pipeline, "is_epic", False) + and current_phase == _pkg.PipelinePhase.PLAN + and next_phase == _pkg.PipelinePhase.APPLY + ): + _pkg._write_apply_phase_handoff( + pipeline, + worktree_repo_path, + approved_phase="plan", + ) + + # Issue #1557 task-2-7: when the just-completed phase is + # APPLY (BRC consensus confirmed), drain the Won't-Do + # handoff JSON before advancing to IMPLEMENT. The drain + # runs out-of-band from the HITL approve POST so a slow + # Jira API never extends that handler's latency. + if current_phase == _pkg.PipelinePhase.APPLY: + _pkg._drain_wontdo_batch_after_apply(pipeline, worktree_repo_path) + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + pipeline.current_phase = next_phase + pipeline.run_epoch = _pkg.datetime.now(_pkg.UTC) + # ``updated_at`` is unconditionally set by ``StateStore.save_pipeline``. + store.save_pipeline(pipeline) + + # Drop the previous phase's in-memory consensus tracker and + # message-store entries (#2502). The other phase-transition + # paths -- ``advance_phase`` REST handler, HITL-revision + # re-run, and the ``recover_pipeline`` resume path -- all + # call this; the auto-advance path used to skip it, leaving + # a stale plan-phase tracker keyed under the bare + # ``pipeline_id`` for ``_get_concurrent_status`` to find and + # report as ``is_complete: True`` long after the implement + # phase had started. ``_write_brc_history`` runs at the + # bottom of each phase iteration with + # ``write_per_slice=False`` (see #2755), so per-slice + # implement-phase transcripts are on the slice integration + # branches, and the work commit picks up only the + # unattributed sibling plus whatever aggregate the writer + # still emits — refine/plan/pr aggregates, and the + # non-slice-implement aggregate that any implement-phase + # run without slice scope lands on work via the ``not + # buckets`` branch — before we wipe the message store here. + from routes.phases import _clear_concurrent_state + + _clear_concurrent_state(pipeline_id) + + _pkg.logger.info( + "Phase advanced (auto), respawning driver thread", + pipeline_id=pipeline_id, + from_phase=current_phase.value, + to_phase=next_phase.value, + ) + + _pkg._spawn_pipeline_run_thread(pipeline_id, repo_path, pipeline.run_epoch) + return + + except _pkg.PipelineNotFoundError as pnf_err: + # `PipelineNotFoundError` can be raised either because the pipeline + # was actually deleted or because of a transient state-store read + # (e.g., empty content while a concurrent commit on the state + # worktree races with the read). Re-verify before treating it as + # deletion: if the pipeline is still on disk after retry, the + # original exception was spurious — bump ``run_epoch`` so the + # finally cleanup detects this thread as superseded and skips the + # destructive worktree teardown, then relaunch ``_run_pipeline`` so + # the next phase keeps making progress. See #2155. + pipeline_still_exists = False + _verify_store = None + try: + _verify_store = _pkg.get_state_store(repo_path) + except Exception as verify_store_err: + # Couldn't even open the state store — treat as transient + # (corrupt-but-present > deletion) so we skip the respawn + # rather than amplifying an infrastructure blip. Note: with + # ``_verify_store=None`` the bump path below short-circuits, + # so worktree preservation depends on whether ``run_epoch`` + # was set before the initial PNFE — this path avoids the + # cascade but does not unconditionally preserve worktrees. + _pkg.logger.warning( + "Failed to obtain state store after PipelineNotFoundError; " + "treating as transient infrastructure failure and skipping respawn", + pipeline_id=pipeline_id, + error=str(verify_store_err), + ) + pipeline_still_exists = True + + if _verify_store is not None: + for _attempt in range(_pkg._PNFE_VERIFY_ATTEMPTS): + _pkg.time.sleep(_pkg._PNFE_VERIFY_INTERVAL) + try: + _verify_store.load_pipeline(pipeline_id) + pipeline_still_exists = True + break + except _pkg.PipelineNotFoundError: + continue + except _pkg.StateValidationError: + # Corrupt JSON or schema mismatch means the file + # exists but is unreadable right now — that's not + # deletion. Treat as transient: better to risk a + # wasted respawn than to nuke the worktrees on a + # transient corruption. + pipeline_still_exists = True + break + except _pkg.StateStoreError as verify_err: + # Other state-store failures (transient git read + # errors, etc.) are also not evidence of deletion. + _pkg.logger.warning( + "State-store error verifying pipeline existence; " + "treating as transient and preserving worktrees", + pipeline_id=pipeline_id, + error=str(verify_err), + ) + pipeline_still_exists = True + break + + if pipeline_still_exists: + # Cap the respawn cascade so a persistent transient can't + # leak threads, overseer containers, and state-branch + # commits without bound. The recovery code is what runs + # exactly when the system is misbehaving — it must not + # amplify the misbehaviour. + if _respawn_attempt >= _pkg._PNFE_RESPAWN_MAX_ATTEMPTS: + _pkg.logger.error( + "Spurious-PipelineNotFoundError recovery exhausted " + "respawn budget; marking pipeline FAILED so an " + "operator can investigate via restart_phase", + pipeline_id=pipeline_id, + attempts=_respawn_attempt, + exc_info=pnf_err, + ) + if _verify_store is not None: + try: + with _pkg.get_pipeline_state_lock(pipeline_id): + _failed_pipeline = _verify_store.load_pipeline(pipeline_id) + _failed_pipeline.status = _pkg.PipelineStatus.FAILED + _failed_pipeline.error = ( + "Transient PipelineNotFoundError recovery " + f"exhausted after {_respawn_attempt} respawns" + ) + _verify_store.save_pipeline(_failed_pipeline) + except Exception as fail_err: + _pkg.logger.warning( + "Failed to mark pipeline FAILED after exhausting respawn budget", + pipeline_id=pipeline_id, + error=str(fail_err), + ) + else: + # Recoverable transient — log at warning so it doesn't + # trip error-rate dashboards every time it self-heals. + _pkg.logger.warning( + "Spurious PipelineNotFoundError during execution — " + "pipeline still exists after retry; relaunching driver " + "thread and preserving worktrees", + pipeline_id=pipeline_id, + attempt=_respawn_attempt, + exc_info=pnf_err, + ) + # Bump run_epoch so the finally cleanup observes this + # thread as superseded (mirrors the advance_phase + # pattern) and skips worktree teardown. Capture the + # pre-bump epoch into the local ``run_epoch`` so the + # finally guard works even when the *initial* load + # raised PNFE (in that case run_epoch was never set + # at line 11393). + bump_succeeded = False + if _verify_store is not None: + try: + with _pkg.get_pipeline_state_lock(pipeline_id): + _bumped = _verify_store.load_pipeline(pipeline_id) + run_epoch = _bumped.run_epoch or _bumped.created_at + _bumped.run_epoch = _pkg.datetime.now(_pkg.UTC) + _verify_store.save_pipeline(_bumped) + bump_succeeded = True + except Exception as bump_err: + _pkg.logger.warning( + "Failed to bump run_epoch during spurious-PNFE " + "recovery; skipping respawn so the existing " + "finally cleanup runs without racing a new thread", + pipeline_id=pipeline_id, + error=str(bump_err), + ) + + if bump_succeeded: + # Exponential backoff between respawn attempts so a + # tight cascade can't fire dozens of respawns per + # second. attempt=0 → 1s, 1 → 2s, 2 → 4s, 3 → 8s, + # 4 → 16s, capped at _PNFE_RESPAWN_BACKOFF_CAP. + _backoff = min(2**_respawn_attempt, _pkg._PNFE_RESPAWN_BACKOFF_CAP) + _pkg.time.sleep(_backoff) + _pkg.threading.Thread( + target=_pkg._run_pipeline, + args=(pipeline_id, repo_path), + kwargs={"_respawn_attempt": _respawn_attempt + 1}, + daemon=True, + name=( + f"pipeline-{pipeline_id}-respawn-" + f"{_respawn_attempt + 1}-{_pkg.time.monotonic_ns()}" + ), + ).start() + else: + _pkg.logger.info( + "Pipeline was deleted during execution, exiting", + pipeline_id=pipeline_id, + exc_info=pnf_err, + ) + except Exception as e: + _pkg.logger.error( + "Pipeline execution failed", pipeline_id=pipeline_id, error=str(e), exc_info=True + ) + persisted_ok = False + try: + store = _pkg.get_state_store(repo_path) + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + + # Don't corrupt a recreated pipeline's state + _fail_epoch = pipeline.run_epoch or pipeline.created_at + if run_epoch and _fail_epoch != run_epoch: + _pkg.logger.info( + "Pipeline was recreated, not marking new run as failed", + pipeline_id=pipeline_id, + ) + else: + pipeline.status = _pkg.PipelineStatus.FAILED + pipeline.error = str(e) + store.save_pipeline(pipeline) + persisted_ok = True + + # Report pipeline failure to collaborator + _pkg.report_pipeline_status( + pipeline, + event_type="pipeline.failed", + message=f"Pipeline failed: {str(e)[:100]}", + ) + _pkg._emit_pipeline_event(pipeline, "pipeline.failed") + except Exception as fail_err: + # If FAILED-marking itself fails (state-store contention, lock + # timeout, etc.), the pipeline stays at ``running`` with no + # error recorded — exactly the silent-wedge symptom in #2219. + # Log so the next occurrence is visible in the orchestrator + # log instead of vanishing. + _pkg.logger.error( + "Failed to mark pipeline FAILED after exception", + pipeline_id=pipeline_id, + original_error=str(e), + mark_error=str(fail_err), + exc_info=True, + ) + # Surface a synthetic ``pipeline.failed`` to the EventBus even + # though the mark-FAILED block raised. Without this, hosts + # blocked on ``/status/wait`` (whose event allowlist requires + # pipeline.failed/completed/cancelled) wait forever on a dead + # runner — the zombie symptom in #2234. ``persisted`` carries + # whether ``save_pipeline`` actually flushed FAILED to disk + # before the inner block raised: True means disk state matches + # the event, False means consumers should treat the event as + # the only authoritative source. + if _pkg._emit_event is not None: + try: + _pkg._emit_event( + _pkg.EventType.PIPELINE_FAILED, + pipeline_id, + data={ + "status": _pkg.PipelineStatus.FAILED.value, + "persisted": persisted_ok, + "original_error": str(e), + "mark_error": str(fail_err), + }, + ) + except Exception as emit_err: + _pkg.logger.warning( + "Failed to emit synthetic pipeline.failed event", + pipeline_id=pipeline_id, + error=str(emit_err), + ) + finally: + # Stop health monitor polling and unsubscribe from events + if health_monitor_timer is not None: + health_monitor_timer.set() + if poll_thread is not None: + poll_thread.join(timeout=5) + if health_monitor_instance is not None: + try: + health_monitor_instance.stop() + _pkg.logger.info("Health monitor stopped", pipeline_id=pipeline_id) + except Exception as hm_stop_err: + _pkg.logger.debug( + "Failed to stop health monitor", + pipeline_id=pipeline_id, + error=str(hm_stop_err), + ) + + # Clean up progress store for this pipeline + try: + from progress_store import get_progress_store + + progress_store = get_progress_store() + if progress_store is not None: + progress_store.clear(pipeline_id) + except Exception as ps_err: + _pkg.logger.debug( + "Failed to clear progress store", + pipeline_id=pipeline_id, + error=str(ps_err), + ) + + # Stop overseer container if it was spawned + if overseer_container_id: + try: + _spawner = _pkg._get_spawner() + _spawner.stop_agent_job( + overseer_container_id, + cleanup_session=True, + timeout=10, + ) + _pkg.logger.info( + "Overseer container stopped", + pipeline_id=pipeline_id, + container_id=overseer_container_id[:12], + ) + except Exception as overseer_err: + _pkg.logger.debug( + "Failed to stop overseer container (may have already exited)", + pipeline_id=pipeline_id, + error=str(overseer_err), + ) + + # Clean up pipeline-level worktrees unless the pipeline has been + # recreated (delete + create with the same ID). In that case the + # new run owns the worktrees and we must not remove them. + try: + _spawner = _pkg._get_spawner() + _store = _pkg.get_state_store(repo_path) + skip_cleanup = False + pipeline_was_restarted = False + try: + current = _store.load_pipeline(pipeline_id) + _cleanup_epoch = current.run_epoch or current.created_at + if run_epoch and _cleanup_epoch != run_epoch: + skip_cleanup = True + pipeline_was_restarted = True + _pkg.logger.info( + "Pipeline was recreated/restarted, skipping worktree cleanup", + pipeline_id=pipeline_id, + old_epoch=run_epoch.isoformat(), + new_epoch=_cleanup_epoch.isoformat(), + ) + elif current.status == _pkg.PipelineStatus.FAILED: + skip_cleanup = True + _pkg.logger.info( + "Pipeline failed, preserving worktrees for retry", + pipeline_id=pipeline_id, + ) + except Exception: + # Pipeline was deleted and not recreated — safe to clean up + pass + + if not skip_cleanup: + try: + _spawner.gateway.delete_worktrees( + container_id=pipeline_id, + force=True, + ) + _pkg.logger.info("Pipeline worktrees cleaned up", pipeline_id=pipeline_id) + except Exception as pipeline_wt_err: + _pkg.logger.warning( + "Failed to clean up pipeline worktrees", + pipeline_id=pipeline_id, + error=str(pipeline_wt_err), + ) + + # Also clean up per-agent session worktrees. Each agent + # registers a gateway session under container_id + # "egg-{pipeline_id}-{role}" and session_create creates a + # worktree keyed to that name. The per-agent cleanup path + # calls delete_session_by_container with the Docker container + # hash (not the session container_id), so those worktrees are + # never removed via the normal per-container cleanup. Sweep + # them here as a safety net. delete_worktrees is a no-op for + # container IDs that have no worktree directory. + # + # NOTE: This uses the "egg-{pipeline_id}-{role}" naming for + # session-created worktrees. Per-agent worktrees from #1481 + # use "{pipeline_id}-{role}" (no "egg-" prefix) and are + # cleaned up by cleanup_pipeline() which scans both container + # labels and the filesystem. + for role in _pkg.AgentRole: + agent_container_id = f"egg-{pipeline_id}-{role.value}" + try: + _spawner.gateway.delete_worktrees( + container_id=agent_container_id, + force=True, + ) + except Exception as agent_wt_err: + _pkg.logger.warning( + "Failed to clean up agent worktrees", + pipeline_id=pipeline_id, + agent_container_id=agent_container_id, + error=str(agent_wt_err), + ) + + except Exception as wt_err: + _pkg.logger.warning( + "Failed to clean up worktrees", + pipeline_id=pipeline_id, + error=str(wt_err), + ) + + # Safety-net: clean up any orphaned containers for this pipeline. + # If the pipeline failed during startup or cleanup timed out, Docker + # containers may persist. This is a no-op when no containers exist. + # Skip when the pipeline was restarted (run_epoch changed) so the + # new thread's containers are not killed. See #1386, #1638. + if not pipeline_was_restarted: + try: + # ``gateway_mode`` is the mode this pipeline ran under; + # the auto-salvage hook needs it to push recovery refs + # under the same policy (#2429 review). + removed = _spawner.cleanup_pipeline( + pipeline_id, + force=True, + preserve_worktrees=skip_cleanup, + salvage_mode=gateway_mode, + salvage_base_branch=pipeline.base_branch, + ) + if removed > 0: + _pkg.logger.info( + "Safety-net cleanup removed orphaned containers", + pipeline_id=pipeline_id, + containers_removed=removed, + ) + except Exception as cleanup_err: + _pkg.logger.warning( + "Safety-net container cleanup failed", + pipeline_id=pipeline_id, + error=str(cleanup_err), + ) diff --git a/orchestrator/routes/pipelines/_run_pipeline_setup.py b/orchestrator/routes/pipelines/_run_pipeline_setup.py new file mode 100644 index 0000000000..90a07e449a --- /dev/null +++ b/orchestrator/routes/pipelines/_run_pipeline_setup.py @@ -0,0 +1,727 @@ +"""run_pipeline setup-block helpers helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _sync_contract_setup( + pipeline, + *, + gateway_mode, + pipeline_id, + pipeline_mode, + repo_path, + source_branch_for_contract_pull, + spawner, + store, + worktree_repo_path, +): + """Create/sync the worktree companion contract (extracted verbatim from + _run_pipeline). Returns (pipeline, done); done=True means the caller must + return immediately (a contract-setup failure already marked the pipeline).""" + if not pipeline.contract_synced: + try: + from egg_contracts.loader import compose_task_description, create_contract + + # Every entry path (GitHub issue, JIRA, free-text) anchors + # the task the same way (#3163): identity first, then the + # operator's submit description. Before #3163 issue + # pipelines deliberately got ``None`` here (#3042 "agents + # fetch the live body"), which left the #3123 binding + # prompt section empty for the most common pipeline type. + issue_url = ( + f"https://github.com/{pipeline.repo}/issues/{pipeline.issue_number}" + if pipeline.issue_number is not None + else None + ) + task_description = compose_task_description( + description=pipeline.prompt, + issue_number=pipeline.issue_number, + issue_url=issue_url, + jira_ticket=pipeline.jira_ticket, + ) + + # When source_branch is set, try to carry over the contract + # (with any resolved HITL decisions) from there instead of + # overwriting with a fresh zero-state contract (#2035). + pulled_contract = False + if source_branch_for_contract_pull: + try: + pulled_contract = _pkg._pull_contract_from_source_branch( + repo_path=worktree_repo_path, + source_branch=source_branch_for_contract_pull, + issue_number=pipeline.issue_number, + pipeline_id=pipeline.id, + spawner=spawner, + gateway_mode=gateway_mode, + task_description=task_description, + ) + except Exception: + _pkg.logger.warning( + "Unexpected error pulling contract from source branch — falling back to fresh contract", + pipeline_id=pipeline_id, + source_branch=source_branch_for_contract_pull, + exc_info=True, + ) + pulled_contract = False + + if not pulled_contract: + if pipeline.issue_number is not None: + create_contract( + issue_number=pipeline.issue_number, + title=f"Issue #{pipeline.issue_number}", + url=issue_url or "", + pipeline_id=pipeline.id, + repo_root=worktree_repo_path, + task_description=task_description, + ) + else: + # ``pipeline.issue_number is None`` covers both + # free-text submits and JIRA-driven pipelines + # (``pipeline.jira_ticket`` set). The event-pump + # never delivers the orchestrator-built spawn + # prompt to the agent, so the contract (read via + # ``egg-contract show`` + the #3123 prompt + # section) is the reliable channel for the + # complete task; the ``title`` arg is only used + # for the ``IssueInfo`` label and is dropped + # without an ``issue_number``, so it is not a + # substitute (#3033). + create_contract( + pipeline_id=pipeline.id, + title=(pipeline.prompt or "")[:100], + task_description=task_description, + repo_root=worktree_repo_path, + ) + + # Write pre-generated drafts for short-flow pipelines so the + # existing plan parser can populate the contract with tasks. + if pipeline.analysis or pipeline.plan: + drafts_dir = worktree_repo_path / ".egg-state" / "drafts" + drafts_dir.mkdir(parents=True, exist_ok=True) + + if pipeline.analysis: + analysis_rel = _pkg._get_draft_path( + "refine", + issue_number=pipeline.issue_number, + pipeline_id=pipeline_id, + ) + if analysis_rel: + (worktree_repo_path / analysis_rel).write_text( + pipeline.analysis, encoding="utf-8" + ) + _pkg.logger.info( + "Wrote pre-generated analysis draft", + pipeline_id=pipeline_id, + path=analysis_rel, + ) + + if pipeline.plan: + plan_rel = _pkg._get_draft_path( + "plan", + issue_number=pipeline.issue_number, + pipeline_id=pipeline_id, + ) + if plan_rel: + (worktree_repo_path / plan_rel).write_text(pipeline.plan, encoding="utf-8") + _pkg.logger.info( + "Wrote pre-generated plan draft", + pipeline_id=pipeline_id, + path=plan_rel, + ) + + # Populate the contract from the plan's yaml-tasks appendix + _inline_plan_populate_result = _pkg._populate_contract_from_plan( + worktree_repo_path, + pipeline_id, + pipeline_mode, + pipeline.issue_number, + ) + # #2627 follow-up: warn-and-continue on non-POPULATED. + # This is the initial-contract creation path (a + # pre-generated plan handed to ``start_pipeline``); + # failing here would block legitimate pipelines that + # recover via the natural plan-phase populator a few + # blocks later. We only attach the structured + # outcome as audit signal. + if _inline_plan_populate_result.outcome != _pkg.PopulateOutcome.POPULATED: + _pkg.logger.warning( + "Pre-generated plan populate produced non-POPULATED outcome", + pipeline_id=pipeline_id, + outcome=_inline_plan_populate_result.outcome.value, + ) + + # Commit all .egg-state/ files so they're on the feature branch + issue_ref = ( + f"issue #{pipeline.issue_number}" + if pipeline.issue_number is not None + else f"pipeline {pipeline_id}" + ) + try: + _pkg._commit_statefiles_to_worktree( + worktree_repo_path, + f"Initialize SDLC contract for {issue_ref}", + pipeline_identifier=_pkg._pipeline_identifier( + pipeline.issue_number, pipeline_id + ), + pipeline_id=pipeline_id, + ) + except Exception as git_err: + # Catch broadly so TimeoutExpired/OSError also produce + # an explicit FAILED state rather than silently + # propagating to the outer handler (#2219). + _pkg.logger.error( + "Failed to commit initial statefiles — aborting pipeline", + pipeline_id=pipeline_id, + error=str(git_err), + ) + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + pipeline.status = _pkg.PipelineStatus.FAILED + pipeline.contract_synced = False + pipeline.error = f"Failed to commit initial statefiles: {git_err}" + store.save_pipeline(pipeline) + return pipeline, True + + # Push contract statefiles to remote so agents see them. + # This MUST succeed before agents start — otherwise agents' + # diffs will include .egg-state/ files they can't push (#1431). + push_succeeded = False + # For prompt-driven pipelines, pipeline.branch is None at this + # point — the branch name is only persisted later when the + # agent container is spawned (line ~6279). Derive it here so + # the push actually happens. The worktree was already created + # on this branch by the gateway. + push_branch = pipeline.branch or f"egg/{pipeline_id}/work" + if not pipeline.branch: + pipeline.branch = push_branch + with _pkg.get_pipeline_state_lock(pipeline_id): + p = store.load_pipeline(pipeline_id) + if not p.branch: + p.branch = push_branch + store.save_pipeline(p) + _pkg.logger.info( + "Recorded generated branch on pipeline (pre-push)", + pipeline_id=pipeline_id, + branch=push_branch, + ) + if worktree_repo_path != repo_path: + push_err_msg = "" + # push_worktree_branch reconciles non-fast-forward + # rejections internally (fetch+rebase+retry), so a + # single call is sufficient — no outer retry needed. + try: + push_result = spawner.gateway.push_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(worktree_repo_path), + branch=push_branch, + mode=gateway_mode, + base_branch=pipeline.base_branch, + ) + push_succeeded = bool(push_result) + if not push_succeeded: + push_err_msg = push_result.describe() + except Exception as push_err: + push_succeeded = False + push_err_msg = str(push_err) + + if not push_succeeded: + _pkg.logger.error( + "Contract init push failed after retry — aborting pipeline", + pipeline_id=pipeline_id, + error=push_err_msg, + ) + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + pipeline.status = _pkg.PipelineStatus.FAILED + pipeline.contract_synced = False + pipeline.error = f"Failed to push contract init to remote: {push_err_msg}" + store.save_pipeline(pipeline) + return pipeline, True + else: + _pkg.logger.warning( + "Skipped contract init push — worktree path equals repo path", + pipeline_id=pipeline_id, + worktree_repo_path=str(worktree_repo_path), + repo_path=str(repo_path), + ) + + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + pipeline.contract_synced = push_succeeded + store.save_pipeline(pipeline, commit=False) + _pkg.logger.info( + "Pipeline contract created in worktree", + pipeline_id=pipeline_id, + mode=pipeline_mode, + ) + except Exception as contract_err: + _pkg.logger.error( + "Failed to create contract in worktree", + pipeline_id=pipeline_id, + error=str(contract_err), + ) + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + pipeline.status = _pkg.PipelineStatus.FAILED + pipeline.error = f"Failed to create contract: {contract_err}" + store.save_pipeline(pipeline) + return pipeline, True + return pipeline, False + + +def _map_host_repos( + pipeline, + *, + host_gid, + host_repo_map, + host_uid, + pipeline_id, + pipeline_repos, + spawner, + worktree_id, + repo_volumes, + worktree_repo_path, +): + """Map host repos -> container volumes + resolve the worktree repo path + (extracted verbatim from _run_pipeline). repo_volumes/worktree_repo_path + come in with their defaults and are returned possibly-updated.""" + if host_repo_map: + try: + # Request repos in owner/repo format if available, else bare names + wt_repos = pipeline_repos if pipeline_repos else list(host_repo_map.keys()) + # When the pipeline specifies a base_branch, pass it through + # so the worktree is branched from that ref instead of the + # repo's default branch. Otherwise let the gateway resolve + # the remote default branch per-repo (see #860). + # Retry worktree creation on transient gateway errors + # (e.g., 500s from concurrent pipeline starts contending + # on per-repo locks). See #1386. + wt_max_attempts = 3 + wt_backoff = 2.0 + wt_result = None + for wt_attempt in range(1, wt_max_attempts + 1): + try: + wt_result = spawner.gateway.create_worktrees( + container_id=worktree_id, + repos=wt_repos, + uid=host_uid, + gid=host_gid, + base_branch=pipeline.base_branch, + ) + break # Success — exit retry loop + except _pkg.GatewayError as gw_err: + is_transient = gw_err.status_code is None or gw_err.status_code >= 500 + if not is_transient or wt_attempt == wt_max_attempts: + # Surface gw_err.details so per-repo failures + # captured by the gateway aren't dropped. See + # #2186. + _pkg.logger.error( + "Worktree creation failed permanently", + pipeline_id=pipeline_id, + attempts=wt_attempt, + status_code=gw_err.status_code, + error_message=gw_err.message, + details=gw_err.details, + ) + detail_suffix = f" (details: {gw_err.details})" if gw_err.details else "" + raise RuntimeError( + f"Failed to create worktrees for pipeline {pipeline_id} " + f"after {wt_max_attempts} attempts: " + f"{gw_err.message}{detail_suffix}" + ) from gw_err + _pkg.logger.warning( + "Worktree creation failed, retrying", + pipeline_id=pipeline_id, + attempt=wt_attempt, + max_attempts=wt_max_attempts, + error=str(gw_err), + details=gw_err.details, + ) + _pkg.time.sleep(wt_backoff) + wt_backoff *= 2 + + if wt_result and wt_result.success and wt_result.worktrees: + # Gateway returns worktrees keyed by the full ``owner/repo`` + # slug (#3393 slice-3, operator ruling #6). The on-disk + # worktree directory (and the container mount target) is + # still the bare repo name at /home/egg/repos/<name>, so + # the path reconstruction below strips the owner prefix + # from each key. + repo_volumes = wt_result.worktrees + + # Derive the orchestrator-accessible worktree path. + # Reviewer containers write verdict/draft/check files into + # the worktree, so the orchestrator must read from there. + # Match against pipeline.repo (full owner/repo slug, which + # is now the map key) explicitly to avoid picking the wrong + # repo in multi-repo pipelines. + matched = False + if pipeline.repo and pipeline.repo in wt_result.worktrees: + repo_short = pipeline.repo.split("/")[-1] + candidate = _pkg.WORKTREE_BASE_DIR / worktree_id / repo_short + if candidate.exists(): + worktree_repo_path = candidate + matched = True + if not matched: + # Fallback: take the first existing worktree path. + # Keys are ``owner/repo``; the on-disk dir is the bare + # leaf, so strip the owner prefix before joining. + for owner_repo in wt_result.worktrees: + candidate = _pkg.WORKTREE_BASE_DIR / worktree_id / owner_repo.split("/")[-1] + if candidate.exists(): + worktree_repo_path = candidate + break + + _pkg.logger.info( + "Worktrees created for pipeline", + pipeline_id=pipeline_id, + worktrees=list(repo_volumes.keys()), + ) + else: + raise RuntimeError( + f"Worktree creation returned no worktrees for pipeline {pipeline_id}: " + f"errors={wt_result.errors}" + ) + + if wt_result.errors: + for err in wt_result.errors: + _pkg.logger.warning("Worktree error", pipeline_id=pipeline_id, error=err) + + except RuntimeError: + raise # Re-raise our own RuntimeError + except Exception as wt_err: + raise RuntimeError( + f"Failed to create worktrees for pipeline {pipeline_id}: {wt_err}" + ) from wt_err + return repo_volumes, worktree_repo_path + + +def _start_phase_setup(pipeline, *, pipeline_id, pipeline_mode, store, worktree_repo_path): + """start_phase==implement safety-net: populate the contract / apply the + plan draft before the implement phase spawns (extracted verbatim from + _run_pipeline). Returns (pipeline, done); done=True -> caller returns.""" + if pipeline.config.start_phase == "implement": + plan_draft_rel = _pkg._get_draft_path( + "plan", + issue_number=pipeline.issue_number, + pipeline_id=pipeline.id, + ) + if plan_draft_rel and (worktree_repo_path / plan_draft_rel).exists(): + # Advance contract.current_phase alongside slice/PR + # ingestion. In the natural flow contract.current_phase + # is mutated by the plan reviewer agent (or the gateway + # phase API) via apply_mutation; with start_phase=implement + # no such reviewer ever runs, so the contract would stay + # at REFINE forever (#2427 sub-bug). We pass + # pipeline.current_phase rather than a hardcoded literal + # so the right value follows automatically if start_phase + # ever supports values other than 'implement'. The + # populator enforces forward-only advancement, so a + # respawn during the PR phase cannot demote the contract. + # Note: the *outer* guard above remains hardcoded to + # ``"implement"``; widening it to other start_phase values + # is a two-line change (this guard plus the matching + # ``initial_phase`` mapping in start_pipeline). + # Catch ``ForestValidationError`` here so a malformed + # plan landing at the safety-net path lands on the + # dedicated empty-contract HITL — the same recovery + # surface the natural plan-complete path uses via + # :func:`_populate_contract_from_plan_safe`'s + # forest-violation translation. Without this catch the + # safety net (which calls the inner directly so the + # ``PlanDraftMissing*`` raises don't fire here) would + # propagate the exception to the outer pipeline + # ``except`` and the operator would see a generic + # ``status: failed`` instead of the actionable + # repopulate/restart-plan/abort decision (#2627 review). + try: + _safety_net_populate_result = _pkg._populate_contract_from_plan( + worktree_repo_path, + pipeline_id, + pipeline_mode, + pipeline.issue_number, + current_phase=pipeline.current_phase, + ) + except _pkg.ForestValidationError as forest_err: + # #3046 — overlap violations map to their own outcome so + # the empty-contract HITL prose matches the discriminator. + _pkg.logger.warning( + "contract_phases_ingest_failed", + pipeline_id=pipeline_id, + reason=forest_err.reason, + source="safety_net", + errors=forest_err.errors, + ) + _safety_net_populate_result = _pkg.PopulateResult( + _pkg._forest_error_to_outcome(forest_err) + ) + # #2627 follow-up: fail-fast whenever the safety-net populate + # did not produce a contract with tasks. Without this guard + # the implement phase spawns into the same empty-contract + # state that #2627 surfaced — the slice-gate at + # implement-phase entry would eventually catch it, but at + # that point the pipeline has already advanced and the + # operator sees the empty-contract divergence after the + # loop is running. Catching it here is earlier and cheaper. + # + # Routes through :func:`_populate_result_is_empty_contract` + # so the two empty-contract call sites (this safety net + # and the natural plan-complete handler below) can't drift + # out of agreement. See that helper's docstring for the + # full discriminator rules. + if _pkg._populate_result_is_empty_contract(_safety_net_populate_result): + # Reason dispatch shared with the plan-complete handler + # via :func:`_populate_outcome_to_hitl_reason` so the + # POPULATED → "populated_but_empty_slices" translation + # (and any future special-cased outcome) can't drift + # between the two call sites (#2627 review follow-up). + _safety_net_reason = _pkg._populate_outcome_to_hitl_reason( + _safety_net_populate_result.outcome + ) + if _safety_net_populate_result.outcome == _pkg.PopulateOutcome.POPULATED: + _safety_net_error = ( + "start_phase=implement safety-net populate " + "completed but produced 0 slices/tasks — refusing " + "to spawn implement-phase agents on an empty " + "contract (#2627)" + ) + else: + _safety_net_error = ( + f"start_phase=implement safety-net populate produced " + f"{_safety_net_populate_result.outcome.value} outcome — " + f"refusing to spawn implement-phase agents on an " + f"empty contract (#2627)" + ) + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + pipeline.status = _pkg.PipelineStatus.FAILED + pipeline.error = _safety_net_error + store.save_pipeline(pipeline) + # Emit the dedicated empty-contract HITL inline so the + # operator sees an actionable decision instead of a + # generic ``status: failed`` with no recovery path + # other than ``restart_phase implement`` (which would + # respawn into the same empty-contract state). + _pkg._emit_empty_contract_hitl( + pipeline_id, + pipeline, + store, + reason=_safety_net_reason, + draft_slice_count=None, + gate="start_phase_implement_safety_net", + phase=pipeline.current_phase, + ) + _pkg.logger.error( + "OVERSEER_ALERT start_phase_implement_safety_net_empty_contract", + pipeline_id=pipeline_id, + outcome=_safety_net_populate_result.outcome.value, + slice_count=_safety_net_populate_result.slice_count, + reason=_safety_net_reason, + ) + _pkg.report_pipeline_status( + pipeline, + event_type="pipeline.failed", + message=f"Pipeline failed: {_safety_net_error[:100]}", + ) + _pkg._emit_pipeline_event(pipeline, "pipeline.failed") + return pipeline, True + + # #3100: the natural plan→implement path enforces the + # #2777 plan pre-flight (``validate_plan_preflight``) at + # the advance_phase site; implement-start submits skip + # that site entirely, so a plan draft without a ``pr:`` + # block previously entered the implement phase and every + # context-PR opener backstop soft-failed with + # ``missing_pr_metadata`` forever. Enforce the same + # validator here — after the empty-contract gate so the + # #2627 HITL routing above is unchanged. + if _pkg._enforce_implement_start_plan_preflight( + pipeline_id, + pipeline, + store, + worktree_repo_path, + plan_draft_rel, + ): + return pipeline, True + return pipeline, False + + +def _sync_source_branch_drafts( + *, gateway_mode, pipeline, pipeline_id, spawner, store, worktree_repo_path +): + """Carry over analysis/plan drafts from the source branch when set (extracted verbatim from _run_pipeline; pure side-effect, no return).""" + if pipeline.source_branch and not (pipeline.plan is not None and pipeline.analysis is not None): + # source_branch is cleared inside _read_source_branch_artifacts + # when artifacts are actually found. + try: + _pkg._read_source_branch_artifacts( + repo_path=worktree_repo_path, + source_branch=pipeline.source_branch, + issue_number=pipeline.issue_number, + pipeline_id=pipeline_id, + store=store, + pipeline=pipeline, + source_artifact_prefix=pipeline.source_artifact_prefix, + spawner=spawner, + gateway_mode=gateway_mode, + ) + except Exception: + _pkg.logger.warning( + "Failed to read artifacts from source branch", + source_branch=pipeline.source_branch, + pipeline_id=pipeline_id, + exc_info=True, + ) + + # Write source-branch artifacts to disk so the safety-net + # _populate_contract_from_plan() call below can find them. + # The inline-plan path writes drafts inside the contract_synced + # block, but that block is skipped on pipeline restarts + # (contract already synced). Writing here ensures the draft + # files exist regardless of contract_synced state. + if pipeline.plan is not None or pipeline.analysis is not None: + drafts_dir = worktree_repo_path / ".egg-state" / "drafts" + drafts_dir.mkdir(parents=True, exist_ok=True) + + if pipeline.plan is not None: + plan_rel = _pkg._get_draft_path( + "plan", + issue_number=pipeline.issue_number, + pipeline_id=pipeline_id, + ) + if plan_rel: + plan_path = worktree_repo_path / plan_rel + plan_path.write_text(pipeline.plan, encoding="utf-8") + _pkg.logger.info( + "Wrote source-branch plan draft to worktree", + pipeline_id=pipeline_id, + path=plan_rel, + ) + + if pipeline.analysis is not None: + analysis_rel = _pkg._get_draft_path( + "refine", + issue_number=pipeline.issue_number, + pipeline_id=pipeline_id, + ) + if analysis_rel: + analysis_path = worktree_repo_path / analysis_rel + analysis_path.write_text(pipeline.analysis, encoding="utf-8") + _pkg.logger.info( + "Wrote source-branch analysis draft to worktree", + pipeline_id=pipeline_id, + path=analysis_rel, + ) + + +def _resolve_worktree_repo( + pipeline, *, gateway_mode, pipeline_id, repo_path, spawner, store, worktree_repo_path +): + """Resolve the per-pipeline worktree repo path + reconcile a stale + worktree (extracted verbatim from _run_pipeline). Returns (pipeline, + done); done=True -> caller returns immediately.""" + if worktree_repo_path != repo_path: + # Determine whether the most recent prior phase completed + # successfully — this controls whether local-ahead commits are + # pushed (success) or discarded (failure). + prior_phase_succeeded = True + current_phase = pipeline.current_phase + phase_order = [ + _pkg.PipelinePhase.REFINE, + _pkg.PipelinePhase.PLAN, + _pkg.PipelinePhase.IMPLEMENT, + ] + current_idx = phase_order.index(current_phase) if current_phase in phase_order else 0 + if current_idx > 0: + prior_phase = phase_order[current_idx - 1] + prior_exec = pipeline.phases.get(prior_phase.value) + if prior_exec and prior_exec.status in ( + _pkg.PipelineStatus.FAILED, + _pkg.PipelineStatus.CANCELLED, + ): + prior_phase_succeeded = False + + # #2979: sync the worktree, pausing for a manual reconcile if + # it diverges and the rebase autoresolve can't reconcile it. + # The helper blocks (AWAITING_HUMAN) on a reconcile HITL and + # resumes the phase start once the operator acks — nothing is + # discarded and the pipeline is never failed for a recoverable + # divergence. + phase_start_sync_outcome, phase_start_sync_aborted = ( + _pkg._sync_worktree_reconciling_divergence( + spawner, + pipeline_id, + store, + repo_path, + worktree_repo_path=worktree_repo_path, + phase=current_phase, + gateway_mode=gateway_mode, + base_branch=pipeline.base_branch, + pipeline_branch=pipeline.branch, + prior_phase_succeeded=prior_phase_succeeded, + ) + ) + if phase_start_sync_aborted: + # Operator aborted the manual reconcile (or the pause + # budget was exhausted). Fail the pipeline; the local + # commits remain pinned under the backup ref for offline + # recovery — nothing was discarded. + _pkg._fail_pipeline_after_divergence_abort( + pipeline_id, + store, + phase=current_phase, + backup_ref=phase_start_sync_outcome.backup_ref, + local_only_commit_shas=phase_start_sync_outcome.local_only_commit_shas, + ) + return pipeline, True + + # When resuming a stale pipeline branch (cancelled run from + # days/weeks ago), rebase origin/<branch> onto origin/<base> + # before any orchestrator/agent commits land — otherwise the + # final PR carries 70+ stale-from-main commits as ancestors + # (#2098). No-op for fresh pipelines and for branches already + # caught up with base. + if pipeline.branch and pipeline.base_branch: + try: + _pkg._rebase_pipeline_branch_onto_base( + spawner, + pipeline_id, + worktree_repo_path, + pipeline_branch=pipeline.branch, + base_branch=pipeline.base_branch, + gateway_mode=gateway_mode, + ) + except _pkg.StalePipelineBranchError as stale_err: + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + pipeline.status = _pkg.PipelineStatus.FAILED + pipeline.error = str(stale_err) + store.save_pipeline(pipeline) + return pipeline, True + + # Remove legacy unprefixed draft files (analysis.md, plan.md) + # that may have been left by earlier pipelines on this branch. + # Uses git rm so deletions are committed directly. See #1559. + cleanup_committed = _pkg._cleanup_stale_generic_drafts(worktree_repo_path) + if cleanup_committed and pipeline.branch: + try: + spawner.gateway.push_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(worktree_repo_path), + branch=pipeline.branch, + mode=gateway_mode, + base_branch=pipeline.base_branch, + ) + except Exception: + _pkg.logger.warning( + "Failed to push stale draft cleanup (continuing)", + pipeline_id=pipeline_id, + ) + return pipeline, False diff --git a/orchestrator/routes/pipelines/_run_pipeline_support.py b/orchestrator/routes/pipelines/_run_pipeline_support.py new file mode 100644 index 0000000000..5bc14923d7 --- /dev/null +++ b/orchestrator/routes/pipelines/_run_pipeline_support.py @@ -0,0 +1,62 @@ +"""run_pipeline health-monitor closure helpers helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _on_health_escalation_impl(escalation, *, health_monitor_instance, pipeline_id): + phase = health_monitor_instance.get_current_phase() + _pkg._send_brc_confirmation_nudge(escalation, pipeline_id, phase) + + +def _health_monitor_poll_impl( + monitor, + stop_event, + interval=30.0, + *, + pipeline_id, + worktree_repo_path, + store, + divergence_alerted_shas, +): + while not stop_event.is_set(): + try: + # Tier 1 no longer sends nudges directly — it raises + # alerts and fires escalation callbacks internally. + # The overseer (Tier 2) decides whether to nudge. + monitor.check_tripwires() + except Exception as poll_err: + _pkg.logger.debug( + "Health monitor poll error", + pipeline_id=pipeline_id, + error=str(poll_err), + ) + + # Branch-divergence detector (#2224 PR 3). Helper + # re-loads pipeline state each tick so a + # base_branch / branch update mid-pipeline is + # picked up. Dedupe set is mutated in place. + _pkg._branch_divergence_tick( + pipeline_id=pipeline_id, + worktree_repo_path=worktree_repo_path, + store=store, + alerted_shas=divergence_alerted_shas, + ) + + # NOTE (#2270 slice-5): the standing-pod overseer respawn loop + # was removed here. The overseer is no longer a respawned + # standing pod — orchestrator-side detection (slice-4 + # ``health_checks.detection_plane``) runs in-process and the + # only agent spawned is the on-demand adjudicator. Any + # surviving restart need is served by the general + # agent-restart machinery (``restart_agent``), not a bespoke + # overseer respawn. This also means a multi-hour zero-agent + # HITL park spawns nothing from this loop (§3). + + stop_event.wait(interval) diff --git a/orchestrator/routes/pipelines/_run_support.py b/orchestrator/routes/pipelines/_run_support.py new file mode 100644 index 0000000000..e09f3a1f0f --- /dev/null +++ b/orchestrator/routes/pipelines/_run_support.py @@ -0,0 +1,376 @@ +"""run-loop support helpers helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +import routes.pipelines as _pkg # noqa: E402,F401 + +if TYPE_CHECKING: + from egg_container import MountSpec # noqa: F401 + from egg_contracts.agent_roles import AgentRole as ContractAgentRole # noqa: F401 + + +def _clear_stale_impasses_for_producers( + repo_path: _pkg.Path, + pipeline_id: str, + producer_roles: "list[ContractAgentRole]", # noqa: UP037 + *, + cleanup_reason: str, +) -> None: + """Drop the ``impasse`` field from each producer's per-pipeline + agent-output file before the next BRC cycle. + + ``save_agent_output`` writes with ``mode="w"`` so a producer that + respawns and reaches its handoff write will overwrite the stale + impasse on its own. But if a producer crashes before writing in the + next iteration (or if the implement roster ever becomes + contract-task-driven, in which case a producer with no remaining + tasks won't spawn at all), the iter-N impasse file would persist + into iter-N+1's ``collect_impasses`` scan and re-trigger routing on + a stale signal — which the ``delegation_attempts`` counter would + then translate into a spurious "second impasse on same task" HITL + escalation. + + Pre-clearing the field keeps ``collect_impasses`` honest about what + came out of the *current* iteration only. Other top-level fields on + the agent output (``handoff_data``, ``role``, anything else) are + preserved. + """ + for role_enum in producer_roles: + try: + existing = _pkg.load_agent_output(repo_path, role_enum, identifier=pipeline_id) + except Exception as exc: # noqa: BLE001 + # Best-effort agent-output file read. Catches OSError on + # the file read, json.JSONDecodeError on parse, and + # pydantic.ValidationError on the role-specific shape. + # Continue (no impasse to clear if the file is unreadable). + _pkg.logger.debug( + "Could not pre-load agent output to clear stale impasse", + pipeline_id=pipeline_id, + role=role_enum.value, + error=str(exc), + ) + continue + if not isinstance(existing, dict) or "impasse" not in existing: + continue + cleaned = {k: v for k, v in existing.items() if k != "impasse"} + try: + _pkg.save_agent_output( + repo_path, + role_enum, + cleaned, + identifier=pipeline_id, + ) + except Exception as exc: # noqa: BLE001 + # Atomic file write of JSON-serialisable dict. Catches + # OSError (write/rename), TypeError/ValueError (non- + # serialisable value sneaking in). Continue — the stale + # impasse will re-trigger routing but the delegation + # counter still bounds the retry. + _pkg.logger.warning( + "Failed to clear stale impasse from agent output", + pipeline_id=pipeline_id, + role=role_enum.value, + error=str(exc), + ) + continue + _pkg.logger.info( + "Cleared stale impasse from agent output", + pipeline_id=pipeline_id, + role=role_enum.value, + cleanup_reason=cleanup_reason, + ) + + +def _pipeline_superseded_by_restart( + store, pipeline_id: str, run_epoch: _pkg.datetime | None +) -> bool: + """True if a newer ``run_epoch`` means another thread now owns this pipeline. + + Reloads pipeline state and compares its ``run_epoch`` against the epoch the + caller runs under (#3315 facet a). Best-effort: a missing epoch or a load + failure returns ``False`` so a transient store hiccup never tears down a + legitimately-running phase. Shared by the ``_run_concurrent_phase`` poll + loop and the slice-path impasse-retry wrapper so the "no escalation when + superseded" property holds on both routes. + """ + if store is None or run_epoch is None: + return False + try: + _epoch_pip = store.load_pipeline(pipeline_id) + except Exception as _epoch_err: # noqa: BLE001 — never wedge the caller + _pkg.logger.debug( + "Epoch supersession check failed; continuing", + pipeline_id=pipeline_id, + error=str(_epoch_err), + ) + return False + current_epoch = _epoch_pip.run_epoch or _epoch_pip.created_at + return current_epoch != run_epoch + + +def _spawn_and_wait( + spawner, + pipeline_id: str, + agent_role: _pkg.AgentRole, + issue_number: int | None, + repo_volumes: dict[str, str], + gateway_mode: str, + repos: list[str], + phase: str, + sandbox_env: dict[str, str], + sandbox_command: list[str], + timeout: int = 3600, + store=None, + certs_volume: str | None = None, + branch: str | None = None, + extra_mounts: list["MountSpec"] | None = None, # noqa: UP037 + spawn_max_retries: int | None = None, + spawn_retry_initial_backoff_seconds: float | None = None, +) -> tuple[int, str]: + """Spawn a container, wait for it to exit, clean up, return (exit_code, logs). + + If ``store`` is provided, the container is recorded in the phase execution + state so that the status endpoint can report it while it runs. + + The container is launched via the shared ``build_sandbox_config()`` path, + which handles GATEWAY_URL, proxy vars, DNS lockdown, extra_hosts, and + .git shadow mounts automatically. + + Args: + repo_volumes: Mapping of repo_name -> host_path for volume mounts. + Each entry is mounted at /home/egg/repos/<name> in the container, + with .git shadowed by /dev/null bind mounts to force gateway git operations. + certs_volume: Docker named volume for gateway CA certs (mounted at + /shared/certs read-only). If None, certs are not mounted. + spawn_max_retries: Override for spawn retry attempts (None uses spawner default). + spawn_retry_initial_backoff_seconds: Override for initial backoff (None uses spawner default). + + Returns: + (exit_code, container_logs) — logs are captured before cleanup on failure. + """ + try: + from agent_model_resolution import DEFAULT_AGENT_MODEL + except ImportError: + from ..agent_model_resolution import ( # type: ignore[import-not-found, no-redef] + DEFAULT_AGENT_MODEL, + ) + + retry_kwargs: dict = {} + if spawn_max_retries is not None: + retry_kwargs["spawn_max_retries"] = spawn_max_retries + if spawn_retry_initial_backoff_seconds is not None: + retry_kwargs["spawn_retry_initial_backoff_seconds"] = spawn_retry_initial_backoff_seconds + + # NOTE: this helper only supports the default Anthropic auth path. It + # does not forward ``upstream``/``upstream_model``, so ``spawn_agent_job`` + # falls back to the Anthropic branch and injects the session-token + # placeholder into ``CLAUDE_CODE_OAUTH_TOKEN`` (#2817). It has no + # production callers today (only test references). If this path is ever + # revived for a LiteLLM agent, plumb ``upstream``/``upstream_model`` + # through here — otherwise Claude Code would send ``x-api-key`` (api_key + # auth) while the placeholder lands in the OAuth header, leaving the + # credential header empty and the session unresolvable. + spawned = spawner.spawn_agent_job( + pipeline_id=pipeline_id, + agent_role=agent_role, + issue_number=issue_number, + mode=gateway_mode, + wait_for_gateway=False, + repos=repos, + phase=phase, + extra_env=sandbox_env, + command=sandbox_command, + repo_volumes=repo_volumes, + branch=branch, + extra_mounts=extra_mounts, + jira_ticket=(sandbox_env.get("EGG_JIRA_TICKET") or None), + **retry_kwargs, + ) + + # Record container and agent in phase execution state + if store is not None: + try: + from models import AgentExecution + + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + phase_execution = pipeline.get_phase_execution(_pkg.PipelinePhase(phase)) + + # Track container — preserve backend-specific fields + # (pod_name, namespace, job_name on K8s) from the spawner. + container_info = spawned.container_info.model_copy( + update={ + "status": _pkg.ContainerStatus.RUNNING, + "started_at": _pkg.datetime.now(_pkg.UTC), + "agent_role": agent_role, + } + ) + phase_execution.containers.append(container_info) + + # Track agent execution. + # + # ``slice_id`` is explicitly ``None`` because this helper has + # no production callers today and is reachable only from + # tests that mock-patch it. If a future change resurrects + # this path for a sliced spawn, the caller MUST plumb a + # ``slice_id`` through here — otherwise the new + # ``(role, slice_id)`` walks added in #2422 will not see + # the record. See PR #2435 review thread. + # This helper hard-codes the default Anthropic auth path (see + # the NOTE above ``spawn_agent_job``), so the resolved model is + # always the built-in default alias. Stamp it for parity with + # ``_run_concurrent_phase`` / ``restart_agent`` (#3174) — if this + # test-only path is ever resurrected for production it will not + # silently regress resolved-model visibility. + agent_execution = AgentExecution( + role=agent_role, + status=_pkg.AgentExecutionStatus.RUNNING, + container_id=spawned.container_info.container_id, + slice_id=None, + started_at=_pkg.datetime.now(_pkg.UTC), + resolved_model=DEFAULT_AGENT_MODEL, + ) + phase_execution.agents.append(agent_execution) + + store.save_pipeline(pipeline) + except Exception as track_err: + _pkg.logger.warning( + "Failed to record container/agent in pipeline state", + container_id=spawned.container_info.container_id[:12], + error=str(track_err), + ) + + backend = spawner.backend + try: + final_info = backend.wait_for_container( + spawned.container_info.container_id, + timeout=timeout, + ) + except ( + _pkg.ContainerNotFoundError, + _pkg.ContainerOperationError, + _pkg.PodNotFoundError, + _pkg.JobOperationError, + ) as e: + _pkg.logger.warning( + "Container lost during wait, marking failed", + container_id=spawned.container_info.container_id, + error=str(e), + ) + final_info = _pkg.ContainerInfo( + container_id=spawned.container_info.container_id, + container_name=spawned.container_info.container_name, + status=_pkg.ContainerStatus.FAILED, + exit_code=-1, + exited_at=_pkg.datetime.now(_pkg.UTC), + ) + + container_logs = "" + if final_info.exit_code != 0: + try: + container_logs = backend.get_container_logs( + spawned.container_info.container_id, + tail=200, + ) + except Exception: + pass + + # Update container and agent status in phase execution + if store is not None: + try: + with _pkg.get_pipeline_state_lock(pipeline_id): + pipeline = store.load_pipeline(pipeline_id) + phase_execution = pipeline.get_phase_execution(_pkg.PipelinePhase(phase)) + + # Update container status + for ci in phase_execution.containers: + if ci.container_id == spawned.container_info.container_id: + ci.status = final_info.status + ci.exited_at = final_info.exited_at + ci.exit_code = final_info.exit_code + break + + # Update agent status + for agent in phase_execution.agents: + if agent.container_id == spawned.container_info.container_id: + agent.completed_at = _pkg.datetime.now(_pkg.UTC) + if final_info.exit_code == 0: + agent.status = _pkg.AgentExecutionStatus.COMPLETE + else: + agent.status = _pkg.AgentExecutionStatus.FAILED + agent.error = f"Container exited with code {final_info.exit_code}" + break + + store.save_pipeline(pipeline) + except Exception as track_err: + _pkg.logger.warning( + "Failed to update container/agent status in pipeline state", + container_id=spawned.container_info.container_id[:12], + error=str(track_err), + ) + + # Always clean up the container + try: + spawner.remove_agent_container( + spawned.container_info.container_id, + force=True, + cleanup_session=True, + ) + except Exception as cleanup_err: + _pkg.logger.warning( + "Failed to clean up container", + container_id=spawned.container_info.container_id[:12], + error=str(cleanup_err), + ) + + return final_info.exit_code, container_logs + + +def _parse_resolution(resolution: str | None) -> tuple[bool, str | None]: + """Parse a HITL phase_gate resolution into (is_approved, feedback). + + Handles both JSON-structured resolutions and legacy bare-string formats. + Used by the AWAITING_HUMAN recovery path in start_pipeline. + + Returns: + (is_approved, feedback): is_approved is True for approve/select/submit_feedback + actions, False for request_changes/change_approach. feedback contains the + revision feedback text (if any) for non-approved resolutions. + """ + if not resolution: + return True, None + + resolution = resolution.strip() + + # JSON-first: try structured payload + try: + payload = _pkg.json.loads(resolution) + if isinstance(payload, dict) and "action" in payload: + action = payload["action"] + feedback_text = payload.get("feedback", "") or None + + if action in ("approve", "select", "submit_feedback"): + return True, None + elif action in ("request_changes", "change_approach"): + return False, feedback_text + # Unknown action — fall through to legacy matching + except _pkg.json.JSONDecodeError, TypeError, AttributeError: + pass + + # Legacy bare-string resolution + if resolution.lower() in _pkg._APPROVE_KEYWORDS: + return True, None + elif resolution.lower() in _pkg._BARE_OPTION_LABELS: + return False, None + elif resolution: + # Free-text feedback — treat as request_changes + return False, resolution + + return True, None diff --git a/orchestrator/routes/pipelines/_salvage.py b/orchestrator/routes/pipelines/_salvage.py new file mode 100644 index 0000000000..a2de9c1328 --- /dev/null +++ b/orchestrator/routes/pipelines/_salvage.py @@ -0,0 +1,71 @@ +"""salvage worktree filters + serializers helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _filter_salvage_worktrees( + worktrees: list[_pkg.Any], + *, + agent_role: str | None, + slice_id: str | None, +) -> list[_pkg.Any]: + """Filter ``enumerate_agent_worktrees`` output by role / slice scope. + + ``agent_role`` and ``slice_id`` may both be ``None`` (return all) or + set together to scope down to one specific worktree. ``agent_role`` + set with ``slice_id=None`` matches non-slice per-agent worktrees. + The pipeline-level worktree (``agent_role=None`` on the worktree) + is included only when the caller did not specify ``agent_role``. + """ + out = [] + for wt in worktrees: + if agent_role is not None and wt.agent_role != agent_role: + continue + if slice_id is not None and wt.slice_id != slice_id: + continue + out.append(wt) + return out + + +def _serialize_commit_report(report: _pkg.Any) -> dict[str, _pkg.Any]: + """Convert a ``WorktreeCommitReport`` to a JSON-safe dict.""" + return { + "worktree_id": report.worktree.worktree_id, + "agent_role": report.worktree.agent_role, + "slice_id": report.worktree.slice_id, + "local_branch": report.worktree.local_branch, + "assigned_branch": report.assigned_branch, + "anchor_ref": report.anchor_ref, + "commits": [ + { + "sha": c.sha, + "summary": c.summary, + "author": c.author, + "authored_at": c.authored_at, + "files_changed": c.files_changed, + } + for c in report.commits + ], + "error": report.error, + } + + +def _serialize_salvage_result(result: _pkg.Any) -> dict[str, _pkg.Any]: + """Convert a ``SalvageResult`` to a JSON-safe dict.""" + return { + "worktree_id": result.worktree_id, + "agent_role": result.agent_role, + "slice_id": result.slice_id, + "recovery_ref": result.recovery_ref, + "head_sha": result.head_sha, + "n_commits": result.n_commits, + "ok": result.ok, + "error": result.error, + } diff --git a/orchestrator/routes/pipelines/_slice_completion.py b/orchestrator/routes/pipelines/_slice_completion.py new file mode 100644 index 0000000000..463acbbb65 --- /dev/null +++ b/orchestrator/routes/pipelines/_slice_completion.py @@ -0,0 +1,135 @@ +"""slice-completion basis validation helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +class SliceCompletionInvariantError(RuntimeError): + """Raised when a slice would be persisted ``COMPLETE`` without a valid + completion basis (#3214). + + The #3214 wedge traced to an interior forest node (``slice-3`` on + pipeline ``issue-3200``) persisted as ``SliceStatus.COMPLETE`` while + its only task was still ``pending``, it had no integration branch, and + it carried its *parent's* commit SHA. ``_persist_slice_status_complete`` + wrote that contradictory state with no validation, so the slice-DAG + driver skipped real work and the chain wedged with no successor — and + it hung ~9h silently because nothing failed loud at the moment of the + bad write. + + A slice has a valid completion basis when ANY of these execution + signals is present: + + * a slice PR is recorded / supplied (``pr_number``); or + * the caller declares a verified ``basis`` — ``"merged"`` (the + integration branch was ancestry-verified merged into its parent) or + ``"consensus_complete"`` (BRC consensus reached, PR not yet opened + or its URL unparseable); or + * the slice forked an integration branch (``integration_base_sha`` is + set — #2871); or + * every task is ``TaskStatus.COMPLETE``. + + The predicate accepts any one signal so it can only flag the slice-3 + state where *all* are absent — a slice marked COMPLETE with zero + evidence it ran. We raise here so that corrupt write fails loud at its + source instead of wedging the forest a phase later. + + #3253 refinement: ``basis="merged"`` is no longer an unconditional + pass. A merged slice went through a PR and left commits its producers + recorded; a ``basis="merged"`` write with **no PR and no produced task + commit** is an empty / never-implemented branch that origin ancestry + mis-detected as merged (the slice-10 case — producers exhausted before + committing, so the integration branch's tip is still its fork base and + is trivially an ancestor of the advanced parent). Such a write is + rejected so the slice is re-run rather than false-completed. + """ + + +def _slice_produced_commits(slice_obj: _pkg.Any) -> bool: + """Return True iff any of the slice's tasks recorded a commit SHA. + + This is the base-SHA-independent "a producer actually committed work" + signal (#3253). It reads *task* commits only — a slice whose producers + all failed before committing has every ``task.commit`` ``None`` (the + AC-4 measurement in the issue-3200 slice-10 incident). It deliberately + ignores ``Slice.commit``: that field can carry the *parent's* SHA on a + false-complete (the #3214 slice-3 carryover), so it is not trustworthy + evidence the slice itself produced anything. + + An empty integration branch (tip still at its fork base, so trivially + an ancestor of an advanced parent) is indistinguishable from a merged + one by origin ancestry alone once the recorded fork base is missing or + stale (#3245). The contract's task-commit record is the durable signal + that survives that ambiguity: no task commit + no slice PR ⇒ the slice + never ran and must be re-run, not completed. + + A slice with *no tasks* returns ``False`` here (``any([])``). Paired with + "origin-detected merged, no PR" that would force such a slice to re-run + indefinitely — but a zero-task slice is unreachable in practice: + plan-derived slices always carry at least one task. The safe direction is + re-run over silently-dropped work, so the edge needs no special-casing + (#3253). + """ + tasks = getattr(slice_obj, "tasks", None) or [] + return any(getattr(t, "commit", None) for t in tasks) + + +def _validate_slice_completion_basis( + slice_obj: _pkg.Any, + *, + pr_number: int | None = None, + basis: str | None = None, +) -> str | None: + """Return ``None`` when ``slice_obj`` may legitimately be marked + ``SliceStatus.COMPLETE``, else a human-readable reason it may not. + + Shared by the write chokepoint (``_persist_slice_status_complete``, + which raises :class:`SliceCompletionInvariantError` on a reason) and + the Layer-A bootstrap read-trust point (which alerts and declines to + trust a contradictory contract-recorded COMPLETE rather than + propagating it into the scheduler). See + :class:`SliceCompletionInvariantError` for the basis rules (#3214). + """ + has_pr = pr_number is not None or getattr(slice_obj, "pr_number", None) is not None + # #3253 — a ``basis="merged"`` slice with no PR and no produced task + # commits is not a merged slice; it is an empty / never-implemented + # integration branch (tip still at its fork base) that origin ancestry + # mis-detected as merged. A genuine merge went through a PR and left + # commits the producers recorded. Reject so the restart re-runs the + # slice instead of false-completing the pipeline with its work missing. + # This guard fires *before* the verified-basis / forked free-passes + # below so a recorded (possibly stale) fork base cannot rescue it. + if basis == "merged" and not has_pr and not _pkg._slice_produced_commits(slice_obj): + return ( + f"slice {getattr(slice_obj, 'id', '?')} would be marked COMPLETE " + f"basis='merged' with no slice PR and no produced task commits — an " + f"empty / never-implemented integration branch is not a merged one " + f"(#3253)" + ) + verified_basis = basis in _pkg._VERIFIED_SLICE_COMPLETION_BASES + # A slice that actually forked its integration branch recorded a base + # SHA (#2871). Its absence — together with no PR, no verified basis, + # and no completed tasks — is the slice-3 false-complete signature: a + # slice marked COMPLETE with zero evidence it ever ran. The predicate + # accepts ANY single execution signal so it can only flag that + # genuinely-contradictory state, never a legitimately-completed slice + # whose other signals happen to be absent (e.g. an unparseable PR URL + # leaves ``pr_number`` None but the slice still forked and reached + # consensus). ``tasks_all_complete`` is the canonical model-side + # predicate so this can't drift from the contract's own notion of + # "work finished". + forked = getattr(slice_obj, "integration_base_sha", None) is not None + if has_pr or verified_basis or forked or slice_obj.tasks_all_complete: + return None + return ( + f"slice {getattr(slice_obj, 'id', '?')} would be marked COMPLETE with no " + f"evidence it ran: no slice PR, no verified merge/consensus basis " + f"(basis={basis!r}), no integration-branch fork base, and tasks not all " + f"complete" + ) diff --git a/orchestrator/routes/pipelines/_slice_state.py b/orchestrator/routes/pipelines/_slice_state.py new file mode 100644 index 0000000000..edb18449b8 --- /dev/null +++ b/orchestrator/routes/pipelines/_slice_state.py @@ -0,0 +1,1094 @@ +"""slice-DAG state helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Literal # noqa: F401 + +import routes.pipelines as _pkg # noqa: E402,F401 + +if TYPE_CHECKING: + try: + from ..container_spawner import ContainerSpawner # noqa: F401 + except ImportError: # pragma: no cover + from container_spawner import ContainerSpawner # type: ignore # noqa: F401 + + +def _resolve_pipeline_worktree_path(pipeline: _pkg.Pipeline, fallback: _pkg.Path) -> _pkg.Path: + """Resolve the on-disk worktree path for *pipeline*. + + Prefers ``WORKTREE_BASE_DIR / pipeline.id / <repo_short>`` when it + exists (the same layout _run_pipeline materialises at spawn time; + see pipelines.py spawn block). Falls back to *fallback* — typically + the state store's ``repo_path`` — when no worktree is materialised. + """ + repo_short = pipeline.repo.split("/")[-1] if pipeline.repo else None + if repo_short: + candidate = _pkg.WORKTREE_BASE_DIR / pipeline.id / repo_short + if candidate.exists(): + return candidate + pipeline_wt_dir = _pkg.WORKTREE_BASE_DIR / pipeline.id + if pipeline_wt_dir.exists(): + # sorted() for deterministic selection when multiple subdirs exist + for sub in sorted(pipeline_wt_dir.iterdir()): + if sub.is_dir() and (sub / ".git").exists(): + return sub + return fallback + + +def _resolve_slice_gate_repo(slice_obj, pipeline: _pkg.Pipeline) -> str | None: + """The repo every implement-phase gate for *slice_obj* is scoped to (#3393). + + Single source of truth for slice → gate-repo resolution (task-6-1): the + test gate, the reviewer diff base, the per-repo check/lint commands, and + the slice agent's cwd all key off this one accessor. It is exactly + :func:`models.resolve_slice_repo` — the slice's own ``repo`` when set, + else the pipeline's primary repo (so a repoless slice, or any slice in an + N=1 pipeline, scopes to the single/primary repo). Returns ``None`` only + for a genuinely repoless pipeline (test scaffolds with no repo at all). + """ + try: + from models import resolve_slice_repo # type: ignore[no-redef] + except ImportError: + from ..models import resolve_slice_repo # type: ignore[no-redef] + return resolve_slice_repo(slice_obj, pipeline) + + +def _resolve_slice_worktree_path( + pipeline: _pkg.Pipeline, slice_repo: str | None, fallback: _pkg.Path +) -> _pkg.Path: + """Resolve the on-disk worktree path for a slice's repo (#3393 task-6-1). + + A multi-repo pipeline materialises one worktree per participating repo + under ``WORKTREE_BASE_DIR / pipeline.id / <repo_short>`` — the same + owner/repo-keyed layout as :func:`_resolve_pipeline_worktree_path`, one + directory per repo. Given a slice's resolved repo (``owner/name``), this + returns that repo's worktree when it exists on disk, else *fallback* + (the pipeline-primary worktree). For an N=1 pipeline the slice's repo IS + the primary, so ``slice_repo`` matches ``pipeline.repo`` and the answer + is byte-identical to the pipeline-primary worktree — callers therefore + only reach here for a genuine secondary-repo slice. + """ + repo_short = slice_repo.split("/")[-1] if slice_repo else None + if repo_short: + candidate = _pkg.WORKTREE_BASE_DIR / pipeline.id / repo_short + if candidate.exists(): + return candidate + return fallback + + +def _is_slice_dag_mode(contract) -> bool: + """Return True when the contract represents a multi-slice DAG (#2777, cq-10). + + Dedupes the bare ``len(contract.slices) > 1`` recompute that + appears at the ``_run_implement_phase_slices`` entry and inside the + run loop's per-slice handling. A single helper means future changes + to "what counts as DAG mode" — e.g. treating a single slice with + explicit dependencies as DAG — only need to land in one place. + The third site under the deleted ``_should_skip_pr_phase_auto_pr`` + is gone since slice-2 of #2777 removed the PR phase. + + Returns False for ``None`` or a contract without a populated + ``slices`` list (monolithic / pre-populate phase pipelines). + """ + if contract is None: + return False + slices = getattr(contract, "slices", None) or [] + return len(slices) > 1 + + +def _resolve_slice_base_branch( + contract, + slice_id: str, + *, + pipeline_id: str, + pipeline_branch: str, + extant_branches: set[str] | None = None, + parent_branch_exists: _pkg.Callable[[str], bool] | None = None, +) -> str: + """Return the parent branch for a slice's integration branch (#2777, cq-9). + + Replaces the deleted slice-1 resolver helper (removed by slice-2 + TASK-2-1) with a single resolver that handles both root and + non-root slices. + + Three-tier resolution (default — ``extant_branches is None``): + + 1. **Eager-persisted parent** (post-slice-4 TASK-4-2). If + ``parent_branch_at_creation`` is set on the slice record, + return it. This is the primary path post-slice-4 — slices + created after the eager persist landed always go through + this arm. + 2. **Dependency-derived parent, gated on parent existence + (#2928)**. For a non-root slice whose + ``parent_branch_at_creation`` is empty (the normal first-run + case), the stack target is its dependency parent's + integration branch ``{issue_branch}/{dependencies[0]}``. When + a ``parent_branch_exists`` callback is provided, the resolver + probes whether that parent branch is still present on origin: + + * parent branch **exists** → return the dependency-derived + parent. This is the correct target for both fresh slices + (whose own integration branch does not exist yet) and + legacy slices. + * parent branch **absent** → the parent slice's PR was merged + into ``work`` and its branch deleted by the cascade, so + ``work`` already contains the parent's commits. Fall back + to ``pipeline_branch``. + * probe **raises** → conservative default: assume the parent + exists and return the derived parent. Never silently swap a + real slice onto ``work`` because of a flaky gateway. + + This replaces the pre-#2928 merge-base check, which probed the + *slice's own* integration branch for a fork point and routed a + ``None`` result (no fork point) to ``pipeline_branch``. That + conflated a FRESH slice (integration branch not yet created — + the common first-run case) with a genuinely orphaned slice, + silently mis-basing fresh slices onto ``work`` whenever + ``work`` had advanced ahead of the parent (the wedge in + #2928). + 3. **Final fallback** to ``pipeline_branch`` (``egg/<id>/work``) + when (a) no eager-persisted parent, (b) the slice is a root + (no dependencies), OR (c) the slice's dependency parent branch + is absent from origin. Root-targeted branches are never + deleted by the cascade so this is always a safe terminal + candidate. + + **Orphan-reconciler mode (``extant_branches`` non-None)**: the + stacked-PR reconciler at ``orchestrator/stacked_pr_reconciler.py`` + needs the resolver to SKIP ancestors whose branches are no longer + on origin (the primary trigger for orphan reconciliation is "parent + branch was deleted by the cascade merge"). When ``extant_branches`` + is supplied, each candidate (including ``parent_branch_at_creation`` + and any walked ancestor) is filtered against the set; if no extant + candidate is found the resolver falls back to ``pipeline_branch`` + (which is always extant — root-targeted branches are never deleted + by the stacked-PR flow). + + Args: + contract: The pipeline contract (must carry ``slices``). + slice_id: The slice whose base branch to resolve. + pipeline_id: Used only for log diagnostics; the resolver does + NOT consult the state store. + pipeline_branch: The pipeline's work branch (``egg/<id>/work``). + Returned for root slices when no + ``parent_branch_at_creation`` is recorded, and as the + final fallback in orphan-reconciler mode and the + merge-base "no fork point" arm. + extant_branches: Optional set of branch names known to exist + on origin. When supplied, the resolver filters every + candidate (recorded parent + walked ancestors) against + this set and skips any that are absent. The reconciler + uses this to escape from the deleted parent branch up the + DAG until an extant ancestor is reached. + parent_branch_exists: Optional callback (#2928) used to + decide whether a non-root slice's dependency parent + branch is still on origin. When provided, the resolver + invokes ``parent_branch_exists(parent_branch)`` with the + dependency-derived parent branch name. ``True`` returns + the derived parent; ``False`` routes to + ``pipeline_branch`` (parent merged + cascade-deleted); a + raised exception is treated conservatively as ``True``. + The default ``_run_one_slice_inner`` caller wires this + against ``spawner.gateway.ls_remote_branch_strict`` — the + strict variant is required so a gateway / network / + policy failure RAISES into this resolver's ``try/except`` + instead of being collapsed to ``False`` (which would + silently route a real slice onto ``pipeline_branch`` on + any gateway flake — re-creating the #2928 wedge). The + stacked-PR reconciler leaves it ``None`` (it has already + verified extant branches via the ``extant_branches`` + set). + + Mutually exclusive with ``extant_branches`` in practice: + the production caller (``_run_one_slice_inner``) passes + only this gate, and the stacked-PR reconciler passes only + ``extant_branches``. If a future caller passed both, this + gate would short-circuit to ``pipeline_branch`` on a + ``False`` return BEFORE the ``extant_branches`` walk + could find an extant ancestor; callers that have already + built the extant set should leave this ``None``. + + Returns: + The branch name to use as the slice integration branch's + parent. Never an empty string. + + Raises: + ValueError: When the requested slice id is absent from the + contract — a structural bug that the slice loop's earlier + forest-validation step should have caught. + """ + slices = getattr(contract, "slices", None) or [] + slice_record = next((s for s in slices if s.id == slice_id), None) + if slice_record is None: + raise ValueError( + f"slice {slice_id!r} not present in contract for pipeline " + f"{pipeline_id!r}; available slices: " + f"{[s.id for s in slices]}" + ) + + def _extant(candidate: str) -> bool: + """True when ``candidate`` passes the orphan-reconciler filter. + + When ``extant_branches`` is None, every non-empty candidate + passes (the default resolver doesn't validate liveness). + """ + if not candidate: + return False + if extant_branches is None: + return True + return candidate in extant_branches + + # (1) Eager-persisted parent (post-slice-4 TASK-4-2). Treated as + # authoritative regardless of root-status: if the persist landed, + # it's the resolved parent — UNLESS the orphan-reconciler caller + # told us this branch was deleted on origin (extant_branches + # filter). + parent_recorded = getattr(slice_record, "parent_branch_at_creation", None) or "" + if parent_recorded and _extant(parent_recorded): + return parent_recorded + + # Build the slice-id → slice-record lookup once for the DAG walk + # below (used in both the default and orphan-reconciler modes). + slices_by_id = {s.id: s for s in slices} + + deps = getattr(slice_record, "dependencies", None) or [] + parent_slice_id = deps[0] if deps else None + + # (2) Root slice — under the new topology (cq-4), the context PR + # is ``egg/<id>/work → main`` so root slices stack directly on the + # work branch rather than a separate ``egg/<id>/context`` branch. + if parent_slice_id is None: + return pipeline_branch + + # (3) Non-root slice — derive from the first dependency. Mirrors + # the existing ``f"{issue_branch}/{parent_slice_id}"`` convention + # at the legacy slice-loop call site. + issue_branch = _pkg._slice_namespace_root(pipeline_branch) + derived_parent = f"{issue_branch}/{parent_slice_id}" + + # #2928: parent-existence gate. When eager-persist did not land + # (``parent_recorded`` empty above) AND a ``parent_branch_exists`` + # callback is provided, decide between the dependency-derived + # parent and ``pipeline_branch`` by probing whether the parent + # slice's integration branch is still on origin — NOT by probing + # the slice's own branch for a fork point. + # + # The pre-#2928 implementation computed + # ``merge_base(integration_branch, derived_parent)`` and routed a + # ``None`` result to ``pipeline_branch``. That conflated a FRESH + # slice (its integration branch is created *after* this resolver + # runs, so it has no fork point on the first run — the common + # case) with a genuinely orphaned slice, silently mis-basing + # fresh slices onto ``work`` whenever ``work`` had advanced ahead + # of the parent (e.g. a stray contract-state commit on ``work``). + # The correct discriminator is parent-branch existence: + # + # * parent exists → stack on it (fresh OR legacy slice). + # * parent absent → the parent PR merged into ``work`` and its + # branch was cascade-deleted, so ``work`` already contains the + # parent's commits → ``pipeline_branch`` is the right base. + # * probe raises → conservative: assume the parent exists and + # return the derived parent; never silently swap a real slice + # onto ``work`` because the gateway was flaky. + if parent_branch_exists is not None: + try: + exists = parent_branch_exists(derived_parent) + except Exception as probe_err: # noqa: BLE001 + _pkg.logger.warning( + "parent_branch_exists probe raised; assuming parent " + "exists and returning dependency-derived parent (#2928)", + pipeline_id=pipeline_id, + slice_id=slice_id, + derived_parent=derived_parent, + error=str(probe_err), + ) + exists = True + if not exists: + _pkg.logger.warning( + "Dependency-parent branch absent on origin; parent " + "appears merged into work — basing slice on pipeline " + "branch (#2928)", + pipeline_id=pipeline_id, + slice_id=slice_id, + derived_parent=derived_parent, + pipeline_branch=pipeline_branch, + ) + return pipeline_branch + + # Default mode (no extant filter): return the immediate parent + # branch synthesised from the slice DAG. This is the unchanged + # pre-extant-kwarg behaviour. + if extant_branches is None: + return f"{issue_branch}/{parent_slice_id}" + + # Orphan-reconciler mode: walk up the DAG via ``dependencies[0]`` + # until an extant ancestor branch is found. The forest constraint + # at ``shared/egg_contracts/models.py:341`` guarantees ≤1 parent + # per slice, so a single traversal pointer suffices. + cursor: str | None = parent_slice_id + while cursor: + candidate = f"{issue_branch}/{cursor}" + if _extant(candidate): + return candidate + cursor_slice = slices_by_id.get(cursor) + if cursor_slice is None: + break + next_deps = getattr(cursor_slice, "dependencies", None) or [] + cursor = next_deps[0] if next_deps else None + + # Every ancestor's branch has been deleted (cascading merge). Fall + # back to the pipeline branch — stable across the stacked-PR flow + # because root-targeted branches are never deleted by the cascade. + return pipeline_branch + + +def _lookup_peer_consensus_tracker_or_none( + pipeline_id: str, slice_id: str | None +) -> _pkg.Any | None: + """Look up a per-slice PeerConsensusTracker; return None on import failure. + + Slice-4 TASK-4-4 helper. The bootstrap classifier needs to inspect + consensus state (``tracker.evaluate()['is_complete']``) for + IN_PROGRESS slices with commits on origin to differentiate case (2) + (consensus not reached → mark spawned) from case (3) (consensus + reached → mark COMPLETE so the slice-PR opener fires). This thin + wrapper centralises the lazy import + None-on-import-failure dance + so the classifier itself stays declarative and easily unit-tested. + """ + try: + from orchestrator.peer_consensus import ( + get_peer_consensus_tracker as _gpct, + ) + except ImportError: + try: + from peer_consensus import ( # type: ignore[no-redef] + get_peer_consensus_tracker as _gpct, + ) + except ImportError: + return None + try: + return _gpct(pipeline_id, slice_id=slice_id) + except Exception: # noqa: BLE001 + return None + + +def _slice_has_pending_decision(slice_id: str, decisions: list[_pkg.Any]) -> bool: + """Return True iff the contract has any unresolved HITL decision. + + Slice-4 TASK-4-4 case (4) helper. The classifier treats a BLOCKED + slice with no pending decision as a state-machine anomaly (the + slice was waiting on a HITL that has since been resolved without + flipping the slice status forward). Surface that to the operator + via OVERSEER_ALERT. + + The contract's :class:`egg_contracts.models.Decision` does NOT + carry a structured ``slice_id`` tag — decisions are scoped by + phase (``decision.phase``) rather than by slice. The conservative + interpretation: any unresolved decision is *potentially* the + reason this slice is BLOCKED, so we return ``True`` (suppress the + "missing-HITL" alert) whenever the contract carries ANY unresolved + decision. The function only returns ``False`` when ZERO unresolved + decisions exist on the contract — at which point a BLOCKED slice + is provably unexplained and the overseer alert is warranted. + + Practically: this errs on the side of NOT alerting (suppressing + a real cross-slice mismatch in favour of skipping a spurious + alert), because alert noise during normal multi-slice HITL flows + is worse than a missed anomaly that the next bootstrap pass will + re-check anyway. + + ``slice_id`` is currently unused; kept in the signature so a + future contract schema bump that adds a structured slice tag can + use the existing call sites verbatim. + """ + del slice_id # contract decisions are not tagged by slice yet + for d in decisions: + if not getattr(d, "resolved", False): + return True + return False + + +def _classify_non_complete_slice( + *, + pipeline_id: str, + slice_obj: _pkg.Any, + issue_branch: str, + pipeline_repo: _pkg.Any, + worktree_repo_path: _pkg.Path, + gateway: _pkg.Any, + gateway_mode: Literal["public", "private"], + consensus_tracker_lookup: _pkg.Callable[[str, str | None], _pkg.Any | None], +) -> str: + """Classify a non-COMPLETE slice for Layer-C bootstrap reconciliation. + + Slice-4 TASK-4-4. Returns one of the five classification labels: + + * ``"fresh"`` — case (1) IN_PROGRESS/PENDING with no commits on + origin. No Layer-C action; the scheduler re-yields READY and + the run loop spawns fresh agents. + * ``"resume"`` — case (2) IN_PROGRESS with commits on origin and + consensus NOT reached. Caller calls + ``scheduler.mark_spawned(slice_id)`` so the run loop does NOT + respawn. + * ``"consensus_complete"`` — case (3) IN_PROGRESS with commits + and ``tracker.evaluate()['is_complete']`` True. Caller marks + the slice COMPLETE so the next loop iteration runs the slice-PR + opener via its idempotent pre-flight. + * ``"blocked"`` — case (4) BLOCKED slice (HITL pending). Caller + preserves status. If no pending HITL is found on the contract, + caller escalates via ``_escalate_blocked_slice_to_hitl`` + which writes a new ``Decision`` to the contract. + * ``"corrupt"`` — case (5) impossible status enum or + contradictory state combination (PENDING with commits, etc.). + Caller escalates via ``_escalate_corrupt_slice_to_hitl`` + which writes a new ``Decision`` to the contract. + + The classifier is intentionally a pure function modulo the + injected ``gateway`` probe + ``consensus_tracker_lookup`` — + unit tests in TASK-4-6 fake both. + """ + try: + from egg_contracts.models import SliceStatus + except ImportError: + return "corrupt" + + status = getattr(slice_obj, "status", None) + if status == SliceStatus.BLOCKED: + # Case 4 — caller (Layer-C loop) validates the HITL via + # ``_slice_has_pending_decision`` and escalates if absent. + # The classifier itself just reports the BLOCKED state. + return "blocked" + + if status not in (SliceStatus.PENDING, SliceStatus.IN_PROGRESS): + # Case 5 — unknown / corrupt status enum value. The + # SliceStatus StrEnum has exactly four members; any other + # value (None, a string that didn't deserialise to the enum, + # a future enum addition we don't recognise yet) is treated + # as corrupt rather than silently re-yielded as READY. + return "corrupt" + + # Probe the slice's integration branch for commits on origin. + integration_branch = f"{issue_branch}/{slice_obj.id}" + has_commits: bool + if pipeline_repo is None: + # Repoless pipelines (test scaffolds) — no origin to consult. + # Treat as no-commits → fresh, which mirrors the default + # scheduler behaviour. + has_commits = False + else: + try: + sha = gateway.get_remote_branch_sha( + pipeline_id, + str(worktree_repo_path), + f"refs/heads/{integration_branch}", + mode=gateway_mode, + ) + has_commits = sha is not None + except Exception as probe_err: # noqa: BLE001 + # Probe failure (gateway down, transient HTTP). Conservative + # default: treat as has_commits=False so the slice is + # re-yielded READY rather than silently mark-spawned with + # no agents alive. + # + # NOTE on asymmetry vs. ``_resolve_slice_base_branch`` + # (slice-4 TASK-4-3, ~line 10510): the resolver defaults + # the *opposite* direction — probe failure → "has fork + # point → derived parent" — because mis-routing onto + # ``pipeline_branch`` on a transient probe error would + # silently change a slice's stack target. Here in Layer C, + # a "fresh" mis-classification just causes the scheduler + # to re-yield the slice as READY (fresh-agent spawn, which + # then sync-then-fetches and continues correctly). The + # asymmetry is deliberate: each direction picks the safer + # default for its own caller. + _pkg.logger.warning( + "Layer-C bootstrap probe raised; treating slice as fresh (slice-4 TASK-4-4)", + pipeline_id=pipeline_id, + slice_id=slice_obj.id, + error=str(probe_err), + ) + has_commits = False + + if not has_commits: + # PENDING-without-commits is the normal fresh slice case. + # IN_PROGRESS-without-commits means a crash between the + # eager-persist (TASK-4-2) and ``create_slice_integration_branch`` + # — also fresh from the scheduler's perspective. + return "fresh" + + if status == SliceStatus.PENDING and has_commits: + # Case 5 — PENDING with commits on origin is a state-machine + # impossibility (the eager-persist (TASK-4-2) flips PENDING → + # IN_PROGRESS in the same contract write that records the + # parent branch BEFORE any commits could land). Treat as + # corrupt. + return "corrupt" + + # IN_PROGRESS with commits — distinguish (2) vs (3) via the + # consensus tracker reconstructed by startup_reconciliation.py + # (slice-4 TASK-4-5). + tracker = consensus_tracker_lookup(pipeline_id, slice_obj.id) + consensus_complete = False + if tracker is not None: + try: + evaluation = tracker.evaluate() + consensus_complete = bool(evaluation.get("is_complete")) + except Exception as eval_err: # noqa: BLE001 + _pkg.logger.warning( + "Layer-C bootstrap tracker.evaluate() raised; treating slice " + "as consensus-incomplete (slice-4 TASK-4-4)", + pipeline_id=pipeline_id, + slice_id=slice_obj.id, + error=str(eval_err), + ) + + return "consensus_complete" if consensus_complete else "resume" + + +def _escalate_layer_c_hitl( + *, + pipeline_id: str, + slice_id: str, + worktree_repo_path: _pkg.Path, + current_phase: _pkg.PipelinePhase | None, + question: str, +) -> None: + """Create an HITL Decision on the contract for a Layer-C anomaly (slice-4 TASK-4-4). + + Shared transport for case (4) blocked-without-HITL and case (5) + corrupt-status escalations. Per the plan task body — "escalate + via ``mcp__sdlc__register_open_question`` (do NOT silently + re-yield as READY — silent classification error is worse than + an operator pause)" — Layer C must create an unresolved + ``Decision`` on the contract that pauses the slice until the + operator picks an option, not just a message-bus broadcast. + + The caller supplies ``worktree_repo_path`` (the per-pipeline + worktree where the live contract lives — Layer C runs inside + ``_run_implement_phase_slices`` which already has it in scope) + and ``current_phase`` (the live pipeline phase, so a Decision + surfaces under the phase the operator is debugging rather than + a hard-coded literal). + + **Lock-nesting invariant (reviewer_code v2 blocker 3)**: the + caller MUST NOT already hold ``get_pipeline_state_lock`` for + this pipeline. Today the Layer-C dispatch loop in + ``_run_implement_phase_slices`` calls this helper at the + top-level slice-loop scope BEFORE any per-slice lock + acquisition (the eager-persist site at + ``_run_one_slice_inner`` is the only nested-lock contract + write today). The current lock IS an ``threading.RLock`` so + re-entry would not deadlock, but if a future refactor narrows + the lock to a plain ``Lock`` (e.g. for monitor visibility), + a Layer-C call from inside another lock-holding scope would + deadlock the entire bootstrap. + + Pattern mirrors ``_persist_hitl_decision`` (above) but loads the + contract from the per-pipeline worktree directly (the caller + has it in scope) so the decision lands on the live contract + that ``/sdlc`` reads. Best-effort: contract-load / save + failures are logged and swallowed (consistent with the rest of + Layer C). The decision is tagged with a ``context`` prefix so a + dispatch handler in + ``routes/decisions.py`` can route on a stable discriminator if + one is added in a follow-up. + """ + try: + from egg_contracts.decisions import ( + find_duplicate_open_question, + find_resolved_question, + next_cq_id, + ) + from egg_contracts.loader import load_contract, save_contract + from egg_contracts.models import Decision, DecisionOption, DecisionType + except ImportError: + try: + from orchestrator.egg_contracts.decisions import ( # type: ignore[no-redef] + find_duplicate_open_question, + find_resolved_question, + next_cq_id, + ) + from orchestrator.egg_contracts.loader import ( # type: ignore[no-redef] + load_contract, + save_contract, + ) + from orchestrator.egg_contracts.models import ( # type: ignore[no-redef] + Decision, + DecisionOption, + DecisionType, + ) + except ImportError: + _pkg.logger.warning( + "Layer-C HITL escalation skipped: egg_contracts not importable (slice-4 TASK-4-4)", + pipeline_id=pipeline_id, + slice_id=slice_id, + ) + return + decision_id: str = "" + try: + with _pkg.get_pipeline_state_lock(pipeline_id): + contract_local = load_contract(pipeline_id, worktree_repo_path) + existing_decisions = contract_local.decisions or [] + decision_phase = current_phase or _pkg.PipelinePhase.IMPLEMENT + # Dedupe/carry-forward — parity with ``register_open_question`` + # (#3374/#3392). The Layer-C question text is deterministic per + # (case, slice, pipeline), so every bootstrap re-run after a + # ``restart_phase`` re-derives the identical question. Without + # this guard each re-run minted a fresh ``cq-N`` (or, against a + # reset-stale contract, re-minted an existing one), making the + # operator re-answer questions they had already answered (#3427). + duplicate = find_duplicate_open_question(existing_decisions, question, decision_phase) + if duplicate is not None: + _pkg.logger.info( + "Layer-C HITL escalation adopted existing open decision (slice-4 TASK-4-4)", + pipeline_id=pipeline_id, + slice_id=slice_id, + decision_id=getattr(duplicate, "id", None), + ) + return + carried = find_resolved_question(existing_decisions, question, decision_phase) + if carried is not None: + _pkg.logger.info( + "Layer-C HITL escalation skipped: identical question " + "already resolved by the operator (slice-4 TASK-4-4)", + pipeline_id=pipeline_id, + slice_id=slice_id, + decision_id=getattr(carried, "id", None), + resolution=str(getattr(carried, "resolution", None))[:200], + ) + return + # Use the canonical ``cq-N`` allocator from + # ``shared/egg_contracts/decisions.py``. Orchestrator-side + # HITL escalations write to the ``cq-N`` namespace; the + # pipeline-side bridge owns ``decision-N``. The split was + # introduced by #2616 to prevent the + # ``len(decisions)+1`` collision between the two + # allocators (see the docstring at + # ``shared/egg_contracts/decisions.py``). + decision_id = next_cq_id(contract_local.decisions) + options = [ + DecisionOption(id="opt-1", label="Mark slice complete and continue"), + DecisionOption(id="opt-2", label="Restart slice from scratch"), + DecisionOption(id="opt-3", label="Cancel pipeline for manual investigation"), + ] + # Use the live pipeline phase rather than a hard-coded + # ``PipelinePhase.IMPLEMENT`` — Layer C fires during + # bootstrap which can run before any phase walk, and + # future slice-DAG topologies may span phases. + # + # The ``or PipelinePhase.IMPLEMENT`` arm (folded into + # ``decision_phase`` above) is defensive: the + # ``Pipeline.current_phase`` field is non-Optional with a + # default at the schema layer (``models.py:1032``), so + # in-tree callers should always populate it. The fallback + # exists for future non-``Pipeline``-shaped callers (e.g. + # contract-only loads during cold-start reconciliation + # that may construct a lighter object) — *not* a known-bug + # papering exercise for current shapes. + contract_local.decisions.append( + Decision( + id=decision_id, + question=question, + type=DecisionType.HITL, + phase=decision_phase, + options=options, + ) + ) + save_contract(contract_local, worktree_repo_path) + _pkg.logger.info( + "Layer-C HITL escalation persisted on contract (slice-4 TASK-4-4)", + pipeline_id=pipeline_id, + slice_id=slice_id, + decision_id=decision_id, + ) + # Durably land the new decision on the work branch so the next + # phase-(re)start worktree reset cannot revert it (#3427). + _pkg.persist_contract_statefiles( + pipeline_id, + worktree_repo_path, + f"Persist Layer-C HITL escalation {decision_id} (#3427)", + ) + except Exception as escalate_err: # noqa: BLE001 + _pkg.logger.warning( + "Layer-C HITL escalation failed (slice-4 TASK-4-4); slice will " + "remain in its current contract status", + pipeline_id=pipeline_id, + slice_id=slice_id, + error=str(escalate_err), + ) + + +def _escalate_corrupt_slice_to_hitl( + *, + pipeline_id: str, + slice_id: str, + worktree_repo_path: _pkg.Path, + current_phase: _pkg.PipelinePhase | None, +) -> None: + """Escalate a Layer-C case-5 corrupt-state slice to HITL (slice-4 TASK-4-4). + + Question text is prefixed with ``[#2777 slice-4 TASK-4-4 case 5]`` + so a future dispatch handler in ``routes/decisions.py`` can route + on the literal substring without a separate context field on the + contract-level ``Decision`` model. + """ + _pkg._escalate_layer_c_hitl( + pipeline_id=pipeline_id, + slice_id=slice_id, + worktree_repo_path=worktree_repo_path, + current_phase=current_phase, + question=( + f"[#2777 slice-4 TASK-4-4 case 5] Slice {slice_id} of pipeline " + f"{pipeline_id} has an impossible status enum value or state " + f"combination (e.g. status not in PENDING/IN_PROGRESS/COMPLETE/" + f"BLOCKED, or PENDING with commits on the integration branch). " + f"Bootstrap reconciliation cannot classify the slice safely. " + f"How should the orchestrator proceed?" + ), + ) + + +def _escalate_blocked_slice_to_hitl( + *, + pipeline_id: str, + slice_id: str, + reason: str, + worktree_repo_path: _pkg.Path, + current_phase: _pkg.PipelinePhase | None, +) -> None: + """Escalate a Layer-C case-4 blocked-without-HITL slice to HITL (slice-4 TASK-4-4). + + Question text is prefixed with ``[#2777 slice-4 TASK-4-4 case 4]`` + so a future dispatch handler in ``routes/decisions.py`` can route + on the literal substring without a separate context field on the + contract-level ``Decision`` model. + """ + _pkg._escalate_layer_c_hitl( + pipeline_id=pipeline_id, + slice_id=slice_id, + worktree_repo_path=worktree_repo_path, + current_phase=current_phase, + question=( + f"[#2777 slice-4 TASK-4-4 case 4] Slice {slice_id} of pipeline " + f"{pipeline_id} is in BLOCKED status, but no PENDING HITL " + f"decision was found on the contract that matches the slice. " + f"{reason}. How should the orchestrator proceed?" + ), + ) + + +def _cross_repo_hold_marker(slice_id: str) -> str: + """Return the stable per-gate discriminator embedded in the hold question.""" + return f"{_pkg._CROSS_REPO_HOLD_MARKER_PREFIX} slice={slice_id}]" + + +def _cross_repo_hold_resolution(contract: _pkg.Any, slice_id: str) -> str | None: + """Return the human's verdict on the cross-repo hold Decision for a slice. + + Scans the (freshly-loaded) contract for the Decision carrying this gate's + :func:`_cross_repo_hold_marker` and, when it is resolved, maps the + operator's SELECTED option to a gate verdict: + + * :data:`cross_repo_merge_gate.RELEASE` — the release option was chosen + (mark the PR ready), else + * :data:`cross_repo_merge_gate.KEEP` — the keep-held option was chosen, OR + the resolution is present but unrecognized (fail-safe: an ambiguous + resolution must NOT auto-ready — cq-1 "human owns the release"). + + Returns ``None`` when the Decision is absent or not yet resolved (keep + waiting). The stored ``Decision.resolution`` may be the option label, the + option id, or a ``{"action":"select","selected":<label>}`` envelope (the + SDLC HITL CLI shape), so we unwrap the envelope and match on both id and a + distinctive keyword. This is the release path that honours the operator's + choice rather than readying on the bare resolved-boolean + (reviewer_code_holistic v1 NACK). + """ + try: + from cross_repo_merge_gate import KEEP, RELEASE + except ImportError: + from ..cross_repo_merge_gate import KEEP, RELEASE # type: ignore[no-redef] + + marker = _pkg._cross_repo_hold_marker(slice_id) + decision = None + for d in getattr(contract, "decisions", None) or []: + if marker in (getattr(d, "question", "") or ""): + decision = d + break + if decision is None or not getattr(decision, "resolved", False): + return None + + raw = getattr(decision, "resolution", None) or "" + # Unwrap the ``{"action":"select","selected":<label>}`` envelope the SDLC + # HITL CLI sends (mirrors routes.decisions._normalize_choice_resolution), + # tolerating a bare string / non-JSON resolution unchanged. + selected = raw + try: + import json as _json + + payload = _json.loads(raw) + if isinstance(payload, dict) and payload.get("action") == "select": + sel = payload.get("selected") + if isinstance(sel, str): + selected = sel + except ValueError, TypeError: + pass + + text = selected.strip().lower() + # #3393 task-5-1/gap-2 (defends the operator's cq-1 fail-safe ruling): + # release ONLY on an EXACT match against the release option's id or + # label. The prior ``"release" in text`` substring check failed OPEN + # — a freeform "Other" resolution that merely CONTAINS the word + # "release" in a negating sense (e.g. "do NOT release yet") would have + # auto-readied a PR the human meant to keep held, a narrower + # reintroduction of the "keep-held is a lie" class reviewer_code_holistic + # NACK'd. Exact equality (after envelope-unwrap + strip + lower) keeps + # the designed path (selecting opt-release / its label) working while + # every ambiguous or negated value falls through to the KEEP fail-safe. + if text in ( + _pkg._CROSS_REPO_HOLD_RELEASE_OPTION_ID.lower(), + _pkg._CROSS_REPO_HOLD_RELEASE_OPTION_LABEL.lower(), + ): + return RELEASE + # Any other resolved value (the keep option, or an unrecognized/freeform + # string) keeps the PR held — never ready on an ambiguous selection. + return KEEP + + +def _register_cross_repo_hold( + *, + pipeline_id: str, + slice_id: str, + repo: str, + pr_number: int, + reason: str, + worktree_repo_path: _pkg.Path, + current_phase: _pkg.PipelinePhase | None, +) -> bool: + """Ensure a cross-repo merge-sequencing HITL hold exists on the contract. + + Idempotent: if a Decision carrying this gate's marker already exists + (pending OR resolved), no new Decision is created. Returns ``True`` + when a hold now exists for the gate (freshly registered or already + present), ``False`` only when registration could not be persisted — + the poll uses the return to decide whether the gate has been handed + off to the HITL release path. Modelled on :func:`_escalate_layer_c_hitl` + (loads the live contract from the per-pipeline worktree, allocates a + ``cq-N`` id, appends an unresolved HITL Decision, saves). The hold + surfaces on ``/status`` via the existing pending-decision collector. + """ + try: + from egg_contracts.decisions import next_cq_id + from egg_contracts.loader import load_contract, save_contract + from egg_contracts.models import Decision, DecisionOption, DecisionType + except ImportError: + try: + from orchestrator.egg_contracts.decisions import ( # type: ignore[no-redef] + next_cq_id, + ) + from orchestrator.egg_contracts.loader import ( # type: ignore[no-redef] + load_contract, + save_contract, + ) + from orchestrator.egg_contracts.models import ( # type: ignore[no-redef] + Decision, + DecisionOption, + DecisionType, + ) + except ImportError: + _pkg.logger.warning( + "Cross-repo hold skipped: egg_contracts not importable (#3393)", + pipeline_id=pipeline_id, + slice_id=slice_id, + ) + return False + + marker = _pkg._cross_repo_hold_marker(slice_id) + reason_text = _pkg._CROSS_REPO_HOLD_REASON_TEXT.get(reason, reason) + try: + with _pkg.get_pipeline_state_lock(pipeline_id): + contract_local = load_contract(pipeline_id, worktree_repo_path) + # Idempotent: a hold Decision for this gate already exists. + for d in contract_local.decisions or []: + if marker in (getattr(d, "question", "") or ""): + return True + decision_id = next_cq_id(contract_local.decisions) + question = ( + f"{marker} Slice {slice_id} of pipeline {pipeline_id} opened PR " + f"{repo}#{pr_number} as a draft behind a cross-repo dependency, " + f"but {reason_text}. Choose how the orchestrator should proceed: " + f"selecting '{_pkg._CROSS_REPO_HOLD_RELEASE_OPTION_LABEL}' marks the PR " + f"ready; selecting '{_pkg._CROSS_REPO_HOLD_KEEP_OPTION_LABEL}' leaves it " + f"draft for you to handle manually." + ) + options = [ + DecisionOption( + id=_pkg._CROSS_REPO_HOLD_RELEASE_OPTION_ID, + label=_pkg._CROSS_REPO_HOLD_RELEASE_OPTION_LABEL, + ), + DecisionOption( + id=_pkg._CROSS_REPO_HOLD_KEEP_OPTION_ID, + label=_pkg._CROSS_REPO_HOLD_KEEP_OPTION_LABEL, + ), + ] + contract_local.decisions.append( + Decision( + id=decision_id, + question=question, + type=DecisionType.HITL, + phase=current_phase or _pkg.PipelinePhase.IMPLEMENT, + options=options, + ) + ) + save_contract(contract_local, worktree_repo_path) + _pkg.logger.info( + "Registered cross-repo merge-sequencing HITL hold (#3393)", + pipeline_id=pipeline_id, + slice_id=slice_id, + repo=repo, + pr_number=pr_number, + reason=reason, + decision_id=decision_id, + ) + return True + except Exception as hold_err: # noqa: BLE001 + _pkg.logger.warning( + "Cross-repo hold registration failed (#3393); PR stays draft, " + "poll will retry next tick", + pipeline_id=pipeline_id, + slice_id=slice_id, + reason=reason, + error=str(hold_err), + ) + return False + + +def _check_slice_evidence_reachability( + pipeline_id: str, + spawner: "ContainerSpawner", # noqa: UP037 + worktree_repo_path: _pkg.Path, + slice_id: str, + integration_branch: str, + *, + gateway_mode: Literal["public", "private"] = "public", + contract: _pkg.Any | None = None, +) -> str | None: + """Verify the slice's cited evidence commits reached the integration branch (#3125). + + The integration branch only advances when a producer pushes + (``consensus_push`` at propose time). A commit recorded by + ``egg-contract complete-task --commit <sha>`` *after* that producer + confirmed — the prescribed HITL unblock flow for a post-confirmation + task reassignment (#3124) — lives only on the agent's local worktree + branch, so the slice would otherwise close and open its PR without + the deliverable while the contract task record points at a commit + nothing retains. + + Runs after slice consensus and before any close side effects (BRC + transcript commit, slice PR). Returns ``None`` when the slice may + close, or a human-readable failure string listing every task row + whose cited commit is not an ancestor of the integration branch tip + — the caller records the slice failure with it, which routes + through the existing cascade + HITL escalation machinery instead of + closing silently. + + Only role-bound task rows are gated (#3339): the check exists for + the producer-scoped #3124 flow, so a ``role=unassigned`` row's + orphan commit is bookkeeping, not a gated deliverable, and must not + fail a consensus-reached slice. See ``cc.evidence_commits``. + + Failure posture mirrors the other completeness checks (#3081 / + #3114): the gate degrades to ``None`` (close proceeds, warning + logged) when the contract cannot be read, the slice id does not + resolve, or the gateway reachability probe cannot be evaluated. + Only a definitive "this cited commit is not on the branch" verdict + fails the close. ``EGG_EVIDENCE_REACHABILITY_GATE`` is the operator + kill switch. + + ``contract`` is an optional pre-loaded contract: the close path + already needs the contract one stretch later for the slice PR data + snapshot, so threading the same load through saves one file read + and one ``get_pipeline_state_lock`` acquisition. When ``None`` + (the default — keeps the gate self-contained for tests), the gate + loads the contract itself under the lock. + """ + try: + import contract_completeness as cc + except ImportError: + from .. import contract_completeness as cc # type: ignore[no-redef] + + if not cc.evidence_gate_enabled(): + _pkg.logger.info( + "Evidence-reachability gate disabled by kill switch (#3125)", + pipeline_id=pipeline_id, + slice_id=slice_id, + ) + return None + + if contract is None: + from egg_contracts.loader import load_contract as _load_contract + + try: + with _pkg.get_pipeline_state_lock(pipeline_id): + contract = _load_contract(pipeline_id, worktree_repo_path) + except Exception as load_err: # noqa: BLE001 + _pkg.logger.warning( + "Evidence-reachability gate skipped: contract load failed (#3125)", + pipeline_id=pipeline_id, + slice_id=slice_id, + error=str(load_err), + ) + return None + + rows = cc.evidence_commits(contract, slice_id) + if rows is None: + _pkg.logger.warning( + "Evidence-reachability gate skipped: slice not found in contract (#3125)", + pipeline_id=pipeline_id, + slice_id=slice_id, + ) + return None + if not rows: + return None + + # De-duplicate while preserving first-seen order: multiple task rows + # can cite the same commit (the prescribed unblock flow #3124 often + # links one commit across two adjacent rows). Each duplicate would + # otherwise burn one merge-base round-trip per dupe. The membership + # join below re-attaches the verdict to every row that cites it. + probe_shas = list(dict.fromkeys(r["commit"] for r in rows)) + unreachable_shas = spawner.gateway.find_unreachable_evidence_commits( + pipeline_id, + str(worktree_repo_path), + commit_shas=probe_shas, + integration_branch=integration_branch, + mode=gateway_mode, + ) + if unreachable_shas is None: + # The probe itself could not be evaluated (gateway/network). + # find_unreachable_evidence_commits already logged the cause. + return None + if not unreachable_shas: + return None + + lost = [r for r in rows if r["commit"] in set(unreachable_shas)] + summary = cc.format_evidence_rows(lost) + _pkg.logger.error( + "Slice close blocked: task records cite commits unreachable from " + "the integration branch (#3125)", + pipeline_id=pipeline_id, + slice_id=slice_id, + integration_branch=integration_branch, + unreachable=summary, + ) + return ( + f"slice {slice_id}: evidence-reachability gate failed — contract task " + f"records cite commits that are not on integration branch " + f"{integration_branch}: {summary}. Cherry-pick (or push) the cited " + f"commits onto {integration_branch}, then re-run the slice close; " + f"set {cc.EVIDENCE_GATE_ENV_VAR}=off to bypass." + ) diff --git a/orchestrator/routes/pipelines/_stacked_pr.py b/orchestrator/routes/pipelines/_stacked_pr.py new file mode 100644 index 0000000000..2d63f5fe67 --- /dev/null +++ b/orchestrator/routes/pipelines/_stacked_pr.py @@ -0,0 +1,251 @@ +"""stacked-PR reconciler launcher helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _start_stacked_pr_reconciler( + pipeline_id: str, + contract_loader: _pkg.Callable[[], _pkg.Any], + gateway, + pipeline, + *, + interval_seconds: float | None = None, + worktree_repo_path: _pkg.Path | None = None, + repo: str | None = None, +) -> tuple[_pkg.threading.Thread, _pkg.threading.Event]: + """Start the periodic stacked-PR reconciler as a daemon thread (#2137 TASK-5-3). + + Returns ``(thread, stop_event)``: caller calls ``stop_event.set()`` + when the implement phase is shutting down so the daemon exits + cleanly. The daemon loops on the configured interval and invokes + :func:`stacked_pr_reconciler.reconcile_once` with callables that + decouple it from the gateway client. + + The list-callables (``list_open_prs`` and ``list_remote_branches``) + forward to ``GatewayClient.list_open_prs`` / + ``GatewayClient.list_remote_branches``. ``list_open_prs`` routes + through the launcher-authed control-plane route + ``/api/v1/gh/list_open_prs`` (#2925); ``list_remote_branches`` routes + through the existing per-agent ``git ls-remote`` allowlist. The rebase + callable forwards to + ``GatewayClient.rebase_onto``, which performs the full local + rebase + ``--force-with-lease`` push + ``gh api PATCH base=…`` + retarget so an orphaned child PR is fully healed on origin + rather than just locally rewritten. + """ + try: + from orchestrator.env_config import get_stacked_pr_reconciler_interval_seconds + except ImportError: + from env_config import ( # type: ignore[no-redef] + get_stacked_pr_reconciler_interval_seconds, + ) + try: + from orchestrator.stacked_pr_reconciler import reconcile_once + except ImportError: + from stacked_pr_reconciler import reconcile_once # type: ignore[no-redef] + # #3393 slice-5: the cross-repo merge-sequencing gate rides the SAME + # reconciler cadence (no new scheduler subsystem). Imported here (not + # top-level) to keep this helper's import surface minimal, mirroring + # the ``reconcile_once`` import above. + try: + import orchestrator.cross_repo_merge_gate as cross_repo_merge_gate + except ImportError: + import cross_repo_merge_gate # type: ignore[no-redef] + try: + from orchestrator.env_config import get_cross_repo_merge_gate_max_attempts + except ImportError: + from env_config import ( # type: ignore[no-redef] + get_cross_repo_merge_gate_max_attempts, + ) + try: + from orchestrator.models import resolve_slice_repo + except ImportError: + from models import resolve_slice_repo # type: ignore[no-redef] + + if interval_seconds is None: + try: + interval_seconds = float(get_stacked_pr_reconciler_interval_seconds()) + except Exception: # noqa: BLE001 + interval_seconds = 30.0 + + stop_event = _pkg.threading.Event() + + # ``repo_path`` must be a filesystem path the gateway's + # ``validate_repo_path`` accepts (``/home/egg/repos/``, + # ``/home/egg/.egg-worktrees/``, etc.) — NOT the git branch + # name. Use the orchestrator-side worktree path that the + # implement loop already owns. + repo_path_str = str(worktree_repo_path) if worktree_repo_path is not None else "" + pr_repo = repo or str(getattr(pipeline, "repo", "") or "") + + # #3393 slice-5: only multi-repo pipelines can have cross-repo + # dependency edges, so the merge gate is a strict no-op for N=1 — + # skip it entirely rather than burning a contract scan per tick. + _gate_enabled = len(getattr(pipeline, "repos", None) or []) > 1 + # Per-run gate bookkeeping (attempts / hold-registered / resolved), + # keyed by dependent slice id; persists across reconciler ticks. + _gate_state: dict[str, _pkg.Any] = {} + try: + _gate_max_attempts = int(get_cross_repo_merge_gate_max_attempts()) + except Exception: # noqa: BLE001 + _gate_max_attempts = cross_repo_merge_gate.DEFAULT_MAX_POLL_ATTEMPTS + _gate_current_phase = getattr(pipeline, "current_phase", None) + + def _poll_cross_repo_merge_gate(contract: _pkg.Any) -> None: + # Drive one cross-repo merge-sequencing pass on the reconciler + # cadence (#3393 slice-5, task-5-1 / task-5-2). Reads upstream PR + # merge-state and auto-readies a dependent draft PR on merge + # (Tier A); registers a HITL hold on the closed-unmerged / timeout + # terminals and for plan-declared beyond-merge-state edges (Tier + # B). All gateway/contract effects are funnelled through the + # injected callables so the gate logic stays pure + unit-tested. + if not _gate_enabled: + return + cross_repo_merge_gate.poll_once( + contract, + resolve_repo=lambda s: resolve_slice_repo(s, pipeline), + get_merge_state=lambda repo_slug, pr_num: gateway.get_pr_merge_state( + pipeline_id, repo_slug, pr_number=pr_num + ), + mark_ready=lambda repo_slug, pr_num: bool( + gateway.mark_pr_ready(pipeline_id, repo_slug, pr_number=pr_num) + ), + register_hold=lambda gate, reason: _pkg._register_cross_repo_hold( + pipeline_id=pipeline_id, + slice_id=gate.slice_id, + repo=gate.repo, + pr_number=gate.pr_number, + reason=reason, + worktree_repo_path=worktree_repo_path, + current_phase=_gate_current_phase, + ), + hold_resolution=lambda gate: _pkg._cross_repo_hold_resolution(contract, gate.slice_id), + state=_gate_state, + max_attempts=_gate_max_attempts, + ) + + def _list_open_prs() -> list[dict[str, _pkg.Any]]: + # Lists open PRs in ``pr_repo`` so ``find_orphaned_child_prs`` + # can detect children whose base branch was deleted (parent + # merged through the GitHub UI). Routes through the launcher-authed + # control-plane endpoint ``/api/v1/gh/list_open_prs`` — the + # orchestrator is the server that manages pipelines, not an agent, + # so it does not register a synthetic agent session or impersonate + # a role (#2922 / #2925). + if not pr_repo: + return [] + try: + return list(gateway.list_open_prs(pipeline_id, pr_repo)) + except Exception as exc: # noqa: BLE001 + _pkg.logger.debug( + "stacked_pr_reconciler: list_open_prs raised — treating as empty", + pipeline_id=pipeline_id, + error=str(exc), + ) + return [] + + def _list_extant_branches() -> set[str]: + # Lists remote branches via ``git ls-remote --heads origin`` + # so the reconciler can detect deleted parents. Routes through + # the existing per-agent ``git ls-remote`` allowlist. The + # synthetic session uses ``agent_role="orchestrator"`` so this + # orchestrator-driven ls-remote is attributed to the orchestrator + # in the audit log instead of a phantom coder (#2919). + if not repo_path_str: + return set() + try: + return set( + gateway.list_remote_branches( + pipeline_id, + repo_path_str, + agent_role="orchestrator", + ) + ) + except Exception as exc: # noqa: BLE001 + _pkg.logger.debug( + "stacked_pr_reconciler: list_remote_branches raised — treating as empty", + pipeline_id=pipeline_id, + error=str(exc), + ) + return set() + + def _rebase_onto(orphan: _pkg.Any) -> bool: + # ``orphan`` is a ``stacked_pr_reconciler.OrphanedChildPR``; + # avoid the import here so this module stays a pure consumer + # of the reconciler's typed interface (the type checker at + # the reconciler boundary already validates the shape). + try: + return bool( + gateway.rebase_onto( + pipeline_id, + repo_path_str, + branch=orphan.branch, + new_base=orphan.intended_new_base, + old_base=orphan.deleted_base, + pr_number=orphan.pr_number, + repo=pr_repo or None, + # Orchestrator-driven heal (rebase + force-push + + # pr-edit); attribute to the orchestrator, not a + # phantom coder (#2919). The force-push targets the + # slice integration branch on a synthetic session, so + # the slice-integration exemption admits it regardless + # of role. + agent_role="orchestrator", + ) + ) + except Exception: # noqa: BLE001 + _pkg.logger.debug( + "stacked_pr_reconciler: rebase_onto raised — counted as failure", + pipeline_id=pipeline_id, + branch=getattr(orphan, "branch", "?"), + ) + return False + + def _loop() -> None: + # Defensive: a slow tick must not pin this thread on a stale + # sleep — Event.wait returns True the moment ``stop_event`` is + # set, so shutdown is bounded by the configured interval. + while not stop_event.wait(interval_seconds): + try: + contract = contract_loader() + if contract is None: + continue + reconcile_once( + contract, + list_open_prs=_list_open_prs, + list_extant_branches=_list_extant_branches, + rebase_onto=_rebase_onto, + ) + # #3393 slice-5: drive the cross-repo merge-sequencing + # gate on the same tick + same freshly-loaded contract. + # No-op for N=1 pipelines. Wrapped in its own try so a + # gate failure never disrupts stacked-PR reconciliation. + try: + _poll_cross_repo_merge_gate(contract) + except Exception as gate_exc: # noqa: BLE001 + _pkg.logger.debug( + "cross_repo_merge_gate tick raised — continuing", + pipeline_id=pipeline_id, + error=str(gate_exc), + ) + except Exception as exc: # noqa: BLE001 + _pkg.logger.debug( + "stacked_pr_reconciler tick raised — continuing", + pipeline_id=pipeline_id, + error=str(exc), + ) + + thread = _pkg.threading.Thread( + target=_loop, + name=f"stacked-pr-reconciler-{pipeline_id}", + daemon=True, + ) + thread.start() + return thread, stop_event diff --git a/orchestrator/routes/pipelines/_statefiles.py b/orchestrator/routes/pipelines/_statefiles.py new file mode 100644 index 0000000000..a184ef653a --- /dev/null +++ b/orchestrator/routes/pipelines/_statefiles.py @@ -0,0 +1,613 @@ +"""statefiles helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim; patched/barrel-resident globals reached via _pkg so +patch("routes.pipelines.<name>") keeps intercepting. +""" + +from __future__ import annotations + +import glob +import json +import subprocess +from pathlib import Path +from typing import Any + +import routes.pipelines as _pkg # noqa: E402,F401 +from models import Pipeline + + +def _commit_statefiles_to_worktree( + worktree_path: Path, + message: str, + pipeline_identifier: int | str | None = None, + *, + pipeline_id: str | None = None, +) -> bool: + """Stage and commit ``.egg-state/`` files in *worktree_path*. + + When *pipeline_identifier* is provided, only files whose names start + with the identifier (followed by ``.`` or ``-``) are staged. This + prevents concurrent pipelines from leaking each other's state files + into unrelated PRs (see #1390). + + Most ``.egg-state/`` files are prefixed with the issue number (drafts, + reviews, BRC history, agent-outputs), but contract files are keyed by + ``pipeline_id`` (e.g. ``issue-1759-v3.json``) and don't share the + issue-number prefix. When *pipeline_id* is provided alongside + *pipeline_identifier*, files matching either prefix are staged — this + closes the gap where plan-phase contract updates were written to disk + but never committed because the glob only saw the issue-number prefix + (see #1829). + + Falls back to staging the entire ``.egg-state/`` directory when both + *pipeline_identifier* and *pipeline_id* are ``None`` (backwards-compat). + + Any pre-existing staged changes in the worktree's index are discarded + on entry — the helper runs ``git read-tree HEAD`` before staging (see + :func:`_read_tree_head` for the cross-worktree-ref-advance defence + from #2626). Only files matching the pipeline scope and present on + disk are committed; callers must not pre-stage state they expect this + helper to preserve. + + The commit is idempotent (skips when nothing is staged). + Raises ``subprocess.CalledProcessError`` on git failure. + Call sites decide whether to abort or continue. + + Returns ``True`` when a commit was actually made, ``False`` when the + helper short-circuited (no .egg-state dir, no prefix match, or + nothing staged after add). Lets call sites skip a follow-up push + that would be a no-op fast-forward (#2548 review suggestion D). + """ + state_dir = worktree_path / ".egg-state" + _pkg.logger.info( + "_commit_statefiles_to_worktree: entering", + worktree_path=str(worktree_path), + pipeline_identifier=str(pipeline_identifier), + pipeline_id=str(pipeline_id), + commit_message=message, + ) + if not state_dir.exists(): + _pkg.logger.info( + "_commit_statefiles_to_worktree: no .egg-state directory — exiting", + worktree_path=str(worktree_path), + pipeline_identifier=str(pipeline_identifier), + pipeline_id=str(pipeline_id), + ) + return False # Nothing to commit yet + + git_base = [ + "git", + "-c", + "core.hooksPath=/dev/null", + "-c", + f"safe.directory={worktree_path}", + "-C", + str(worktree_path), + ] + + if pipeline_identifier is not None or pipeline_id is not None: + # Scope to files belonging to this pipeline only (#1390). + # Use prefix-anchored patterns with delimiter boundaries to avoid + # substring false positives (e.g. pipeline 4 matching pipeline 42). + # Union both prefixes so issue-number-prefixed files (drafts, + # reviews, BRC history) and pipeline-id-keyed files (contracts) + # are all staged (#1829). + prefixes: list[str] = [] + if pipeline_identifier is not None: + prefixes.append(str(pipeline_identifier)) + if pipeline_id is not None and pipeline_id not in prefixes: + prefixes.append(pipeline_id) + + matched_set: set[str] = set() + for pid in prefixes: + escaped = glob.escape(pid) + pattern_dot = str(state_dir / "**" / f"{escaped}.*") + pattern_dash = str(state_dir / "**" / f"{escaped}-*") + for f in glob.glob(pattern_dot, recursive=True) + glob.glob( + pattern_dash, recursive=True + ): + if Path(f).is_file(): + matched_set.add(f) + matched = sorted(matched_set) + _pkg.logger.info( + "_commit_statefiles_to_worktree: glob match results", + pipeline_identifier=str(pipeline_identifier), + pipeline_id=str(pipeline_id), + prefixes=prefixes, + match_count=len(matched), + matched_paths=[str(Path(f).relative_to(worktree_path)) for f in matched[:20]], + truncated=len(matched) > 20, + ) + if not matched: + return False # No state files for this pipeline yet + + rel_paths = [str(Path(f).relative_to(worktree_path)) for f in matched] + _pkg._read_tree_head(git_base) + # Restore scope is intentionally broader than the staging glob: + # the helper operates over all of ``.egg-state/`` to maintain + # HEAD↔disk parity (so other readers — e.g. peer-artifact loads — + # see what HEAD says). Each pipeline has its own worktree, so + # broader scope cannot resurrect a sibling-pipeline file. + _pkg._restore_missing_state_files_from_head(git_base, worktree_path, pipeline_id) + subprocess.run( + [*git_base, "add", "--force", "--"] + rel_paths, + capture_output=True, + text=True, + check=True, + timeout=30, + ) + else: + _pkg._read_tree_head(git_base) + # Restore scope matches the staging scope here — both span all of + # ``.egg-state/`` — so the broader restore is trivially safe. + _pkg._restore_missing_state_files_from_head(git_base, worktree_path, pipeline_id) + subprocess.run( + [*git_base, "add", "--force", ".egg-state/"], + capture_output=True, + text=True, + check=True, + timeout=30, + ) + + # Only commit if there are staged changes (idempotent on re-runs). + # No pathspec: match the diff scope to the commit scope below so the + # early-out fires iff the commit would have nothing to write. A + # scoped diff (``-- .egg-state/``) paired with the unscoped commit + # below would short-circuit when only non-``.egg-state/`` content + # is staged, dropping that content on the floor instead of + # committing it. Nothing in this code path stages outside + # ``.egg-state/`` today, so this is belt-and-suspenders, but the + # two scopes must stay symmetric to keep the invariant local. + result = subprocess.run( + [*git_base, "diff", "--cached", "--quiet"], + capture_output=True, + text=True, + check=False, + timeout=30, + ) + if result.returncode == 0: + _pkg.logger.info( + "_commit_statefiles_to_worktree: nothing staged — skipping commit", + pipeline_identifier=str(pipeline_identifier), + commit_message=message, + ) + return False # Nothing to commit + + _pkg.logger.info( + "_commit_statefiles_to_worktree: staged changes detected — committing", + pipeline_identifier=str(pipeline_identifier), + commit_message=message, + ) + # Commit WITHOUT a trailing ``-- .egg-state/`` pathspec. ``git commit`` + # with a pathspec defaults to ``--only`` semantics, which auto-stages + # working-tree changes (including unstaged *deletions*) for the + # matching paths — i.e. ``git commit -- .egg-state/`` silently picks + # up files that disappeared from disk even though the explicit + # ``git add`` above only staged the on-disk hits from the glob. This + # surfaces as two distinct failure shapes that share the same + # mechanism (HEAD references a draft that is not on disk locally): + # #2625, where agents push drafts to ``origin/<branch>`` from their + # own worktrees so the orchestrator's local checkout sits at a HEAD + # containing files it never materialised; and #2626, where the + # agent-side ``git update-ref`` recovery (plumbing, no per-worktree + # branch lock) advances the shared pipeline-branch ref under the + # orchestrator's worktree, leaving every agent-pushed file looking + # like a staged deletion. In both cases the pathspec form turned a + # benign working-tree gap into a delete-commit against agent-pushed + # work. Without the pathspec, only the explicit ``git add`` staging + # above is committed. + subprocess.run( + [*git_base, "commit", "--no-verify", "-m", message], + capture_output=True, + text=True, + check=True, + timeout=30, + ) + _pkg.logger.info( + "_commit_statefiles_to_worktree: commit succeeded", + pipeline_identifier=str(pipeline_identifier), + commit_message=message, + ) + return True + + +def persist_contract_statefiles( + pipeline_id: str, + worktree_path: Path, + message: str, + *, + pipeline: Pipeline | None = None, +) -> bool: + """Durably persist a contract decision write: commit + push to the work branch. + + Contract HITL decisions (``cq-N`` registrations and resolutions) are + written to the shared pipeline worktree's contract file with no git + commit; the file was only serialized to the work branch at slice/phase + checkpoints. Both phase-(re)start syncs — the gateway's worktree-reuse + reset and ``_sync_worktree_with_remote`` step 4 — run + ``git reset --hard origin/<work>``, so any decision write that had not + been committed AND pushed by then was silently reverted, letting the + bootstrap reconciler re-mint the same ``cq-N`` ids and clobber + just-resolved operator decisions (#3427). Committing and pushing at + write time makes the reset target already contain the decision. + + Best-effort by design: failures are logged and swallowed — the write is + still live on the worktree file and the next checkpoint commit retries. + Returns ``True`` only when the state was committed and pushed (or there + was nothing new to commit). + """ + try: + if pipeline is None: + _, pipeline = _pkg._resolve_pipeline(pipeline_id, _pkg.get_repo_path()) + identifier = _pkg._pipeline_identifier(getattr(pipeline, "issue_number", None), pipeline_id) + committed = _pkg._commit_statefiles_to_worktree( + worktree_path, + message, + identifier, + pipeline_id=pipeline_id, + ) + if not committed: + return True # Nothing new on disk — already durable. + branch = getattr(pipeline, "branch", None) + if not branch: + _pkg.logger.warning( + "Contract decision write committed but pipeline has no work " + "branch to push to; the commit is local-only and a worktree " + "reset may still discard it (#3427)", + pipeline_id=pipeline_id, + ) + return False + gateway_mode, _ = _pkg._compute_gateway_mode(pipeline) + _pkg._get_spawner().gateway.push_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(worktree_path), + branch=branch, + mode=gateway_mode, + base_branch=getattr(pipeline, "base_branch", None), + ) + _pkg.logger.info( + "Contract decision write persisted to work branch (#3427)", + pipeline_id=pipeline_id, + branch=branch, + commit_message=message, + ) + return True + except Exception as persist_err: # noqa: BLE001 — best-effort durability + _pkg.logger.warning( + "Failed to durably persist contract decision write; the decision " + "is live on the worktree file but will not survive a worktree " + "reset until the next checkpoint commit (#3427)", + pipeline_id=pipeline_id, + error=str(persist_err), + ) + return False + + +def _ensure_statefiles_on_branch( + worktree_repo_path: Path, + pipeline: Pipeline, +) -> bool: + """Verify the contract file exists in the worktree and re-create if missing. + + This is a safety net for short-flow pipelines where the initial contract + push may have failed or where subsequent pushes diverged. + + Returns True if the contract exists (or was successfully restored), + False if restoration failed. + """ + from egg_contracts.loader import contract_exists, create_contract, get_contract_path + + # Contract lookup uses pipeline.id directly (canonical key). + if contract_exists(pipeline.id, worktree_repo_path): + return True + + canonical_path = get_contract_path(pipeline.id, worktree_repo_path) + + _pkg.logger.warning( + "Contract file missing from worktree — attempting restoration", + pipeline_id=pipeline.id, + expected_path=str(canonical_path), + ) + + try: + # Mirror the primary creation site: the composed task statement + # (identity anchor + submit description, #3163) lands on the + # restored contract too, for every entry path. + from egg_contracts.loader import compose_task_description + + issue_url = ( + f"https://github.com/{pipeline.repo}/issues/{pipeline.issue_number}" + if pipeline.issue_number is not None + else None + ) + task_description = compose_task_description( + description=pipeline.prompt, + issue_number=pipeline.issue_number, + issue_url=issue_url, + jira_ticket=pipeline.jira_ticket, + ) + if pipeline.issue_number is not None: + create_contract( + issue_number=pipeline.issue_number, + title=f"Issue #{pipeline.issue_number}", + url=issue_url or "", + pipeline_id=pipeline.id, + repo_root=worktree_repo_path, + task_description=task_description, + ) + else: + create_contract( + pipeline_id=pipeline.id, + title=(pipeline.prompt or "")[:100], + task_description=task_description, + repo_root=worktree_repo_path, + ) + + # Restore plan/analysis drafts from remote if missing locally. + # These were pushed during init but may be lost from the worktree + # after agent activity during the implement phase (#1454). + if pipeline.branch: + git_base = [ + "git", + "-c", + "core.hooksPath=/dev/null", + "-c", + f"safe.directory={worktree_repo_path}", + "-C", + str(worktree_repo_path), + ] + # Ensure remote-tracking ref is fresh before reading from it. + try: + subprocess.run( + [*git_base, "fetch", "origin", pipeline.branch], + capture_output=True, + text=True, + timeout=30, + check=False, + ) + except Exception: + pass # Best-effort; git show may still work with cached ref + for draft_phase in ("plan", "refine"): + draft_rel = _pkg._get_draft_path( + draft_phase, + issue_number=pipeline.issue_number, + pipeline_id=pipeline.id, + ) + if not draft_rel: + continue + draft_path = worktree_repo_path / draft_rel + if draft_path.exists(): + continue + try: + result = subprocess.run( + [*git_base, "show", f"origin/{pipeline.branch}:{draft_rel}"], + capture_output=True, + text=True, + timeout=15, + check=False, + ) + if result.returncode == 0 and result.stdout: + draft_path.parent.mkdir(parents=True, exist_ok=True) + draft_path.write_text(result.stdout, encoding="utf-8") + _pkg.logger.info( + "Restored draft from remote branch", + pipeline_id=pipeline.id, + draft_path=draft_rel, + ) + except Exception as e: + _pkg.logger.warning( + "Could not restore draft from remote", + pipeline_id=pipeline.id, + draft_path=draft_rel, + error=str(e), + ) + + # Final fallback: write plan/analysis from pipeline model if still + # missing after remote restoration attempt. This handles the case + # where the draft was never pushed to the remote (#1460). + for draft_phase, field_value in [("plan", pipeline.plan), ("refine", pipeline.analysis)]: + if not field_value: + continue + draft_rel = _pkg._get_draft_path( + draft_phase, + issue_number=pipeline.issue_number, + pipeline_id=pipeline.id, + ) + if not draft_rel: + continue + draft_path = worktree_repo_path / draft_rel + if draft_path.exists(): + continue + draft_path.parent.mkdir(parents=True, exist_ok=True) + draft_path.write_text(field_value, encoding="utf-8") + _pkg.logger.info( + "Restored draft from pipeline model (remote unavailable)", + pipeline_id=pipeline.id, + draft_path=draft_rel, + ) + + # Re-populate tasks and PR metadata from plan draft if available. + # Without this, recreated contracts lose the planner-generated PR + # title/description and fall back to the generic pipeline ID title. + # See: https://github.com/jwbron/egg/issues/1432 + _restore_populate_result = _pkg._populate_contract_from_plan( + worktree_repo_path, + pipeline.id, + pipeline.mode.value if pipeline.mode else "issue", + pipeline.issue_number, + ) + # #2627 follow-up: this is a best-effort restoration path on a + # recreated contract — failure here is recoverable on later + # pipeline steps, so we just log the structured outcome. + if _restore_populate_result.outcome != _pkg.PopulateOutcome.POPULATED: + _pkg.logger.info( + "Restored-contract populate produced non-POPULATED outcome", + pipeline_id=pipeline.id, + outcome=_restore_populate_result.outcome.value, + ) + + # File-staging identifier still uses _pipeline_identifier convention. + identifier = _pkg._pipeline_identifier(pipeline.issue_number, pipeline.id) + _pkg._commit_statefiles_to_worktree( + worktree_repo_path, + f"Restore missing contract for {identifier}", + pipeline_identifier=identifier, + pipeline_id=pipeline.id, + ) + _pkg.logger.info( + "Contract file restored successfully", + pipeline_id=pipeline.id, + ) + return True + except Exception as restore_err: + _pkg.logger.error( + "Failed to restore contract file", + pipeline_id=pipeline.id, + error=str(restore_err), + ) + return False + + +def _detect_default_branch(worktree_repo_path: Path) -> str: + """Detect the remote's default branch from a worktree. + + Tries in order: + 1. origin/HEAD symbolic ref (most reliable) + 2. origin/main + 3. origin/master + 4. Fallback to "main" + + Returns: + The branch name (e.g., "main" or "master"), without the "origin/" prefix. + """ + # Try origin/HEAD symbolic ref + try: + result = subprocess.run( + ["git", "symbolic-ref", "refs/remotes/origin/HEAD", "--short"], + capture_output=True, + text=True, + cwd=str(worktree_repo_path), + timeout=10, + check=False, + ) + if result.returncode == 0 and result.stdout.strip(): + ref = result.stdout.strip() # e.g. "origin/main" + return ref.removeprefix("origin/") + except Exception: + pass + + # Try origin/main + try: + result = subprocess.run( + ["git", "rev-parse", "--verify", "origin/main"], + capture_output=True, + text=True, + cwd=str(worktree_repo_path), + timeout=10, + check=False, + ) + if result.returncode == 0: + return "main" + except Exception: + pass + + # Try origin/master + try: + result = subprocess.run( + ["git", "rev-parse", "--verify", "origin/master"], + capture_output=True, + text=True, + cwd=str(worktree_repo_path), + timeout=10, + check=False, + ) + if result.returncode == 0: + return "master" + except Exception: + pass + + _pkg.logger.warning( + "Could not detect default branch, falling back to 'main'", + worktree_path=str(worktree_repo_path), + ) + return "main" + + +def _resolve_origin_ref(base_branch: str | None) -> str: + """Return ``origin/<branch>``, falling back to ``origin/main``. + + Centralises the ``f"origin/{base_branch}" if base_branch else "origin/main"`` + pattern so every orient-prompt / diff-command call site honours the + resolved base branch consistently. + """ + ref = (base_branch or "main").strip() or "main" + # Tolerate callers that already passed ``origin/<x>`` by mistake. + if ref.startswith("origin/"): + return ref + return f"origin/{ref}" + + +def _fetch_pr_state(pr_number: int, repo: str | None = None) -> dict[str, Any]: + """Fetch PR state, base/head refs, and fork-hint via ``gh pr view``. + + Returns a dict with keys ``state`` (str, e.g. "OPEN"/"MERGED"/"CLOSED"), + ``base_ref`` (str or None), ``head_ref`` (str or None), ``head_sha`` + (str or None), ``is_fork`` (bool), ``changed_files`` (int), and + ``head_repository_name_with_owner`` (str or None). Returns an empty + dict when ``gh`` is unavailable or the PR cannot be looked up. + """ + if pr_number is None: + return {} + fields = ( + "state,baseRefName,headRefName,headRefOid,isCrossRepository," + "changedFiles,headRepositoryOwner,headRepository" + ) + cmd = ["gh", "pr", "view", str(pr_number), "--json", fields] + if repo: + cmd.extend(["--repo", repo]) + try: + result = subprocess.run( + cmd, + capture_output=True, + text=True, + timeout=20, + check=False, + ) + except Exception as exc: # pragma: no cover - defensive + _pkg.logger.warning( + "_fetch_pr_state: gh pr view raised", + pr_number=pr_number, + repo=repo, + error=str(exc), + ) + return {} + if result.returncode != 0: + _pkg.logger.warning( + "_fetch_pr_state: gh pr view failed", + pr_number=pr_number, + repo=repo, + returncode=result.returncode, + stderr=result.stderr.strip()[:200], + ) + return {} + try: + data = json.loads(result.stdout) + except json.JSONDecodeError, ValueError: + return {} + + head_repo = data.get("headRepository") or {} + head_owner = data.get("headRepositoryOwner") or {} + head_repo_name = head_repo.get("name") if isinstance(head_repo, dict) else None + head_owner_login = head_owner.get("login") if isinstance(head_owner, dict) else None + head_repo_full = ( + f"{head_owner_login}/{head_repo_name}" if head_owner_login and head_repo_name else None + ) + return { + "state": data.get("state"), + "base_ref": data.get("baseRefName"), + "head_ref": data.get("headRefName"), + "head_sha": data.get("headRefOid"), + "is_fork": bool(data.get("isCrossRepository")), + "changed_files": data.get("changedFiles") or 0, + "head_repository_name_with_owner": head_repo_full, + } diff --git a/orchestrator/routes/pipelines/_status_view.py b/orchestrator/routes/pipelines/_status_view.py new file mode 100644 index 0000000000..445b09c001 --- /dev/null +++ b/orchestrator/routes/pipelines/_status_view.py @@ -0,0 +1,391 @@ +"""pipeline status view + slice diff summary helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Literal # noqa: F401 + +import routes.pipelines as _pkg # noqa: E402,F401 + +if TYPE_CHECKING: + try: + from ..container_spawner import ContainerSpawner # noqa: F401 + except ImportError: # pragma: no cover + from container_spawner import ContainerSpawner # type: ignore # noqa: F401 + + +def _get_pr_info(pipeline: _pkg.Pipeline) -> tuple[str | None, int | None]: + """Extract context-PR URL and number from the pipeline contract. + + Returns ``(pr_url, pr_number)`` or ``(None, None)`` when no PR has + been opened. Under #2777 the PR phase was removed and the context + PR opens up-front via ``_open_context_pr_at_implement_start`` which + persists ``context_pr_number`` to ``contract.pr.context_pr_number``; + we read that directly. ``pr_url`` is also persisted on the pipeline + record by ``_open_context_pr_at_implement_start`` for downstream + consumers (the JIRA reassess sweep at ``jira_reassess.py``). + """ + # ``Pipeline.pr_url`` / ``Pipeline.pr_number`` are populated by the + # up-front opener; they are the canonical surface for callers that + # used to read ``phases["pr"].artifacts["pr_url"]``. + pr_url = getattr(pipeline, "pr_url", None) + pr_number = getattr(pipeline, "pr_number", None) + if not pr_url: + return None, None + if pr_number is None: + match = _pkg.re.search(r"/pull/(\d+)", pr_url) + pr_number = int(match.group(1)) if match else None + return pr_url, pr_number + + +def _consensus_block(consensus_state: dict) -> dict: + """Slim a tracker ``get_state()`` snapshot down to the status payload. + + Keeps the fields operators act on (per-role phases + confirmed + flags, the blocking set, and the unresolved-NACK details: who + NACKed whom, on which version, and why; #3481) and drops the + bulky ``approval_matrix`` / ``review_graph`` dumps. + + BRC trackers only emit dict-format agent entries (the legacy + AgentReadiness object came from the now-deleted ConsensusEvaluator, + cq-5 of #2777). + """ + return { + "agents": dict(consensus_state.get("agents", {})), + "is_complete": consensus_state.get("is_complete", False), + "blocking_agents": consensus_state.get("blocking_agents", []), + "has_unresolved_nacks": consensus_state.get("has_unresolved_nacks", False), + "unresolved_nacks": consensus_state.get("unresolved_nacks", []), + "protocol": consensus_state.get("protocol", "brc"), + } + + +def _get_concurrent_status(pipeline: _pkg.Pipeline, slice_id: str | None = None) -> dict | None: + """Get concurrent execution monitoring data for a pipeline. + + Returns None if concurrent execution is not enabled for this pipeline. + Returns a dict with the following structure when concurrent mode is active:: + + { + "enabled": True, + "max_concurrent_agents": int, + "messages": {"total": int, "by_type": {"PROGRESS": int, ...}}, + "consensus": { + "agents": {"coder": {"state": "READY", ...}, ...}, + "is_complete": bool, + "blocking_agents": ["role", ...] # agents not yet READY + }, + "agents": [{"role": str, "status": str}, ...] # from phase execution + } + + Dependencies on other concurrent-mode modules (message_store, consensus) are + imported lazily and degrade gracefully to empty structures when unavailable. + + ``slice_id``: in a slice-DAG implement phase each slice runs its own + BRC consensus, keyed ``{pipeline_id}/{slice_id}``. The bare pipeline + id has no tracker, so a non-slice lookup reported a misleading + cross-slice reconstruction (#2761). Callers querying a per-slice + agent's consensus must pass that agent's ``slice_id``; the consensus + block then reflects exactly that slice's tracker. When omitted, only + pipeline-level (non-slice) consensus is reported in ``consensus``; + a slice-DAG pipeline queried without a slice yields no ``consensus`` + block rather than a fabricated one. Instead, live slice-scoped + trackers are surfaced under ``slice_consensus`` keyed by slice_id + (#3481), so operators still see each active round's real state. + """ + try: + from concurrent_executor import is_concurrent_execution + except ImportError: + from ..concurrent_executor import is_concurrent_execution # type: ignore[no-redef] + + current_phase = pipeline.current_phase.value if pipeline.current_phase else None + if not is_concurrent_execution(pipeline, phase=current_phase): + return None + + config = pipeline.config + result: dict = { + "enabled": True, + "max_concurrent_agents": getattr(config, "max_concurrent_agents", 6), + } + + # Message store provides aggregate counts of inter-agent messages by type. + # This module is implemented in phase-1 of the concurrent execution feature; + # ImportError is expected until that phase lands. + try: + from message_store import get_message_store + except ImportError: + try: + from ..message_store import get_message_store # type: ignore[no-redef] + except ImportError: + _pkg.logger.debug("Message store not available for status") + get_message_store = None # type: ignore[assignment] + + if get_message_store is not None: + store = get_message_store() + msg_status = store.get_status(pipeline.id) + result["messages"] = { + "total": msg_status.get("total", 0), + "by_type": msg_status.get("by_type", {}), + } + else: + result["messages"] = {"total": 0, "by_type": {}} + + # Consensus evaluator tracks per-agent readiness states and determines + # whether all agents agree the phase is complete. Implemented in phase-3; + # blocking_agents lists roles that are not yet READY (WORKING or BLOCKED). + # BRC peer consensus (preferred) or legacy readiness-based + try: + try: + from peer_consensus import get_peer_consensus_tracker + except ImportError: + from ..peer_consensus import get_peer_consensus_tracker # type: ignore[no-redef] + + tracker = get_peer_consensus_tracker(pipeline.id, slice_id) + if not tracker: + # Attempt lazy reconstruction from message store for concurrent + # pipelines. ``slice_id`` scopes the replay to one slice's + # tracker; without it, only pipeline-level messages replay so a + # slice-DAG pipeline does not reconstruct cross-slice (#2761). + try: + from review_graph import get_review_graph_for_phase + + try: + from peer_consensus import reconstruct_tracker_from_messages + except ImportError: + from ..peer_consensus import ( + reconstruct_tracker_from_messages, # type: ignore[no-redef] + ) + + if is_concurrent_execution(pipeline, pipeline.current_phase): + graph = get_review_graph_for_phase( + pipeline.current_phase.value, repo=pipeline.repo + ) + tracker = reconstruct_tracker_from_messages( + pipeline.id, + graph, + slice_id=slice_id, + phase=pipeline.current_phase.value, + ) + except ImportError: + pass # Fall through to legacy evaluator + except Exception as e: + _pkg.logger.warning( + "Tracker reconstruction failed", + error=str(e), + pipeline_id=pipeline.id, + slice_id=slice_id, + ) + if tracker: + consensus_state = tracker.get_state() + else: + # No BRC tracker available (slice-scoped query for a slice with + # no tracker yet, or a non-concurrent pipeline). The legacy + # ConsensusEvaluator was removed under cq-5 of #2777, so there + # is no fallback evaluator to consult. Report no consensus + # block; callers (e.g. the MCP get_consensus_status tool) fall + # back to message-based inference per the existing #1229 path. + consensus_state = None + except ImportError: + _pkg.logger.debug("Peer consensus tracker not available for status") + consensus_state = None + + if consensus_state is not None: + result["consensus"] = _pkg._consensus_block(consensus_state) + else: + # Don't populate consensus with empty placeholder — callers (e.g. the + # MCP get_consensus_status tool) use truthiness to decide whether to + # fall back to message-based inference. An empty-but-truthy dict + # prevents that fallback from triggering (see issue #1229). + pass + + # Slice-id-less observability (#3481): in a slice-DAG implement phase + # the live trackers are keyed ``{pipeline_id}/{slice_id}``, so the + # pipeline-level lookup above finds nothing and an operator querying + # without a slice scope saw no structured consensus at all; the only + # way to see tracker state was tailing orchestrator pod logs. Surface + # each active slice's real snapshot, explicitly keyed by slice. This + # is NOT the #2761 cross-slice "soup" (that was mingling every + # slice's messages into ONE inferred tracker); the pipeline-level + # ``consensus`` block above still never reflects a slice tracker. + if slice_id is None: + try: + try: + from peer_consensus import get_slice_trackers + except ImportError: + from ..peer_consensus import get_slice_trackers # type: ignore[no-redef] + + slice_trackers = get_slice_trackers(pipeline.id) + except ImportError: + slice_trackers = {} + slice_consensus: dict[str, dict] = {} + for sid in sorted(slice_trackers): + try: + slice_consensus[sid] = _pkg._consensus_block(slice_trackers[sid].get_state()) + except Exception as e: # noqa: BLE001 - one bad slice must not hide the rest + _pkg.logger.warning( + "Slice consensus snapshot failed", + pipeline_id=pipeline.id, + slice_id=sid, + error=str(e), + ) + if slice_consensus: + result["slice_consensus"] = slice_consensus + + # Agent lifecycle info from the phase execution record — shows which agents + # are spawned for the current phase and their container-level status. + # Includes ``container_id`` and server-computed ``elapsed_seconds`` so the + # sandboxed overseer can anchor stall-duration math on the live container's + # ``started_at`` rather than pre-restart message-bus events (issue #2084). + current_phase_name = pipeline.current_phase.value + phase_exec = pipeline.phases.get(current_phase_name) + agents_info: list[dict[str, _pkg.Any]] = [] + if phase_exec and hasattr(phase_exec, "agents"): + now = _pkg.datetime.now(_pkg.UTC) + for agent in phase_exec.agents: + if hasattr(agent, "role"): + role = agent.role.value if hasattr(agent.role, "value") else str(agent.role) + else: + role = str(agent) + if hasattr(agent, "status"): + status = agent.status.value if hasattr(agent.status, "value") else "unknown" + else: + status = "unknown" + + entry: dict[str, _pkg.Any] = {"role": role, "status": status} + + container_id = getattr(agent, "container_id", None) + if isinstance(container_id, str) and container_id: + entry["container_id"] = container_id + + started_at = getattr(agent, "started_at", None) + started_dt: _pkg.datetime | None = None + if isinstance(started_at, _pkg.datetime): + started_dt = started_at + elif isinstance(started_at, str) and started_at: + try: + started_dt = _pkg.datetime.fromisoformat(started_at) + except ValueError: + started_dt = None + if started_dt is not None: + if started_dt.tzinfo is None: + started_dt = started_dt.replace(tzinfo=_pkg.UTC) + entry["started_at"] = started_dt.isoformat() + entry["elapsed_seconds"] = max(0, int((now - started_dt).total_seconds())) + + agents_info.append(entry) + + # When the persisted phase-agent list is empty, backfill the + # running-pod view from live Job labels (#3230). Under the + # orchestrator-owned event loop (#3164) on-demand one-shot pods are + # never persisted into ``phase_exec.agents``, so without this the + # overseer's stall-duration math and the dashboard see "0 running + # agents" while role pods are demonstrably ``Running``. Empty stays + # empty when no pod is live, so legitimate between-spawn quiescence is + # not misreported as a cohort. + if not agents_info: + agents_info = _pkg._live_event_agents(pipeline.id, slice_id) + result["agents"] = agents_info + + return result + + +def _build_slice_diff_summary( + pipeline, + spawner: "ContainerSpawner", # noqa: UP037 + worktree_repo_path: _pkg.Path, + integration_branch: str, + parent_branch: str, + gateway_mode: Literal["public", "private"] = "public", +) -> tuple[list[str] | None, str | None]: + """Compute commit subjects + diffstat for a slice PR body (#3115). + + The slice PR body's task list is plan-derived — it describes intent, + not what the pushed branch actually contains. This helper reads the + real git state so ``create_slice_pr`` can render a ``## What's in + this PR`` section: the slice's commit subjects + (``git log origin/<parent>..origin/<head>``) and a diffstat against + the merge base (``git diff --stat origin/<parent>...origin/<head>``, + three-dot to match GitHub's PR diff semantics). + + Both remote-tracking refs are refreshed first via + ``GatewayClient.fetch_branch`` — the slice's agents push directly to + origin, so the orchestrator worktree's tracking refs may lag (same + pattern as :func:`_commit_slice_brc_history_to_integration_branch`, + which runs immediately before this in the slice loop). ``gateway_mode`` + must be threaded from the pipeline-computed mode at the call site; + defaulting to ``public`` against a private/internal repo causes the + gateway to refuse the session and the whole diff section silently + no-ops. + + Strictly best-effort: returns ``(None, None)`` on any failure + (fetch, git error, timeout) and never raises — a missing diff + summary must not block slice PR creation. + """ + pipeline_id = pipeline.id + try: + for branch in (parent_branch, integration_branch): + # ``fetch_branch`` swallows exceptions and returns False; + # a stale parent ref degrades the diffstat, it doesn't + # break it, so we just continue. + spawner.gateway.fetch_branch( + pipeline_id, + str(worktree_repo_path), + args=[f"+refs/heads/{branch}:refs/remotes/origin/{branch}"], + mode=gateway_mode, + ) + + git_base = [ + "git", + "-c", + "core.hooksPath=/dev/null", + "-c", + f"safe.directory={worktree_repo_path}", + "-C", + str(worktree_repo_path), + ] + span = f"origin/{parent_branch}..origin/{integration_branch}" + log_proc = _pkg.subprocess.run( + [*git_base, "log", "--no-merges", "--format=%s", span], + capture_output=True, + text=True, + check=False, + timeout=30, + ) + commit_subjects = ( + [line.strip() for line in log_proc.stdout.splitlines() if line.strip()] + if log_proc.returncode == 0 + else None + ) + # ``--stat=100,80,40``: 100-col output, then git truncates past + # 40 entries with an ellipsis line — a slice touching hundreds + # of files must not produce a body longer than the task dump + # this section exists to displace. + diff_proc = _pkg.subprocess.run( + [ + *git_base, + "diff", + "--stat=100,80,40", + f"origin/{parent_branch}...origin/{integration_branch}", + ], + capture_output=True, + text=True, + check=False, + timeout=60, + ) + diffstat = diff_proc.stdout.strip() if diff_proc.returncode == 0 else None + if not commit_subjects and not diffstat: + return None, None + return commit_subjects or None, diffstat or None + except Exception as err: # noqa: BLE001 + _pkg.logger.warning( + "Slice diff summary failed (slice PR opens without it) (#3115)", + pipeline_id=pipeline_id, + integration_branch=integration_branch, + parent_branch=parent_branch, + error=str(err), + ) + return None, None diff --git a/orchestrator/routes/pipelines/_status_wait.py b/orchestrator/routes/pipelines/_status_wait.py new file mode 100644 index 0000000000..51962bbb94 --- /dev/null +++ b/orchestrator/routes/pipelines/_status_wait.py @@ -0,0 +1,147 @@ +"""status-wait cursor + host-wait tracking helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim from the pipelines barrel; barrel-resident and +test-patched globals are reached via ``_pkg`` so +``patch("routes.pipelines.<name>")`` keeps intercepting. +""" + +from __future__ import annotations + +import routes.pipelines as _pkg # noqa: E402,F401 + + +def _track_host_wait_start() -> None: + if _pkg._inflight_host_waits is not None: + try: + _pkg._inflight_host_waits.inc() + except Exception: # pragma: no cover + pass + + +def _track_host_wait_end() -> None: + if _pkg._inflight_host_waits is not None: + try: + _pkg._inflight_host_waits.dec() + except Exception: # pragma: no cover + pass + + +def _parse_status_wait_cursor( + raw: str | None, +) -> tuple[bool, str | None, int | None]: + """Parse a ``/status/wait`` cursor. + + Returns ``(ok, msg_since_id, event_since_seq)`` where either half + may be ``None`` (meaning "snap to tip on this source"). ``ok`` + is False only for a syntactically malformed cursor — the route + returns 400 in that case. An empty / missing cursor is treated + as "snap to tip on both sources" (``ok=True, None, None``). + """ + if raw is None or raw == "": + return True, None, None + match = _pkg._STATUS_WAIT_CURSOR_RE.match(raw) + if not match: + return False, None, None + msg_part = match.group(1) + evt_part = match.group(2) + msg_since_id = msg_part if msg_part else None + event_since_seq: int | None = None + if evt_part: + try: + event_since_seq = int(evt_part) + except ValueError: # pragma: no cover — the regex guarantees digits/- + event_since_seq = None + return True, msg_since_id, event_since_seq + + +def _build_status_wait_cursor( + msg_tip_id: str | None, + event_tip_seq: int, +) -> str: + """Format a cursor for a ``/status/wait`` response. + + Both halves are emitted — the consumer treats empty halves as + "snap to tip" on the next call, matching ``_parse_status_wait_cursor``. + """ + msg_part = msg_tip_id or "" + return f"msg:{msg_part}|evt:{event_tip_seq}" + + +def _message_store_tip_id(pipeline_id: str) -> str | None: + """Best-effort read of the message-store tip ID for a pipeline. + + Used to build the initial / terminal cursor when the route + returns without matching a message. Returns ``None`` when the + store has no messages yet — the caller formats this as the + empty ``msg:`` half of the compound cursor. + + Three distinct conditions all collapse to ``None`` here and + callers cannot distinguish between them: + + 1. **Store import failure** — the message-store module is not + loadable in this process (test harness without Redis, + packaging skew). Pre-PR / post-#2464: same behavior. + 2. **Transient ``get_latest_id`` failure** — e.g., + :class:`redis.RedisError` from ``XREVRANGE`` on a connection + blip. ``RedisMessageStore.get_latest_id`` already catches + this and returns ``None``, so we see "no tip". This conflates + a transient error with a genuinely empty store; #2464's fix + at the call site (``_message_store_tip_id() or msg_since_id`` + removal) drops the consumer's cursor on this transient as + well, which is a small behavioral regression vs. pre-PR + graceful-degradation behavior. Acceptable in practice + because transient Redis errors degrade many other paths + simultaneously, but worth knowing. + 3. **Empty store** — the ``/status/wait`` post-clear case the + PR is fixing. Returning ``None`` lets the route emit an + empty ``msg:`` half so the consumer doesn't re-feed the + dead cursor. + """ + try: + store = _pkg._get_message_store()() + except Exception: # pragma: no cover — store may not be importable + return None + try: + return store.get_latest_id(pipeline_id) + except Exception: + return None + + +def _build_minimal_status_envelope( + pipeline: _pkg.Pipeline, + cursor: str, +) -> dict[str, _pkg.Any]: + """Compute the small envelope used on both wait paths. + + Ships ``current_phase`` / ``status`` / ``phase_elapsed_seconds`` + so dashboards can refresh cheaply on a timeout without paying + for a second round-trip. ``concurrent.consensus`` is also + included (R5 mitigation from the refine phase) so the host + does not miss a BRC state change during a quiet interval. + """ + phase_key = pipeline.current_phase.value if pipeline.current_phase else "" + phase_data = pipeline.phases.get(phase_key, None) + envelope: dict[str, _pkg.Any] = { + "current_phase": phase_key, + "status": pipeline.status.value if pipeline.status else "", + "cursor": cursor, + } + if phase_data is not None: + started_at = getattr(phase_data, "started_at", None) + if started_at: + try: + if isinstance(started_at, str): + started_dt = _pkg.datetime.fromisoformat(started_at) + else: + started_dt = started_at + if started_dt.tzinfo is None: + started_dt = started_dt.replace(tzinfo=_pkg.UTC) + elapsed = int((_pkg.datetime.now(_pkg.UTC) - started_dt).total_seconds()) + envelope["phase_elapsed_seconds"] = max(0, elapsed) + except ValueError, TypeError, AttributeError: + pass + + concurrent_data = _pkg._get_concurrent_status(pipeline) + if concurrent_data and "consensus" in concurrent_data: + envelope["concurrent"] = {"consensus": concurrent_data["consensus"]} + return envelope diff --git a/orchestrator/routes/pipelines/_worktree_sync.py b/orchestrator/routes/pipelines/_worktree_sync.py new file mode 100644 index 0000000000..4f3593e02e --- /dev/null +++ b/orchestrator/routes/pipelines/_worktree_sync.py @@ -0,0 +1,1393 @@ +"""worktree sync helpers for routes/pipelines (#3312 slice-4). + +Extracted verbatim; patched/barrel-resident globals reached via _pkg so +patch("routes.pipelines.<name>") keeps intercepting. +""" + +from __future__ import annotations + +import subprocess +import time +from pathlib import Path +from typing import ( + TYPE_CHECKING, + Literal, + NamedTuple, # noqa: F401 +) + +import routes.pipelines as _pkg # noqa: E402,F401 + +if TYPE_CHECKING: + try: + from ..container_spawner import ContainerSpawner # noqa: F401 + except ImportError: # pragma: no cover + from container_spawner import ContainerSpawner # type: ignore # noqa: F401 + + +class WorktreeSyncOutcome(NamedTuple): + """Structured outcome from :func:`_sync_worktree_with_remote` (#2792, #2979). + + Phase-boundary callers inspect ``diverged_unreconciled`` to decide + whether to pause the pipeline for a manual reconcile. Best-effort + callers can ignore the return value entirely — every field has a + safe default and the sync still does the same in-band work whether + or not the outcome is consumed. + + ``case`` is the same discriminator the function emits to its + ``worktree_sync_outcome`` log line, so the field can be cross- + referenced against operator-grep patterns. + + ``diverged_unreconciled`` is True when local and remote had truly + diverged (ahead AND behind) and the rebase autoresolve could not + reconcile them. Since #2979 the helper does **not** hard-reset in + that case — the rebase autoresolve already aborted (restoring the + worktree to the clean local HEAD with the orchestrator's committed + work intact), so the helper leaves the worktree there and reports + the unreconciled divergence so the caller can pause for a manual + reconcile rather than discarding committed work. + + ``backup_ref`` is the full ref name (``refs/egg-backup/sync-recovery/ + <pipeline_id>/<unix_ts>``) pinning the local HEAD when divergence is + unreconciled — a stable handle the operator can inspect/reset to. + ``None`` means the (best-effort) backup write failed; the commits are + still on the live HEAD, and the local-only SHAs go into the WARN log + inline so they're at least in the audit trail (see the helper body). + + ``local_only_commit_shas`` is the list of local-only short SHAs (with + summaries) that are on HEAD but not yet on origin. Empty when the + rev-list itself failed; the divergence is still reported, but the + operator can't be given the exact commit list inline. + + ``rebase_category`` / ``rebase_detail`` carry the failing rebase's + ``PushResult.category`` / ``detail`` (conflicting paths, the rebase + argv, and a git-output excerpt) when ``diverged_unreconciled`` is + True. They exist so the reconcile HITL can show the operator *what* + failed instead of an unfalsifiable generic claim (#3416) — the log + lines carry the same data but roll; the decision persists. + """ + + case: str + diverged_unreconciled: bool = False + backup_ref: str | None = None + local_only_commit_shas: tuple[str, ...] = () + rebase_category: str | None = None + rebase_detail: str | None = None + + +def _build_sync_recovery_backup_ref(pipeline_id: str, unix_ts: int) -> str: + """Return the canonical ``refs/egg-backup/sync-recovery/<pid>/<ts>`` name (#2792). + + Pulled out so the test, the writer, and any future opportunistic + pruner share a single ref-name convention. The slash-segment + layout lets ``git for-each-ref refs/egg-backup/sync-recovery/<pid>`` + enumerate just this pipeline's backups. + """ + return f"refs/egg-backup/sync-recovery/{pipeline_id}/{unix_ts}" + + +def _collect_local_only_commits( + git_base: list[str], + *, + pipeline_id: str, + branch: str, + remote_branch: str, +) -> tuple[str, ...]: + """Enumerate local-only commits between HEAD and ``origin/<remote_branch>``. + + Returns a tuple of ``"<short-sha> <summary>"`` strings, oldest first. + A failure (subprocess error, nonzero rc, parse error) returns an + empty tuple and emits a WARN — the hard-reset fallback proceeds + with an unknown discard list rather than blocking on best-effort + forensic enumeration (#2792 section 5). + """ + try: + result = subprocess.run( + [ + *git_base, + "rev-list", + "--reverse", + "--pretty=format:%h %s", + "--no-commit-header", + f"origin/{remote_branch}..HEAD", + ], + capture_output=True, + text=True, + timeout=10, + check=False, + ) + if result.returncode != 0: + _pkg.logger.warning( + "Failed to enumerate local-only commits before hard reset", + pipeline_id=pipeline_id, + branch=branch, + remote_branch=remote_branch, + rc=result.returncode, + stderr=result.stderr.strip()[:200], + ) + return () + lines = [ln.strip() for ln in result.stdout.splitlines() if ln.strip()] + return tuple(lines) + except Exception as exc: + _pkg.logger.warning( + "Local-only commit enumeration raised before hard reset", + pipeline_id=pipeline_id, + branch=branch, + remote_branch=remote_branch, + error=str(exc), + ) + return () + + +def _create_sync_recovery_backup_ref( + git_base: list[str], + *, + pipeline_id: str, + ref_name: str, +) -> bool: + """Pin current HEAD under ``ref_name`` via ``git update-ref`` (#2792). + + Returns True on success. On failure logs WARN and returns False; + the caller proceeds with the destructive reset regardless — the + backup is best-effort, the reset is the reconcile primitive. + """ + try: + result = subprocess.run( + [*git_base, "update-ref", ref_name, "HEAD"], + capture_output=True, + text=True, + timeout=10, + check=False, + ) + if result.returncode != 0: + _pkg.logger.warning( + "Failed to create sync-recovery backup ref", + pipeline_id=pipeline_id, + ref_name=ref_name, + rc=result.returncode, + stderr=result.stderr.strip()[:200], + ) + return False + return True + except Exception as exc: + _pkg.logger.warning( + "Sync-recovery backup-ref write raised", + pipeline_id=pipeline_id, + ref_name=ref_name, + error=str(exc), + ) + return False + + +def _sync_worktree_with_remote( + spawner: "ContainerSpawner", # noqa: UP037 + pipeline_id: str, + worktree_repo_path: Path, + prior_phase_succeeded: bool = True, + gateway_mode: Literal["public", "private"] = "public", + base_branch: str | None = None, + *, + pipeline_branch: str | None = None, +) -> WorktreeSyncOutcome: + """Sync a worktree with its remote branch (best-effort). + + After an orchestrator restart or a phase boundary, the local worktree + branch may be behind the remote: commits pushed during previous phases + (contracts, drafts, statefiles) exist on origin but not in the local + checkout. This function fetches those commits and reconciles the + worktree so that all downstream code (contract loading, draft reading, + populator, etc.) sees the full pipeline state. + + ``pipeline_branch`` is the **remote** branch name to reconcile against. + Since #2399 the pipeline tip lives at ``egg/<pid>/work`` on origin so + slice integration branches at ``egg/<pid>/slice-N`` can coexist as + siblings; ``pipeline.branch`` already carries that ``/work`` suffix + (set by :func:`_ensure_pipeline_work_ref` at submission time), so + callers should pass ``pipeline_branch=pipeline.branch`` directly — + the local worktree branch and the remote ref now match. Without an + explicit ``pipeline_branch``, the function reads + ``git branch --show-current`` and looks up ``origin/<that-name>``, + which always misses on real pipelines and exits at + ``case=no_remote_tracking`` (#2367). Callers with a pipeline in + scope MUST pass ``pipeline_branch=pipeline.branch``. When omitted, + the function falls back to the local branch name for backward + compatibility with non-pipeline scripts. + + When local is ahead of remote: + - If the prior phase succeeded, push local commits to remote first. + On a successful push, reset to origin (a no-op fast-forward that + keeps the worktree clean). If the push FAILS, the local commits + are preserved as-is and the function returns without resetting — + ``remote_ahead == 0`` means origin holds nothing to incorporate, so + a ``reset --hard origin`` would only discard completed, committed + work (e.g. agent-registered HITL contract decisions) before the + phase_gate decision bridge could surface them (#2972). + - If the prior phase failed or was killed, discard local commits and + reset to remote (discards incomplete work). + + When local has diverged (ahead AND behind), rebase local commits onto + ``origin/{pipeline_branch}`` via the same helper used by the + gateway-side push-reject reconcile path. ``--ff-only`` cannot + reconcile real divergence by definition, so the pre-#2337 + implementation silently left the worktree stale and downstream + populator/decision-sync paths consumed the stale state. + + When the rebase itself fails (#2792, made non-destructive in #2979), + the autoresolve has already run ``git rebase --abort`` — which + restores the worktree to the clean local HEAD and reapplies the + autostash, so the orchestrator's committed work is intact on HEAD. + The helper does **not** hard-reset (the pre-#2979 behaviour, which + discarded that committed work to a backup ref and FAILed the + pipeline). It pins HEAD under ``refs/egg-backup/sync-recovery/ + <pipeline_id>/<unix_ts>`` as a stable operator handle and returns + ``diverged_unreconciled=True`` so phase-boundary callers pause the + pipeline for a manual reconcile (AWAITING_HUMAN) rather than + consuming the un-reconciled state or discarding work. + + Every return path emits at least one ``worktree_sync_outcome`` log + line with a ``case`` discriminator so production logs name which + path fired. The ``rev_list_failed`` and ``divergence_unreconciled`` + cases bail non-destructively (no ``reset --hard``); only the + local-behind and prior-phase-failed-discard cases reach the step-4 + reset, neither of which can lose committed work that isn't already + on origin. + + Safe to call on every pipeline start because it is idempotent when the + local branch is already up to date. + + Returns a :class:`WorktreeSyncOutcome` describing what the helper + did. Most callers can ignore the return value; phase-boundary + callers inspect ``diverged_unreconciled`` to decide whether to pause + the pipeline for a manual reconcile (#2979). + """ + base_branch_for_reconcile = base_branch + git_base = [ + "git", + "-c", + "core.hooksPath=/dev/null", + "-c", + f"safe.directory={worktree_repo_path}", + "-C", + str(worktree_repo_path), + ] + + # Step 1: Authenticated fetch via gateway (gateway holds GitHub credentials) + fetch_ok = spawner.gateway.fetch_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(worktree_repo_path), + mode=gateway_mode, + ) + if not fetch_ok: + _pkg.logger.info( + "worktree_sync_outcome", + pipeline_id=pipeline_id, + case="fetch_failed", + ) + return WorktreeSyncOutcome(case="fetch_failed") + + # Step 2: Determine current branch + try: + result = subprocess.run( + [*git_base, "branch", "--show-current"], + capture_output=True, + text=True, + timeout=10, + check=False, + ) + branch = result.stdout.strip() + if not branch: + _pkg.logger.info( + "worktree_sync_outcome", + pipeline_id=pipeline_id, + case="detached_head", + ) + return WorktreeSyncOutcome(case="detached_head") + except Exception as branch_err: + _pkg.logger.info( + "worktree_sync_outcome", + pipeline_id=pipeline_id, + case="branch_detect_failed", + error=str(branch_err), + ) + return WorktreeSyncOutcome(case="branch_detect_failed") + + # ``branch`` is the **local** branch name (e.g. ``egg/<pid>/work`` on + # orchestrator worktrees). ``remote_branch`` is the remote-side name + # we look up on origin and push/reset against. When the caller + # passes ``pipeline_branch`` (the canonical, agent-facing branch), + # use it for every remote-side ref so the ``/work`` suffix mismatch + # in #2367 cannot strand a pipeline in ``no_remote_tracking``. + remote_branch = pipeline_branch or branch + + # Step 3: Verify remote tracking branch exists + try: + result = subprocess.run( + [*git_base, "rev-parse", "--verify", f"origin/{remote_branch}"], + capture_output=True, + text=True, + timeout=10, + check=False, + ) + if result.returncode != 0: + _pkg.logger.info( + "worktree_sync_outcome", + pipeline_id=pipeline_id, + branch=branch, + remote_branch=remote_branch, + case="no_remote_tracking", + ) + return WorktreeSyncOutcome(case="no_remote_tracking") + except Exception as rev_parse_err: + _pkg.logger.info( + "worktree_sync_outcome", + pipeline_id=pipeline_id, + branch=branch, + remote_branch=remote_branch, + case="rev_parse_failed", + error=str(rev_parse_err), + ) + return WorktreeSyncOutcome(case="rev_parse_failed") + + # Step 3b: Check divergence between local and remote. + local_ahead = 0 + remote_ahead = 0 + rev_list_ok = False + try: + result = subprocess.run( + [ + *git_base, + "rev-list", + "--left-right", + "--count", + f"HEAD...origin/{remote_branch}", + ], + capture_output=True, + text=True, + timeout=10, + check=False, + ) + parts = result.stdout.strip().split() + if result.returncode == 0 and len(parts) == 2: + local_ahead = int(parts[0]) + remote_ahead = int(parts[1]) + rev_list_ok = True + else: + _pkg.logger.warning( + "worktree_sync_outcome", + pipeline_id=pipeline_id, + branch=branch, + remote_branch=remote_branch, + case="rev_list_failed", + rc=result.returncode, + stdout=result.stdout.strip()[:200], + ) + # #2979: the ahead/behind counts are unknown, so a Step-4 + # ``reset --hard origin`` here could discard local-only + # commits that are NOT on origin — a destructive reset over + # un-provably-pushed work with no backup ref. Bail + # non-destructively instead, leaving the worktree untouched. + return WorktreeSyncOutcome(case="rev_list_failed") + except Exception as rev_list_err: + _pkg.logger.warning( + "worktree_sync_outcome", + pipeline_id=pipeline_id, + branch=branch, + remote_branch=remote_branch, + case="rev_list_failed", + error=str(rev_list_err), + ) + # #2979: unknown ahead/behind counts — bail non-destructively + # rather than fall through to the Step-4 ``reset --hard`` (which + # would risk discarding un-pushed local work without a backup). + return WorktreeSyncOutcome(case="rev_list_failed") + + # Step 3c: Handle local-ahead commits. + if local_ahead == 0 and remote_ahead == 0 and rev_list_ok: + # Local and remote are already in sync — skip the no-op reset entirely + # so the outcome is distinguishable from a true behind-remote sync. + _pkg.logger.info( + "worktree_sync_outcome", + pipeline_id=pipeline_id, + branch=branch, + remote_branch=remote_branch, + case="already_in_sync", + local_ahead=0, + remote_ahead=0, + ) + return WorktreeSyncOutcome(case="already_in_sync") + + if local_ahead > 0 and remote_ahead == 0: + # Local is strictly ahead of remote (no divergence). + if prior_phase_succeeded: + # Prior phase completed successfully — push local work to remote + # before resetting, so it's not lost. Pushing to ``remote_branch`` + # (not the local ``/work`` name) so the agent-facing branch + # receives the commits — the gateway builds + # ``HEAD:refs/heads/{branch}`` from this argument. + push_result = spawner.gateway.push_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(worktree_repo_path), + branch=remote_branch, + mode=gateway_mode, + base_branch=base_branch_for_reconcile, + ) + if push_result: + # Push succeeded — local and remote are now in sync. + # Re-fetch to update the remote tracking ref so that + # origin/{remote_branch} reflects the pushed commits. + spawner.gateway.fetch_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(worktree_repo_path), + mode=gateway_mode, + ) + _pkg.logger.info( + "worktree_sync_outcome", + pipeline_id=pipeline_id, + branch=branch, + remote_branch=remote_branch, + case="local_ahead_pushed", + local_ahead=local_ahead, + remote_ahead=remote_ahead, + ) + return WorktreeSyncOutcome(case="local_ahead_pushed") + else: + # Push failed. ``remote_ahead == 0`` in this branch, so + # origin holds nothing the worktree lacks — resetting to + # origin here would discard the completed, committed local + # work (e.g. the agent-registered HITL contract decisions + # the pre-sync ``_commit_statefiles_to_worktree`` just + # committed) for ZERO reconcile benefit, then advance + # silently. That is exactly how #2972 dropped a refiner's + # ``register_open_question`` / ``request_feedback`` items + # before the phase_gate decision bridge could surface them: + # the prior code fell through to the Step-4 ``reset --hard`` + # and returned ``reset_succeeded`` (``hard_reset_performed`` + # False), so no operator signal fired. Preserve the local + # commits instead — they remain in the worktree for + # downstream reads (the decision bridge, populator) and for + # the next push attempt. The WARNING below is the loud + # breadcrumb that the tip is unpushed; unlike the divergence + # path (``remote_ahead > 0``) there is no remote work to + # rebase onto, so non-destructive preservation is correct. + _pkg.logger.warning( + "worktree_sync_outcome", + pipeline_id=pipeline_id, + branch=branch, + remote_branch=remote_branch, + case="local_ahead_push_failed", + local_ahead=local_ahead, + remote_ahead=remote_ahead, + category=push_result.category, + error=push_result.detail, + ) + return WorktreeSyncOutcome(case="local_ahead_push_failed") + else: + # Prior phase failed — incomplete local work will be discarded by + # the step-4 reset. Emit a distinct case so operators can grep + # this branch of the taxonomy without inferring it from + # reset_succeeded with local_ahead > 0. + _pkg.logger.info( + "worktree_sync_outcome", + pipeline_id=pipeline_id, + branch=branch, + remote_branch=remote_branch, + case="local_ahead_discarded", + local_ahead=local_ahead, + remote_ahead=remote_ahead, + ) + # Fall through to reset (Step 4) — discards incomplete local work + # from a failed/killed prior phase. (The successful-phase + # push-failure case returns above without resetting so completed + # work is never silently dropped — #2972.) + + elif local_ahead > 0 and remote_ahead > 0: + # True divergence. Reconcile by rebasing local commits onto + # origin/{branch} via the same helper used by the gateway-side + # push-reject reconcile path (#2337). --ff-only cannot reconcile + # real divergence by definition, so the pre-#2337 implementation + # silently left the worktree stale. + # + # ⚠️ When ``base_branch_for_reconcile`` is None, + # ``_build_rebase_cmd`` falls back to the plain + # ``git rebase origin/{branch}`` form — the same form that + # triggered #2222 main-contamination on the gateway-side + # push-reject path. That fallback is the contamination vector: + # with HEAD at current main and origin/{branch} on a stale + # snapshot, the plain form replays merge-base..HEAD on the + # stale tip, producing a PR full of duplicate-by-content + # commits. Callers should always thread ``pipeline.base_branch`` + # so the helper emits the safer + # ``--onto origin/{branch} origin/{base_branch}`` form. Logging + # the None case so the next person debugging contamination has a + # breadcrumb. + if base_branch_for_reconcile is None: + _pkg.logger.warning( + "worktree_sync divergence_rebase with base_branch=None — " + "falling back to bare-rebase form (#2222 contamination risk)", + pipeline_id=pipeline_id, + branch=branch, + remote_branch=remote_branch, + ) + _pkg.logger.info( + "Local and remote have diverged — rebasing local onto origin", + pipeline_id=pipeline_id, + branch=branch, + remote_branch=remote_branch, + local_ahead=local_ahead, + remote_ahead=remote_ahead, + ) + rebase_outcome = _pkg._rebase_with_agent_output_autoresolve( + git_base=git_base, + pipeline_id=pipeline_id, + branch=remote_branch, + base_branch=base_branch_for_reconcile, + ) + if rebase_outcome.ok: + _pkg.logger.info( + "worktree_sync_outcome", + pipeline_id=pipeline_id, + branch=branch, + remote_branch=remote_branch, + case="divergence_rebased", + local_ahead=local_ahead, + remote_ahead=remote_ahead, + ) + return WorktreeSyncOutcome(case="divergence_rebased") + _pkg.logger.error( + "worktree_sync_outcome", + pipeline_id=pipeline_id, + branch=branch, + remote_branch=remote_branch, + case="divergence_rebase_failed", + local_ahead=local_ahead, + remote_ahead=remote_ahead, + category=rebase_outcome.category, + detail=rebase_outcome.detail, + ) + + # #2979: non-destructive divergence reconcile. The rebase + # autoresolve could not reconcile the divergence — a conflict on + # a path outside ``.egg-state/agent-outputs/``. In normal + # operation this is now unreachable: #2979 stopped agents from + # git-pushing ``.egg-state/contracts/`` (they mutate contracts + # through the contract API), so the orchestrator is the sole + # writer of the only non-agent-outputs path both sides touched on + # the work branch, and the rebase only ever replays disjoint + # paths. When it *does* fire (an unexpected residual conflict, a + # restart mid-flight), the autoresolve has already run + # ``git rebase --abort``, which restored the worktree to the + # clean local HEAD and reapplied the autostash — the + # orchestrator's committed work is intact on HEAD. + # + # #2792/#2797 used to ``git reset --hard origin`` here, discarding + # that committed work (operator-bound contract decisions included) + # to a backup ref the operator had to spelunk, then FAIL the + # pipeline. Instead, leave the worktree at local HEAD and report + # the unreconciled divergence; the caller pauses the pipeline for + # a manual reconcile (AWAITING_HUMAN, not FAILED). Downstream + # consumers — populator, decision-sync, plan-complete — never run + # against the un-reconciled state because the pause halts the + # phase before them, which is the silent-stale-read failure #2337 + # raised an error for, addressed without discarding work. + # + # Pin HEAD under a backup ref anyway: a stable, enumerable handle + # the operator can ``git log`` / ``git reset`` against, and a + # guard against any later worktree mutation. Best-effort — a + # failed write inlines the SHAs into the WARN log for the audit + # trail (the commits remain on the live HEAD regardless). + # Nanosecond precision so two reconcile attempts within the same + # second on the same pipeline cannot collide on the ref name. + local_only = _collect_local_only_commits( + git_base, + pipeline_id=pipeline_id, + branch=branch, + remote_branch=remote_branch, + ) + unix_ts = time.time_ns() + backup_ref = _build_sync_recovery_backup_ref(pipeline_id, unix_ts) + backup_ok = _create_sync_recovery_backup_ref( + git_base, + pipeline_id=pipeline_id, + ref_name=backup_ref, + ) + if not backup_ok and local_only: + _pkg.logger.warning( + "Divergence-reconcile backup ref write failed; local-only " + "SHAs inlined for audit (commits remain on the live HEAD)", + pipeline_id=pipeline_id, + branch=branch, + remote_branch=remote_branch, + local_only_commit_shas=list(local_only), + ) + _pkg.logger.warning( + "worktree_sync_outcome", + pipeline_id=pipeline_id, + branch=branch, + remote_branch=remote_branch, + case="divergence_unreconciled", + local_ahead=local_ahead, + remote_ahead=remote_ahead, + backup_ref=backup_ref if backup_ok else None, + local_only_commit_count=len(local_only), + rebase_category=rebase_outcome.category, + ) + return WorktreeSyncOutcome( + case="divergence_unreconciled", + diverged_unreconciled=True, + backup_ref=backup_ref if backup_ok else None, + local_only_commit_shas=local_only, + rebase_category=rebase_outcome.category, + rebase_detail=rebase_outcome.detail, + ) + + # Step 4: Reset local branch to remote. + # This handles: local behind remote (origin strictly ahead — nothing + # local to lose) and the prior-phase-failed local-ahead discard (the + # incomplete work is intentionally dropped). The already-in-sync case + # returns early above; the rev-list-failed and unreconciled-divergence + # cases now bail non-destructively before reaching here (#2979), so + # this reset never runs over un-provably-pushed committed work. + try: + result = subprocess.run( + [*git_base, "reset", "--hard", f"origin/{remote_branch}"], + capture_output=True, + text=True, + timeout=30, + check=False, + ) + if result.returncode != 0: + _pkg.logger.warning( + "worktree_sync_outcome", + pipeline_id=pipeline_id, + branch=branch, + remote_branch=remote_branch, + case="reset_failed", + local_ahead=local_ahead, + remote_ahead=remote_ahead, + error=result.stderr.strip(), + ) + return WorktreeSyncOutcome(case="reset_failed") + _pkg.logger.info( + "worktree_sync_outcome", + pipeline_id=pipeline_id, + branch=branch, + remote_branch=remote_branch, + case="reset_succeeded", + local_ahead=local_ahead, + remote_ahead=remote_ahead, + ) + return WorktreeSyncOutcome(case="reset_succeeded") + except Exception as sync_err: + _pkg.logger.warning( + "worktree_sync_outcome", + pipeline_id=pipeline_id, + branch=branch, + remote_branch=remote_branch, + case="reset_failed", + local_ahead=local_ahead, + remote_ahead=remote_ahead, + error=str(sync_err), + ) + return WorktreeSyncOutcome(case="reset_failed") + + +class StalePipelineBranchError(RuntimeError): + """Raised when ``origin/<pipeline.branch>`` is behind base and the + rebase to bring it up to date hit a conflict. + + Phase-startup callers convert this into a FAILED pipeline with a + clear ``error`` so the operator knows to manually rebase or start + fresh — vastly preferable to silently producing a PR with 70+ + cherry-picked-variant commits buried in it (#2098). + """ + + +def _rebase_pipeline_branch_onto_base( + spawner: "ContainerSpawner", # noqa: UP037 + pipeline_id: str, + worktree_repo_path: Path, + pipeline_branch: str, + base_branch: str, + gateway_mode: Literal["public", "private"] = "public", +) -> None: + """Rebase a stale ``origin/<pipeline_branch>`` onto ``origin/<base_branch>``. + + When ``submit_task`` resumes a pipeline whose branch has been sitting + on the remote for days/weeks while ``main`` advanced, the existing + pipeline branch tip carries old-SHA copies of commits that have since + been rebased onto main. Without this helper, the first orchestrator + push hits non-fast-forward, the reconcile path rebases ``HEAD`` onto + the stale tip, and every downstream commit inherits 70+ stale-from- + main commits as ancestors — producing a final PR diff that buries + the actual feature work under contamination (#2098). + + This helper runs on the orchestrator-side worktree and treats it as + scratch space for the rebase: + + 1. Skip when ``pipeline_branch`` doesn't exist on the remote (fresh + run — there's nothing to rebase). + 2. Skip when ``origin/<pipeline_branch>`` is not behind + ``origin/<base_branch>`` (already up to date). + 3. Skip when ``HEAD`` is an ancestor of *neither* + ``origin/<pipeline_branch>`` *nor* ``origin/<base_branch>``. Two + real resume paths satisfy the ancestry check: + + (a) **Preserved worktree** (canonical #2098 case): the + orchestrator-side worktree was kept across a cancel/resubmit, + so ``HEAD`` carries state-file commits that were already + pushed to ``origin/<branch>``. ``HEAD`` is a strict ancestor + of ``origin/<branch>``. + (b) **Fresh worktree**: the worktree volume was wiped between + cancel and resubmit (e.g. orchestrator redeploy onto a fresh + PVC), so the gateway recreated it from ``origin/<base>``. + ``HEAD == origin/<base>`` is a (trivial) ancestor of + ``origin/<base>``; resetting to ``origin/<branch>`` discards + no unique commits because every base commit is preserved as + the rebase target. + + If neither ancestry holds, ``HEAD`` carries truly unpublished + work and we defer rather than overwrite it. + 4. Reset the worktree to ``origin/<pipeline_branch>``, ``git rebase + origin/<base_branch>``, and force-push the rebased tip. Git's + built-in cherry-pick-skip drops commits already content-equivalent + to ones on the new base. + 5. On conflict: abort the rebase, restore the worktree to + ``origin/<base_branch>``, and raise ``StalePipelineBranchError`` + so phase startup fails fast with an actionable error. + + Best-effort fetch+rev-list errors are logged and swallowed so a + transient gateway hiccup doesn't block pipeline startup; only a + rebase that *started* but couldn't finish raises. + """ + if not pipeline_branch or not base_branch or pipeline_branch == base_branch: + return + + git_base = [ + "git", + "-c", + "core.hooksPath=/dev/null", + "-c", + f"safe.directory={worktree_repo_path}", + "-C", + str(worktree_repo_path), + ] + + def _run_git( + args: list[str], + timeout: int, + ) -> subprocess.CompletedProcess[str] | None: + """Run a git command and convert ``TimeoutExpired`` / ``OSError`` + into a ``None`` return so callers can decide what to do. + + Mirrors the defensive pattern in ``_sync_worktree_with_remote``. + """ + try: + return subprocess.run( + [*git_base, *args], + capture_output=True, + text=True, + timeout=timeout, + check=False, + ) + except (subprocess.TimeoutExpired, OSError) as exc: + _pkg.logger.warning( + "rebase-on-resume: git command failed to run", + pipeline_id=pipeline_id, + branch=pipeline_branch, + git_args=args, + error=str(exc), + ) + return None + + # Step 1: Fetch both refs through the gateway so we have current + # origin/<branch> and origin/<base> tips locally. fetch_worktree_branch + # already runs `git fetch origin` (no refspec) which updates all + # remote-tracking refs in one call. + fetch_ok = spawner.gateway.fetch_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(worktree_repo_path), + mode=gateway_mode, + ) + if not fetch_ok: + _pkg.logger.warning( + "rebase-on-resume: fetch failed, skipping rebase check", + pipeline_id=pipeline_id, + branch=pipeline_branch, + ) + return + + # Step 2: Verify origin/<pipeline_branch> exists. Fresh pipelines + # haven't pushed yet, so there's nothing to rebase. + verify_branch = _run_git(["rev-parse", "--verify", f"origin/{pipeline_branch}"], timeout=10) + if verify_branch is None or verify_branch.returncode != 0: + return + + verify_base = _run_git(["rev-parse", "--verify", f"origin/{base_branch}"], timeout=10) + if verify_base is None or verify_base.returncode != 0: + _pkg.logger.warning( + "rebase-on-resume: origin/<base_branch> not resolvable, skipping", + pipeline_id=pipeline_id, + branch=pipeline_branch, + base_branch=base_branch, + ) + return + + # Step 3: Is the pipeline branch actually behind base? If not, no-op. + behind = _run_git( + [ + "rev-list", + "--count", + f"origin/{pipeline_branch}..origin/{base_branch}", + ], + timeout=10, + ) + if behind is None or behind.returncode != 0: + _pkg.logger.warning( + "rebase-on-resume: rev-list failed, skipping", + pipeline_id=pipeline_id, + branch=pipeline_branch, + stderr=(behind.stderr.strip() if behind is not None else None), + ) + return + try: + behind_count = int(behind.stdout.strip() or "0") + except ValueError: + behind_count = 0 + if behind_count == 0: + return + + # Step 4: Confirm reset-to-origin/<branch> is lossless before we + # overwrite HEAD. Three worktree states are handled: + # + # (a) Preserved-worktree resume (#2098 canonical): the orchestrator- + # side worktree was kept across a cancel/resubmit, so HEAD + # carries state-file commits that were already pushed to + # origin/<branch>. HEAD is a strict ancestor of + # origin/<branch> — resetting drops nothing. + # (b) Fresh-worktree resume: the worktree volume was wiped between + # cancel and resubmit (e.g. orchestrator redeploy onto a fresh + # PVC, manual cleanup), so the gateway recreated the worktree + # from origin/<base>. HEAD == origin/<base>; resetting to + # origin/<branch> discards no unique commits because every + # commit on origin/<base> is preserved as the rebase target. + # (c) Confused-HEAD resume (#2222): the worktree carries a local- + # only commit (e.g. a half-pushed statefiles commit) on top of + # a stale origin/<branch> tip — HEAD is on neither ref. The + # previous behaviour was to "defer to push-reconcile", but the + # reconcile path's _build_rebase_cmd fallback is the + # contamination producer in #2222. Recover by hard-resetting + # to origin/<base>: any local-only work is dropped (it would + # be re-created by agents on the next phase, vastly preferable + # to a contaminated PR). + def _head_on(ref: str) -> bool: + result = _run_git(["merge-base", "--is-ancestor", "HEAD", ref], timeout=10) + return result is not None and result.returncode == 0 + + if not (_head_on(f"origin/{pipeline_branch}") or _head_on(f"origin/{base_branch}")): + _pkg.logger.warning( + "rebase-on-resume: HEAD on neither origin/<branch> nor origin/<base> — " + "resetting to origin/<base> to avoid push-reconcile contamination (#2222)", + pipeline_id=pipeline_id, + branch=pipeline_branch, + base_branch=base_branch, + behind_base=behind_count, + ) + # Step 4a: hard-reset to ``origin/<base>`` first. Note that step 5 + # immediately overwrites HEAD again with ``reset --hard + # origin/<branch>`` in the success path, so this reset's effect on + # HEAD is short-lived — its purpose is to act as a safe-state floor: + # if step 5 itself fails (network blip, ref vanishes), we leave the + # worktree on a known-good ref (``origin/<base>``) instead of the + # ambiguous pre-recovery state that prompted the rescue. Don't + # "simplify" by dropping this — the back-to-back hard resets are + # intentional. + recovery_reset = _run_git(["reset", "--hard", f"origin/{base_branch}"], timeout=30) + if recovery_reset is None or recovery_reset.returncode != 0: + _pkg.logger.warning( + "rebase-on-resume: recovery reset to origin/<base> failed, skipping", + pipeline_id=pipeline_id, + branch=pipeline_branch, + base_branch=base_branch, + stderr=(recovery_reset.stderr.strip() if recovery_reset is not None else None), + ) + return + + _pkg.logger.info( + "rebase-on-resume: pipeline branch is behind base, attempting rebase", + pipeline_id=pipeline_id, + branch=pipeline_branch, + base_branch=base_branch, + behind_base=behind_count, + ) + + # Step 5: Reset the worktree to the stale pipeline branch tip so we + # can rebase it onto current base. + reset_to_branch = _run_git(["reset", "--hard", f"origin/{pipeline_branch}"], timeout=30) + if reset_to_branch is None or reset_to_branch.returncode != 0: + _pkg.logger.warning( + "rebase-on-resume: reset to pipeline branch failed, skipping", + pipeline_id=pipeline_id, + branch=pipeline_branch, + stderr=(reset_to_branch.stderr.strip() if reset_to_branch is not None else None), + ) + return + + # Step 6: Rebase onto current base. Plain ``git rebase + # origin/<base>`` — git's cherry-pick-skip drops content-equivalent + # commits already on base (the 70+ stale-variant commits in #2098). + rebase = _run_git(["rebase", f"origin/{base_branch}"], timeout=120) + if rebase is None or rebase.returncode != 0: + # Conflict, timeout, or other rebase failure. Abort the rebase, + # restore the worktree to origin/<base> so it isn't left mid- + # rebase for downstream callers, and raise so the operator gets + # an actionable error rather than a contaminated PR. ``rebase + # is None`` covers the timeout case where ``_run_git`` already + # logged the underlying exception. + _run_git(["rebase", "--abort"], timeout=30) + _run_git(["reset", "--hard", f"origin/{base_branch}"], timeout=30) + stderr_text = rebase.stderr.strip() if rebase is not None else "rebase command timed out" + _pkg.logger.error( + "rebase-on-resume: rebase failed — aborting pipeline start", + pipeline_id=pipeline_id, + branch=pipeline_branch, + base_branch=base_branch, + stderr=stderr_text, + timed_out=rebase is None, + ) + raise StalePipelineBranchError( + f"origin/{pipeline_branch} is {behind_count} commits behind " + f"origin/{base_branch} and rebasing it failed. " + f"Manually rebase the branch (or delete it to start fresh) " + f"and resubmit. Stderr: {stderr_text}" + ) + + # Git emits ``warning: skipped previously applied commit <sha>`` on + # stderr for every cherry-pick-equivalent it dropped. Counting them + # gives operators a quick sanity check that the helper actually + # discarded the stale-from-main commits (vs. e.g. silently no-op'd). + skipped_via_rebase = sum( + 1 for line in rebase.stderr.splitlines() if "skipped previously applied commit" in line + ) + + # Step 7: Force-push the rebased branch. ``force=True`` is required + # because the rebased tip has different SHAs from origin/<branch>; + # this is exactly the contamination we just removed, so overwriting + # is the desired behavior. + push_result = spawner.gateway.push_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(worktree_repo_path), + branch=pipeline_branch, + mode=gateway_mode, + base_branch=base_branch, + force=True, + ) + if not push_result.ok: + # Restore HEAD to origin/<base> so the worktree is in a known + # state for downstream callers (the rebased commits stay in the + # local reflog if needed for recovery). + _run_git(["reset", "--hard", f"origin/{base_branch}"], timeout=30) + _pkg.logger.error( + "rebase-on-resume: force-push of rebased branch failed", + pipeline_id=pipeline_id, + branch=pipeline_branch, + category=push_result.category, + detail=push_result.detail, + ) + raise StalePipelineBranchError( + f"Rebased {pipeline_branch} onto origin/{base_branch} but " + f"force-push to remote failed ({push_result.category}): " + f"{push_result.detail}" + ) + + # Re-fetch so origin/<pipeline_branch> reflects the rebased tip for + # any subsequent rev-parse in the same pipeline-start path. + spawner.gateway.fetch_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(worktree_repo_path), + mode=gateway_mode, + ) + _pkg.logger.info( + "rebase-on-resume: rebased and force-pushed pipeline branch", + pipeline_id=pipeline_id, + branch=pipeline_branch, + base_branch=base_branch, + dropped_stale_commits=behind_count, + skipped_via_rebase=skipped_via_rebase, + ) + + +def _refresh_pipeline_branch_against_current_base( + spawner: "ContainerSpawner", # noqa: UP037 + pipeline_id: str, + worktree_repo_path: Path, + pipeline_branch: str, + base_branch: str, + gateway_mode: Literal["public", "private"] = "public", +) -> bool: + """Rebase ``origin/<pipeline_branch>`` onto current ``origin/<base_branch>`` + immediately before opening the PR (#2224 PR 2). + + ``_rebase_pipeline_branch_onto_base`` runs at the start of each + phase iteration to clean up stale branch state on resume. Nothing + between branch-cut and PR-open refreshes against + ``origin/<base_branch>``; if ``base_branch`` advances *during* the + PR phase's own work, the resulting PR is behind. This helper + closes that gap. + + The pipeline branch is the only ref this helper writes to: the + rebase replays pipeline-branch commits onto current + ``origin/<base_branch>``, and the force-push targets + ``pipeline_branch``. ``base_branch`` is read-only here — no + commits are ever pushed to it, even when it happens to be + ``main``. + + On success, force-pushes the rebased branch so the open PR's head + SHA reflects the rebase. + + On *any* failure (rebase conflict, push rejection, transient gateway + error), restores the worktree to ``origin/<pipeline_branch>``, + logs at WARNING, and returns ``False`` — the caller still opens the + PR against the un-rebased tip. This is intentional: a merge conflict + at PR-open time is better surfaced to the human reviewer than + swallowed by failing the whole pipeline. + + Returns ``True`` when a rebase was performed and pushed; ``False`` + when no rebase was needed or any step failed (in which case the + caller proceeds with the un-rebased tip). + """ + if not pipeline_branch or not base_branch or pipeline_branch == base_branch: + return False + + git_base = [ + "git", + "-c", + "core.hooksPath=/dev/null", + "-c", + f"safe.directory={worktree_repo_path}", + "-C", + str(worktree_repo_path), + ] + + def _run_git( + args: list[str], + timeout: int, + ) -> subprocess.CompletedProcess[str] | None: + try: + return subprocess.run( + [*git_base, *args], + capture_output=True, + text=True, + timeout=timeout, + check=False, + ) + except (subprocess.TimeoutExpired, OSError) as exc: + _pkg.logger.warning( + "pr-open rebase: git command failed", + pipeline_id=pipeline_id, + branch=pipeline_branch, + git_args=args, + error=str(exc), + ) + return None + + # Step 1: Fetch fresh refs. Without this we'd rebase against the + # base tip we saw at branch-cut, defeating the whole point. + fetch_ok = spawner.gateway.fetch_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(worktree_repo_path), + mode=gateway_mode, + ) + if not fetch_ok: + _pkg.logger.warning( + "pr-open rebase: fetch failed, skipping", + pipeline_id=pipeline_id, + branch=pipeline_branch, + ) + return False + + # Step 2: Verify both refs resolve. + verify_branch = _run_git(["rev-parse", "--verify", f"origin/{pipeline_branch}"], timeout=10) + if verify_branch is None or verify_branch.returncode != 0: + return False + verify_base = _run_git(["rev-parse", "--verify", f"origin/{base_branch}"], timeout=10) + if verify_base is None or verify_base.returncode != 0: + return False + + # Step 3: No-op when the branch is already up-to-date with current base + # (no commits behind). Saves a force-push when none is needed. + behind = _run_git( + [ + "rev-list", + "--count", + f"origin/{pipeline_branch}..origin/{base_branch}", + ], + timeout=10, + ) + if behind is None or behind.returncode != 0: + return False + try: + behind_count = int((behind.stdout or "0").strip() or "0") + except ValueError: + behind_count = 0 + if behind_count == 0: + return False + + # Step 4: Compute the merge-base so we can use the safe + # ``--onto <new_base> <upstream>`` form (HEAD is the implicit branch + # being rebased after the step-5 reset). The merge-base is the + # commit where the branch diverged from base_branch; using it as + # ``<upstream>`` tells git "replay only the commits unique to HEAD + # onto <new_base>" — no base-branch commits get absorbed into the + # branch's linear history, which is the contamination shape #2222 + # hardened against in the push-reconcile path. + merge_base_proc = _run_git( + ["merge-base", f"origin/{pipeline_branch}", f"origin/{base_branch}"], + timeout=15, + ) + if merge_base_proc is None or merge_base_proc.returncode != 0: + _pkg.logger.warning( + "pr-open rebase: merge-base resolution failed, skipping", + pipeline_id=pipeline_id, + branch=pipeline_branch, + base_branch=base_branch, + stderr=(merge_base_proc.stderr.strip() if merge_base_proc is not None else None), + ) + return False + merge_base = (merge_base_proc.stdout or "").strip() + if not merge_base: + return False + + # Step 5: Reset the worktree to the current branch tip so the + # rebase operates on the right starting state. The reset target + # is ``origin/<pipeline_branch>`` — fresh from fetch in step 1 — + # so we are not rebasing on top of stale local state. + # + # Unlike ``_rebase_pipeline_branch_onto_base`` (resume-time helper), + # there is no ``_head_on(...)`` ancestry guard before this reset. + # That is intentional at this PR-open call site: any local-ahead + # commits at this point are orchestrator housekeeping commits that + # are orphan-by-design (the agents' work is already on + # ``origin/<branch>`` via the per-cycle push) so nothing needs to + # be preserved here. + reset = _run_git(["reset", "--hard", f"origin/{pipeline_branch}"], timeout=30) + if reset is None or reset.returncode != 0: + _pkg.logger.warning( + "pr-open rebase: reset to origin/<branch> failed, skipping", + pipeline_id=pipeline_id, + branch=pipeline_branch, + stderr=(reset.stderr.strip() if reset is not None else None), + ) + return False + + # Step 6: Rebase using the safe ``--onto <new_base> <upstream>`` + # form. HEAD is the implicit branch being rebased (set by the + # step-5 reset above). The closest argv-shape prior art is + # ``gateway_client._build_rebase_cmd`` — that one rebases in the + # opposite direction (replay HEAD onto a stale branch tip) but uses + # the same explicit-upstream pattern that pins the replay range to + # ``<upstream>..HEAD`` and so sidesteps the bare-form contamination + # shape behind #2222. + rebase = _run_git( + [ + "rebase", + "--onto", + f"origin/{base_branch}", + merge_base, + ], + timeout=120, + ) + if rebase is None or rebase.returncode != 0: + # Conflict, timeout, or any failure: abort cleanly and restore + # to origin/<branch> so the caller can still open the PR + # against the un-rebased tip. Unlike the resume-time helper, + # we *don't* raise here — pipeline failure for a merge conflict + # at PR-open time is worse than a slightly-behind PR. + _run_git(["rebase", "--abort"], timeout=30) + _run_git(["reset", "--hard", f"origin/{pipeline_branch}"], timeout=30) + stderr_text = rebase.stderr.strip() if rebase is not None else "rebase command timed out" + _pkg.logger.warning( + "pr-open rebase: rebase failed, opening PR against un-rebased tip", + pipeline_id=pipeline_id, + branch=pipeline_branch, + base_branch=base_branch, + behind_base=behind_count, + stderr=stderr_text, + timed_out=rebase is None, + ) + return False + + # Step 7: Force-push the rebased tip so origin/<branch> matches the + # SHAs the PR will be opened against. + push_result = spawner.gateway.push_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(worktree_repo_path), + branch=pipeline_branch, + mode=gateway_mode, + base_branch=base_branch, + force=True, + ) + if not push_result.ok: + # Best-effort restore so the worktree state is predictable for + # downstream callers; the PR still opens against the pre-rebase + # remote tip (which is what origin/<branch> still reflects). + _run_git(["reset", "--hard", f"origin/{pipeline_branch}"], timeout=30) + _pkg.logger.warning( + "pr-open rebase: force-push of rebased branch failed, " + "opening PR against un-rebased remote tip", + pipeline_id=pipeline_id, + branch=pipeline_branch, + category=push_result.category, + detail=push_result.detail, + ) + return False + + # Re-fetch so origin/<branch> reflects the pushed tip locally. + spawner.gateway.fetch_worktree_branch( + pipeline_id=pipeline_id, + repo_path=str(worktree_repo_path), + mode=gateway_mode, + ) + _pkg.logger.info( + "pr-open rebase: rebased and force-pushed pipeline branch", + pipeline_id=pipeline_id, + branch=pipeline_branch, + base_branch=base_branch, + behind_base_at_start=behind_count, + ) + return True + + +def _read_tree_head(git_base: list[str]) -> None: + """Refresh the index from HEAD without touching the working tree. + + Defends ``_commit_statefiles_to_worktree`` against a cross-worktree + branch-ref advance. When an agent runs the gateway-allowed recovery + primitive ``git update-ref refs/heads/<pipeline-branch> <sha>`` from + a sibling worktree (see ``sandbox/agent-config/rules/branch-recovery.md`` + and the detached-HEAD hint in ``gateway/gateway.py``), the shared local + branch ref advances out from under this worktree. ``update-ref`` does + not honour per-worktree locks, so this worktree's HEAD symref + silently jumps to the agent's commit while the index and working tree + stay at the prior state. Without this refresh, the stale index + reports every agent-pushed file as a *staged deletion* against HEAD, + and the subsequent ``git commit`` lands them as a real delete commit + (the symptom in #2626). ``read-tree HEAD`` repoints the index to + the new HEAD without touching the working tree; the immediately + following ``git add --force`` then stages only the orchestrator's + on-disk writes. + """ + subprocess.run( + [*git_base, "read-tree", "HEAD"], + capture_output=True, + text=True, + check=True, + timeout=30, + ) + + +def _restore_missing_state_files_from_head( + git_base: list[str], + worktree_path: Path, + pipeline_id: str | None = None, +) -> None: + """Materialize tracked ``.egg-state/`` files that HEAD has but disk doesn't. + + Companion to :func:`_read_tree_head`: the same cross-worktree + ``update-ref`` advance behind #2626 leaves the working tree stale + relative to the just-advanced HEAD. The #2626 fix protected the + *commit* (no delete-commit lands), but downstream readers go through + the working tree — :func:`_populate_contract_from_plan` reads the + plan draft via ``Path(...).read_text()``, which fails with the + natural ``PlanDraftMissingOnLocalError`` even though HEAD itself + carries the agent-pushed draft (the #2721 symptom; recovery in the + field was ``git checkout HEAD -- .egg-state/drafts/ + .egg-state/agent-outputs/``). + + ``git ls-files -z --deleted -- .egg-state/`` lists tracked files + that are missing on disk. ``-z`` switches the output to + NUL-separated raw bytes so paths with non-ASCII chars, newlines, or + quote chars survive parsing intact (with the default + ``core.quotePath=true`` the non-``-z`` form C-quote-encodes those + paths and ``splitlines()`` would misparse them). Must be called + AFTER :func:`_read_tree_head` so the index reflects HEAD; otherwise + a stale index can leave the delete-list incomplete. ``git checkout + HEAD --pathspec-from-file=- --pathspec-file-nul`` then restores each + missing path in both the index and the working tree (the index + reset is a no-op because read-tree HEAD already aligned it); piping + the NUL-separated list via stdin sidesteps any ARG_MAX limit on the + argv path even for pathological ``.egg-state/`` populations. + Confined to ``.egg-state/`` so the restoration cannot resurrect a + sibling-pipeline file the orchestrator deliberately removed + elsewhere in the tree. + + Fail-open: any subprocess error logs and returns silently — the + downstream populator still has its own missing-draft guard, so a + failure here cannot silently hide a true draft-missing case. + """ + try: + deleted = subprocess.run( + [*git_base, "ls-files", "-z", "--deleted", "--", ".egg-state/"], + capture_output=True, + check=False, + timeout=30, + ) + except (subprocess.TimeoutExpired, OSError) as ls_err: + _pkg.logger.warning( + "_restore_missing_state_files_from_head: ls-files probe failed", + worktree_path=str(worktree_path), + pipeline_id=pipeline_id, + error=str(ls_err), + ) + return + if deleted.returncode != 0: + _pkg.logger.warning( + "_restore_missing_state_files_from_head: ls-files probe failed", + worktree_path=str(worktree_path), + pipeline_id=pipeline_id, + returncode=deleted.returncode, + stderr=deleted.stderr.decode("utf-8", errors="replace").strip()[:200], + ) + return + missing_paths = [p for p in deleted.stdout.split(b"\0") if p] + if not missing_paths: + return + pathspec_stdin = b"\0".join(missing_paths) + b"\0" + try: + restore = subprocess.run( + [ + *git_base, + "checkout", + "HEAD", + "--pathspec-from-file=-", + "--pathspec-file-nul", + ], + input=pathspec_stdin, + capture_output=True, + check=False, + timeout=30, + ) + except (subprocess.TimeoutExpired, OSError) as checkout_err: + _pkg.logger.warning( + "_restore_missing_state_files_from_head: checkout failed", + worktree_path=str(worktree_path), + pipeline_id=pipeline_id, + missing_count=len(missing_paths), + error=str(checkout_err), + ) + return + if restore.returncode != 0: + _pkg.logger.warning( + "_restore_missing_state_files_from_head: checkout failed", + worktree_path=str(worktree_path), + pipeline_id=pipeline_id, + missing_count=len(missing_paths), + returncode=restore.returncode, + stderr=restore.stderr.decode("utf-8", errors="replace").strip()[:200], + ) + return + _pkg.logger.info( + "_restore_missing_state_files_from_head: restored tracked-but-missing files", + worktree_path=str(worktree_path), + pipeline_id=pipeline_id, + restored_count=len(missing_paths), + restored_sample=[p.decode("utf-8", errors="replace") for p in missing_paths[:5]], + ) diff --git a/orchestrator/tests/test_advance_phase_thread.py b/orchestrator/tests/test_advance_phase_thread.py index 987a813283..6685ebb291 100644 --- a/orchestrator/tests/test_advance_phase_thread.py +++ b/orchestrator/tests/test_advance_phase_thread.py @@ -295,7 +295,11 @@ def _auto_advance_block(self) -> str: """Extract the auto-advance block from _run_pipeline's source.""" from routes import pipelines - source = inspect.getsource(pipelines._run_pipeline) + # #3312 slice-4: _run_pipeline moved into its own submodule, so its + # free barrel-global references are rewritten as ``_pkg.<name>``. + # Strip that decomposition-only prefix so these structural + # assertions (written against the pre-split text) keep matching. + source = inspect.getsource(pipelines._run_pipeline).replace("_pkg.", "") assert self._BLOCK_MARKER in source, ( f"Could not find {self._BLOCK_MARKER!r} in _run_pipeline source. " "Tests rely on this token bracketing the auto-advance block; " @@ -393,12 +397,17 @@ class TestRecoverPipelineClearsConcurrentState: _BLOCK_MARKER = "TEST_MARKER: recover_advance_clear" def _recover_advance_block(self) -> str: - """Extract the recover-advance clear block from start_pipeline.""" + """Extract the recover-advance clear block from start_pipeline's body.""" from routes import pipelines - source = inspect.getsource(pipelines.start_pipeline) + # #3312 slice-4: the ``@pipelines_bp.route`` decorator stays on the + # thin ``start_pipeline`` wrapper (decision-8) while its body moved to + # ``_start_pipeline_body`` in ``_routes_lifecycle.py``; introspect the + # body and strip the ``_pkg.`` barrel-reference prefix so this marker + # (and the pre-split assertions below) keep resolving. + source = inspect.getsource(pipelines._start_pipeline_body).replace("_pkg.", "") assert self._BLOCK_MARKER in source, ( - f"Could not find {self._BLOCK_MARKER!r} in start_pipeline source. " + f"Could not find {self._BLOCK_MARKER!r} in _start_pipeline_body source. " "Tests rely on this token bracketing the recover_pipeline advance " "branch's clear call; if the block was moved or removed, update " "both source and tests." @@ -450,7 +459,29 @@ class TestPostBrcBandSwallowsErrors: def _run_pipeline_source(self) -> str: from routes import pipelines - return inspect.getsource(pipelines._run_pipeline) + # _run_pipeline's setup blocks are being decomposed into helper + # functions (#3312 slice-4); include them so call-site coverage + # assertions still see the calls that moved out of the barrel body. + # The moved code (and its broad ``except``) is verbatim, so the pinned + # counts and try/except regexes below stay valid. + _EXTRACTED_HELPERS = ( + "_sync_contract_setup", + "_run_hitl_gate_converge", + "_run_plan_advance", + "_run_pending_phase_init", + "_run_implement_advance", + ) + parts = [inspect.getsource(pipelines._run_pipeline)] + for _name in _EXTRACTED_HELPERS: + _fn = getattr(pipelines, _name, None) + if _fn is not None: + parts.append(inspect.getsource(_fn)) + # #3312 slice-4: _run_pipeline (and its extracted helpers) now live in + # submodules whose barrel-global references are rewritten as + # ``_pkg.<name>``; strip that decomposition-only prefix so the + # try/except structural regexes (written against the pre-split text) + # keep matching the post-BRC call sites. + return "\n".join(parts).replace("_pkg.", "") def test_sync_worktree_with_remote_is_wrapped(self): """The post-BRC worktree-sync call was unwrapped — a gateway HTTP diff --git a/orchestrator/tests/test_ble001_narrowing_audit.py b/orchestrator/tests/test_ble001_narrowing_audit.py index ae610c3953..e29635e149 100644 --- a/orchestrator/tests/test_ble001_narrowing_audit.py +++ b/orchestrator/tests/test_ble001_narrowing_audit.py @@ -45,7 +45,17 @@ sys.path.insert(0, str(_orchestrator_path)) -_PIPELINES_SRC = (_orchestrator_path / "routes" / "pipelines.py").read_text(encoding="utf-8") +# ``routes/pipelines`` was decomposed from a single ``pipelines.py`` module +# into a sub-package (#3312 slice-4); concatenate every source file under the +# package so the BLE001-narrowing audit still sees the full module surface +# regardless of which submodule now holds each narrowed except-site. +_PIPELINES_PKG = _orchestrator_path / "routes" / "pipelines" +if _PIPELINES_PKG.is_dir(): + _PIPELINES_SRC = "\n".join( + p.read_text(encoding="utf-8") for p in sorted(_PIPELINES_PKG.rglob("*.py")) + ) +else: # pre-split fallback + _PIPELINES_SRC = (_orchestrator_path / "routes" / "pipelines.py").read_text(encoding="utf-8") def test_cascade_alert_gateway_import_uses_narrow_importerror() -> None: diff --git a/orchestrator/tests/test_hitl_revision.py b/orchestrator/tests/test_hitl_revision.py index 9bc96448cd..20816e2062 100644 --- a/orchestrator/tests/test_hitl_revision.py +++ b/orchestrator/tests/test_hitl_revision.py @@ -24,6 +24,26 @@ from routes.pipelines import _APPROVE_KEYWORDS, _BARE_OPTION_LABELS +def _pipelines_package_source() -> str: + """Concatenated source of every module in the routes/pipelines package. + + #3312 slice-4 decomposed ``pipelines.py`` into a sub-package, so + ``inspect.getsource(pipelines)`` now returns only the barrel + ``__init__.py``. Source-token assertions must scan the whole package. + The submodules reach barrel globals via the ``_pkg.`` indirection; + stripping it restores the pre-split token space so the assertions + below keep pinning the same code shapes. + """ + import inspect + from pathlib import Path + + from routes import pipelines + + pkg_dir = Path(inspect.getfile(pipelines)).parent + combined = "\n".join(p.read_text() for p in sorted(pkg_dir.glob("*.py"))) + return combined.replace("_pkg.", "") + + class TestApproveKeywords: """Verify _APPROVE_KEYWORDS correctly classifies resolutions.""" @@ -173,17 +193,14 @@ def test_plan_phase_always_triggers_contract_population(self): _populate_contract_from_plan should be called for plan phase regardless of review_cycles or hitl_review_cycles count. """ - import inspect from routes.pipelines import _populate_contract_from_plan # Verify the function exists and is callable assert callable(_populate_contract_from_plan) - # Read the source of _run_pipeline to verify the guard was removed - from routes import pipelines - - source = inspect.getsource(pipelines) + # Read the pipelines package source to verify the guard was removed + source = _pipelines_package_source() # The old guard was: if current_phase.value == "plan" and phase_execution.review_cycles == 0: # The new guard should just be: if current_phase.value == "plan": @@ -215,11 +232,7 @@ def test_config_serialization(self): def test_circuit_breaker_reads_hitl_config(self): """The pipeline source should reference config.max_hitl_review_cycles, not max_review_cycles.""" - import inspect - - from routes import pipelines - - source = inspect.getsource(pipelines) + source = _pipelines_package_source() # The old code was: max_hitl_cycles = pipeline.config.max_review_cycles # It should now be: max_hitl_cycles = pipeline.config.max_hitl_review_cycles assert "pipeline.config.max_hitl_review_cycles" in source @@ -659,11 +672,7 @@ class TestNoForceAdvance: """ def _pipelines_source(self) -> str: - import inspect - - from routes import pipelines - - return inspect.getsource(pipelines) + return _pipelines_package_source() def test_force_advance_log_removed(self): """The "advancing despite feedback" force-advance log must be gone.""" @@ -982,11 +991,7 @@ def test_sync_function_is_callable(self): def test_sync_called_for_hitl_gate_phases(self): """Verify the pipeline source calls sync for both refine and plan phases.""" - import inspect - - from routes import pipelines - - source = inspect.getsource(pipelines) + source = _pipelines_package_source() # The sync should be called when current_phase.value is in _HITL_GATE_PHASES assert "_sync_pipeline_decisions_to_contract" in source diff --git a/orchestrator/tests/test_overseer_model.py b/orchestrator/tests/test_overseer_model.py index 1cf2c502bf..6f6a6a19c6 100644 --- a/orchestrator/tests/test_overseer_model.py +++ b/orchestrator/tests/test_overseer_model.py @@ -121,7 +121,14 @@ def _spawn_path_source() -> str: ``routes/pipelines.py`` (``_spawn_overseer_agent``). Concatenate both so the resolver assertion holds wherever the spawn path currently lives. """ - return _spawner_source() + "\n" + (_orchestrator_path / "routes" / "pipelines.py").read_text() + return ( + _spawner_source() + + "\n" + + "\n".join( + p.read_text(encoding="utf-8") + for p in sorted((_orchestrator_path / "routes" / "pipelines").rglob("*.py")) + ) + ) # A model name that is not any tier default and routes through the LiteLLM diff --git a/orchestrator/tests/test_pipeline_role_to_reviewer_type_mapping.py b/orchestrator/tests/test_pipeline_role_to_reviewer_type_mapping.py index 9bcd66fae7..024479d508 100644 --- a/orchestrator/tests/test_pipeline_role_to_reviewer_type_mapping.py +++ b/orchestrator/tests/test_pipeline_role_to_reviewer_type_mapping.py @@ -98,7 +98,10 @@ class TestNoRedundantMappingDictNearLine: """ def setup_method(self) -> None: - self.source = PIPELINES_PATH.read_text(encoding="utf-8") + self.source = "\n".join( + p.read_text(encoding="utf-8") + for p in sorted((PIPELINES_PATH.parent / "pipelines").rglob("*.py")) + ) def test_no_explicit_dict_for_new_role_names(self) -> None: # Reject any dict literal that maps a new role-name string key to diff --git a/orchestrator/tests/test_pipelines_apply.py b/orchestrator/tests/test_pipelines_apply.py index df645aa16e..16957e5a57 100644 --- a/orchestrator/tests/test_pipelines_apply.py +++ b/orchestrator/tests/test_pipelines_apply.py @@ -100,10 +100,20 @@ # Source-text reads for structural invariants. These always run — # they read the .py file directly rather than importing the module. +# ``routes/pipelines`` was decomposed from a single ``pipelines.py`` module into +# a sub-package (#3312 slice-4); concatenate every source file under the package +# so these structural source-text invariants still see the full module surface +# regardless of which submodule now holds each symbol. +_PIPELINES_PKG_PATH = Path(__file__).parent.parent / "routes" / "pipelines" _PIPELINES_SRC_PATH = Path(__file__).parent.parent / "routes" / "pipelines.py" -_PIPELINES_SRC: str = ( - _PIPELINES_SRC_PATH.read_text(encoding="utf-8") if _PIPELINES_SRC_PATH.exists() else "" -) +if _PIPELINES_PKG_PATH.is_dir(): + _PIPELINES_SRC: str = "\n".join( + p.read_text(encoding="utf-8") for p in sorted(_PIPELINES_PKG_PATH.rglob("*.py")) + ) +elif _PIPELINES_SRC_PATH.exists(): + _PIPELINES_SRC = _PIPELINES_SRC_PATH.read_text(encoding="utf-8") +else: + _PIPELINES_SRC = "" # ----------------------------------------------------------------------------- # WontDoEntry dataclass diff --git a/orchestrator/tests/test_pipelines_origin_main_parameterization.py b/orchestrator/tests/test_pipelines_origin_main_parameterization.py index 6a4a3b819d..719f1bbbd3 100644 --- a/orchestrator/tests/test_pipelines_origin_main_parameterization.py +++ b/orchestrator/tests/test_pipelines_origin_main_parameterization.py @@ -86,7 +86,11 @@ def test_helper_has_multiple_call_sites(self) -> None: diff commands, recovery paths). If this count drops below 5, the parameterization has likely been partially reverted. """ - source = inspect.getsource(pipelines_module) + # After the #3312 slice-4 decomposition the call sites live across the + # routes/pipelines/ package submodules (reached via ``_pkg.``), so scan + # the whole package directory rather than just the barrel module source. + _pkg_dir = Path(pipelines_module.__file__).parent + source = "".join(p.read_text() for p in sorted(_pkg_dir.glob("*.py"))) # Count invocations, NOT the def line. invocation_count = source.count("_resolve_origin_ref(") # Definition (``def _resolve_origin_ref(``) contributes 1 match; diff --git a/orchestrator/tests/test_populate_contract_audit_events.py b/orchestrator/tests/test_populate_contract_audit_events.py index 878723d932..af6fb25ceb 100644 --- a/orchestrator/tests/test_populate_contract_audit_events.py +++ b/orchestrator/tests/test_populate_contract_audit_events.py @@ -1231,11 +1231,16 @@ class TestPlanCompleteCallSiteWireUp: @staticmethod def _run_pipeline_source() -> str: + # #3312 slice-4: the plan-complete branch moved out of + # ``_run_pipeline`` into ``_run_plan_advance`` (the per-phase + # handler split anticipated by the fragility note above). The + # submodule reaches barrel globals via ``_pkg.``; stripping it + # restores the pre-split token space these assertions pin. import inspect - from routes.pipelines import _run_pipeline + from routes.pipelines import _run_plan_advance - return inspect.getsource(_run_pipeline) + return inspect.getsource(_run_plan_advance).replace("_pkg.", "") def test_call_site_uses_populate_result_is_empty_contract_helper(self): """The call-site condition must route through the shared helper, @@ -1348,11 +1353,16 @@ class TestSafetyNetForestViolationLandsOnEmptyContractHitl: @staticmethod def _run_pipeline_source() -> str: + # #3312 slice-4: the ``start_phase=implement`` safety net moved + # out of ``_run_pipeline`` into ``_start_phase_setup`` (the + # decomposition anticipated by the fragility note above). The + # submodule reaches barrel globals via ``_pkg.``; stripping it + # restores the pre-split token space these assertions pin. import inspect - from routes.pipelines import _run_pipeline + from routes.pipelines import _start_phase_setup - return inspect.getsource(_run_pipeline) + return inspect.getsource(_start_phase_setup).replace("_pkg.", "") def test_safety_net_catches_forest_validation_error(self): """The safety-net inner call must be wrapped in a try/except for diff --git a/orchestrator/tests/test_slice_loop_import_seam.py b/orchestrator/tests/test_slice_loop_import_seam.py index 25505c58e2..0b7b69838d 100644 --- a/orchestrator/tests/test_slice_loop_import_seam.py +++ b/orchestrator/tests/test_slice_loop_import_seam.py @@ -28,7 +28,10 @@ import ast from pathlib import Path -_PIPELINES = Path(__file__).resolve().parent.parent / "routes" / "pipelines.py" +_PIPELINES_PKG = Path(__file__).resolve().parent.parent / "routes" / "pipelines" +_PIPELINES_SRC = "\n".join( + p.read_text(encoding="utf-8") for p in sorted(_PIPELINES_PKG.rglob("*.py")) +) # Functions that run inside the implement-phase slice loop. A bare # ``from orchestrator.X import Y`` inside any of these will crash the @@ -102,7 +105,7 @@ def test_slice_loop_orchestrator_imports_are_dual_guarded() -> None: must be wrapped in ``try/except ImportError`` so the pod runtime (flat layout, no ``orchestrator/`` package) falls back to the top-level form. Catches the recurrence pattern that #2901 fixes.""" - tree = ast.parse(_PIPELINES.read_text()) + tree = ast.parse(_PIPELINES_SRC) failures: dict[str, list[int]] = {} seen: set[str] = set() for node in ast.walk(tree): @@ -114,7 +117,7 @@ def test_slice_loop_orchestrator_imports_are_dual_guarded() -> None: missing = _SLICE_LOOP_FUNCS - seen assert not missing, ( - f"Slice-loop functions not found in {_PIPELINES}: {sorted(missing)}. " + f"Slice-loop functions not found in {_PIPELINES_PKG}: {sorted(missing)}. " "Update _SLICE_LOOP_FUNCS or the function names if the refactor " "of routes/pipelines.py moved them." ) diff --git a/orchestrator/tests/test_start_pipeline.py b/orchestrator/tests/test_start_pipeline.py index 182c1fb9f8..90408fc22f 100644 --- a/orchestrator/tests/test_start_pipeline.py +++ b/orchestrator/tests/test_start_pipeline.py @@ -1446,7 +1446,10 @@ class TestSandboxJiraEnvBuilderSourceSnippet: """ def test_source_exports_egg_jira_ticket_and_project(self): - src = (Path(__file__).parent.parent / "routes" / "pipelines.py").read_text() + src = "\n".join( + p.read_text(encoding="utf-8") + for p in sorted((Path(__file__).parent.parent / "routes" / "pipelines").rglob("*.py")) + ) assert 'sandbox_env["EGG_JIRA_TICKET"]' in src assert 'sandbox_env["EGG_JIRA_PROJECT"]' in src @@ -1454,7 +1457,10 @@ def test_source_never_exports_jira_secrets(self): """A regression check: the spawn-env assembly must never put ``JIRA_BASE_URL`` / ``JIRA_USERNAME`` / ``JIRA_API_TOKEN`` into ``sandbox_env`` (risk R7).""" - src = (Path(__file__).parent.parent / "routes" / "pipelines.py").read_text() + src = "\n".join( + p.read_text(encoding="utf-8") + for p in sorted((Path(__file__).parent.parent / "routes" / "pipelines").rglob("*.py")) + ) for forbidden in ("JIRA_BASE_URL", "JIRA_USERNAME", "JIRA_API_TOKEN"): # Scan for any write to sandbox_env[<forbidden>]. assert f'sandbox_env["{forbidden}"]' not in src, ( @@ -1478,7 +1484,10 @@ def test_source_never_exports_egg_branch(self): slice agents are downgraded to the pipeline tip, breaking every slice-coder push (the original bug). """ - src = (Path(__file__).parent.parent / "routes" / "pipelines.py").read_text() + src = "\n".join( + p.read_text(encoding="utf-8") + for p in sorted((Path(__file__).parent.parent / "routes" / "pipelines").rglob("*.py")) + ) assert 'sandbox_env["EGG_BRANCH"]' not in src, ( "orchestrator/routes/pipelines.py writes EGG_BRANCH into " "sandbox_env; the spawner is the single source of truth and " diff --git a/scripts/check-hardcoded-ports.py b/scripts/check-hardcoded-ports.py index 0346af65c2..836ad4761f 100644 --- a/scripts/check-hardcoded-ports.py +++ b/scripts/check-hardcoded-ports.py @@ -38,8 +38,10 @@ # Shell scripts cannot import Python modules "gateway/entrypoint.sh", "orchestrator/entrypoint.sh", - # Gateway Python module has its own DEFAULT_PORT (source of truth for gateway) - "gateway/gateway.py", + # Gateway Python package has its own DEFAULT_PORT (source of truth for + # gateway). Was the monolithic gateway/gateway.py before the #3312 + # decomposition split it into a sub-package. + "gateway/gateway/", # Gateway tests may need hardcoded values "gateway/tests/", # Squid configuration template diff --git a/scripts/file-size-allowlist.yaml b/scripts/file-size-allowlist.yaml index 7bf13b0a2e..bb96b21c9d 100644 --- a/scripts/file-size-allowlist.yaml +++ b/scripts/file-size-allowlist.yaml @@ -19,8 +19,12 @@ caps: soft_lines: 800 soft_bytes: 60000 +# The file-size decomposition program (#3312, continued by #3450/#3447) is +# COMPLETE: every giant it targeted has been decomposed into a sub-package +# whose barrel + submodules are all under the global cap. New entries should +# only be added (with a tracking issue) for files awaiting their own +# decomposition; the ratchet flags any allowlisted file that has dropped +# back under the cap as stale. files: - orchestrator/routes/pipelines.py: - issue: "2248" orchestrator/concurrent_executor.py: issue: "3498" diff --git a/tests/sandbox/egg_agent_tools/test_handlers_brc.py b/tests/sandbox/egg_agent_tools/test_handlers_brc.py index cf3f8a258c..f3deb02a7c 100644 --- a/tests/sandbox/egg_agent_tools/test_handlers_brc.py +++ b/tests/sandbox/egg_agent_tools/test_handlers_brc.py @@ -1424,8 +1424,10 @@ class TestBrcHistoryTypesDriftGuard: def test_handler_whitelist_matches_writer_set(self): import re - pipelines_path = ROOT / "orchestrator" / "routes" / "pipelines.py" - source = pipelines_path.read_text() + pipelines_pkg = ROOT / "orchestrator" / "routes" / "pipelines" + source = "\n".join( + p.read_text(encoding="utf-8") for p in sorted(pipelines_pkg.rglob("*.py")) + ) match = re.search( r"^BRC_HISTORY_TYPES\s*=\s*frozenset\s*\(\s*\{(?P<body>.*?)\}\s*\)", source, @@ -1433,7 +1435,7 @@ def test_handler_whitelist_matches_writer_set(self): ) assert match is not None, ( "Could not locate ``BRC_HISTORY_TYPES = frozenset({...})`` " - f"literal in {pipelines_path}; the drift-guard regex needs " + f"literal in {pipelines_pkg}; the drift-guard regex needs " "updating to track the new shape." ) writer_types = frozenset(re.findall(r'"([A-Z_]+)"', match.group("body")))