Escalate a green-gate red to HITL, mirroring the evidence gate

docs/architecture/slice-dag.md

@@ -556,9 +556,12 @@ shape:
        withholds the slice PR), `log` runs the checks and logs the
        verdict without blocking, `off` returns before the runner Job is
        spawned at all; fail-open on infra errors, including
-       infra-signature-tagged reds inside check execution, #3417; the
-       gate can also write to the integration branch — see
-       "Green-gate autofix" below) — calls
+       infra-signature-tagged reds inside check execution, #3417; a red
+       verdict that survives to block lands an unresolved HITL
+       `Decision` via `_escalate_green_gate_to_hitl`, #3572 parity with
+       the evidence gate above, so the block is an operator question
+       rather than a silently parked slice; the gate can also write to
+       the integration branch — see "Green-gate autofix" below) — calls
        `GatewayClient.create_slice_pr` with `base` resolved from the
        slice's DAG parent (root → latest completed chain tip, else the
        pipeline branch (#3541); child → parent's
@@ -614,7 +617,7 @@ shape:
        operator-facing slice failure message only when every genuine
        red's own re-run went green; otherwise it stays in the
        structured log. Any failure to commit or push blocks the
-       slice exactly like an unfixed red.
+       slice exactly like an unfixed red, HITL escalation included.
     4. After the wave completes, `scheduler.poll_cascades()` drains any
        expired cascades and emits the orchestrator-side
        `OVERSEER_ALERT` for each (see "Failure cascade").
@@ -1112,7 +1115,7 @@ on parse failure. The green-gate knobs below are read directly via
 | `EGG_ORCH_STACKED_PR_RECONCILER_INTERVAL_SECONDS` | float | 30.0 | Reconciler polling cadence for orphaned child PRs. |
 | `EGG_ORCH_CROSS_REPO_MERGE_GATE_MAX_ATTEMPTS` | int | 240 | Poll-attempt budget for the cross-repo merge-sequencing gate (#3393) before a never-merging upstream escalates to a HITL hold; ~2h at the default reconciler cadence. See [Cross-repo merge-sequencing hold](#cross-repo-merge-sequencing-hold-two-tier). |
 | `EGG_SLICE_BASE_ANCESTRY_GATE` | str | `on` | Operator kill switch for the admission-time base-ancestry gate (#3541 — see [Root linearization & the base-ancestry gate](#root-linearization--the-base-ancestry-gate-3541)): any of `off`/`0`/`false`/`no` (case-insensitive, whitespace-tolerant) disables the gate; any other value (including unset) leaves it enabled. |
-| `EGG_SLICE_GREEN_GATE` | str | `on` | Per-slice green gate rollout switch (#3398): `off` skips the gate entirely; `log` runs the repo's configured checks at the slice tip and logs a red verdict without blocking; `on` (the default) withholds the slice PR on a red verdict. Case-insensitive, with aliases — `on` also accepts `1`/`true`/`yes`, `log` also accepts `log-only`/`log_only`, and `off` also accepts `0`/`false`/`no`. Weakening the gate takes an explicit, correctly-spelled `off` or `log`; unset or unrecognised values resolve to `on`, so a typo cannot silently drop a deployment below the product default. Expect the first reds to be gate wiring rather than slice code — a stale contract snapshot reddening contract-hygiene tests (#3301), or `make test`'s changeset narrowing resolving its baseline against `git merge-base` in a fresh worktree — and note either of those reds *every* slice close until fixed (a missing prebuilt-deps snapshot is not in this list: the runner exits non-zero and the gate fails open, costing coverage rather than throughput); the failure message names the branch to fix and quotes `EGG_SLICE_GREEN_GATE=off` as the bypass, and the slice's commits stay on the integration branch (a red gate withholds the PR, it does not discard work). Latency is identical under `log` and `on` — both run the checks and wait for the runner pod, so slice-close latency grows by the check duration (bounded by `EGG_SLICE_GREEN_GATE_TIMEOUT_SECONDS`). The worst case is not a slow suite but a runner pod that never schedules — the wait is `timeout` plus a 120s scheduling grace (~32 min at the defaults) before failing open, so a capacity-starved cluster pays that per slice close. A *partially* delayed pod is quieter and more common: the runner's deadline is the **Job's** `activeDeadlineSeconds`, counted from the Job's `startTime` (before any pod is bound), so time spent Pending or pulling is subtracted from the in-pod check budget rather than added to the wait — the scheduling grace widens only the orchestrator's wait. A pod delayed N seconds gets N fewer seconds to run checks, and a `DeadlineExceeded` kill emits no verdict line, so the gate fails open with no verdict at all. Capacity starvation therefore raises the rate of spurious no-verdict fail-opens as well as dead time ([#3622](https://github.com/jwbron/egg/issues/3622)) — under the `on` default that is a slice close you believed was gated and wasn't, though the direction is always *under*-blocking, never a false red. Grep for *both* fail-open log lines when diagnosing one: the Job controller deletes the active pod on `DeadlineExceeded` rather than leaving it terminal, so the wait normally times out ("runner pod did not reach a terminal state"); a poll that catches the pod reporting `Failed` mid-termination reads partial output and lands on "no parseable verdict from runner" instead. |
+| `EGG_SLICE_GREEN_GATE` | str | `on` | Per-slice green gate rollout switch (#3398): `off` skips the gate entirely; `log` runs the repo's configured checks at the slice tip and logs a red verdict without blocking; `on` (the default) withholds the slice PR on a red verdict. Case-insensitive, with aliases — `on` also accepts `1`/`true`/`yes`, `log` also accepts `log-only`/`log_only`, and `off` also accepts `0`/`false`/`no`. Weakening the gate takes an explicit, correctly-spelled `off` or `log`; unset or unrecognised values resolve to `on`, so a typo cannot silently drop a deployment below the product default. Expect the first reds to be gate wiring rather than slice code — a stale contract snapshot reddening contract-hygiene tests (#3301), or `make test`'s changeset narrowing resolving its baseline against `git merge-base` in a fresh worktree — and note either of those reds *every* slice close until fixed (a missing prebuilt-deps snapshot is not in this list: the runner exits non-zero and the gate fails open, costing coverage rather than throughput); recovery is operator-driven rather than automatic: a red verdict lands an unresolved HITL `Decision` on the contract (`_escalate_green_gate_to_hitl`, #3572 parity) whose continue / restart-slice / cancel options are *recorded* for the operator rather than dispatched — nothing routes on the `[#3398 green-gate]` marker today ([#3634](https://github.com/jwbron/egg/issues/3634)), so picking one answers the question without taking the action, and the fix and restart stay manual. What the Decision buys is visibility: the block surfaces in `/sdlc` instead of only as a FAILED phase an operator has to notice. Retries of one *unanswered* red adopt the open decision rather than stacking duplicates, but a red that recurs *after* the operator resolved it raises a fresh decision — precisely because resolving one has no mechanical effect, a gate-wiring red that survives a `restart_phase` must ask again rather than silently leave `pending_decisions` empty. The failure message names the branch to fix and quotes `EGG_SLICE_GREEN_GATE=off` as the bypass, and the slice's commits stay on the integration branch (a red gate withholds the PR, it does not discard work). Latency is identical under `log` and `on` — both run the checks and wait for the runner pod, so slice-close latency grows by the check duration (bounded by `EGG_SLICE_GREEN_GATE_TIMEOUT_SECONDS`). The worst case is not a slow suite but a runner pod that never schedules — the wait is `timeout` plus a 120s scheduling grace (~32 min at the defaults) before failing open, so a capacity-starved cluster pays that per slice close. A *partially* delayed pod is quieter and more common: the runner's deadline is the **Job's** `activeDeadlineSeconds`, counted from the Job's `startTime` (before any pod is bound), so time spent Pending or pulling is subtracted from the in-pod check budget rather than added to the wait — the scheduling grace widens only the orchestrator's wait. A pod delayed N seconds gets N fewer seconds to run checks, and a `DeadlineExceeded` kill emits no verdict line, so the gate fails open with no verdict at all. Capacity starvation therefore raises the rate of spurious no-verdict fail-opens as well as dead time ([#3622](https://github.com/jwbron/egg/issues/3622)) — under the `on` default that is a slice close you believed was gated and wasn't, though the direction is always *under*-blocking, never a false red. Grep for *both* fail-open log lines when diagnosing one: the Job controller deletes the active pod on `DeadlineExceeded` rather than leaving it terminal, so the wait normally times out ("runner pod did not reach a terminal state"); a poll that catches the pod reporting `Failed` mid-termination reads partial output and lands on "no parseable verdict from runner" instead. |
 | `EGG_SLICE_GREEN_GATE_SKIP_CHECKS` | str (comma-separated) | `security` | Configured check *names* (from `repositories.yaml` `checks`) the gate skips. |
 | `EGG_SLICE_GREEN_GATE_TIMEOUT_SECONDS` | int | 1800 | Wall-clock budget for the check-runner pod (spawn-to-terminal); a hung suite degrades to fail-open rather than wedging the slice close. Enforced as a Job-level `activeDeadlineSeconds` (`timeout + 60`) counted from Job start, so it is a ceiling on scheduling *plus* checks, not on checks alone — see the `EGG_SLICE_GREEN_GATE` row and [#3622](https://github.com/jwbron/egg/issues/3622). |
 | `EGG_SLICE_GREEN_GATE_INFRA_FAIL_OPEN` | str | `on` | Infra-red fail-open (#3417): the runner tags red checks whose full output matches an exact infra signature (the sandbox git wrapper's gateway-down / missing-env / session-auth errors, the kernel's ENOSPC message) or whose process died by SIGKILL; a verdict where *every* red check is infra-tagged fails open instead of blocking, and mixed verdicts block on the genuine reds only. `off`/`0`/`false`/`no` restores strict every-red-blocks behavior; `on`/`1`/`true`/`yes` (and unset, and an empty or whitespace-only value) enable it silently; any other value resolves to `on` and logs a warning, since the typo direction here is strict → lenient. |

orchestrator/routes/pipelines/__init__.py

@@ -1405,6 +1405,7 @@ def stream_pipeline(pipeline_id: str) -> Response:
     _parent_branch_probe_impl,
     _persist_slice_status_complete_impl,
     _slice_close_evidence_gate,
+    _slice_close_green_gate,
 )
 from ._run_phase import (  # noqa: E402,F401
     _run_phase_execution,
@@ -1454,6 +1455,7 @@ def stream_pipeline(pipeline_id: str) -> Response:
     _escalate_blocked_slice_to_hitl,
     _escalate_corrupt_slice_to_hitl,
     _escalate_evidence_gate_to_hitl,
+    _escalate_green_gate_to_hitl,
     _escalate_layer_c_hitl,
     _is_slice_dag_mode,
     _latest_completed_chain_tip,

orchestrator/routes/pipelines/_run_implement.py

@@ -907,38 +907,29 @@ def _run_one_slice_inner(
                     scheduler.record_failure(slice_id)
                     return 1, evidence_failure
 
-                # #3398 — per-slice green gate: execute the repo's
-                # configured checks (repositories.yaml, via
-                # get_repo_checks) against the integration-branch tip
-                # in a sandboxed one-shot runner and, in "on" mode,
-                # refuse to open the slice PR while any check is red.
-                # Closes the trust-vs-verify gap in the propose-time
-                # checks_passed self-report. Same posture as the
-                # evidence gate above: fail-open on infra errors,
-                # fail-closed only on a definitive red verdict;
-                # EGG_SLICE_GREEN_GATE is the operator switch
-                # (off / log / on), defaulting to "on": the checks
-                # run on every slice close and a definitive red
-                # withholds the PR. "log" runs them without
-                # blocking; "off" is the escape hatch, and is
-                # quoted in the failure message itself.
-                if pipeline.repo:
-                    try:
-                        import slice_green_gate as _green_gate
-                    except ImportError:
-                        from .. import slice_green_gate as _green_gate  # type: ignore[no-redef]
-
-                    green_gate_failure = _green_gate.run_slice_green_gate(
-                        pipeline_id,
-                        spawner,
-                        slice_id,
-                        integration_branch,
-                        pipeline.repo,
-                        gateway_mode=gateway_mode,  # type: ignore[arg-type]
-                    )
-                    if green_gate_failure is not None:
-                        scheduler.record_failure(slice_id)
-                        return 1, green_gate_failure
+                # #3398 — per-slice green gate: run the repo's
+                # configured checks against the integration-branch tip
+                # and, in "on" mode (the default), refuse to open the
+                # slice PR while any check is red. Same posture as the
+                # evidence gate above on every axis, escalation
+                # included: fail-open on infra errors, fail-closed only
+                # on a definitive red verdict, and on that red the
+                # helper lands an unresolved HITL Decision before
+                # returning so the block is an operator question rather
+                # than a silently parked slice. See
+                # ``_slice_close_green_gate`` for the full posture.
+                green_gate_failure = _pkg._slice_close_green_gate(
+                    pipeline_id,
+                    spawner,
+                    worktree_repo_path,
+                    slice_id,
+                    integration_branch,
+                    gateway_mode=gateway_mode,  # type: ignore[arg-type]
+                    pipeline=pipeline,
+                )
+                if green_gate_failure is not None:
+                    scheduler.record_failure(slice_id)
+                    return 1, green_gate_failure
 
                 # Snapshot the slice's PR data from the same loaded
                 # contract — no second lock acquire, no second file

orchestrator/routes/pipelines/_run_implement_support.py

@@ -489,3 +489,77 @@ def _slice_close_evidence_gate(
             current_phase=getattr(pipeline, "current_phase", None),
         )
     return contract_post, evidence_failure
+
+
+def _slice_close_green_gate(
+    pipeline_id,
+    spawner,
+    worktree_repo_path,
+    slice_id,
+    integration_branch,
+    *,
+    gateway_mode,
+    pipeline,
+) -> str | None:
+    """Run the per-slice green gate (#3398) and escalate a red to HITL.
+
+    Runs after the #3125 evidence gate and before any close side effect:
+    execute the repo's configured checks (``repositories.yaml``, via
+    ``get_repo_checks``) against the integration-branch tip in a
+    sandboxed one-shot runner and, in ``on`` mode, refuse to open the
+    slice PR while any check is red. Closes the trust-vs-verify gap in
+    the propose-time ``checks_passed`` self-report. Only repo-backed
+    pipelines are gated; the failure is ``None`` otherwise.
+
+    Same posture as ``_slice_close_evidence_gate`` above, on every axis
+    that matters: fail-open on infra errors, fail-closed only on a
+    definitive red verdict, and on that definitive red this helper lands
+    an unresolved HITL ``Decision`` on the contract before returning.
+    The caller's ``record_failure`` only arms the descendant cascade;
+    nothing re-drives the close, so without the Decision a
+    consensus-complete slice parks until an operator notices the failed
+    phase. ``EGG_SLICE_GREEN_GATE`` is the operator switch (``off`` /
+    ``log`` / ``on``), defaulting to ``on``.
+
+    The escalation embeds only ``failure_headline(failure)``; see that
+    function and ``_escalate_green_gate_to_hitl`` for why the per-check
+    output tails must stay out of the question text.
+    """
+    if not pipeline.repo:
+        return None
+
+    try:
+        import slice_green_gate as _green_gate
+    except ImportError:
+        # Absolute fallback, not a relative one. ``slice_green_gate``
+        # lives at ``orchestrator/slice_green_gate.py``, so ``from ..``
+        # names ``routes.slice_green_gate``, which exists under neither
+        # layout. ``from ...`` would resolve under the repo-root layout
+        # — the only layout where this fallback fires, since the flat
+        # ``import slice_green_gate`` above succeeds under the deployed
+        # one — but it is a package-depth assertion this module cannot
+        # make: under the deployed layout it reaches beyond top-level,
+        # which its own ``import routes.pipelines`` implies. The
+        # absolute form names the module under the layout that runs it
+        # and stays inert under the one that doesn't. Mirrors the
+        # ``global_slice_admit`` pair in ``_run_implement.py``, the
+        # module this helper was extracted from.
+        from orchestrator import slice_green_gate as _green_gate  # type: ignore[no-redef]
+
+    green_gate_failure = _green_gate.run_slice_green_gate(
+        pipeline_id,
+        spawner,
+        slice_id,
+        integration_branch,
+        pipeline.repo,
+        gateway_mode=gateway_mode,
+    )
+    if green_gate_failure is not None:
+        _pkg._escalate_green_gate_to_hitl(
+            pipeline_id=pipeline_id,
+            slice_id=slice_id,
+            failure_headline=_green_gate.failure_headline(green_gate_failure),
+            worktree_repo_path=worktree_repo_path,
+            current_phase=getattr(pipeline, "current_phase", None),
+        )
+    return green_gate_failure