Escalate a green-gate red to HITL, mirroring the evidence gate

docs/architecture/slice-dag.md

@@ -555,7 +555,11 @@ shape:
        rollout via `EGG_SLICE_GREEN_GATE` — default `on` (a red verdict
        withholds the slice PR), `log` runs the checks and logs the
        verdict without blocking; fail-open on infra errors, including
-       infra-signature-tagged reds inside check execution, #3417.
+       infra-signature-tagged reds inside check execution, #3417; a red
+       verdict that survives to block lands an unresolved HITL
+       `Decision` via `_escalate_green_gate_to_hitl`, #3572 parity with
+       the evidence gate above, so the block is an operator question
+       rather than a silently parked slice.
        **The gate can also write to the integration branch**: when every
        genuine red carries an optional `fix:` command in
        `repositories.yaml` (e.g. `lint: {fix: make lint-fix}`), the
@@ -568,7 +572,7 @@ shape:
        *every* configured check against the all-fixes-applied tree
        (`final_verification.all_ok`) plus a no-new-untracked-files
        check. Any failure to commit or push blocks the slice exactly
-       like an unfixed red) — calls
+       like an unfixed red, HITL escalation included) — calls
        `GatewayClient.create_slice_pr` with `base` resolved from the
        slice's DAG parent (root → latest completed chain tip, else the
        pipeline branch (#3541); child → parent's
@@ -1072,7 +1076,7 @@ on parse failure. The green-gate knobs below are read directly via
 | `EGG_ORCH_STACKED_PR_RECONCILER_INTERVAL_SECONDS` | float | 30.0 | Reconciler polling cadence for orphaned child PRs. |
 | `EGG_ORCH_CROSS_REPO_MERGE_GATE_MAX_ATTEMPTS` | int | 240 | Poll-attempt budget for the cross-repo merge-sequencing gate (#3393) before a never-merging upstream escalates to a HITL hold; ~2h at the default reconciler cadence. See [Cross-repo merge-sequencing hold](#cross-repo-merge-sequencing-hold-two-tier). |
 | `EGG_SLICE_BASE_ANCESTRY_GATE` | str | `on` | Operator kill switch for the admission-time base-ancestry gate (#3541 — see [Root linearization & the base-ancestry gate](#root-linearization--the-base-ancestry-gate-3541)): any of `off`/`0`/`false`/`no` (case-insensitive, whitespace-tolerant) disables the gate; any other value (including unset) leaves it enabled. |
-| `EGG_SLICE_GREEN_GATE` | str | `on` | Per-slice green gate rollout switch (#3398): `off` skips the gate entirely; `log` runs the repo's configured checks at the slice tip and logs a red verdict without blocking; `on` (the default) withholds the slice PR on a red verdict. Case-insensitive, with aliases — `on` also accepts `1`/`true`/`yes`, `log` also accepts `log-only`/`log_only`, and `off` also accepts `0`/`false`/`no`. Weakening the gate takes an explicit, correctly-spelled `off` or `log`; unset or unrecognised values resolve to `on`, so a typo cannot silently drop a deployment below the product default. Expect the first reds to be gate wiring rather than slice code — a stale contract snapshot reddening contract-hygiene tests (#3301), or `make test`'s changeset narrowing resolving its baseline against `git merge-base` in a fresh worktree — and note either of those reds *every* slice close until fixed (a missing prebuilt-deps snapshot is not in this list: the runner exits non-zero and the gate fails open, costing coverage rather than throughput); the failure message names the branch to fix and quotes `EGG_SLICE_GREEN_GATE=off` as the bypass, and the slice's commits stay on the integration branch (a red gate withholds the PR, it does not discard work). Latency is identical under `log` and `on` — both run the checks and wait for the runner pod, so slice-close latency grows by the check duration (bounded by `EGG_SLICE_GREEN_GATE_TIMEOUT_SECONDS`). The worst case is not a slow suite but a runner pod that never schedules — the wait is `timeout` plus a 120s scheduling grace (~32 min at the defaults) before failing open, so a capacity-starved cluster pays that per slice close. A *partially* delayed pod is quieter and more common: the runner's deadline is the **Job's** `activeDeadlineSeconds`, counted from the Job's `startTime` (before any pod is bound), so time spent Pending or pulling is subtracted from the in-pod check budget rather than added to the wait — the scheduling grace widens only the orchestrator's wait. A pod delayed N seconds gets N fewer seconds to run checks, and a `DeadlineExceeded` kill emits no verdict line, so the gate fails open with no verdict at all. Capacity starvation therefore raises the rate of spurious no-verdict fail-opens as well as dead time ([#3622](https://github.com/jwbron/egg/issues/3622)) — under the `on` default that is a slice close you believed was gated and wasn't, though the direction is always *under*-blocking, never a false red. Grep for *both* fail-open log lines when diagnosing one: the Job controller deletes the active pod on `DeadlineExceeded` rather than leaving it terminal, so the wait normally times out ("runner pod did not reach a terminal state"); a poll that catches the pod reporting `Failed` mid-termination reads partial output and lands on "no parseable verdict from runner" instead. |
+| `EGG_SLICE_GREEN_GATE` | str | `on` | Per-slice green gate rollout switch (#3398): `off` skips the gate entirely; `log` runs the repo's configured checks at the slice tip and logs a red verdict without blocking; `on` (the default) withholds the slice PR on a red verdict. Case-insensitive, with aliases — `on` also accepts `1`/`true`/`yes`, `log` also accepts `log-only`/`log_only`, and `off` also accepts `0`/`false`/`no`. Weakening the gate takes an explicit, correctly-spelled `off` or `log`; unset or unrecognised values resolve to `on`, so a typo cannot silently drop a deployment below the product default. Expect the first reds to be gate wiring rather than slice code — a stale contract snapshot reddening contract-hygiene tests (#3301), or `make test`'s changeset narrowing resolving its baseline against `git merge-base` in a fresh worktree — and note either of those reds *every* slice close until fixed (a missing prebuilt-deps snapshot is not in this list: the runner exits non-zero and the gate fails open, costing coverage rather than throughput); recovery is operator-driven rather than automatic: a red verdict lands an unresolved HITL `Decision` on the contract (`_escalate_green_gate_to_hitl`, #3572 parity) offering continue / restart-slice / cancel, so the block surfaces in `/sdlc` instead of only as a FAILED phase an operator has to notice; the failure message names the branch to fix and quotes `EGG_SLICE_GREEN_GATE=off` as the bypass, and the slice's commits stay on the integration branch (a red gate withholds the PR, it does not discard work). Latency is identical under `log` and `on` — both run the checks and wait for the runner pod, so slice-close latency grows by the check duration (bounded by `EGG_SLICE_GREEN_GATE_TIMEOUT_SECONDS`). The worst case is not a slow suite but a runner pod that never schedules — the wait is `timeout` plus a 120s scheduling grace (~32 min at the defaults) before failing open, so a capacity-starved cluster pays that per slice close. A *partially* delayed pod is quieter and more common: the runner's deadline is the **Job's** `activeDeadlineSeconds`, counted from the Job's `startTime` (before any pod is bound), so time spent Pending or pulling is subtracted from the in-pod check budget rather than added to the wait — the scheduling grace widens only the orchestrator's wait. A pod delayed N seconds gets N fewer seconds to run checks, and a `DeadlineExceeded` kill emits no verdict line, so the gate fails open with no verdict at all. Capacity starvation therefore raises the rate of spurious no-verdict fail-opens as well as dead time ([#3622](https://github.com/jwbron/egg/issues/3622)) — under the `on` default that is a slice close you believed was gated and wasn't, though the direction is always *under*-blocking, never a false red. Grep for *both* fail-open log lines when diagnosing one: the Job controller deletes the active pod on `DeadlineExceeded` rather than leaving it terminal, so the wait normally times out ("runner pod did not reach a terminal state"); a poll that catches the pod reporting `Failed` mid-termination reads partial output and lands on "no parseable verdict from runner" instead. |
 | `EGG_SLICE_GREEN_GATE_SKIP_CHECKS` | str (comma-separated) | `security` | Configured check *names* (from `repositories.yaml` `checks`) the gate skips. |
 | `EGG_SLICE_GREEN_GATE_TIMEOUT_SECONDS` | int | 1800 | Wall-clock budget for the check-runner pod (spawn-to-terminal); a hung suite degrades to fail-open rather than wedging the slice close. Enforced as a Job-level `activeDeadlineSeconds` (`timeout + 60`) counted from Job start, so it is a ceiling on scheduling *plus* checks, not on checks alone — see the `EGG_SLICE_GREEN_GATE` row and [#3622](https://github.com/jwbron/egg/issues/3622). |
 | `EGG_SLICE_GREEN_GATE_INFRA_FAIL_OPEN` | str | `on` | Infra-red fail-open (#3417): the runner tags red checks whose full output matches an exact infra signature (the sandbox git wrapper's gateway-down / missing-env / session-auth errors, the kernel's ENOSPC message) or whose process died by SIGKILL; a verdict where *every* red check is infra-tagged fails open instead of blocking, and mixed verdicts block on the genuine reds only. `off`/`0`/`false`/`no` restores strict every-red-blocks behavior; `on`/`1`/`true`/`yes` (and unset, and an empty or whitespace-only value) enable it silently; any other value resolves to `on` and logs a warning, since the typo direction here is strict → lenient. |

orchestrator/routes/pipelines/__init__.py

@@ -1405,6 +1405,7 @@ def stream_pipeline(pipeline_id: str) -> Response:
     _parent_branch_probe_impl,
     _persist_slice_status_complete_impl,
     _slice_close_evidence_gate,
+    _slice_close_green_gate,
 )
 from ._run_phase import (  # noqa: E402,F401
     _run_phase_execution,
@@ -1454,6 +1455,7 @@ def stream_pipeline(pipeline_id: str) -> Response:
     _escalate_blocked_slice_to_hitl,
     _escalate_corrupt_slice_to_hitl,
     _escalate_evidence_gate_to_hitl,
+    _escalate_green_gate_to_hitl,
     _escalate_layer_c_hitl,
     _is_slice_dag_mode,
     _latest_completed_chain_tip,

orchestrator/routes/pipelines/_run_implement.py

@@ -907,38 +907,29 @@ def _run_one_slice_inner(
                     scheduler.record_failure(slice_id)
                     return 1, evidence_failure
 
-                # #3398 — per-slice green gate: execute the repo's
-                # configured checks (repositories.yaml, via
-                # get_repo_checks) against the integration-branch tip
-                # in a sandboxed one-shot runner and, in "on" mode,
-                # refuse to open the slice PR while any check is red.
-                # Closes the trust-vs-verify gap in the propose-time
-                # checks_passed self-report. Same posture as the
-                # evidence gate above: fail-open on infra errors,
-                # fail-closed only on a definitive red verdict;
-                # EGG_SLICE_GREEN_GATE is the operator switch
-                # (off / log / on), defaulting to "on": the checks
-                # run on every slice close and a definitive red
-                # withholds the PR. "log" runs them without
-                # blocking; "off" is the escape hatch, and is
-                # quoted in the failure message itself.
-                if pipeline.repo:
-                    try:
-                        import slice_green_gate as _green_gate
-                    except ImportError:
-                        from .. import slice_green_gate as _green_gate  # type: ignore[no-redef]
-
-                    green_gate_failure = _green_gate.run_slice_green_gate(
-                        pipeline_id,
-                        spawner,
-                        slice_id,
-                        integration_branch,
-                        pipeline.repo,
-                        gateway_mode=gateway_mode,  # type: ignore[arg-type]
-                    )
-                    if green_gate_failure is not None:
-                        scheduler.record_failure(slice_id)
-                        return 1, green_gate_failure
+                # #3398 — per-slice green gate: run the repo's
+                # configured checks against the integration-branch tip
+                # and, in "on" mode (the default), refuse to open the
+                # slice PR while any check is red. Same posture as the
+                # evidence gate above on every axis, escalation
+                # included: fail-open on infra errors, fail-closed only
+                # on a definitive red verdict, and on that red the
+                # helper lands an unresolved HITL Decision before
+                # returning so the block is an operator question rather
+                # than a silently parked slice. See
+                # ``_slice_close_green_gate`` for the full posture.
+                green_gate_failure = _pkg._slice_close_green_gate(
+                    pipeline_id,
+                    spawner,
+                    worktree_repo_path,
+                    slice_id,
+                    integration_branch,
+                    gateway_mode=gateway_mode,  # type: ignore[arg-type]
+                    pipeline=pipeline,
+                )
+                if green_gate_failure is not None:
+                    scheduler.record_failure(slice_id)
+                    return 1, green_gate_failure
 
                 # Snapshot the slice's PR data from the same loaded
                 # contract — no second lock acquire, no second file

orchestrator/routes/pipelines/_run_implement_support.py

@@ -489,3 +489,64 @@ def _slice_close_evidence_gate(
             current_phase=getattr(pipeline, "current_phase", None),
         )
     return contract_post, evidence_failure
+
+
+def _slice_close_green_gate(
+    pipeline_id,
+    spawner,
+    worktree_repo_path,
+    slice_id,
+    integration_branch,
+    *,
+    gateway_mode,
+    pipeline,
+) -> str | None:
+    """Run the per-slice green gate (#3398) and escalate a red to HITL.
+
+    Runs after the #3125 evidence gate and before any close side effect:
+    execute the repo's configured checks (``repositories.yaml``, via
+    ``get_repo_checks``) against the integration-branch tip in a
+    sandboxed one-shot runner and, in ``on`` mode, refuse to open the
+    slice PR while any check is red. Closes the trust-vs-verify gap in
+    the propose-time ``checks_passed`` self-report. Only repo-backed
+    pipelines are gated; the failure is ``None`` otherwise.
+
+    Same posture as ``_slice_close_evidence_gate`` above, on every axis
+    that matters: fail-open on infra errors, fail-closed only on a
+    definitive red verdict, and on that definitive red this helper lands
+    an unresolved HITL ``Decision`` on the contract before returning.
+    The caller's ``record_failure`` only arms the descendant cascade;
+    nothing re-drives the close, so without the Decision a
+    consensus-complete slice parks until an operator notices the failed
+    phase. ``EGG_SLICE_GREEN_GATE`` is the operator switch (``off`` /
+    ``log`` / ``on``), defaulting to ``on``.
+
+    The escalation embeds only ``failure_headline(failure)``; see that
+    function and ``_escalate_green_gate_to_hitl`` for why the per-check
+    output tails must stay out of the question text.
+    """
+    if not pipeline.repo:
+        return None
+
+    try:
+        import slice_green_gate as _green_gate
+    except ImportError:
+        from ... import slice_green_gate as _green_gate  # type: ignore[no-redef]
+
+    green_gate_failure = _green_gate.run_slice_green_gate(
+        pipeline_id,
+        spawner,
+        slice_id,
+        integration_branch,
+        pipeline.repo,
+        gateway_mode=gateway_mode,
+    )
+    if green_gate_failure is not None:
+        _pkg._escalate_green_gate_to_hitl(
+            pipeline_id=pipeline_id,
+            slice_id=slice_id,
+            failure_headline=_green_gate.failure_headline(green_gate_failure),
+            worktree_repo_path=worktree_repo_path,
+            current_phase=getattr(pipeline, "current_phase", None),
+        )
+    return green_gate_failure

orchestrator/routes/pipelines/_slice_state.py

@@ -1004,6 +1004,62 @@ def _escalate_evidence_gate_to_hitl(
     )
 
 
+def _escalate_green_gate_to_hitl(
+    *,
+    pipeline_id: str,
+    slice_id: str,
+    failure_headline: str,
+    worktree_repo_path: _pkg.Path,
+    current_phase: _pkg.PipelinePhase | None,
+) -> None:
+    """Escalate a green-gate slice-close failure to HITL (#3398).
+
+    Exact parity with ``_escalate_evidence_gate_to_hitl`` above, one gate
+    later. The green gate (#3398) began blocking when its default flipped
+    to ``on``; before this, a red verdict on a consensus-complete slice
+    reproduced the pre-#3572 shape the evidence gate had already left
+    behind. ``record_failure`` arms the descendant cascade and the phase
+    goes FAILED, but nothing lands on ``contract.decisions``, so the
+    block is not resolvable through ``/sdlc`` / ``provide_input`` and
+    recovery meant an operator noticing a failed phase and re-running the
+    entire confirmed wave via ``restart_phase``. On a gate-wiring red
+    (which by construction recurs on every close) that re-run hits the
+    same red.
+
+    Question text is prefixed with ``[#3398 green-gate]`` so a future
+    dispatch handler in ``routes/decisions.py`` can route on the literal
+    substring, mirroring the Layer-C case-4/case-5 and evidence-gate
+    wrappers above.
+
+    ``failure_headline`` MUST be
+    ``slice_green_gate.failure_headline(failure)``, not the full failure
+    string: the headline is deterministic per incident (slice id +
+    integration branch + red check names) while the blocks after it
+    carry per-check output tails that vary between closes. The
+    ``_escalate_layer_c_hitl`` dedupe/carry-forward guard (#3427) matches
+    on question text, so embedding the tails would mint a fresh ``cq-N``
+    on every close retry and phase restart and re-ask the operator a
+    question they had already answered.
+    """
+    _pkg._escalate_layer_c_hitl(
+        pipeline_id=pipeline_id,
+        slice_id=slice_id,
+        worktree_repo_path=worktree_repo_path,
+        current_phase=current_phase,
+        question=(
+            f"[#3398 green-gate] Slice {slice_id} of pipeline "
+            f"{pipeline_id} reached full consensus, but its close is "
+            f"blocked by the per-slice green gate (#3398): {failure_headline} "
+            f"The slice's commits are on the integration branch; the gate "
+            f"withholds the slice PR rather than discarding work. Fix the "
+            f"named checks at the integration tip and restart the slice, or "
+            f"set EGG_SLICE_GREEN_GATE=off to bypass the gate. The per-check "
+            f"output tails are in the phase failure message and the runner "
+            f"logs. How should the orchestrator proceed?"
+        ),
+    )
+
+
 def _cross_repo_hold_marker(slice_id: str) -> str:
     """Return the stable per-gate discriminator embedded in the hold question."""
     return f"{_pkg._CROSS_REPO_HOLD_MARKER_PREFIX} slice={slice_id}]"