Replace implicit consensus READY with agent restart

docs/guides/concurrent-execution.md

@@ -23,8 +23,6 @@ Relevant `PipelineConfig` fields:
 | `consensus_timeout_minutes` | `30` | Timeout before HITL escalation |
 | `agent_idle_timeout_minutes` | `60` | Idle agent timeout before termination |
 
-Agent containers also respect `EGG_CONSENSUS_WRAPPER_TIMEOUT` (default: `300` seconds) — see [Consensus Wrapper](#consensus-wrapper) below.
-
 ## Agent Startup Protocol
 
 When concurrent execution starts for the implement phase, the `ConcurrentPhaseExecutor` (in `orchestrator/concurrent_executor.py`) spawns the following roles simultaneously using a `ThreadPoolExecutor`:
@@ -49,28 +47,24 @@ Each agent is registered in the consensus evaluator before spawning begins.
 
 ## Consensus Wrapper
 
-All concurrent agent containers are wrapped with a shell safety net defined in `orchestrator/consensus_wrapper.py`. The wrapper runs after the Claude process exits and enforces lifecycle compliance for agents that exit before the orchestrator stops them.
+All concurrent agent containers are wrapped with a shell script defined in `orchestrator/consensus_wrapper.py`. The wrapper detects when Claude exits without the orchestrator confirming consensus and restarts the agent with a recovery prompt instead of silently marking it as ready.
 
 **How it works:**
 
-1. Claude runs inside the wrapper script.
-2. When Claude exits cleanly (code 0), the wrapper auto-signals `READY` via `egg-orch signal readiness` (a no-op if the agent already signaled).
-3. The wrapper then polls the consensus endpoint in a loop, sleeping `EGG_MESSAGE_POLL_INTERVAL` seconds between checks, until consensus is reached or `EGG_CONSENSUS_WRAPPER_TIMEOUT` expires.
-4. When consensus is reached (`is_complete: true`), the wrapper exits with Claude's original exit code.
-5. If Claude exits non-zero (crashed), the wrapper does **not** signal `READY` and exits immediately with the same code.
-
-**Environment variables:**
-
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `EGG_CONSENSUS_WRAPPER_TIMEOUT` | `300` | Seconds the wrapper polls for consensus before giving up |
-| `EGG_MESSAGE_POLL_INTERVAL` | `30` | Seconds between wrapper poll iterations |
+1. Claude runs inside the wrapper script with the original task prompt.
+2. If Claude exits non-zero (crashed), the wrapper exits immediately with the same code — no restart.
+3. If Claude exits cleanly (code 0), the wrapper restarts Claude with a **recovery prompt** that explains the agent was restarted because it exited without signaling `READY`. The recovery prompt instructs the agent to poll for messages, assess state, and explicitly signal `READY` or continue working.
+4. Restarts are capped at `MAX_CONSENSUS_RESTARTS` (default: 2). After each restart, the wrapper checks if consensus was reached. If so, it exits cleanly.
+5. After exhausting all restarts, the wrapper exits with code 1, triggering the orchestrator's agent failure path (HITL decision with retry/abort/continue options).
 
-**Implicit READY on clean exit (orchestrator-side):**
+**Key design principle:** Agents must **explicitly** participate in consensus. The wrapper never auto-signals `READY` on behalf of an agent — it restarts the agent so it can assess state and signal for itself.
 
-In addition to the wrapper, the orchestrator's `_run_concurrent_phase()` monitors container exit codes. When a container exits with code 0 and has not yet signaled `READY`, the orchestrator auto-registers it as `READY` with the reason `"Container exited cleanly (implicit READY)"`. This ensures agents that complete their work and exit without explicitly signaling do not block consensus indefinitely.
+**Configuration:**
 
-**Agents should still follow the protocol.** The wrapper and implicit READY are safety nets for cases where Claude's exit is unavoidable (e.g., max turns reached, context exhausted). Well-behaved agents explicitly signal `READY` and enter a polling loop — this allows them to react to late-arriving messages before the orchestrator stops the container.
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `max_restarts` | `2` | Maximum restart attempts (passed to `build_consensus_wrapped_command()`) |
+| `EGG_MESSAGE_POLL_INTERVAL` | `30` | Seconds between message polls during restarts |
 
 ## Message Bus
 

orchestrator/consensus_wrapper.py

@@ -2,96 +2,191 @@
 
 When agents run in concurrent mode, they must stay alive after completing
 their work to participate in consensus. This module provides a shell wrapper
-that catches early Claude exits and keeps the container alive polling for
-consensus, as a safety net for agents that exit without following the
-stay-alive protocol.
+that detects early Claude exits and restarts the agent with a recovery prompt
+instead of blindly marking consensus as approved.
+
+If the agent exits without signaling READY, the wrapper restarts Claude with
+a prompt that explains what happened and instructs it to assess state, then
+either signal READY or continue working. Restarts are capped at
+``MAX_CONSENSUS_RESTARTS`` (default 2). After exhausting restarts the wrapper
+exits with code 1 so the orchestrator's failure path handles escalation.
 """
 
 import shlex
 
-# Shell script that wraps the Claude CLI invocation. After Claude exits,
-# if EGG_CONCURRENT_MODE is set and exit was clean (code 0), it auto-signals
-# READY (if the agent didn't) and polls until consensus is reached or a
-# timeout expires. Non-zero exits are treated as failures — no READY signal.
+# Default maximum number of times the wrapper will restart Claude after a
+# clean exit without consensus being reached.
+MAX_CONSENSUS_RESTARTS = 2
+
+# Default maximum number of poll cycles to wait for consensus when the agent
+# already signaled READY. With a default poll interval of 30s, this gives
+# 10 * 30 = 300 seconds (5 minutes) for other agents to finish.
+MAX_READY_POLL_CYCLES = 10
+
+# Recovery prompt given to Claude when it is restarted by the wrapper.
+# Placeholders: {restart_number}, {max_restarts}
+_RECOVERY_PROMPT = (
+    "## CONSENSUS RECOVERY — You were restarted by the consensus wrapper\n\n"
+    "You exited your previous session without the orchestrator confirming "
+    "consensus. This is restart {restart_number} of {max_restarts}.\n\n"
+    "**What happened**: Your agent process exited cleanly, but the consensus "
+    "protocol requires you to remain alive until ALL agents signal READY and "
+    "the orchestrator stops your container. Because you exited early, the "
+    "wrapper restarted you so you can finish the protocol.\n\n"
+    "**What you must do now**:\n"
+    "1. Poll for messages: `egg-orch message poll`\n"
+    "2. Check if your work is complete — review any new commits or feedback "
+    "from other agents.\n"
+    "3. If your work is done, signal READY: "
+    '`egg-orch signal readiness --state READY --reason "Work complete"`\n'
+    "4. If there is new feedback or work to address, handle it first, then "
+    "signal READY.\n"
+    "5. **Stay alive** — keep polling with `egg-orch message poll` in a loop. "
+    "Do NOT exit. The orchestrator will send SIGTERM when consensus is reached.\n\n"
+    "**If you exit again without signaling READY, you will be restarted again "
+    "(up to the maximum). After that, your role will be left without consensus "
+    "and the orchestrator will need to handle it.**\n"
+)
+
+# Shell script that wraps the Claude CLI invocation. After Claude exits:
+# - Non-concurrent mode: exit normally.
+# - Non-zero exit: treat as failure, no restart.
+# - Clean exit (code 0): restart Claude with a recovery prompt (up to
+#   MAX_RESTARTS times). After max restarts, exit 1 to trigger the
+#   orchestrator's agent failure path (HITL decision).
 _CONSENSUS_WRAPPER_TEMPLATE = r"""
 #!/bin/bash
 set -uo pipefail
 
-# Run the Claude agent
-{claude_command}
+MAX_RESTARTS={max_restarts}
+RESTART_COUNT=0
+
+run_claude() {{
+    local prompt="$1"
+    {claude_command_prefix} "$prompt"
+    return $?
+}}
+
+# --- Initial run ---
+run_claude {initial_prompt}
 CLAUDE_EXIT=$?
 
 # If not in concurrent mode, exit normally
 if [ "${{EGG_CONCURRENT_MODE:-}}" != "true" ]; then
     exit $CLAUDE_EXIT
 fi
 
-# Only signal READY on clean exit. A non-zero exit means the agent crashed
-# or errored — its work may be incomplete, so we must NOT claim readiness.
+# Non-zero exit means the agent crashed — do not restart or signal READY.
 if [ "$CLAUDE_EXIT" -ne 0 ]; then
-    echo "[consensus-wrapper] Agent failed (code $CLAUDE_EXIT). NOT signaling READY."
+    echo "[consensus-wrapper] Agent failed (code $CLAUDE_EXIT). NOT restarting."
     exit $CLAUDE_EXIT
 fi
 
-# Clean exit — auto-signal READY as a safety net.
-# If the agent already signaled READY, this is a no-op update.
-echo "[consensus-wrapper] Agent exited cleanly. Auto-signaling READY..."
-egg-orch signal readiness --state READY \
-    --reason "Agent process exited cleanly, auto-signaling READY" \
-    2>/dev/null || true
+# --- Check if consensus is already complete or agent already signaled READY ---
+# If the agent signaled READY but then exited (e.g., context exhaustion),
+# restarting is unnecessary. Query pipeline status for consensus state.
+MAX_READY_POLLS={max_ready_polls}
+RESPONSE=$(egg-orch pipeline status --json 2>/dev/null || echo "{{}}")
+IS_COMPLETE=$(echo "$RESPONSE" | python3 -c \
+    "import sys,json; d=json.load(sys.stdin); print(d.get('data',{{}}).get('concurrent',{{}}).get('consensus',{{}}).get('is_complete',False))" \
+    2>/dev/null || echo "False")
+if [ "$IS_COMPLETE" = "True" ]; then
+    echo "[consensus-wrapper] Consensus already reached. Exiting."
+    exit 0
+fi
 
-# Stay alive polling for consensus until reached or timeout.
-POLL_INTERVAL="${{EGG_MESSAGE_POLL_INTERVAL:-30}}"
-TIMEOUT="${{EGG_CONSENSUS_WRAPPER_TIMEOUT:-300}}"
-ELAPSED=0
+# Check if this agent already signaled READY (uses EGG_AGENT_ROLE env var)
+AGENT_ROLE="${{EGG_AGENT_ROLE:-}}"
+if [ -n "$AGENT_ROLE" ]; then
+    AGENT_STATE=$(echo "$RESPONSE" | python3 -c \
+        "import sys,json; d=json.load(sys.stdin); agents=d.get('data',{{}}).get('concurrent',{{}}).get('consensus',{{}}).get('agents',{{}}); print(agents.get('$AGENT_ROLE',{{}}).get('state',''))" \
+        2>/dev/null || echo "")
+    if [ "$AGENT_STATE" = "READY" ]; then
+        echo "[consensus-wrapper] Agent already signaled READY. Skipping restart, waiting for consensus..."
+        POLL_INTERVAL="${{EGG_MESSAGE_POLL_INTERVAL:-30}}"
+        WAIT_COUNT=0
+        while [ "$WAIT_COUNT" -lt "$MAX_READY_POLLS" ]; do
+            WAIT_COUNT=$((WAIT_COUNT + 1))
+            sleep "$POLL_INTERVAL"
+            RESPONSE=$(egg-orch pipeline status --json 2>/dev/null || echo "{{}}")
+            IS_COMPLETE=$(echo "$RESPONSE" | python3 -c \
+                "import sys,json; d=json.load(sys.stdin); print(d.get('data',{{}}).get('concurrent',{{}}).get('consensus',{{}}).get('is_complete',False))" \
+                2>/dev/null || echo "False")
+            if [ "$IS_COMPLETE" = "True" ]; then
+                echo "[consensus-wrapper] Consensus reached. Exiting."
+                exit 0
+            fi
+        done
+        echo "[consensus-wrapper] Agent was READY but consensus not reached. Exiting cleanly."
+        exit 0
+    fi
+fi
 
-echo "[consensus-wrapper] Entering consensus wait loop (timeout=${{TIMEOUT}}s)..."
-while [ "$ELAPSED" -lt "$TIMEOUT" ]; do
-    # Check consensus via readiness signal response
-    RESPONSE=$(egg-orch signal readiness --state READY \
-        --reason "Waiting for consensus" --json 2>/dev/null || echo "{{}}")
+# --- Restart loop for clean exits without consensus ---
+while [ "$RESTART_COUNT" -lt "$MAX_RESTARTS" ]; do
+    RESTART_COUNT=$((RESTART_COUNT + 1))
+    echo "[consensus-wrapper] Agent exited without consensus. Restarting ($RESTART_COUNT/$MAX_RESTARTS)..."
 
-    IS_COMPLETE=$(echo "$RESPONSE" | python3 -c \
-        "import sys,json; d=json.load(sys.stdin); print(d.get('data',{{}}).get('consensus',{{}}).get('is_complete',False))" \
-        2>/dev/null || echo "False")
+    # Build recovery prompt with restart context
+    RECOVERY_PROMPT=$(cat <<'RECOVERY_EOF'
+{recovery_prompt_template}
+RECOVERY_EOF
+)
+    # Substitute restart number into the prompt
+    RECOVERY_PROMPT=$(echo "$RECOVERY_PROMPT" | sed "s/{{restart_number}}/$RESTART_COUNT/g; s/{{max_restarts}}/$MAX_RESTARTS/g")
 
-    if [ "$IS_COMPLETE" = "True" ]; then
-        echo "[consensus-wrapper] Consensus reached. Exiting."
+    run_claude "$RECOVERY_PROMPT"
+    CLAUDE_EXIT=$?
+
+    if [ "$CLAUDE_EXIT" -ne 0 ]; then
+        echo "[consensus-wrapper] Agent failed on restart $RESTART_COUNT (code $CLAUDE_EXIT). Stopping."
         exit $CLAUDE_EXIT
     fi
 
-    # Poll for messages (may contain work that should have been handled)
-    egg-orch message poll 2>/dev/null || true
+    # Check if consensus was reached during the restart
+    RESPONSE=$(egg-orch pipeline status --json 2>/dev/null || echo "{{}}")
+    IS_COMPLETE=$(echo "$RESPONSE" | python3 -c \
+        "import sys,json; d=json.load(sys.stdin); print(d.get('data',{{}}).get('concurrent',{{}}).get('consensus',{{}}).get('is_complete',False))" \
+        2>/dev/null || echo "False")
 
-    sleep "$POLL_INTERVAL"
-    ELAPSED=$((ELAPSED + POLL_INTERVAL))
+    if [ "$IS_COMPLETE" = "True" ]; then
+        echo "[consensus-wrapper] Consensus reached after restart $RESTART_COUNT. Exiting."
+        exit 0
+    fi
 done
 
-echo "[consensus-wrapper] Consensus not reached within ${{TIMEOUT}}s. Exiting."
-exit $CLAUDE_EXIT
+# --- Max restarts exhausted: shut down with failure ---
+echo "[consensus-wrapper] Max restarts ($MAX_RESTARTS) exhausted. Agent never signaled READY. Exiting with failure."
+exit 1
 """
 
 
 def build_consensus_wrapped_command(
     prompt_text: str,
     model: str = "opus",
     max_turns: int = 200,
+    max_restarts: int = MAX_CONSENSUS_RESTARTS,
+    max_ready_polls: int = MAX_READY_POLL_CYCLES,
 ) -> list[str]:
-    """Build a shell command that runs Claude with a consensus wait wrapper.
+    """Build a shell command that runs Claude with a consensus restart wrapper.
 
-    The wrapper ensures that after Claude exits, the container stays alive
-    polling for consensus rather than disappearing and triggering the
-    orchestrator's fallback path.
+    The wrapper detects when Claude exits without consensus and restarts it
+    with a recovery prompt instead of auto-signaling READY. This ensures
+    agents explicitly participate in consensus rather than having it faked.
 
     Args:
         prompt_text: The prompt to pass to the Claude CLI.
         model: Claude model to use.
         max_turns: Maximum number of tool-call turns.
+        max_restarts: Maximum restart attempts before exiting with failure.
+        max_ready_polls: Maximum poll cycles to wait when agent already
+            signaled READY (avoids unnecessary restarts).
 
     Returns:
         Command list suitable for container spawning (bash -c "...").
     """
-    claude_parts = [
+    # Build the claude command prefix (everything except the prompt argument)
+    claude_prefix_parts = [
         "claude",
         "--dangerously-skip-permissions",
         "--print",
@@ -102,9 +197,16 @@ def build_consensus_wrapped_command(
         model,
         "--max-turns",
         str(max_turns),
-        prompt_text,
     ]
-    claude_command = " ".join(shlex.quote(p) for p in claude_parts)
-    script = _CONSENSUS_WRAPPER_TEMPLATE.format(claude_command=claude_command)
+    claude_command_prefix = " ".join(shlex.quote(p) for p in claude_prefix_parts)
+    initial_prompt = shlex.quote(prompt_text)
+
+    script = _CONSENSUS_WRAPPER_TEMPLATE.format(
+        claude_command_prefix=claude_command_prefix,
+        initial_prompt=initial_prompt,
+        max_restarts=max_restarts,
+        max_ready_polls=max_ready_polls,
+        recovery_prompt_template=_RECOVERY_PROMPT,
+    )
 
     return ["bash", "-c", script]

orchestrator/routes/coordinator.py

@@ -303,8 +303,8 @@ def spawn_agent(pipeline_id: str) -> tuple[Response, int]:
             )
 
             # Wrap the Claude invocation in a consensus shell wrapper that
-            # keeps the container alive polling for consensus if Claude exits
-            # before the orchestrator stops the container.
+            # restarts the agent with a recovery prompt if Claude exits
+            # before the orchestrator confirms consensus.
             agent_command = build_consensus_wrapped_command(agent_prompt)
 
             # Spawn the container

orchestrator/routes/pipelines.py

@@ -703,6 +703,12 @@ def update_pipeline(pipeline_id: str) -> tuple[Response, int]:
                     pipeline_id=pipeline_id,
                     error=str(e),
                 )
+                # Reload pipeline so the response reflects current state
+                # rather than the stale pre-cleanup object.
+                try:
+                    pipeline = store.load_pipeline(pipeline_id)
+                except Exception:
+                    pass  # Use stale pipeline if reload also fails
 
         logger.info("Pipeline updated", pipeline_id=pipeline_id)
 
@@ -4779,33 +4785,15 @@ def _update_agents_complete() -> None:
                             error=str(e),
                         )
                 else:
-                    # Clean exit (code 0): treat as implicit READY if the
-                    # agent didn't explicitly signal.  This prevents early
-                    # exits from blocking consensus indefinitely.
-                    try:
-                        from consensus import ReadinessState, get_consensus_evaluator
-
-                        evaluator = get_consensus_evaluator()
-                        current = evaluator.evaluate(pipeline_id)
-                        agent_state = current.get("agents", {}).get(exec_info.role.value)
-                        if agent_state is None or agent_state.state != ReadinessState.READY:
-                            evaluator.update_readiness(
-                                pipeline_id,
-                                exec_info.role.value,
-                                ReadinessState.READY,
-                                reason="Container exited cleanly (implicit READY)",
-                            )
-                            logger.info(
-                                "Auto-registered READY for cleanly exited container",
-                                pipeline_id=pipeline_id,
-                                role=exec_info.role.value,
-                            )
-                    except Exception as e:
-                        logger.warning(
-                            "Failed to auto-register READY for exited container",
-                            role=exec_info.role.value,
-                            error=str(e),
-                        )
+                    # Clean exit (code 0): the consensus wrapper inside the
+                    # container handles restarts if the agent didn't signal
+                    # READY. We do NOT auto-register READY here — agents
+                    # must explicitly participate in consensus.
+                    logger.info(
+                        "Container exited cleanly, wrapper handles consensus",
+                        pipeline_id=pipeline_id,
+                        role=exec_info.role.value,
+                    )
 
         # 5. All containers exited — fall back to exit-code-based result
         if len(exited_containers) >= len(active_executions):