Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 13 additions & 11 deletions docs/architecture/brc-memory.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,11 @@ memory across invocations, so something else has to carry the reviewer's
distilled understanding of the codebase, the change under review, and the
verdicts already issued on each producer. That something is this file.

Slice-1 lands the **writer** in `brc_ack` / `brc_nack` and gates everything
behind `EGG_BRC_MEMORY` so production pipelines stay inert. The **reader**
(`compose_event_prompt` consuming the per-producer `last_reviewed_commit_sha`
to parameterise an adversarial git-log delta) lands in slice-3.
Slice-1 lands the **writer** in `brc_ack` / `brc_nack` gated by `EGG_BRC_MEMORY`.
The **reader** (`compose_event_prompt` consuming the per-producer
`last_reviewed_commit_sha` to parameterise an adversarial git-log delta)
lands in slice-3. Slice-4 flipped the default to `full` so both writer and
reader are active in production.

## File path

Expand Down Expand Up @@ -133,12 +134,13 @@ The writer is gated by `EGG_BRC_MEMORY`, which takes one of three values:

| Mode | Writes | Reads | Notes |
|------|--------|-------|-------|
| `off` | no | no | **Default.** Slice-1 ships inert in production. |
| `write-only` | yes | no | Slice-1 rollout posture: handlers populate the file but no other code path reads it. The read path lands in slice-3. |
| `full` | yes | yes | Slice-3+ end state: handlers populate the file and the event-pump consults it on re-entry. |
| `full` | yes | yes | **Default since slice-4.** Handlers populate the file and the event-pump consults it on re-entry. |
| `write-only` | yes | no | Slice-1/2/3 rollout posture: handlers populate the file but no other code path reads it. |
| `off` | no | no | One-release rollback escape hatch; writes are no-ops. |

The default of `off` ensures slice-1 produces no observable behavior change
in production. Slice-4 is responsible for flipping the rollout dial.
Slice-4 flipped the unset-env default from `off` to `full`. Operators that
need to roll back the reader for one release can set `EGG_BRC_MEMORY=write-only`
explicitly.

## Atomic-write contract

Expand Down Expand Up @@ -248,8 +250,8 @@ The acceptance set codified by slice-1 task-1-6:
a partial state (asserted via fault injection in slice-1 tests).
- Path constructor raises on empty `EGG_AGENT_ROLE` **before** creating any
file or directory.
- `EGG_BRC_MEMORY=off` produces no file (the default behavior in
production).
- `EGG_BRC_MEMORY=off` produces no file (the one-release rollback escape
hatch; `full` is the production default since slice-4).
- The `.egg-state/agent-outputs/<role>/` subdirectory is created if absent
on first write.
- Handler return values are unchanged for callers in every case.
Expand Down
2 changes: 1 addition & 1 deletion docs/development/STRUCTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ orchestrator/
├── action_guards.py # Formal BRC state machine action guards (preconditions for propose/ack/nack/confirm/withdraw)
├── approval_matrix.py # Per-reviewer ACK/NACK matrix for BRC consensus
├── attestation_schemas.py # Attestation payload validation for BRC proposals
├── consensus_wrapper.py # Shell wrapper templates: legacy capped-restart path + event-pump path (EGG_BRC_EVENT_PUMP, default off) that replaces model-driven re-entry with a deterministic wrapper loop
├── consensus_wrapper.py # Shell wrapper template: deterministic event-pump loop (sole path since slice-4; legacy capped-restart template and EGG_BRC_EVENT_PUMP flag deleted)
├── dag_visualizer.py # ASCII DAG visualization for pipeline status
├── decision_queue.py # HITL decision queue
├── events.py # Event bus for pipeline events
Expand Down
9 changes: 4 additions & 5 deletions docs/guides/concurrent-execution.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,8 +81,8 @@ All concurrent agent containers are wrapped with a shell script defined in `orch
2. The wrapper polls `egg-orch brc get-state --json`. If `role_complete` is true, it calls `egg-orch consensus confirmed` and exits with code 0.
3. Otherwise it polls `egg-orch brc next-action --json`. The action is one of `WAIT` (block in `egg-orch message wait-loop` with a conditional filter — see [Wait-filter construction](../architecture/orchestrator.md#wait-filter-construction-pre-confirm-vs-post-confirm)) or `INVOKE` (spawn the agent with a one-shot event prompt).
4. When the agent exits cleanly after an `INVOKE`, the wrapper loops back to step 2 — no restart, no recovery prompt. Clean exit between events is the expected steady state.
5. If the agent exits with a transient-crash signal (134/SIGABRT, 136/SIGFPE, 137/SIGKILL/OOM, 139/SIGSEGV, 255/Bun segfault), the wrapper restarts the pod with exponential backoff (initial 5 s, doubling, capped at 30 s). These signals are infrastructure-level pod failures that the wrapper still distinguishes from clean event-pump exits via the surviving `is_transient_crash` / `is_buffer_overflow` / `is_startup_failure` classifiers.
6. If the agent exits with a non-transient non-zero code, the wrapper exits with the same code, triggering the orchestrator's agent failure path. The pre-#2908 `MAX_CONSENSUS_RESTARTS = 3` cap and the `_RECOVERY_SYSTEM_PROMPT` recovery-restart cycle no longer exist; the surviving liveness guarantee is the **idle / no-progress safety budget** (`EGG_BRC_IDLE_BUDGET_MIN`, default 30 minutes) which emits an `OVERSEER_ALERT` at threshold and `2 ×` threshold but never transitions the pipeline to FAILED.
5. If the agent exits non-zero (transient signal-class crash, exit 1, or anything else), the wrapper increments `AGENT_FAIL_STREAK`, sleeps 1 s, and resumes the loop — there is no per-crash restart, no exponential backoff, and no exit-code branching on the `propose|ack|nack` arm. The `is_transient_crash` / `is_buffer_overflow` / `is_startup_failure` helpers are still defined in the wrapper bash template but are **not invoked** today; they are retained as named placeholders against a possible future classifier-gated fast-fail path (see `orchestrator/consensus_wrapper.py` L145–150).
6. The pre-#2908 `MAX_CONSENSUS_RESTARTS = 3` cap and the `_RECOVERY_SYSTEM_PROMPT` recovery-restart cycle no longer exist. The wrapper does not propagate non-zero agent exits — they accrue against the **idle / no-progress safety budget** (`EGG_BRC_IDLE_BUDGET_MIN`, default 30 minutes), which is the sole operator-visible escalation surface. At threshold and `2 ×` threshold the wrapper emits an `OVERSEER_ALERT`; the pipeline is never transitioned to FAILED by this path.

**Key design principle:** Agents must **explicitly** participate in consensus. The wrapper never auto-signals `READY` on behalf of an agent — it dispatches the agent to handle each actionable event and lets the agent issue ACK / NACK / PROPOSE / CONFIRM verdicts itself. Sequencing is the wrapper's job; *judgment* (what to review, what to fix, when to re-propose) stays with the model.

Expand All @@ -94,8 +94,7 @@ All concurrent agent containers are wrapped with a shell script defined in `orch
|-----------|---------|-------------|
| `max_turns` | `1000` | Maximum tool-call turns per agent run (set high so per-event agent invocations have headroom; the wrapper-side loop never reaches this cap because per-event exits are short-lived). |
| `max_ready_polls` | `10` | Maximum poll cycles (each ~30 s) for the legacy "already-confirmed" guard preserved for race-window safety when the wrapper observes `role_complete` between polls. |
| `TRANSIENT_RESTART_BACKOFF_INITIAL` | `5` | Initial backoff delay (seconds) before restarting after a transient infrastructure crash (134/136/137/139/255). Doubles after each crash restart, capped at 30 s. Clean per-event exits do not trigger backoff. |
| `STARTUP_FAILURE_WINDOW_SECONDS` | `30` | Window (seconds) during which exit code 1 is classified as a transient startup failure and retried. Set to `0` to disable. |
| `STARTUP_FAILURE_WINDOW_SECONDS` | `30` | Window (seconds) preserved from the legacy `is_startup_failure` classifier. The helper is still defined but is not invoked by the post-slice-4 `propose|ack|nack` arm, so this knob has no runtime effect today; it is retained against a future classifier-gated fast-fail path. |
| `EGG_BRC_IDLE_BUDGET_MIN` | `30` | Idle / no-progress safety budget in minutes. Replaces the pre-#2908 3-restart FAIL cap; at threshold and `2 ×` threshold the wrapper emits `mcp__progress__overseer_alert` (anomaly `stuck-phase-transition`) without transitioning the pipeline to FAILED. |
| `EGG_MESSAGE_POLL_INTERVAL` | `30` | Seconds between heartbeat emissions and message polls. |

Expand Down Expand Up @@ -879,7 +878,7 @@ The additional API call in step 5 is negligible — it only runs on the terminal

### Transient Crash Recovery

Before an agent failure reaches the orchestrator's `handle_agent_crash()` path, the consensus wrapper attempts to recover from transient runtime crashes (segfaults, OOM kills, SIGABRT). Exit codes 134, 136, 137, 139, and 255 are classified as transient and trigger a restart with exponential backoff (starting at 5 s, doubling up to 30 s). If the transient-crash restart succeeds and the agent re-enters the wrapper's event-pump loop (resuming `egg-orch brc get-state` → `next-action`), the failure is fully recovered at the wrapper level — the orchestrator never sees a failure event. Repeated transient crashes inside the same wrapper run are bounded by the **idle / no-progress safety budget** (`EGG_BRC_IDLE_BUDGET_MIN`, default 30 minutes): the wrapper emits an `OVERSEER_ALERT` at threshold and at `2 ×` threshold but does **not** mark the pipeline FAILED. The pre-#2908 `MAX_CONSENSUS_RESTARTS = 3` hard cap was deleted in slice-4 alongside the legacy template. See [Agent Recovery: Consensus Wrapper](../reference/agent-recovery.md#consensus-wrapper-transient-crash-recovery) for the full exit code classification.
Before an agent failure reaches the orchestrator's `handle_agent_crash()` path, the consensus wrapper absorbs non-zero agent exits at the wrapper level rather than propagating them. In the post-slice-4 event-pump arm there is **no per-crash restart and no exponential backoff** — every non-zero exit from a `propose|ack|nack` agent invocation is handled identically: `AGENT_FAIL_STREAK++`, `sleep 1 s`, resume the loop. Because the agent is one-shot per event, "resume the loop" means the wrapper polls `egg-orch brc next-action` again on the next iteration. If the next actionable event yields a successful invocation, the failure is fully absorbed at the wrapper level — the orchestrator never sees a failure event. Repeated failures inside the same wrapper run are bounded by the **idle / no-progress safety budget** (`EGG_BRC_IDLE_BUDGET_MIN`, default 30 minutes): the wrapper emits an `OVERSEER_ALERT` at threshold and at `2 ×` threshold but does **not** mark the pipeline FAILED. The pre-#2908 `MAX_CONSENSUS_RESTARTS = 3` hard cap and the `TRANSIENT_RESTART_BACKOFF_INITIAL` backoff constant were deleted in slice-4 alongside the legacy template. The `is_transient_crash` / `is_buffer_overflow` / `is_startup_failure` helpers in `consensus_wrapper.py` remain defined but are not invoked by the post-slice-4 arm. See [Agent Recovery: Consensus Wrapper](../reference/agent-recovery.md#crash-handling-in-the-event-pump-wrapper) for the full exit-code table.

### Agent Failure During Consensus

Expand Down
2 changes: 1 addition & 1 deletion docs/guides/sdlc-pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -1388,7 +1388,7 @@ Phase completion in concurrent mode uses a consensus-based approach:
5. Timeout (per-phase: refine 30 / plan 60 / implement 90 by default; configurable via `consensus_timeout_minutes_<phase>` or the legacy global `consensus_timeout_minutes`) publishes a non-blocking `OVERSEER_ALERT` (subject `consensus-timeout: <agent_role> [<priority>]`) rather than gating on a HITL decision — see [issue #2264](https://github.com/jwbron/egg/issues/2264)
- The `/sdlc` skill surfaces the alert (Check agent logs / Acknowledge / Cancel pipeline)
- The orchestrator continues polling for consensus under the post-timeout budget; operators can intervene with `cancel_task`, `restart_phase`, or `provide_input`
6. If a container exits cleanly without signaling `READY`, the consensus wrapper restarts it with a recovery prompt (up to `MAX_CONSENSUS_RESTARTS`, default 3). Each restart publishes a medium-priority `OVERSEER_ALERT` (anomaly `agent-restart`) so the operator sees recovery attempts in real time. After exhausting restarts, the wrapper performs a final consensus check — if consensus has already been reached (`is_complete=True`), it exits with code 0 (success). Only if consensus is genuinely incomplete does it exit with code 1. For **producer** roles this triggers a hard-fail: the orchestrator re-queries consensus once (race-window guard), and if consensus is still incomplete it transitions the pipeline to FAILED and emits a high-priority `OVERSEER_ALERT` (anomaly `producer-permanent-death`); use `restart_phase` to resume or `cancel_task` to abort. For **reviewer** roles the existing single-agent failure path applies (HITL decision: retry, abort, or continue without). See [Concurrent Execution: Consensus Wrapper](concurrent-execution.md#consensus-wrapper).
6. The consensus wrapper drives the agent lifecycle as a deterministic event-pump: it invokes the agent one-shot per actionable BRC event (`propose|ack|nack`) and blocks between events on `egg-orch message wait-loop`. If no actionable event arrives within the idle budget (`EGG_BRC_IDLE_BUDGET_MIN`, default 30 min), the wrapper emits an `OVERSEER_ALERT` (anomaly `stuck-phase-transition`) and keeps blocking — it never exits with code 1 on idle alone. The old capped-restart model (`MAX_CONSENSUS_RESTARTS`, recovery system prompt) was deleted in slice-4 (#2908). See [Concurrent Execution: Consensus Wrapper](concurrent-execution.md#consensus-wrapper).
7. **Consensus gates phase advancement unconditionally.** When all containers have exited — whether with failures or cleanly — the orchestrator performs a final consensus recheck before returning success. If BRC consensus is incomplete, the phase fails (exit code 1) regardless of individual container exit codes. This prevents a PR from being opened when agents exit code 0 without completing the full BRC lifecycle. See [Concurrent Execution: All-Container-Exit Consensus Recovery](concurrent-execution.md#all-container-exit-consensus-recovery).

**Readiness states**:
Expand Down
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ This index helps both humans and LLMs navigate the documentation efficiently.
| [Why egg Works](architecture/collaboration-effectiveness.md) | How the public, sandboxed, async model delivers safety and collaboration |
| [Integration-Test Trust Boundary](architecture/integration-test-trust-boundary.md) | Test execution contexts (in-sandbox-agent / trusted-CI-runner / human-operator) and fixture tiers; authoritative reference for plan-phase Trust-Boundary Audit (#2594) |
| [BRC Memory Artifact](architecture/brc-memory.md) | Per-role-per-pipeline distilled memory file written by `brc_ack`/`brc_nack`; schema, three `EGG_BRC_MEMORY` modes, atomic-write contract, fail-closed path constructor, distill-on-write cap at 20, and role-allowlist coverage ([#2908](https://github.com/jwbron/egg/issues/2908) slice-1) |
| [BRC Event-Pump Wrapper](architecture/orchestrator.md#brc-event-pump-wrapper-slice-2-behind-egg_brc_event_pump) | Deterministic wrapper loop (gated by `EGG_BRC_EVENT_PUMP`, default off until slice-4); wrapper-side heartbeat + gateway-session keep-alive migration; idle/no-progress safety budget replacing the 3-restart FAIL cap. Wait-side companion: [agent-wait-patterns §10](reference/agent-wait-patterns.md#10-brc-event-pump-wrapper-slice-2-behind-egg_brc_event_pump) ([#2908](https://github.com/jwbron/egg/issues/2908) slice-2) |
| [BRC Consensus Wrapper](architecture/orchestrator.md#brc-consensus-wrapper) | Deterministic event-pump wrapper loop (default on since slice-4); wrapper-side heartbeat + gateway-session keep-alive; idle/no-progress safety budget replacing the 3-restart FAIL cap. Wait-side companion: [agent-wait-patterns §10](reference/agent-wait-patterns.md#10-brc-consensus-wrapper-event-pump-model) ([#2908](https://github.com/jwbron/egg/issues/2908) slices 1–4) |
| [BRC Per-Event Prompt Composer + Preamble Collapse](architecture/orchestrator.md#brc-per-event-prompt-composer--preamble-collapse-slice-3) | Per-event prompt assembled by `compose_event_prompt` (10 KB envelope, full git-log delta for adversarial re-review, tail-position memory excerpt per od-6 Option B); `_build_brc_preamble` collapse; `EGG_BRC_MEMORY` mode matrix. Wait-side companion: [agent-wait-patterns §10.9](reference/agent-wait-patterns.md#109-brc-per-event-prompt-composer--preamble-collapse-slice-3) ([#2908](https://github.com/jwbron/egg/issues/2908) slice-3) |

### Development
Expand Down
Loading
Loading