From 3e65387b473345f67d2fdc6820dd9e19b401c80c Mon Sep 17 00:00:00 2001 From: jwbron <8340608+jwbron@users.noreply.github.com> Date: Wed, 29 Apr 2026 01:59:26 +0000 Subject: [PATCH 1/2] docs: update docs for slice-DAG implement phase (#2137) [doc-updater] Update documentation to reflect changes from d1a213901 (PR #2220): - README.md: describe slice-DAG model in the Implement phase step - STRUCTURE.md: add slice_scheduler.py and stacked_pr_reconciler.py entries; update dependency_graph.py to mention its generification - orchestrator-cli.md: add the five new EGG_ORCH_*SLICE* and EGG_ORCH_STACKED_PR_RECONCILER_* env vars to the reference table - sdlc-pipeline.md: introduce slice-DAG paragraph in Multi-Agent Orchestration section with link to architecture/slice-dag.md - concurrent-execution.md: correct "sharing the pipeline branch" claim (implement now uses per-slice integration branches) Triggered by: https://github.com/jwbron/egg/pull/2220 Authored-by: egg --- README.md | 2 +- docs/development/STRUCTURE.md | 4 +++- docs/guides/concurrent-execution.md | 4 +++- docs/guides/sdlc-pipeline.md | 4 +++- docs/reference/orchestrator-cli.md | 5 +++++ 5 files changed, 15 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 76f7a953d9..faab923398 100644 --- a/README.md +++ b/README.md @@ -102,7 +102,7 @@ Human gate Human gate Human merge 1. **Refine**: Agents analyze the task, research the codebase, produce requirements. Reviewers validate. Human approves before planning. 2. **Plan**: Architect recommends approach, task planner breaks it into discrete tasks with acceptance criteria, risk analyst flags concerns. Human approves before any code is written. -3. **Implement**: Coder writes code, tester writes tests and runs linters/type-checkers, documenter updates docs. Code and contract reviewers provide line-level feedback; security and concurrency lens reviewers add targeted cross-file analysis and block consensus on a NACK. Cycles continue until all checks pass and BRC consensus is reached. +3. **Implement**: The plan's tasks are split into a **DAG of independent slices** — each slice runs as its own agent team on its own integration branch with its own BRC consensus and stacked PR. Slices whose dependencies are satisfied run concurrently (up to `EGG_ORCH_MAX_PARALLEL_SLICES`, default 5); slices with unmet dependencies wait in subsequent waves. Within each slice, the coder writes code, the tester writes tests and runs linters/type-checkers, and the documenter updates docs. Code and contract reviewers provide line-level feedback; security and concurrency lens reviewers add targeted cross-file analysis and block consensus on a NACK. Cycles continue until all checks pass and BRC consensus is reached for that slice. See [Slice-DAG Implement Phase](docs/architecture/slice-dag.md) for the full model. 4. **PR**: Orchestrator auto-creates the PR from plan metadata. Only a human can merge via GitHub UI. Within each phase, specialized agents run concurrently via BRC (enabled by default for refine, plan, and implement). Here's what a completed pipeline looks like: diff --git a/docs/development/STRUCTURE.md b/docs/development/STRUCTURE.md index 9e5f975d20..b04143f3ed 100644 --- a/docs/development/STRUCTURE.md +++ b/docs/development/STRUCTURE.md @@ -98,6 +98,8 @@ orchestrator/ ├── kubernetes_spawner.py # Agent Job lifecycle (replaces ContainerSpawner) ├── kubernetes_monitor.py # k8s Job state monitoring (replaces ContainerMonitor) ├── concurrent_executor.py # Concurrent phase executor (spawns all agents simultaneously) +├── slice_scheduler.py # Wave-based scheduler for the implement-phase slice DAG: computes execution waves, caps concurrency, two-tier max_cycles accounting, failure-cascade detection (#2137) +├── stacked_pr_reconciler.py # Stacked-PR rebase reconciler: detects child slice PRs whose base branch was deleted after a parent merge and retargets them via gateway rebase_onto (#2137) ├── action_guards.py # Formal BRC state machine action guards (preconditions for propose/ack/nack/confirm/withdraw) ├── approval_matrix.py # Per-reviewer ACK/NACK matrix for BRC consensus ├── attestation_schemas.py # Attestation payload validation for BRC proposals @@ -324,7 +326,7 @@ shared/ │ ├── agent_roles.py # Multi-agent role definitions (all agent and reviewer roles) │ ├── orchestrator.py # Multi-agent orchestration dispatch logic │ ├── orchestration.py # Agent execution state management -│ ├── dependency_graph.py # Agent dependency resolution for parallel execution +│ ├── dependency_graph.py # Generic dependency graph (PEP-695 typed): used for agent-role DAGs and for the implement-phase slice DAG (#2137 generification) │ ├── plan_parser.py # Plan document parsing with task extraction and phase dependency normalization │ ├── agent_recovery.py # Failed agent recovery logic │ ├── checkpoints.py # Checkpoint data models diff --git a/docs/guides/concurrent-execution.md b/docs/guides/concurrent-execution.md index 26f9fefe83..edb83ad6da 100644 --- a/docs/guides/concurrent-execution.md +++ b/docs/guides/concurrent-execution.md @@ -1,6 +1,8 @@ # Concurrent Execution Mode -Concurrent execution mode runs all agents for the current pipeline phase simultaneously — all sharing the pipeline branch — rather than sequentially in dependency-ordered waves. Agents communicate via the orchestrator message bus and signal readiness for phase completion via a consensus protocol. BRC consensus is active by default for the **refine**, **plan**, and **implement** phases. Additional phases (such as `review`) can be added via the `concurrent_phases` config. +Concurrent execution mode runs all agents for the current pipeline phase simultaneously rather than sequentially in dependency-ordered waves. Agents communicate via the orchestrator message bus and signal readiness for phase completion via a consensus protocol. BRC consensus is active by default for the **refine**, **plan**, and **implement** phases. Additional phases (such as `review`) can be added via the `concurrent_phases` config. + +**Implement-phase note**: the implement phase no longer runs as a single team on a shared branch. Instead, the plan's tasks are split into a DAG of independent **slices** — each slice runs its own concurrent agent team on its own integration branch. Concurrent execution within each slice follows the BRC protocol described here. See [Slice-DAG Implement Phase](../architecture/slice-dag.md) for the slice-level orchestration model. This is distinct from the standard wave-based parallel execution (Tier 2), where agents run in dependency order but multiple independent agents execute in parallel within each wave. diff --git a/docs/guides/sdlc-pipeline.md b/docs/guides/sdlc-pipeline.md index ea2c405500..d17c4d0e79 100644 --- a/docs/guides/sdlc-pipeline.md +++ b/docs/guides/sdlc-pipeline.md @@ -294,7 +294,9 @@ The contract tracks per-reviewer verdicts for debugging: ### Multi-Agent Orchestration -The implement phase uses concurrent BRC execution, where specialized agents run simultaneously and coordinate via the message bus. +The implement phase runs as a **DAG of independent slices** (#2137). Each slice has its own integration branch (`egg/issue-N/slice-M`), agent team, BRC consensus, and stacked PR targeting the parent slice's branch (or the pipeline branch for root slices). The `SliceScheduler` computes execution waves — slices whose dependencies are satisfied run concurrently (capped at `EGG_ORCH_MAX_PARALLEL_SLICES`, default 5); dependent slices wait in subsequent waves. See [Slice-DAG Implement Phase](../architecture/slice-dag.md) for the full model including forest validation, two-tier `max_cycles` accounting, failure cascade, and the stacked-PR reconciler. + +Within each slice, concurrent BRC execution runs: specialized agents run simultaneously and coordinate via the message bus. **Agent Roles:** diff --git a/docs/reference/orchestrator-cli.md b/docs/reference/orchestrator-cli.md index a0d0205ef1..a97e7dbe90 100644 --- a/docs/reference/orchestrator-cli.md +++ b/docs/reference/orchestrator-cli.md @@ -79,6 +79,11 @@ Agent role can be omitted when `EGG_AGENT_ROLE` is set. | `EGG_ORCH_STATE_STORE_PROBE_INTERVAL` | Cadence (seconds) of the background state-store self-heal probe. Default `15`. Lowering tightens wedge-detection at the cost of more frequent `git` calls; raising it does the inverse. The staleness watchdog flips `/api/v1/ready` to 503 when cache age exceeds `interval × 2`, so this setting also controls the readiness-flap window. Values above ~30s can exceed the readinessProbe's boot tolerance (`initialDelaySeconds + periodSeconds × failureThreshold = 35s`). | | `EGG_ORCH_WAITRESS_THREADS` | Waitress WSGI thread pool size. Default `16`, minimum `4`. Values `< 4` cause the orchestrator to `sys.exit(78)` (EX_CONFIG) at boot with an ERROR log. Each blocking long-poll occupies one thread; size the pool above the concurrent-agent count plus short-request headroom. The `egg_inflight_long_polls` Prometheus gauge exposes saturation. See [Agent Wait Patterns §7](agent-wait-patterns.md#7-egg_orch_waitress_threads--thread-pool--long-poll-coupling). | | `EGG_HEARTBEAT_RATE_LIMIT` | Per-`(pipeline_id, agent_role)` `HEARTBEAT` rate cap (messages per minute). Default `20`. Exceeding returns HTTP 429 with a `Retry-After` header; the CLI surfaces 429 as exit 3 (permanent). See [Agent Wait Patterns §5](agent-wait-patterns.md#5-egg_heartbeat_rate_limit--per-role-heartbeat-cap). | +| `EGG_ORCH_MAX_PARALLEL_SLICES` | Maximum number of implement-phase slices that may run concurrently within a single pipeline wave. Default `5`. Backed by `SliceScheduler.max_parallel_slices`; reduce when container or gateway resources are constrained. | +| `EGG_ORCH_SLICE_LOCAL_MAX_CYCLES` | Per-slice BRC re-proposal ceiling before HITL escalation. Default `3`. Part of the two-tier cycle cap model (#2137 decision-9). | +| `EGG_ORCH_SLICE_GLOBAL_MAX_CYCLES` | Pipeline-wide cap on the summed total of slice re-proposal cycles. Default `10`. Either the local or global cap tripping escalates to HITL. | +| `EGG_ORCH_SLICE_FAILURE_GRACE_SECONDS` | Grace window (seconds) between a slice failure and the orchestrator marking the downstream subtree `BLOCKED_ON_FAILED_DEPENDENCY`. Default `60`. Allows HITL resolution before the cascade fires. | +| `EGG_ORCH_STACKED_PR_RECONCILER_INTERVAL_SECONDS` | Polling cadence (seconds) of the stacked-PR reconciler that detects child slice PRs whose base branch was deleted after a parent merge. Default `30`. | | `AGENT_ANCHOR_ID` | Agent anchor ID (`{role}-{short_container_id}`), auto-set by container spawner | | `EGG_LIFECYCLE_SECRET` | Bearer token required for lifecycle-control endpoints (HITL resolve/cancel, pipeline CRUD, phase overrides, container spawn/stop). Stored at `~/.config/egg/lifecycle-secret`. Must be exported in the human's shell to run `egg-orch decision resolve`, `egg-orch pipeline delete`, etc. Agent pods never receive it (see #1769). | From fe6018a4f13047448f5ee0ed7a2843a23e27825f Mon Sep 17 00:00:00 2001 From: "egg-reviewer[bot]" <261018737+egg-reviewer[bot]@users.noreply.github.com> Date: Wed, 29 Apr 2026 02:16:39 +0000 Subject: [PATCH 2/2] docs: resolve slice-DAG/shared-branch contradictions raised in PR #2231 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address review feedback from egg-reviewer: - concurrent-execution.md: qualify "shared branch" claims at L49 (top-level branch model paragraph), L862 (per-agent worktree architecture bullet), L867 (push coordination step), and L877 (reviewer worktree sync) so the whole guide consistently teaches the slice model for the implement phase rather than contradicting the new top-of-file note. - sdlc-pipeline.md: rewrite the "Shared Pipeline Branch" subsection (now "Branch Model") and the "Commit conflicts" troubleshooting bullet to match the slice-DAG model introduced earlier in the same file. - orchestrator-cli.md: add "API live, not yet wired in the run loop — see [Slice-DAG Implement Phase]; #2199" caveat to the EGG_ORCH_SLICE_LOCAL_MAX_CYCLES and EGG_ORCH_SLICE_GLOBAL_MAX_CYCLES rows so the reference table doesn't oversell current behavior. Authored-by: egg --- docs/guides/concurrent-execution.md | 8 ++++---- docs/guides/sdlc-pipeline.md | 28 ++++++++++++++++------------ docs/reference/orchestrator-cli.md | 4 ++-- 3 files changed, 22 insertions(+), 18 deletions(-) diff --git a/docs/guides/concurrent-execution.md b/docs/guides/concurrent-execution.md index edb83ad6da..196ec0ed8d 100644 --- a/docs/guides/concurrent-execution.md +++ b/docs/guides/concurrent-execution.md @@ -46,7 +46,7 @@ When concurrent execution starts, the `ConcurrentPhaseExecutor` (in `orchestrato | `plan` | `architect`, `task_planner`, `risk_analyst`, `reviewer_plan` | | `implement` | `coder`, `tester`, `documenter`, `reviewer_code`, `reviewer_code_holistic`, `reviewer_contract`, `reviewer_security`, `reviewer_concurrency` | -**Shared branch**: All agents operate on the pipeline's shared branch (e.g., `egg/issue-123`). Agents coordinate commits via the message bus to sequence their work and avoid conflicts. +**Branch model**: For **refine** and **plan**, all agents operate on the pipeline's shared branch (e.g., `egg/issue-123`) and coordinate commits via the message bus to sequence their work and avoid conflicts. For **implement**, each slice runs on its own integration branch (`egg/issue-N/slice-M`); the shared-branch coordination described below applies *within* a slice's agent team — see [Slice-DAG Implement Phase](../architecture/slice-dag.md). **Environment injection**: Each concurrent agent receives: @@ -859,12 +859,12 @@ Each concurrent agent runs in its own isolated git worktree. This prevents agent **Architecture:** - Each agent pod receives a unique worktree created by the gateway, keyed by Job name (not pipeline ID) -- All agents push to the same shared pipeline branch (e.g., `egg/issue-{N}`) +- For **refine** and **plan**, all agents push to the same shared pipeline branch (e.g., `egg/issue-{N}`); for **implement**, all agents within a slice push to that slice's integration branch (`egg/issue-{N}/slice-{M}`) — see [Slice-DAG Implement Phase](../architecture/slice-dag.md) - Git worktrees share the object store — only working tree files are duplicated, so disk overhead is marginal **Push coordination (pull-before-push):** 1. Agent finishes work, commits in its own worktree -2. Agent pushes to the shared branch via the gateway +2. Agent pushes to the team's branch (pipeline branch for refine/plan, slice integration branch for implement) via the gateway 3. If push is rejected (another agent pushed first) → `git pull --rebase` → retry push 4. Rebase **cannot conflict** because agents have mutually exclusive file write permissions (see [Agent Roles Reference](../reference/agent-roles.md)) @@ -874,7 +874,7 @@ This works because role restrictions guarantee non-overlapping file sets (coder ### Reviewer Worktree Sync -Per-agent worktrees are created at phase start from the pipeline branch. When a producer pushes commits and proposes, the reviewer's worktree does not automatically have those commits. To address this, the BRC preamble instructs reviewers to sync their worktree before reviewing: +Per-agent worktrees are created at phase start from the team's branch — the pipeline branch for refine/plan, the slice integration branch for each implement slice. When a producer pushes commits and proposes, the reviewer's worktree does not automatically have those commits. To address this, the BRC preamble instructs reviewers to sync their worktree before reviewing: ```bash git fetch origin && git merge origin/{branch} --no-edit diff --git a/docs/guides/sdlc-pipeline.md b/docs/guides/sdlc-pipeline.md index d17c4d0e79..f4e1f052c6 100644 --- a/docs/guides/sdlc-pipeline.md +++ b/docs/guides/sdlc-pipeline.md @@ -1359,14 +1359,18 @@ all changes. **Reviewer (code/contract)**: Reviews committed code or contract artifacts. Polls for `PROGRESS` from coder. Signals `READY` after review is complete. -### Shared Pipeline Branch +### Branch Model -All concurrent agents operate on the pipeline's shared branch (e.g., `egg/issue-999`). -Rather than each agent having an isolated worktree branch, all agents commit directly -to a single shared history. Agents coordinate via the message bus to sequence commits -and avoid conflicts — for example, the coder signals `HANDOFF` when its changes are -committed so downstream agents (tester, documenter) know it is safe to pull and build -on top. +For the **refine** and **plan** phases, all concurrent agents operate on the pipeline's +shared branch (e.g., `egg/issue-999`) — they commit directly to a single shared history +and coordinate via the message bus to sequence commits and avoid conflicts (for example, +the coder signals `HANDOFF` when its changes are committed so downstream agents know it +is safe to pull and build on top). + +For the **implement** phase, the pipeline branch is no longer shared across the whole +team. Tasks are split into a DAG of slices, and each slice runs on its own integration +branch (`egg/issue-999/slice-M`); the shared-history coordination above applies *within* +a slice's agent team. See [Slice-DAG Implement Phase](../architecture/slice-dag.md). ### Failure Handling @@ -1438,11 +1442,11 @@ identify blocked or stuck agents. **Message bus empty**: Verify the pipeline has `concurrent_execution: true` in its config. The message bus is only active for concurrent pipelines. -**Commit conflicts**: Since all concurrent agents share a single branch, agents -coordinate commits via the message bus to avoid conflicts. If an agent encounters a -conflict when pushing, it should pull, rebase, and retry. If conflicts persist, the -agent signals `BLOCKED` and a HITL decision is created. Consider adding role-based -file restrictions to minimize overlap. +**Commit conflicts**: Within a single team's branch (the pipeline branch for refine/plan, +or a slice's integration branch for implement), concurrent agents coordinate commits via +the message bus to avoid conflicts. If an agent encounters a conflict when pushing, it +should pull, rebase, and retry. If conflicts persist, the agent signals `BLOCKED` and a +HITL decision is created. Consider adding role-based file restrictions to minimize overlap. ## Agent MCP tools (`EGG_MCP_TOOLS` flag) diff --git a/docs/reference/orchestrator-cli.md b/docs/reference/orchestrator-cli.md index a97e7dbe90..c76f54e0b1 100644 --- a/docs/reference/orchestrator-cli.md +++ b/docs/reference/orchestrator-cli.md @@ -80,8 +80,8 @@ Agent role can be omitted when `EGG_AGENT_ROLE` is set. | `EGG_ORCH_WAITRESS_THREADS` | Waitress WSGI thread pool size. Default `16`, minimum `4`. Values `< 4` cause the orchestrator to `sys.exit(78)` (EX_CONFIG) at boot with an ERROR log. Each blocking long-poll occupies one thread; size the pool above the concurrent-agent count plus short-request headroom. The `egg_inflight_long_polls` Prometheus gauge exposes saturation. See [Agent Wait Patterns §7](agent-wait-patterns.md#7-egg_orch_waitress_threads--thread-pool--long-poll-coupling). | | `EGG_HEARTBEAT_RATE_LIMIT` | Per-`(pipeline_id, agent_role)` `HEARTBEAT` rate cap (messages per minute). Default `20`. Exceeding returns HTTP 429 with a `Retry-After` header; the CLI surfaces 429 as exit 3 (permanent). See [Agent Wait Patterns §5](agent-wait-patterns.md#5-egg_heartbeat_rate_limit--per-role-heartbeat-cap). | | `EGG_ORCH_MAX_PARALLEL_SLICES` | Maximum number of implement-phase slices that may run concurrently within a single pipeline wave. Default `5`. Backed by `SliceScheduler.max_parallel_slices`; reduce when container or gateway resources are constrained. | -| `EGG_ORCH_SLICE_LOCAL_MAX_CYCLES` | Per-slice BRC re-proposal ceiling before HITL escalation. Default `3`. Part of the two-tier cycle cap model (#2137 decision-9). | -| `EGG_ORCH_SLICE_GLOBAL_MAX_CYCLES` | Pipeline-wide cap on the summed total of slice re-proposal cycles. Default `10`. Either the local or global cap tripping escalates to HITL. | +| `EGG_ORCH_SLICE_LOCAL_MAX_CYCLES` | Per-slice BRC re-proposal ceiling before HITL escalation. Default `3`. Part of the two-tier cycle cap model (#2137 decision-9). *API live, not yet wired in the run loop — see [Slice-DAG Implement Phase](../architecture/slice-dag.md); #2199.* | +| `EGG_ORCH_SLICE_GLOBAL_MAX_CYCLES` | Pipeline-wide cap on the summed total of slice re-proposal cycles. Default `10`. Either the local or global cap tripping escalates to HITL. *API live, not yet wired in the run loop — see [Slice-DAG Implement Phase](../architecture/slice-dag.md); #2199.* | | `EGG_ORCH_SLICE_FAILURE_GRACE_SECONDS` | Grace window (seconds) between a slice failure and the orchestrator marking the downstream subtree `BLOCKED_ON_FAILED_DEPENDENCY`. Default `60`. Allows HITL resolution before the cascade fires. | | `EGG_ORCH_STACKED_PR_RECONCILER_INTERVAL_SECONDS` | Polling cadence (seconds) of the stacked-PR reconciler that detects child slice PRs whose base branch was deleted after a parent merge. Default `30`. | | `AGENT_ANCHOR_ID` | Agent anchor ID (`{role}-{short_container_id}`), auto-set by container spawner |