Skip to content

Enforce spawn dependency ordering and fix coordinator bugs - #1085

Merged
jwbron merged 3 commits into
mainfrom
egg/fix-pipeline-collaboration-failures
Mar 14, 2026
Merged

Enforce spawn dependency ordering and fix coordinator bugs#1085
jwbron merged 3 commits into
mainfrom
egg/fix-pipeline-collaboration-failures

Conversation

@james-in-a-box

@james-in-a-box james-in-a-box Bot commented Mar 14, 2026

Copy link
Copy Markdown
Contributor

Enforce agent spawn ordering and fix coordinator bugs

During the issue-1059 pipeline run, reviewers consistently finished
before the primary agents they were meant to review, provide_input
returned 405 on every HITL resolution, and concurrent agents diverged
onto separate branches. These failures stem from missing guardrails in
the coordinator spawn path and a wrong URL in the MCP tool handler.

This PR adds dependency validation and contract checks to the spawn
endpoint (extending the phase-advance contract guard from #1072 to
cover spawns), fixes the provide_input URL, appends consensus
protocol instructions to agent prompts, and switches
ConcurrentPhaseExecutor to a shared pipeline branch.

Issue: #1081

Test plan:

  • pytest orchestrator/tests/test_coordinator_routes_functional.py — 75 pass, including 4 new tests for dependency and contract enforcement at spawn time
  • pytest orchestrator/tests/test_concurrent_integration.py — 2 new tests for shared branch behavior
  • pytest orchestrator/tests/test_coordinator_mcp_functional.pytest_provide_input verifies /resolve suffix

Authored-by: egg

Fix five coordination issues observed during the issue-1059 pipeline run:
provide_input returning 405 due to missing /resolve in URL, reviewers
finishing before primary agents due to no dependency enforcement, missing
contract validation at spawn time, no consensus protocol reminder in
agent prompts, and ConcurrentPhaseExecutor giving each agent a separate
branch instead of sharing the pipeline branch.

Closes #1081
@james-in-a-box

This comment has been minimized.

@james-in-a-box

Copy link
Copy Markdown
Contributor Author
Autofix tracking
{"Lint/Python": 1}

@james-in-a-box

This comment has been minimized.

@james-in-a-box james-in-a-box Bot changed the title Fix pipeline collaboration failures Enforce spawn dependency ordering and fix coordinator bugs Mar 14, 2026

@egg-reviewer egg-reviewer Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: Enforce spawn dependency ordering and fix coordinator bugs

Reviewed all 7 changed files (281 additions, 13 deletions). The changes are well-scoped and address real failures observed in issue-1059. The MCP URL fix, contract enforcement at spawn time, and dependency validation are all correct. Tests are comprehensive.

MCP URL fix (mcp_tools.py)

Correct fix. The decision resolution endpoint is POST /api/v1/pipelines/{id}/decisions/{id}/resolve. The old URL without /resolve maps to a GET-only handler, which explains the 405s seen in production. Test updated to match.

Contract enforcement at spawn (`routes/coordinator.py:212-227)

Correct extension of the existing phase-advance guard. Blocks spawns in IMPLEMENT/PR phases when contract_synced is false. Consistent with the phase-advance contract enforcement from #1072. Default value of contract_synced=True in the Pipeline model means existing tests pass without modification unless explicitly testing this path — which the new tests do correctly.

Dependency validation (routes/coordinator.py:229-259)

The implementation is correct and handles cross-phase dependencies properly. agents_spawned persists across phases in CoordinatorState, so plan-phase agents (TASK_PLANNER, RISK_ANALYST) remain visible during implement phase. This allows reviewer roles with cross-phase dependencies to correctly check against the full spawn history.

The ordering of checks — contract enforcement before dependency validation before phase-role validation — is logical: fail fast on the cheapest checks first.

One note on the exception handler (lines 254-259): except (ValueError, KeyError) silently allows spawn when get_role_definition fails. The debug-level log is appropriate for forward compatibility with roles not yet registered in egg_contracts, but consider whether logger.warning would be more appropriate here since this bypasses a safety check. If a role definition is misconfigured and raises unexpectedly, the dependency check is silently skipped.

Shared branch (concurrent_executor.py:95-100)

The switch from per-role branches (egg/issue-{N}/{role}) to a shared pipeline branch is the correct direction — divergent branches were causing the integration failures described in the issue. The dependency ordering added in this same PR mitigates the primary risk by ensuring agents that write to the same areas run sequentially.

Advisory: Agents that CAN run in parallel on the shared branch (e.g., TESTER+DOCUMENTER in wave 2, REVIEWER_CODE+REVIEWER_CONTRACT in wave 4) may hit non-fast-forward push failures when pushing concurrently. The gateway has no push serialization or retry mechanism. In practice this is low-risk since these role pairs write to disjoint file sets, but it's worth documenting. If an agent's push fails due to a concurrent push from its sibling, the agent has no built-in recovery path.

Consensus protocol prompt (routes/coordinator.py:338-344)

The stay-alive instructions are appended unconditionally to every coordinator-spawned agent. This is fine for orchestrated pipelines where the coordinator is always active, though it's slightly redundant with the concurrent mode section already in CLAUDE.md. The directness of injecting it into the prompt should improve agent compliance, which was the problem.

Test coverage

Good test additions:

  • TestSpawnDependencyValidation (2 tests): Correctly verifies blocking when deps are incomplete and allowing when deps are satisfied.
  • TestSpawnContractEnforcement (2 tests): Correctly verifies blocking in implement phase without contract and allowing in refine phase.
  • TestGetWorktreeBranch (2 tests): Verifies shared branch behavior.
  • Existing tests updated to satisfy new dependency checks — this is correct and shows the new validation integrates properly with the existing test suite.

Minor observation on test_get_worktree_branch_fallback: setting pipeline.branch = "" works (empty string is falsy) but None is the more realistic falsy value given the model type str | None. The fallback also produces "egg/issue-999" which is identical to the cleared pipeline branch value, making it harder to confirm the fallback path is actually computing the branch rather than using a cached value. Consider using a distinct issue_number to make the test more discriminating.

Summary

No blocking issues. The fixes address the three failures described in the issue (reviewer ordering, provide_input 405, branch divergence) with correct implementations and good test coverage.

— Authored by egg

@james-in-a-box

This comment has been minimized.

@james-in-a-box

This comment has been minimized.

@james-in-a-box james-in-a-box Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Both suggestions addressed in c078126:

  1. Logger level: Changed logger.debuglogger.warning in the dependency check exception handler (coordinator.py:256). The reviewer is right that bypassing a safety check warrants warning-level visibility.

  2. Fallback test: Updated test_get_worktree_branch_fallback to use pipeline.branch = None (more realistic falsy value matching the str | None type) and a distinct issue_number = 777 so the asserted branch (egg/issue-777) differs from the default pipeline branch, proving the fallback path actually computes the value.

— Authored by egg

@egg-reviewer egg-reviewer Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review: c078126 (feedback addressing commit)

Reviewed the delta since 5ec697e. One commit: c078126 Address review feedback: raise log level, improve fallback test.

Changes verified

1. Logger level (coordinator.py:257): logger.debuglogger.warning in the dependency check exception handler. Correct. The surrounding comment also updated to explain why warning level is appropriate. No other safety-check bypass paths remain at debug level — the remaining logger.debug at line 288 (phase-role mapping fallback) is a distinct case where the phase legitimately has no role definitions, not a dependency bypass.

2. Fallback test (test_concurrent_integration.py): pipeline.branch = ""pipeline.branch = None (matches str | None model type). Added pipeline.issue_number = 777 so the asserted branch (egg/issue-777) is distinct from the helper's default (egg/issue-999), proving the fallback actually computes from issue_number rather than using a cached or coincidental value. Both changes make the test more discriminating.

Verdict

Both suggestions from the previous review are properly addressed. No new issues introduced. The delta is minimal and correct.

— Authored by egg

@james-in-a-box

Copy link
Copy Markdown
Contributor Author

egg review completed. View run logs

4 previous review(s) hidden.

@jwbron
jwbron merged commit 476c39d into main Mar 14, 2026
29 checks passed
github-actions Bot pushed a commit that referenced this pull request Mar 14, 2026
…updater]

Update documentation to reflect changes from 476c39d (PR #1085):
- concurrent-execution.md: Replace "per-agent worktree isolation" section
  with "shared pipeline branch" — agents now share `egg/issue-{N}` instead
  of having per-role branches (`egg/issue-{N}/{role}`)
- coordinator.md: Document two new 409 spawn validation cases: contract
  enforcement (no contract in implement/pr phase) and dependency ordering
  (reviewer/downstream roles blocked until primary dependencies complete)
- README.md: Sync concurrent execution mode description to match

Triggered by: #1085

Authored-by: egg
jwbron added a commit that referenced this pull request Mar 14, 2026
…updater] (#1091)

* docs: update concurrent branch model and coordinator spawn docs [doc-updater]

Update documentation to reflect changes from 476c39d (PR #1085):
- concurrent-execution.md: Replace "per-agent worktree isolation" section
  with "shared pipeline branch" — agents now share `egg/issue-{N}` instead
  of having per-role branches (`egg/issue-{N}/{role}`)
- coordinator.md: Document two new 409 spawn validation cases: contract
  enforcement (no contract in implement/pr phase) and dependency ordering
  (reviewer/downstream roles blocked until primary dependencies complete)
- README.md: Sync concurrent execution mode description to match

Triggered by: #1085

Authored-by: egg

* docs: fix stale per-agent worktree refs and improve 409 error docs

* docs: fix stale per-agent worktree ref in orchestrator README

---------

Co-authored-by: jwbron <8340608+jwbron@users.noreply.github.com>
Co-authored-by: egg-reviewer[bot] <261018737+egg-reviewer[bot]@users.noreply.github.com>
jwbron added a commit that referenced this pull request Mar 14, 2026
The dependency blocking added in #1085 prevented agents from spawning
until their declared dependencies completed. This is wrong — agents
should run concurrently and communicate via the messaging system.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
jwbron added a commit that referenced this pull request Mar 14, 2026
…#1100)

* Fix orchestrator crash: add missing egg_agent module to Dockerfile

The egg_agent shared module was added in #1088 but the orchestrator
Dockerfile was not updated to copy it into the container image, causing
an ImportError crash loop on startup.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Update Dockerfile comment to include egg_agent in shared modules list

* Remove spawn dependency enforcement to allow concurrent agents

The dependency blocking added in #1085 prevented agents from spawning
until their declared dependencies completed. This is wrong — agents
should run concurrently and communicate via the messaging system.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Fix stale container/agent status after pipeline cancellation

When a pipeline is cancelled, cleanup_pipeline() removes Docker containers
but the pipeline state store still shows them as "running". This adds
_mark_pipeline_records_cancelled() which syncs the persisted state after
container cleanup — marking containers as REMOVED, agents as FAILED, and
coordinator spawn records as cancelled.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Fix cancel state sync: reload pipeline before marking records

The previous approach mutated the stale pipeline object returned by
update_pipeline(), which could overwrite coordinator state changes made
between the status update and container cleanup. Now reloads the latest
state from the store after containers are killed, ensuring we mark the
most up-to-date records and don't clobber concurrent writes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Address review feedback: fix error message, rename function, clean up docs

* Fix spawned agents missing EGG_ORCHESTRATOR_URL env var

container_spawner.py set EGG_PIPELINE_ID but never set
EGG_ORCHESTRATOR_URL, causing the sandbox entrypoint to bail out
with "Orchestrator mode enabled but missing URL or pipeline_id"
(exit 124). The other two spawn paths (sandbox_template.py and
routes/pipelines.py) both set it correctly — this was the only
one missing it.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Increase cancel_task MCP endpoint timeout from 30s to 120s

Cancellation involves stopping containers which regularly exceeds the
default 30s request timeout, causing spurious "timed out" errors.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Fix test to expect timeout=120 in cancel_task

* Align orchestrator URL construction across spawn paths

Use ORCHESTRATOR_EXTERNAL_IP (static IP) instead of
ORCHESTRATOR_CONTAINER_NAME (hostname) in container_spawner.py for
public mode, matching the pattern in routes/pipelines.py.

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: egg-reviewer[bot] <261018737+egg-reviewer[bot]@users.noreply.github.com>
jwbron added a commit that referenced this pull request Mar 14, 2026
* Fix orchestrator crash: add missing egg_agent module to Dockerfile

The egg_agent shared module was added in #1088 but the orchestrator
Dockerfile was not updated to copy it into the container image, causing
an ImportError crash loop on startup.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Update Dockerfile comment to include egg_agent in shared modules list

* Remove spawn dependency enforcement to allow concurrent agents

The dependency blocking added in #1085 prevented agents from spawning
until their declared dependencies completed. This is wrong — agents
should run concurrently and communicate via the messaging system.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Fix stale container/agent status after pipeline cancellation

When a pipeline is cancelled, cleanup_pipeline() removes Docker containers
but the pipeline state store still shows them as "running". This adds
_mark_pipeline_records_cancelled() which syncs the persisted state after
container cleanup — marking containers as REMOVED, agents as FAILED, and
coordinator spawn records as cancelled.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Fix cancel state sync: reload pipeline before marking records

The previous approach mutated the stale pipeline object returned by
update_pipeline(), which could overwrite coordinator state changes made
between the status update and container cleanup. Now reloads the latest
state from the store after containers are killed, ensuring we mark the
most up-to-date records and don't clobber concurrent writes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Replace implicit consensus READY with agent restart

The consensus wrapper was blindly auto-signaling READY when an agent
exited cleanly without participating in consensus. This masked the
real problem: the agent didn't follow the consensus protocol.

Instead of faking consensus, the wrapper now restarts Claude with a
recovery prompt that explains what happened and instructs the agent
to assess state and explicitly signal READY or continue working.
Restarts are capped at 2 (configurable). After exhausting restarts,
the wrapper enters a passive wait loop but does NOT auto-signal READY.

Also removes the orchestrator-level implicit READY registration in
the concurrent phase poll loop (pipelines.py) — agents must explicitly
participate in consensus.

Issue: #1081

* Fix checks: apply automated formatting fixes

* Exit with failure after max restarts instead of passive wait

After exhausting restart attempts, the wrapper now exits with code 1
instead of entering a passive wait loop. There's nothing to wait for —
no process inside the container can make progress. The non-zero exit
triggers the orchestrator's agent failure path (HITL decision).

* Update docs for restart-based consensus wrapper

Update concurrent-execution.md to reflect the new behavior: the wrapper
restarts the agent with a recovery prompt instead of auto-signaling
READY. Remove references to EGG_CONSENSUS_WRAPPER_TIMEOUT (no longer
used) and implicit READY (removed from orchestrator).

* Address review feedback on consensus restart PR

* Fix consensus wrapper to use pipeline status endpoint

All consensus checks were querying egg-orch message status, which returns
message bus statistics — not consensus data. Changed to egg-orch pipeline
status with the correct nested path (data.concurrent.consensus) matching
the real API response.

Also addressed:
- Added MAX_READY_POLL_CYCLES (default 10) as a separate constant for the
  READY polling loop instead of reusing MAX_RESTARTS (default 2)
- Extracted claude mock creation to shared _make_mock_claude() helper
- Updated all test mocks to use pipeline status and the correct response
  structure

* Add behavioral test for READY polling path

---------

Co-authored-by: James Wiesebron <jameswiesebron@khanacademy.org>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: egg-reviewer[bot] <261018737+egg-reviewer[bot]@users.noreply.github.com>
Co-authored-by: egg <egg@localhost>
Co-authored-by: jwbron <8340608+jwbron@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant