Enforce spawn dependency ordering and fix coordinator bugs - #1085
Conversation
Fix five coordination issues observed during the issue-1059 pipeline run: provide_input returning 405 due to missing /resolve in URL, reviewers finishing before primary agents due to no dependency enforcement, missing contract validation at spawn time, no consensus protocol reminder in agent prompts, and ConcurrentPhaseExecutor giving each agent a separate branch instead of sharing the pipeline branch. Closes #1081
This comment has been minimized.
This comment has been minimized.
Autofix tracking{"Lint/Python": 1} |
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Review: Enforce spawn dependency ordering and fix coordinator bugs
Reviewed all 7 changed files (281 additions, 13 deletions). The changes are well-scoped and address real failures observed in issue-1059. The MCP URL fix, contract enforcement at spawn time, and dependency validation are all correct. Tests are comprehensive.
MCP URL fix (mcp_tools.py)
Correct fix. The decision resolution endpoint is POST /api/v1/pipelines/{id}/decisions/{id}/resolve. The old URL without /resolve maps to a GET-only handler, which explains the 405s seen in production. Test updated to match.
Contract enforcement at spawn (`routes/coordinator.py:212-227)
Correct extension of the existing phase-advance guard. Blocks spawns in IMPLEMENT/PR phases when contract_synced is false. Consistent with the phase-advance contract enforcement from #1072. Default value of contract_synced=True in the Pipeline model means existing tests pass without modification unless explicitly testing this path — which the new tests do correctly.
Dependency validation (routes/coordinator.py:229-259)
The implementation is correct and handles cross-phase dependencies properly. agents_spawned persists across phases in CoordinatorState, so plan-phase agents (TASK_PLANNER, RISK_ANALYST) remain visible during implement phase. This allows reviewer roles with cross-phase dependencies to correctly check against the full spawn history.
The ordering of checks — contract enforcement before dependency validation before phase-role validation — is logical: fail fast on the cheapest checks first.
One note on the exception handler (lines 254-259): except (ValueError, KeyError) silently allows spawn when get_role_definition fails. The debug-level log is appropriate for forward compatibility with roles not yet registered in egg_contracts, but consider whether logger.warning would be more appropriate here since this bypasses a safety check. If a role definition is misconfigured and raises unexpectedly, the dependency check is silently skipped.
Shared branch (concurrent_executor.py:95-100)
The switch from per-role branches (egg/issue-{N}/{role}) to a shared pipeline branch is the correct direction — divergent branches were causing the integration failures described in the issue. The dependency ordering added in this same PR mitigates the primary risk by ensuring agents that write to the same areas run sequentially.
Advisory: Agents that CAN run in parallel on the shared branch (e.g., TESTER+DOCUMENTER in wave 2, REVIEWER_CODE+REVIEWER_CONTRACT in wave 4) may hit non-fast-forward push failures when pushing concurrently. The gateway has no push serialization or retry mechanism. In practice this is low-risk since these role pairs write to disjoint file sets, but it's worth documenting. If an agent's push fails due to a concurrent push from its sibling, the agent has no built-in recovery path.
Consensus protocol prompt (routes/coordinator.py:338-344)
The stay-alive instructions are appended unconditionally to every coordinator-spawned agent. This is fine for orchestrated pipelines where the coordinator is always active, though it's slightly redundant with the concurrent mode section already in CLAUDE.md. The directness of injecting it into the prompt should improve agent compliance, which was the problem.
Test coverage
Good test additions:
TestSpawnDependencyValidation(2 tests): Correctly verifies blocking when deps are incomplete and allowing when deps are satisfied.TestSpawnContractEnforcement(2 tests): Correctly verifies blocking in implement phase without contract and allowing in refine phase.TestGetWorktreeBranch(2 tests): Verifies shared branch behavior.- Existing tests updated to satisfy new dependency checks — this is correct and shows the new validation integrates properly with the existing test suite.
Minor observation on test_get_worktree_branch_fallback: setting pipeline.branch = "" works (empty string is falsy) but None is the more realistic falsy value given the model type str | None. The fallback also produces "egg/issue-999" which is identical to the cleared pipeline branch value, making it harder to confirm the fallback path is actually computing the branch rather than using a cached value. Consider using a distinct issue_number to make the test more discriminating.
Summary
No blocking issues. The fixes address the three failures described in the issue (reviewer ordering, provide_input 405, branch divergence) with correct implementations and good test coverage.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Agreed. Both suggestions addressed in c078126:
-
Logger level: Changed
logger.debug→logger.warningin the dependency check exception handler (coordinator.py:256). The reviewer is right that bypassing a safety check warrants warning-level visibility. -
Fallback test: Updated
test_get_worktree_branch_fallbackto usepipeline.branch = None(more realistic falsy value matching thestr | Nonetype) and a distinctissue_number = 777so the asserted branch (egg/issue-777) differs from the default pipeline branch, proving the fallback path actually computes the value.
— Authored by egg
There was a problem hiding this comment.
Re-review: c078126 (feedback addressing commit)
Reviewed the delta since 5ec697e. One commit: c078126 Address review feedback: raise log level, improve fallback test.
Changes verified
1. Logger level (coordinator.py:257): logger.debug → logger.warning in the dependency check exception handler. Correct. The surrounding comment also updated to explain why warning level is appropriate. No other safety-check bypass paths remain at debug level — the remaining logger.debug at line 288 (phase-role mapping fallback) is a distinct case where the phase legitimately has no role definitions, not a dependency bypass.
2. Fallback test (test_concurrent_integration.py): pipeline.branch = "" → pipeline.branch = None (matches str | None model type). Added pipeline.issue_number = 777 so the asserted branch (egg/issue-777) is distinct from the helper's default (egg/issue-999), proving the fallback actually computes from issue_number rather than using a cached or coincidental value. Both changes make the test more discriminating.
Verdict
Both suggestions from the previous review are properly addressed. No new issues introduced. The delta is minimal and correct.
— Authored by egg
|
egg review completed. View run logs 4 previous review(s) hidden. |
…updater] Update documentation to reflect changes from 476c39d (PR #1085): - concurrent-execution.md: Replace "per-agent worktree isolation" section with "shared pipeline branch" — agents now share `egg/issue-{N}` instead of having per-role branches (`egg/issue-{N}/{role}`) - coordinator.md: Document two new 409 spawn validation cases: contract enforcement (no contract in implement/pr phase) and dependency ordering (reviewer/downstream roles blocked until primary dependencies complete) - README.md: Sync concurrent execution mode description to match Triggered by: #1085 Authored-by: egg
…updater] (#1091) * docs: update concurrent branch model and coordinator spawn docs [doc-updater] Update documentation to reflect changes from 476c39d (PR #1085): - concurrent-execution.md: Replace "per-agent worktree isolation" section with "shared pipeline branch" — agents now share `egg/issue-{N}` instead of having per-role branches (`egg/issue-{N}/{role}`) - coordinator.md: Document two new 409 spawn validation cases: contract enforcement (no contract in implement/pr phase) and dependency ordering (reviewer/downstream roles blocked until primary dependencies complete) - README.md: Sync concurrent execution mode description to match Triggered by: #1085 Authored-by: egg * docs: fix stale per-agent worktree refs and improve 409 error docs * docs: fix stale per-agent worktree ref in orchestrator README --------- Co-authored-by: jwbron <8340608+jwbron@users.noreply.github.com> Co-authored-by: egg-reviewer[bot] <261018737+egg-reviewer[bot]@users.noreply.github.com>
The dependency blocking added in #1085 prevented agents from spawning until their declared dependencies completed. This is wrong — agents should run concurrently and communicate via the messaging system. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…#1100) * Fix orchestrator crash: add missing egg_agent module to Dockerfile The egg_agent shared module was added in #1088 but the orchestrator Dockerfile was not updated to copy it into the container image, causing an ImportError crash loop on startup. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Update Dockerfile comment to include egg_agent in shared modules list * Remove spawn dependency enforcement to allow concurrent agents The dependency blocking added in #1085 prevented agents from spawning until their declared dependencies completed. This is wrong — agents should run concurrently and communicate via the messaging system. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix stale container/agent status after pipeline cancellation When a pipeline is cancelled, cleanup_pipeline() removes Docker containers but the pipeline state store still shows them as "running". This adds _mark_pipeline_records_cancelled() which syncs the persisted state after container cleanup — marking containers as REMOVED, agents as FAILED, and coordinator spawn records as cancelled. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix cancel state sync: reload pipeline before marking records The previous approach mutated the stale pipeline object returned by update_pipeline(), which could overwrite coordinator state changes made between the status update and container cleanup. Now reloads the latest state from the store after containers are killed, ensuring we mark the most up-to-date records and don't clobber concurrent writes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Address review feedback: fix error message, rename function, clean up docs * Fix spawned agents missing EGG_ORCHESTRATOR_URL env var container_spawner.py set EGG_PIPELINE_ID but never set EGG_ORCHESTRATOR_URL, causing the sandbox entrypoint to bail out with "Orchestrator mode enabled but missing URL or pipeline_id" (exit 124). The other two spawn paths (sandbox_template.py and routes/pipelines.py) both set it correctly — this was the only one missing it. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Increase cancel_task MCP endpoint timeout from 30s to 120s Cancellation involves stopping containers which regularly exceeds the default 30s request timeout, causing spurious "timed out" errors. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix test to expect timeout=120 in cancel_task * Align orchestrator URL construction across spawn paths Use ORCHESTRATOR_EXTERNAL_IP (static IP) instead of ORCHESTRATOR_CONTAINER_NAME (hostname) in container_spawner.py for public mode, matching the pattern in routes/pipelines.py. --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: egg-reviewer[bot] <261018737+egg-reviewer[bot]@users.noreply.github.com>
* Fix orchestrator crash: add missing egg_agent module to Dockerfile The egg_agent shared module was added in #1088 but the orchestrator Dockerfile was not updated to copy it into the container image, causing an ImportError crash loop on startup. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Update Dockerfile comment to include egg_agent in shared modules list * Remove spawn dependency enforcement to allow concurrent agents The dependency blocking added in #1085 prevented agents from spawning until their declared dependencies completed. This is wrong — agents should run concurrently and communicate via the messaging system. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix stale container/agent status after pipeline cancellation When a pipeline is cancelled, cleanup_pipeline() removes Docker containers but the pipeline state store still shows them as "running". This adds _mark_pipeline_records_cancelled() which syncs the persisted state after container cleanup — marking containers as REMOVED, agents as FAILED, and coordinator spawn records as cancelled. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix cancel state sync: reload pipeline before marking records The previous approach mutated the stale pipeline object returned by update_pipeline(), which could overwrite coordinator state changes made between the status update and container cleanup. Now reloads the latest state from the store after containers are killed, ensuring we mark the most up-to-date records and don't clobber concurrent writes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Replace implicit consensus READY with agent restart The consensus wrapper was blindly auto-signaling READY when an agent exited cleanly without participating in consensus. This masked the real problem: the agent didn't follow the consensus protocol. Instead of faking consensus, the wrapper now restarts Claude with a recovery prompt that explains what happened and instructs the agent to assess state and explicitly signal READY or continue working. Restarts are capped at 2 (configurable). After exhausting restarts, the wrapper enters a passive wait loop but does NOT auto-signal READY. Also removes the orchestrator-level implicit READY registration in the concurrent phase poll loop (pipelines.py) — agents must explicitly participate in consensus. Issue: #1081 * Fix checks: apply automated formatting fixes * Exit with failure after max restarts instead of passive wait After exhausting restart attempts, the wrapper now exits with code 1 instead of entering a passive wait loop. There's nothing to wait for — no process inside the container can make progress. The non-zero exit triggers the orchestrator's agent failure path (HITL decision). * Update docs for restart-based consensus wrapper Update concurrent-execution.md to reflect the new behavior: the wrapper restarts the agent with a recovery prompt instead of auto-signaling READY. Remove references to EGG_CONSENSUS_WRAPPER_TIMEOUT (no longer used) and implicit READY (removed from orchestrator). * Address review feedback on consensus restart PR * Fix consensus wrapper to use pipeline status endpoint All consensus checks were querying egg-orch message status, which returns message bus statistics — not consensus data. Changed to egg-orch pipeline status with the correct nested path (data.concurrent.consensus) matching the real API response. Also addressed: - Added MAX_READY_POLL_CYCLES (default 10) as a separate constant for the READY polling loop instead of reusing MAX_RESTARTS (default 2) - Extracted claude mock creation to shared _make_mock_claude() helper - Updated all test mocks to use pipeline status and the correct response structure * Add behavioral test for READY polling path --------- Co-authored-by: James Wiesebron <jameswiesebron@khanacademy.org> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: egg-reviewer[bot] <261018737+egg-reviewer[bot]@users.noreply.github.com> Co-authored-by: egg <egg@localhost> Co-authored-by: jwbron <8340608+jwbron@users.noreply.github.com>
Enforce agent spawn ordering and fix coordinator bugs
During the issue-1059 pipeline run, reviewers consistently finished
before the primary agents they were meant to review,
provide_inputreturned 405 on every HITL resolution, and concurrent agents diverged
onto separate branches. These failures stem from missing guardrails in
the coordinator spawn path and a wrong URL in the MCP tool handler.
This PR adds dependency validation and contract checks to the spawn
endpoint (extending the phase-advance contract guard from #1072 to
cover spawns), fixes the
provide_inputURL, appends consensusprotocol instructions to agent prompts, and switches
ConcurrentPhaseExecutorto a shared pipeline branch.Issue: #1081
Test plan:
pytest orchestrator/tests/test_coordinator_routes_functional.py— 75 pass, including 4 new tests for dependency and contract enforcement at spawn timepytest orchestrator/tests/test_concurrent_integration.py— 2 new tests for shared branch behaviorpytest orchestrator/tests/test_coordinator_mcp_functional.py—test_provide_inputverifies/resolvesuffixAuthored-by: egg