Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,7 @@ The orchestrator exposes an MCP server (port 9850) for controlling pipelines fro
- `list_containers` / `get_container_logs`: debugging
- `send_message` / `get_consensus_status`: agent coordination
- `validate_config`: validate a pipeline config without creating a pipeline
- `start_pipeline`: recover a non-RUNNING pipeline (FAILED, AWAITING_HUMAN with resolved decisions, or PENDING)
- `advance_phase` / `start_phase` / `complete_phase`: phase management
- `populate_contract`: populate SDLC contract from plan draft

Expand Down
5 changes: 3 additions & 2 deletions docs/architecture/orchestrator.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,10 @@ On orchestrator restart, orphaned container state is automatically recovered:

1. **RUNNING pipelines**: For each pipeline showing `status=RUNNING`, the reconciliation process recovers orphaned state:
- **Un-spawned PENDING phase** (crash-between-submit-and-spawn): If the current phase is `PENDING` with `started_at=None` and no containers or agents, the orchestrator crashed before `_run_pipeline` reached `executor.spawn_all`. The pipeline is immediately marked `FAILED` with an actionable message; no container scanning is performed for this pipeline. Operators restart via `POST /pipelines/{id}/start`.
- Scans only the **current phase** for stale containers. This applies when the phase has containers or agents — the un-spawned PENDING case above is handled separately. Containers from prior phases are intentionally terminated and their absence is expected — checking all phases caused false `FAILED` transitions when the orchestrator restarted mid-pipeline.
- **Live-pod safety net** (#2411): before any stale-record scan, queries Kubernetes for pods labeled `egg.pipeline.id=<id>`. If **any** pods are alive for the pipeline, it is left `RUNNING` and the per-agent scan below is skipped — record drift between persisted state and the new orchestrator process's view of pods is expected after a restart and is reconciled by the running orchestrator. If the pod query itself fails, the pipeline is also left `RUNNING` (fail-safe: defer to the running orchestrator's reconciliation rather than risk a false-positive `FAILED` transition). Only when the pipeline has **zero live pods** does reconciliation fall through to the per-agent stale scan below.
- **Stale-record scan scope** (zero-live-pod path only): scans only the **current phase** for stale containers. This applies when the phase has containers or agents — the un-spawned PENDING case above is handled separately. Containers from prior phases are intentionally terminated and their absence is expected — checking all phases caused false `FAILED` transitions when the orchestrator restarted mid-pipeline.
- Any agent in the current phase whose pod is absent from the live Kubernetes pod set is marked `FAILED`.
- If at least one stale entry is found, the pipeline itself is marked `FAILED` with an error message instructing operators to restart via `POST /pipelines/{id}/start`.
- If at least one stale entry is found, the pipeline itself is marked `FAILED` with an error message instructing operators to restart via `POST /pipelines/{id}/start` (the `start_pipeline` MCP tool).

2. **AWAITING_HUMAN pipelines**: For each pipeline showing `status=AWAITING_HUMAN` with no pending decisions (orphaned after a restart where the decision was already resolved), the pipeline is marked `FAILED` with an error message instructing operators to restart via `POST /pipelines/{id}/start`. The restart endpoint will automatically recover by parsing the latest phase_gate resolution and either advancing to the next phase (approved) or resetting the current phase for re-run (request_changes/change_approach).

Expand Down
Loading