diff --git a/docs/architecture/orchestrator.md b/docs/architecture/orchestrator.md index bbc2b6d6bc..f2a41544b7 100644 --- a/docs/architecture/orchestrator.md +++ b/docs/architecture/orchestrator.md @@ -36,7 +36,13 @@ The orchestrator persists pipeline state using a dedicated git worktree on an or This differs from agent worktrees (managed by the gateway for agent isolation). The orchestrator manages its own state worktree independently. -See `orchestrator/state_store.py` for implementation details. +**Startup reconciliation:** + +On orchestrator restart, orphaned container state is automatically recovered. For each pipeline showing `status=RUNNING`, any agent/container whose container ID is absent from the live Docker container set is marked `FAILED`. If at least one stale entry is found, the pipeline itself is marked `FAILED` with an error message instructing operators to restart via `POST /pipelines/{id}/start`. + +This prevents pipelines from being stuck in a `RUNNING` state indefinitely after a crash. Operators (or CI systems) can detect the `FAILED` status and restart the pipeline using the existing restart endpoint, which preserves worktrees and phase state while re-spawning containers. + +See `orchestrator/state_store.py` and `orchestrator/startup_reconciliation.py` for implementation details. ## Network Mode diff --git a/docs/development/STRUCTURE.md b/docs/development/STRUCTURE.md index 8f751164cc..89f73a215e 100644 --- a/docs/development/STRUCTURE.md +++ b/docs/development/STRUCTURE.md @@ -95,6 +95,7 @@ orchestrator/ ├── resilience.py # Retry and error recovery ├── sandbox_template.py # Sandbox container template ├── sse.py # Server-Sent Events streaming for pipeline visualization +├── startup_reconciliation.py # Startup reconciliation for orphaned containers ├── state_store.py # Git-backed pipeline state ├── status_reporter.py # Real-time status reporter for collaborators ├── unified_sse.py # Unified SSE stream for all pipelines