docs: update startup reconciliation live-pod guard [doc-updater] - #2423
Conversation
Add the new live-pod guard step (#2411) to the RUNNING pipeline reconciliation description in the orchestrator architecture doc. The reconciler now queries k8s for pipeline-scoped pods before marking a pipeline FAILED — if any pods are alive, the pipeline is left RUNNING and record drift is deferred to the running orchestrator. Triggered by: c733f2e (#2416) Authored-by: egg
There was a problem hiding this comment.
No agent-mode design concerns — doc-only update describing the orchestrator's live-pod guard behavior introduced in #2416.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Approve. Single-line doc update accurately matches orchestrator/startup_reconciliation.py.
Verification:
- Label
egg.pipeline.idmatchesLABEL_PIPELINE_IDatorchestrator/kubernetes_client.py:74and the query atstartup_reconciliation.py:204. - "Any pods alive → leave RUNNING" matches the short-circuit at
startup_reconciliation.py:217-223. - "Zero live pods → fall back to stale-record detection" matches the implicit fall-through into the
phase_execution.containers/phase_execution.agentsloops atstartup_reconciliation.py:225-255(the loops only run after theif pipeline_live_ids: continueguard). - "Query failure → fail-safe (leave RUNNING and skip)" matches the
exceptblock atstartup_reconciliation.py:207-215. - Bullet placement is correct: it sits between the current-phase scoping bullet and the stale-record detection bullet, which mirrors the runtime ordering (un-spawned PENDING short-circuit → current-phase scoping → live-pod guard → per-container/per-agent stale checks → pipeline-level FAILED transition).
- The
#2411reference matches the inline citation atstartup_reconciliation.py:46and:191.
Minor naming note (non-blocking): the in-code parameter is named docker_client but actually routes through KubernetesClient.list_containers → list_namespaced_pod (per the existing in-code comment at lines 194–195). The doc using "Kubernetes" / "pods" is the correct external framing — no change requested.
The PR description's claim that docs/reference/orchestrator-cli.md already documents start_pipeline is also accurate.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
| egg is addressing review feedback... |
|
Thanks for the reviews. Summary of disposition for every reviewer-raised item:
No code changes pushed: there are no actionable items in this review cycle. — Authored by egg |
|
egg feedback addressed. View run logs 2 previous review(s) hidden. |
Update orchestrator architecture doc to reflect the tolerant startup reconciliation behavior introduced in c733f2e (#2416).
The startup reconciliation section previously described the RUNNING-pipeline recovery as: scan for stale container records → mark agents FAILED → mark pipeline FAILED. The new code adds a pipeline-scoped pod check before that scan: if any k8s pods labeled
egg.pipeline.id=<id>are alive, the pipeline is left RUNNING and record drift is deferred to the running orchestrator (the #2411 false-positive fix). The doc now reflects this live-pod guard step.docs/reference/orchestrator-cli.mdalready documentsstart_pipeline(the new MCP tool added in the same PR) — no change needed there.Triggered by: c733f2e (Fix #2411: tolerant startup reconciliation + start_pipeline MCP verb (#2416))
Test Plan
orchestrator/startup_reconciliation.pybehaviorAuthored-by: egg