Skip to content

docs: update startup reconciliation live-pod guard [doc-updater] - #2423

Merged
jwbron merged 1 commit into
mainfrom
egg/doc-update-tolerant-startup-reconciliation
May 6, 2026
Merged

docs: update startup reconciliation live-pod guard [doc-updater]#2423
jwbron merged 1 commit into
mainfrom
egg/doc-update-tolerant-startup-reconciliation

Conversation

@james-in-a-box

@james-in-a-box james-in-a-box Bot commented May 6, 2026

Copy link
Copy Markdown
Contributor

Update orchestrator architecture doc to reflect the tolerant startup reconciliation behavior introduced in c733f2e (#2416).

The startup reconciliation section previously described the RUNNING-pipeline recovery as: scan for stale container records → mark agents FAILED → mark pipeline FAILED. The new code adds a pipeline-scoped pod check before that scan: if any k8s pods labeled egg.pipeline.id=<id> are alive, the pipeline is left RUNNING and record drift is deferred to the running orchestrator (the #2411 false-positive fix). The doc now reflects this live-pod guard step.

docs/reference/orchestrator-cli.md already documents start_pipeline (the new MCP tool added in the same PR) — no change needed there.

Triggered by: c733f2e (Fix #2411: tolerant startup reconciliation + start_pipeline MCP verb (#2416))

Test Plan

  • Automated: doc-only change, no code paths affected
  • Manual: verify the updated description matches orchestrator/startup_reconciliation.py behavior

Authored-by: egg

Add the new live-pod guard step (#2411) to the RUNNING pipeline
reconciliation description in the orchestrator architecture doc.
The reconciler now queries k8s for pipeline-scoped pods before
marking a pipeline FAILED — if any pods are alive, the pipeline
is left RUNNING and record drift is deferred to the running
orchestrator.

Triggered by: c733f2e (#2416)

Authored-by: egg

@egg-reviewer egg-reviewer Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No agent-mode design concerns — doc-only update describing the orchestrator's live-pod guard behavior introduced in #2416.

— Authored by egg

@james-in-a-box

This comment has been minimized.

@egg-reviewer egg-reviewer Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. Single-line doc update accurately matches orchestrator/startup_reconciliation.py.

Verification:

  • Label egg.pipeline.id matches LABEL_PIPELINE_ID at orchestrator/kubernetes_client.py:74 and the query at startup_reconciliation.py:204.
  • "Any pods alive → leave RUNNING" matches the short-circuit at startup_reconciliation.py:217-223.
  • "Zero live pods → fall back to stale-record detection" matches the implicit fall-through into the phase_execution.containers / phase_execution.agents loops at startup_reconciliation.py:225-255 (the loops only run after the if pipeline_live_ids: continue guard).
  • "Query failure → fail-safe (leave RUNNING and skip)" matches the except block at startup_reconciliation.py:207-215.
  • Bullet placement is correct: it sits between the current-phase scoping bullet and the stale-record detection bullet, which mirrors the runtime ordering (un-spawned PENDING short-circuit → current-phase scoping → live-pod guard → per-container/per-agent stale checks → pipeline-level FAILED transition).
  • The #2411 reference matches the inline citation at startup_reconciliation.py:46 and :191.

Minor naming note (non-blocking): the in-code parameter is named docker_client but actually routes through KubernetesClient.list_containers → list_namespaced_pod (per the existing in-code comment at lines 194–195). The doc using "Kubernetes" / "pods" is the correct external framing — no change requested.

The PR description's claim that docs/reference/orchestrator-cli.md already documents start_pipeline is also accurate.

— Authored by egg

@james-in-a-box

This comment has been minimized.

@james-in-a-box

james-in-a-box Bot commented May 6, 2026

Copy link
Copy Markdown
Contributor Author
egg is addressing review feedback...

@james-in-a-box

james-in-a-box Bot commented May 6, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the reviews. Summary of disposition for every reviewer-raised item:

egg-reviewer[bot] (agent-mode-design) — APPROVED

  • No items raised. Nothing to address.

egg-reviewer[bot] (review) — COMMENTED ("Approve")

  • Verification of label, short-circuit, fall-through, except block, bullet placement, and #2411 citation — disagree (no change requested; reviewer confirmed accuracy).
  • Minor naming note re: in-code parameter docker_client vs. doc framing "Kubernetes"/"pods" — disagree (reviewer explicitly stated "The doc using 'Kubernetes' / 'pods' is the correct external framing — no change requested").

No code changes pushed: there are no actionable items in this review cycle.

— Authored by egg

@james-in-a-box

james-in-a-box Bot commented May 6, 2026

Copy link
Copy Markdown
Contributor Author

egg feedback addressed. View run logs

2 previous review(s) hidden.

@jwbron
jwbron merged commit de96d01 into main May 6, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Orchestrator startup-reconciliation marks live pipelines FAILED; no MCP recovery path

1 participant