Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/guides/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -387,6 +387,7 @@ Normal response:
"components": {
"state_store": {"/home/egg/repos/egg": {"status": "ok"}},
"state_store_summary": "ok",
"message_store": {"status": "ok"},
"docker": "unknown"
},
"healthy_since": "2026-04-27T12:00:00+00:00",
Expand All @@ -396,6 +397,8 @@ Normal response:

`components.state_store` is a per-repo map keyed by repo path (#2176), so multi-repo deployments surface every wedged repo in a single response rather than just the first one the probe loop hit. Each value is `{"status": "ok"}` or `{"status": "error", "error": "<git error>"}`. `components.state_store_summary` is the human-readable aggregate (`"ok"`, `"probe-skipped: ..."`, or `"N/M repos wedged: <paths>"`) — useful for log lines and skip cases where the per-repo map is empty.

`components.message_store` surfaces whether the auto→memory fallback fired. In the common case it is `{"status": "ok"}`. If `EGG_MESSAGE_STORE_BACKEND` is unset or `"auto"` and Redis was unreachable at startup, the orchestrator falls back to the in-memory store and this field becomes `{"status": "degraded", "reason": "MESSAGE_STORE_AUTO_FALLBACK_TO_MEMORY"}`; the top-level `status` is `"degraded"` in that case. The in-memory store works for single-instance deployments, but a worker restart between BRC events will drop in-flight messages (see #3076). Setting `EGG_MESSAGE_STORE_BACKEND=memory` explicitly (dev/test intent) does **not** set degraded — only the unintentional auto fallback does.

`healthy_since` is the timestamp of the most recent healthy → unhealthy → healthy transition (or process start if the orchestrator has been healthy since boot); use it to distinguish "stable since boot" from "just recovered." Transitions are recorded at BG-thread cadence (every 15s by default, tunable via `EGG_ORCH_STATE_STORE_PROBE_INTERVAL`) via the probe's `on_observation` callback, and `/api/v1/health` request hits also drive the tracker on the staleness-corrected value — so wedge cycles between sporadic operator/dashboard hits are still observed, and a wedged BG thread can surface as an unhealthy transition that the BG itself cannot record. `probe.age_seconds` is how long ago the background probe last ran — values consistently above ~2× the probe interval indicate the BG thread itself has wedged, and `/api/v1/ready` will flip to 503 even if the cached observation was healthy.

If the state-store worktree is wedged (for example, after a state-volume reset that left a stale `.git/worktrees/` admin dir), `status` becomes `"degraded"` and `components.state_store[<repo>]["error"]` shows the underlying git error for each wedged repo (`components.state_store_summary` carries the aggregate "N/M repos wedged: <paths>" string). The probe is **curative**: each tick of the background thread attempts to remove the stale admin dir and retry `git worktree add` on **every** wedged repo, so a wedge on repo A no longer hides an independent wedge on repo B. The curative cadence (15s) is now independent of kubelet probe traffic — operators do not need to do anything; under normal conditions the orchestrator self-heals within a few BG-thread cycles. See [Orchestrator status: degraded](#orchestrator-status-degraded-state-store-wedge) if degraded persists.
Expand Down
Loading