Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
684 changes: 684 additions & 0 deletions .egg-state/brc-history/3288-implement-slice-2.json

Large diffs are not rendered by default.

853 changes: 853 additions & 0 deletions .egg-state/brc-history/3288-implement-slice-2.md

Large diffs are not rendered by default.

306 changes: 136 additions & 170 deletions docs/architecture/brc-memory.md

Large diffs are not rendered by default.

23 changes: 12 additions & 11 deletions docs/architecture/context-discipline.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,8 +184,9 @@ JSON file and reads it back:

Mid-phase restarts additionally need the BRC *message record* to survive so a
reseeded session can re-pull it and re-derive the #3189 anchors
([#3200](https://github.com/jwbron/egg/issues/3200) slice-7); `_write_brc_history`
historically persisted only at phase transitions.
([#3200](https://github.com/jwbron/egg/issues/3200)). `_write_brc_history`
persists the record mid-phase, not only at phase transitions, so the record is
available to a reseeded session before the next phase boundary.

## Feature flags

Expand Down Expand Up @@ -218,13 +219,13 @@ value and is NOT blindly forwarded — it still needs explicit per-pod wiring.
`EGG_RESEED_THRESHOLD` was similarly per-pod-only but is now always injected
by `_ExecutorEventSpawner.spawn_event` ([#3284](https://github.com/jwbron/egg/issues/3284)).

**Slice-9 master flag (`EGG_CONTEXT_DISCIPLINE`, shipped).** The terminal slice
([#3200](https://github.com/jwbron/egg/issues/3200) slice-9) introduced a single
master context-discipline flag that gates the whole discipline (protected-root /
queryable-environment split + threshold reseed + JIT pull) for **every**
event-pump role — producers and reviewers alike — each inlining only its own
contract and its own anchors via the role-parameterized `render_protected_root`.
It is a **kill-switch**, not the preserved fallback build:
**Master flag (`EGG_CONTEXT_DISCIPLINE`).** A single master context-discipline
flag ([#3200](https://github.com/jwbron/egg/issues/3200)) gates the whole
discipline (protected-root / queryable-environment split + threshold reseed +
JIT pull) for **every** event-pump role — producers and reviewers alike — each
inlining only its own contract and its own anchors via the role-parameterized
`render_protected_root`. It is a **kill-switch**, not the preserved fallback
build:

- **ON** → every role takes the new path; the mechanism is uniform and only the
*content* of the root differs by role.
Expand All @@ -248,8 +249,8 @@ wrapper-bash standalone case where `egg_agent` is off `PYTHONPATH`).

## Measurement (emit-only)

The discipline is "build + measure": slice-10 emits per-event measurement
surfaces from the occupancy field and the reseed decisions — window occupancy
The discipline is "build + measure": per-event measurement surfaces are emitted
from the occupancy field and the reseed decisions — window occupancy
per event, peak utilization under resume (the **primary** metric), single-event
working set vs the real window (the recursion-escalation signal), reseed
frequency per phase (the cost case rests on this being low), root-cache hit
Expand Down
666 changes: 262 additions & 404 deletions docs/architecture/orchestrator.md

Large diffs are not rendered by default.

111 changes: 54 additions & 57 deletions docs/architecture/overseer-calibration-corpus.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
# Overseer Calibration Corpus & Detector Test Harness

> **Status:** Contract definition for the overseer overhaul ([#2270](https://github.com/jwbron/egg/issues/2270), slice 1).
> This is **deliverable #1** and the regression bedrock every downstream detector plugs into.
> It satisfies acceptance criterion **AC-3**. Slices 4, 7, and 8 have all landed — the corpus xfails for those slices have flipped to strict assertions.
> The regression bedrock every overseer detector plugs into
> ([#2270](https://github.com/jwbron/egg/issues/2270)).

The overseer overhaul replaces a noisy, respawning watcher pod with an
orchestrator-side **detection plane** of deterministic detectors (slices 4, 7, 8 — all delivered).
The overseer architecture uses an orchestrator-side **detection plane** of
deterministic detectors instead of a noisy, respawning watcher pod.
The single biggest risk in that work is *recalibration regressions* — a detector
that goes quiet on a real fault, or one that floods operators with false alarms.
The **calibration corpus** is how we make that risk testable: a fixed set of
The **calibration corpus** is how that risk is made testable: a fixed set of
labelled event-stream snapshots that every detector is asserted against, so a
miscalibration shows up as a failing test instead of a 3 a.m. page.

No production detector code changes in slice 1. Slice 1 ships **only** the corpus
fixtures, the harness, and this contract.
The corpus is a test-fixtures package — it holds the corpus fixtures, the
harness, and the contract detectors are asserted against; it contains no
production detector code.

---

Expand Down Expand Up @@ -57,8 +57,7 @@ Each row (`CorpusRow`) carries:
| `incident` | Human-readable incident name |
| `pins` | The issue / defect ids this row pins |
| `expected` | The expected detector verdict: `None` for known-normal, or the expected `ExpectedFinding` for known-bad |
| `detector_key` | Which detector this row exercises (lets later slices flip their own rows; see §5) |
| `delivered_in_slice` | For known-bad rows, the slice (one of 4/7/8) that delivers the detector; `None` for known-normal |
| `detector_key` | Which detector this row exercises (resolves the row to its detector; see §5) |
| `notes` | Why this row exists / what it pins |

The fault class is **not** a standalone row field — it lives on the row's
Expand All @@ -76,7 +75,7 @@ Finding{
finding_class, # the fault class detected
severity, # info | low | medium | high
evidence, # the snapshot facts that triggered it
recommended_action, # from the bounded corrective vocabulary (slice 6)
recommended_action, # from the bounded corrective vocabulary
requires_adjudication, # bool — escalate to the on-demand adjudicator agent?
}
```
Expand All @@ -85,14 +84,14 @@ The corpus's `expected` label is the narrower `ExpectedFinding`
(`finding_class`, `severity`, `requires_adjudication`); the harness matches a
detector's `Finding` against it structurally via `match_finding`.

`requires_adjudication=True` is the seam to the adjudication plane (slice 4): the
`requires_adjudication=True` is the seam to the adjudication plane: the
orchestrator spawns a **normal** on-demand OVERSEER agent with the `Finding` +
evidence + snapshot, and consumes its structured verdict. The corpus does not
test the agent; it tests that detectors set this flag correctly.

---

## 2. The AC-3 contract: None-on-normal / Finding-on-bad
## 2. The detector contract: None-on-normal / Finding-on-bad

This is the whole point of the corpus, and it is exactly two rules:

Expand All @@ -107,17 +106,17 @@ This is the whole point of the corpus, and it is exactly two rules:
> `known-bad` row tagged for that detector.*

A "calibration test" for a detector is simply: run it over its corpus rows and
assert rule 1 and rule 2. **Corpus-tested == shippable.** Every §5 detector
(slice 8) and every calibrated fix (slice 7) ships *only* if it passes this
contract — that is the structural guard against a new false-positive flood.
assert rule 1 and rule 2. **Corpus-tested == shippable.** A detector ships *only*
if it passes this contract — that is the structural guard against a new
false-positive flood.

---

## 3. The seed corpus rows

Slice 1 lands these rows. They are the concrete failure modes that motivated the
overhaul; each is encoded in **both** a `known-normal` and a `known-bad` variant
where the distinction is the whole point.
These rows are the concrete failure modes that motivated the detection plane;
each is encoded in **both** a `known-normal` and a `known-bad` variant where the
distinction is the whole point.

| Row | `finding_class` | Why it exists |
|-----|-----------------|---------------|
Expand All @@ -128,11 +127,11 @@ where the distinction is the whole point.
| #2222/#2224 branch-divergence | `branch_divergence` | Detector must use **ancestor-of-`origin/main` OR patch-id** match, not `(#NNNN)` subject-regex matching |
| #2948 transient kubelet eviction | `container_death` | A transient evict/reschedule must NOT cascade to a producer-permanent-death `FAILED` |

Downstream slices add rows for their own detectors (the full §5 survey in slice 8
was the largest addition — 25 coverage-gap detectors: 24 across 8 modules in
Beyond this seed set, each detector contributes the rows it owns. The §5 survey
is the largest contributor — 25 coverage-gap detectors: 24 across 8 modules in
``health_checks/tier1/`` plus ``detect_overseer_self_health`` in
``overseer/self_monitor.py``). The seed set establishes the shape and the contract;
later slices extend, never reshape.
``overseer/self_monitor.py``. New rows extend the corpus without reshaping the
record shape or the contract.

---

Expand All @@ -152,45 +151,43 @@ one bucket:
| `undelivered` | Rows whose detector is not yet registered (counted for visibility) |

From these the scoreboard derives `precision` (TP / (TP + FP)) and `recall`
(TP / (TP + FN)). The slice-1 acceptance signal is `false_positive == 0` and
`precision == 1.0` (the baseline null detector never over-fires); with all
slices (4/7/8) delivered, recall is at 1.0 and precision must remain there.
This precision/recall tally is distinct from the per-row pytest pass/`xfail`
bookkeeping in the harness (see §5).
(TP / (TP + FN)). The acceptance signal is `false_positive == 0` and
`precision == 1.0` (a registered detector never over-fires), with `recall == 1.0`
once every detector the corpus pins is registered. This precision/recall tally is
distinct from the per-row pytest pass/`xfail` bookkeeping in the harness (see §5).

---

## 5. The xfail → strict flip convention (red → green workflow)

Slice 1 defined corpus rows for detectors that slices 4, 7, and 8 had not built
yet. Those rows cannot pass against code that does not exist, so the harness
marks each known-bad row whose detector is unregistered as **`xfail`**
(expected-to-fail). This is deliberate: the corpus is the *spec*, written before
the detector.
A corpus row can exist for a detector that is not yet built — the corpus is the
*spec*, and a row can be written before its detector. Such a row cannot pass
against code that does not exist, so the harness marks each known-bad row whose
detector is unregistered as **`xfail`** (expected-to-fail).

The marker is applied **automatically and conditionally** in `_row_param`: a row
is xfailed iff `resolve_detector(row.detector_key) is None` and the row is
known-bad. The xfail is registered with `strict=False` — intentional for slice-1
build-greenness, so the baseline run is green (`10 passed, 6 xfailed`) without
any `xpass` failing the build. Nobody hand-marks individual rows.

The workflow each downstream slice follows is **red → green**:

1. **Red (slice 1).** The row for a future detector exists; because no detector
resolves for its `detector_key`, the harness marks it `xfail`. It documents the
exact `known_normal`/`known_bad` behaviour the future detector must satisfy. The
build stays green because `xfail` is tolerated.
2. **Implement (slice 4/7/8).** The owning slice builds the detector and registers
it via `register_detector`. The moment it resolves, `_row_param` no longer
applies the xfail marker — the row becomes a strict assertion automatically (the
marker "evaporates" because the detector resolves).
known-bad. The xfail is registered with `strict=False`, so a build with
unimplemented detectors stays green (e.g. `10 passed, 6 xfailed`) without any
`xpass` failing the build. Nobody hand-marks individual rows.

A row therefore moves **red → green** as its detector is built:

1. **Red.** The row exists but no detector resolves for its `detector_key`, so the
harness marks it `xfail`. It documents the exact `known_normal`/`known_bad`
behaviour the detector must satisfy, and the build stays green because `xfail`
is tolerated.
2. **Implement.** The detector is built and registered via `register_detector`.
The moment it resolves, `_row_param` no longer applies the xfail marker — the
row becomes a strict assertion automatically (the marker "evaporates" because
the detector resolves).
3. **Green.** From then on the row is a hard regression gate: any later change that
breaks the detector fails the build.

> **Convention for downstream slices:** when you land a detector, register it so
> the harness resolves it for the row's `detector_key`; the rows then flip to
> strict on their own in the *same* change. A detector merged without registering
> leaves its rows stuck at `xfail` — treat that as an incomplete slice.
> **Convention:** when you land a detector, register it so the harness resolves it
> for the row's `detector_key`; the rows then flip to strict on their own in the
> *same* change. A detector merged without registering leaves its rows stuck at
> `xfail` — treat that as incomplete.

This keeps the corpus honest: every detector the design promises has a row from
day one (red), and every detector that ships is permanently regression-locked
Expand All @@ -208,15 +205,15 @@ day one (red), and every detector that ships is permanently regression-locked
assert-over-rows driver that runs each detector-under-test across its corpus rows
and emits the §4 scoreboard.
- **Detectors:** pure `EventStreamSnapshot -> Optional[Finding]` functions in the
slice-4 framework extending `health_checks/tier1/`. Each registers the rows it
owns and is run against them by the harness.
detection-plane framework extending `health_checks/tier1/`. Each registers the
rows it owns and is run against them by the harness.
- **Adjudication seam:** detectors only set `requires_adjudication`; the
orchestrator (slice 4) owns spawning the on-demand OVERSEER agent and consuming
its verdict. The corpus tests detectors, not the agent.
orchestrator owns spawning the on-demand OVERSEER agent and consuming its
verdict. The corpus tests detectors, not the agent.

## Related

- Issue [#2270](https://github.com/jwbron/egg/issues/2270) — overseer overhaul umbrella (AC-3, deliverable #1)
- Issue [#2270](https://github.com/jwbron/egg/issues/2270) — overseer overhaul umbrella
- [Health Check Framework](../../orchestrator/health_checks/README.md) — the Tier-1 framework the detection plane extends
- `docs/architecture/overseer.md` — overseer architecture (delivered shape; written in slice 9)
- `docs/architecture/overseer.md` — overseer architecture
- #3230 (false stall), #2242 (heartbeat-stall), #2222/#2224 (branch-divergence), #2948 (transient eviction) — the calibration motivating cases
Loading
Loading