-
Notifications
You must be signed in to change notification settings - Fork 1
feat(b-0494): circuit breaker viz — slice-2 live bus snapshot #3134
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| { | ||
| "generatedAt": "2026-05-14T13:16:13.824Z", | ||
| "source": "tools/bus/export-cb-snapshot.ts", | ||
| "busDir": "/tmp/zeta-bus", | ||
| "envelopeCount": 73, | ||
| "entries": [ | ||
| { | ||
| "model": "Otto", | ||
| "harness": "Claude Code", | ||
| "state": "CLOSED", | ||
| "consecutiveFailures": 0, | ||
| "threshold": 5, | ||
| "lastCheck": "2026-05-14T13:14:21.460Z", | ||
| "note": "Active work detected — normal operation" | ||
| }, | ||
| { | ||
| "model": "Alexa", | ||
| "harness": "Kiro / Qwen", | ||
| "state": "CLOSED", | ||
| "consecutiveFailures": 0, | ||
| "threshold": 5, | ||
| "lastCheck": "2026-05-14T13:16:13.824Z", | ||
| "note": "No recent bus activity — assuming healthy" | ||
| }, | ||
| { | ||
| "model": "Lior", | ||
| "harness": "Gemini", | ||
| "state": "CLOSED", | ||
| "consecutiveFailures": 0, | ||
| "threshold": 5, | ||
| "lastCheck": "2026-05-14T13:16:13.824Z", | ||
| "note": "No recent bus activity — assuming healthy" | ||
| }, | ||
| { | ||
| "model": "Vera", | ||
| "harness": "Codex / GPT", | ||
| "state": "CLOSED", | ||
| "consecutiveFailures": 0, | ||
| "threshold": 5, | ||
| "lastCheck": "2026-05-13T22:38:20.254Z", | ||
| "note": "Active work detected — normal operation" | ||
| }, | ||
| { | ||
| "model": "Riven", | ||
| "harness": "Grok", | ||
| "state": "CLOSED", | ||
| "consecutiveFailures": 0, | ||
| "threshold": 5, | ||
| "lastCheck": "2026-05-14T13:16:13.824Z", | ||
| "note": "No recent bus activity — assuming healthy" | ||
| } | ||
| ] | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
93 changes: 93 additions & 0 deletions
93
docs/backlog/P1/B-0494-circuit-breaker-live-bus-snapshot-2026-05-14.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| --- | ||
| id: B-0494 | ||
| priority: P1 | ||
| status: open | ||
| title: "Circuit breaker viz — slice-2: wire renderCircuitBreakerTab() to live bus snapshot" | ||
| type: feature | ||
| origin: B-0435 slice-2 (noted in PR #3133 body) | ||
| created: 2026-05-14 | ||
| last_updated: 2026-05-14 | ||
| depends_on: [B-0435] | ||
| composes_with: | ||
| - B-0401 | ||
| - B-0435 | ||
| - docs/backlog/P1/B-0213-broadcast-bus-production-hardening-2026-05-13.md | ||
| tags: [demo, circuit-breaker, alignment-ui, github-pages, html, js, bus] | ||
| --- | ||
|
|
||
| # B-0494 — Circuit breaker viz: slice-2 live bus snapshot | ||
|
|
||
| ## What | ||
|
|
||
| B-0435 slice-1 (PR #3133, merged 2026-05-14) ships the circuit breaker panel with | ||
| **static mock data**. Slice-2 wires `renderCircuitBreakerTab()` to read a committed | ||
| JSON snapshot generated from the live `/tmp/zeta-bus/` envelopes, so the panel | ||
| reflects actual agent activity rather than hardcoded values. | ||
|
|
||
| ## Approach | ||
|
|
||
| Two-part change: | ||
|
|
||
| ### Part A — `tools/bus/export-cb-snapshot.ts` | ||
|
|
||
| A TypeScript script (run via `bun tools/bus/export-cb-snapshot.ts`) that: | ||
|
|
||
| 1. Reads all non-expired envelopes from `/tmp/zeta-bus/` | ||
| 2. Groups by agent identity (`from` field, normalising surface-tagged variants | ||
| back to identity level, e.g. `otto-cli` → `otto`) | ||
| 3. Derives circuit-breaker state per agent: | ||
| - `CLOSED` — no idle heartbeats, or recent claim/work-assignment activity | ||
| - `HALF_OPEN` — some idle heartbeats (1–4) in the window | ||
| - `OPEN` — ≥5 consecutive idle heartbeats (matches `threshold: 5` in the UI) | ||
| 4. Writes output to `demo/circuit-breaker-snapshot.json` | ||
|
|
||
| ### Part B — `demo/index.html` update | ||
|
|
||
| Change `renderCircuitBreakerTab()` from synchronous (calls `buildCbMockData()`) | ||
| to async: first tries `fetch('./circuit-breaker-snapshot.json', {cache:'no-cache'})`, | ||
| falls back to `buildCbMockData()` if the file is absent or the fetch fails. | ||
| `buildCbMockData()` remains as the authoritative fallback. | ||
|
|
||
| ### Part C — committed initial snapshot | ||
|
|
||
| Run `bun tools/bus/export-cb-snapshot.ts` once and commit the output as | ||
| `demo/circuit-breaker-snapshot.json` so GitHub Pages visitors see real data | ||
| from the build moment rather than the mock. | ||
|
|
||
| ## Acceptance criteria | ||
|
|
||
| - [ ] `tools/bus/export-cb-snapshot.ts` exists and runs without errors via `bun` | ||
| - [ ] `demo/circuit-breaker-snapshot.json` is committed (generated by the script) | ||
| - [ ] `renderCircuitBreakerTab()` in `demo/index.html` tries the snapshot first, | ||
| falls back to `buildCbMockData()` — no visible change when snapshot absent | ||
| - [ ] Panel renders correctly in both paths (snapshot present and absent) | ||
| - [ ] `dotnet build -c Release` → 0 warnings, 0 errors | ||
| - [ ] `bun tsc --noEmit tools/bus/export-cb-snapshot.ts` passes (TypeScript clean) | ||
|
|
||
| ## Not in scope | ||
|
|
||
| - A live relay / HTTP server to serve fresh bus data to GitHub Pages visitors | ||
| (requires CORS / deployment plumbing — future slice) | ||
| - Automated refresh of the snapshot in CI (future slice; for now, snapshot is | ||
| committed from a local run) | ||
| - Adding new bus topics for richer circuit-breaker signals (B-0213 territory) | ||
|
|
||
| ## Pre-start checklist (backlog-item-start-gate) | ||
|
|
||
| **Prior-art search (2026-05-14):** | ||
|
|
||
| - Surfaces searched: `tools/bus/` (bus.ts, types.ts, claim.ts), `demo/index.html` | ||
| (renderCircuitBreakerTab, buildCbMockData), backlog for circuit-breaker + | ||
| live-bus + snapshot keywords | ||
| - Queries run: grep for "circuit-breaker-snapshot", "export-cb", "live.*bus.*demo" | ||
| - Results: No prior snapshot script or fetch path exists; `buildCbMockData()` is | ||
| the only current data source; `types.ts` defines the envelope schema | ||
| - Prior-art gap confirmed: output is net-new on both script and HTML sides | ||
|
|
||
| **Dependency check:** | ||
|
|
||
| - `depends_on: [B-0435]` — slice-1 merged (PR #3133, 2026-05-14) ✓ | ||
| - `composes_with: B-0213` — bus hardening is a sibling, not a blocker | ||
| - No blockers; all scaffolding in place from slice-1 | ||
|
|
||
| **Claim acquired:** otto-cli, 2026-05-14, branch `feat/b-0494-circuit-breaker-live-bus-snapshot` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,186 @@ | ||
| #!/usr/bin/env bun | ||
| /** | ||
| * export-cb-snapshot.ts — derive circuit-breaker state from live bus envelopes | ||
| * | ||
| * Reads non-expired envelopes from /tmp/zeta-bus/, groups by agent identity, | ||
| * derives CLOSED/HALF_OPEN/OPEN state, and writes demo/circuit-breaker-snapshot.json. | ||
| * | ||
| * Usage: | ||
| * bun tools/bus/export-cb-snapshot.ts [--bus-dir <path>] [--out <path>] | ||
| * | ||
| * Defaults: | ||
| * --bus-dir /tmp/zeta-bus | ||
| * --out demo/circuit-breaker-snapshot.json (relative to repo root) | ||
| * | ||
| * Run from any directory; paths resolve relative to this file's location. | ||
| * | ||
| * B-0494 slice-2. | ||
| */ | ||
|
|
||
| import { readdir, readFile, writeFile } from "fs/promises"; | ||
| import { join, resolve, dirname } from "path"; | ||
| import type { MessageEnvelope, SenderAgentId } from "./types.ts"; | ||
|
|
||
| // ── constants ───────────────────────────────────────────────────────────────── | ||
|
|
||
| const REPO_ROOT = resolve(dirname(import.meta.path), "../.."); | ||
| const DEFAULT_BUS_DIR = "/tmp/zeta-bus"; | ||
| const DEFAULT_OUT = join(REPO_ROOT, "demo/circuit-breaker-snapshot.json"); | ||
|
|
||
| /** Circuit-breaker trips at this many consecutive idle heartbeats. */ | ||
| const THRESHOLD = 5; | ||
|
|
||
| /** Canonical identity → display metadata. Order determines output order. */ | ||
| const AGENT_META: Record<string, { model: string; harness: string }> = { | ||
| otto: { model: "Otto", harness: "Claude Code" }, | ||
| alexa: { model: "Alexa", harness: "Kiro / Qwen" }, | ||
| lior: { model: "Lior", harness: "Gemini" }, | ||
| vera: { model: "Vera", harness: "Codex / GPT" }, | ||
| riven: { model: "Riven", harness: "Grok" }, | ||
| }; | ||
|
|
||
| /** Known identity prefixes in longest-match order. */ | ||
| const IDENTITIES = Object.keys(AGENT_META); | ||
|
|
||
| // ── helpers ─────────────────────────────────────────────────────────────────── | ||
|
|
||
| /** Normalise a surface-tagged sender ID back to identity level. | ||
| * e.g. "otto-cli" → "otto", "lior-gemini" → "lior", "otto" → "otto" | ||
| */ | ||
| function toIdentity(from: SenderAgentId): string | null { | ||
| for (const id of IDENTITIES) { | ||
| if (from === id || from.startsWith(id + "-")) return id; | ||
| } | ||
| return null; | ||
| } | ||
|
|
||
| async function readEnvelopes(busDir: string): Promise<MessageEnvelope[]> { | ||
| const now = Date.now(); | ||
| let files: string[]; | ||
| try { | ||
| files = await readdir(busDir); | ||
| } catch { | ||
| return []; | ||
| } | ||
| const envelopes: MessageEnvelope[] = []; | ||
| for (const file of files) { | ||
| if (!file.endsWith(".json")) continue; | ||
| try { | ||
| const raw = JSON.parse(await readFile(join(busDir, file), "utf8")) as MessageEnvelope; | ||
| if (new Date(raw.expiresAt).getTime() > now) { | ||
| envelopes.push(raw); | ||
| } | ||
| } catch { | ||
| // corrupted or partial file — skip | ||
| } | ||
| } | ||
| return envelopes; | ||
| } | ||
|
|
||
| // ── circuit-breaker derivation ──────────────────────────────────────────────── | ||
|
|
||
| type CbState = "CLOSED" | "HALF_OPEN" | "OPEN"; | ||
|
|
||
| interface CbEntry { | ||
| model: string; | ||
| harness: string; | ||
| state: CbState; | ||
| consecutiveFailures: number; | ||
| threshold: number; | ||
| lastCheck: string; | ||
| note: string; | ||
| } | ||
|
|
||
| function deriveEntry( | ||
| identity: string, | ||
| meta: { model: string; harness: string }, | ||
| envelopes: MessageEnvelope[] | ||
| ): CbEntry { | ||
| // Collect envelopes from this identity (any surface variant) | ||
| const own = envelopes | ||
| .filter(e => toIdentity(e.from) === identity) | ||
| .sort((a, b) => new Date(b.timestamp).getTime() - new Date(a.timestamp).getTime()); | ||
|
|
||
| const lastCheck = own[0]?.timestamp ?? new Date().toISOString(); | ||
|
|
||
| if (own.length === 0) { | ||
| return { | ||
| model: meta.model, | ||
| harness: meta.harness, | ||
| state: "CLOSED", | ||
| consecutiveFailures: 0, | ||
| threshold: THRESHOLD, | ||
| lastCheck, | ||
| note: "No recent bus activity — assuming healthy", | ||
| }; | ||
| } | ||
|
|
||
| // Count idle heartbeats in the non-expired window | ||
| const idleHeartbeats = own.filter( | ||
| e => e.topic === "heartbeat" && (e.payload as { status: string }).status === "idle" | ||
| ); | ||
| const consecutiveIdle = idleHeartbeats.length; | ||
|
|
||
|
AceHack marked this conversation as resolved.
Outdated
|
||
| // Any positive signal: claim, work-assignment, or working heartbeat | ||
| const hasWorkSignal = own.some( | ||
| e => | ||
| e.topic === "claim" || | ||
| e.topic === "work-assignment" || | ||
|
AceHack marked this conversation as resolved.
Outdated
|
||
| (e.topic === "heartbeat" && | ||
| (e.payload as { status: string }).status === "working") | ||
| ); | ||
|
|
||
| let state: CbState; | ||
| let note: string; | ||
|
|
||
| if (consecutiveIdle >= THRESHOLD) { | ||
| state = "OPEN"; | ||
| note = `Tripped — ${consecutiveIdle} idle heartbeats exceeded threshold (${THRESHOLD})`; | ||
| } else if (consecutiveIdle > 0) { | ||
| state = "HALF_OPEN"; | ||
| note = `${consecutiveIdle} idle heartbeat(s) — watching; threshold ${THRESHOLD}`; | ||
| } else if (hasWorkSignal) { | ||
| state = "CLOSED"; | ||
| note = "Active work detected — normal operation"; | ||
| } else { | ||
| state = "CLOSED"; | ||
| note = "Bus activity present; no idle pattern detected"; | ||
| } | ||
|
|
||
| return { | ||
| model: meta.model, | ||
| harness: meta.harness, | ||
| state, | ||
| consecutiveFailures: consecutiveIdle, | ||
|
AceHack marked this conversation as resolved.
Outdated
|
||
| threshold: THRESHOLD, | ||
| lastCheck, | ||
| note, | ||
| }; | ||
| } | ||
|
|
||
| // ── main ────────────────────────────────────────────────────────────────────── | ||
|
|
||
| const args = process.argv.slice(2); | ||
| const busDir = args.includes("--bus-dir") | ||
| ? (args[args.indexOf("--bus-dir") + 1] ?? DEFAULT_BUS_DIR) | ||
| : DEFAULT_BUS_DIR; | ||
| const outPath = args.includes("--out") | ||
| ? (args[args.indexOf("--out") + 1] ?? DEFAULT_OUT) | ||
| : DEFAULT_OUT; | ||
|
|
||
| const envelopes = await readEnvelopes(busDir); | ||
|
|
||
| const entries: CbEntry[] = Object.entries(AGENT_META).map(([id, meta]) => | ||
| deriveEntry(id, meta, envelopes) | ||
| ); | ||
|
|
||
| const snapshot = { | ||
| generatedAt: new Date().toISOString(), | ||
| source: "tools/bus/export-cb-snapshot.ts", | ||
| busDir, | ||
| envelopeCount: envelopes.length, | ||
| entries, | ||
| }; | ||
|
|
||
| await writeFile(outPath, JSON.stringify(snapshot, null, 2) + "\n"); | ||
| console.log(`Wrote ${entries.length} entries (${envelopes.length} envelopes) → ${outPath}`); | ||
|
AceHack marked this conversation as resolved.
Outdated
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.