HEL-3113: supervise rolling telemetry-hole review cadence - #9
Conversation
|
Closing and reopening to retrigger the pull_request CI workflow, which did not enqueue on the initial open event. No branch or head change. |
…uarantined corrupt DB The gateway dispatcher's telemetry review tick ran unconditionally before the per-board dispatch tick and read the same board DB file. When that file was corrupt, both paths hit the same sqlite corruption error, but only the per-board dispatch tick had corruption-aware handling (fingerprint quarantine + traceback-free logging via _is_corrupt_board_db_error). The telemetry review tick's bare 'except Exception: logger.exception(...)' ran first and unconditionally logged a full traceback for the exact scenario test_gateway_dispatcher_disables_corrupt_board_without_traceback asserts must stay traceback-free. Fix: move the telemetry review tick after _tick_once() so it observes the per-board dispatch outcomes, skip the review entirely when no board produced a successful dispatch result this tick (i.e. every board's DB looked unusable), and route any remaining exception through the same _is_corrupt_board_db_error() classification the dispatch tick already uses so a genuinely corrupt board never gets a second, redundant full-traceback log line. Fixes exact-head CI failure on PR #9 (test slice 8/8): tests/hermes_cli/test_kanban_core_functionality.py::test_gateway_dispatcher_disables_corrupt_board_without_traceback[sqlite] tests/hermes_cli/test_kanban_core_functionality.py::test_gateway_dispatcher_disables_corrupt_board_without_traceback[guard] Linear: HEL-3113
TRC VERDICT: GO-WITH-CHANGES — PR #9 (HEL-3113) at exact head
|
| session | model | in | out | reasoning | est cost |
|---|---|---|---|---|---|
20260730_032600_26a095 |
anthropic/claude-fable-5 | 536,978 | 19,960 | 0 | $5.5328 (est) |
| TOTAL | — | 536,978 | 19,960 | $5.5328 |
profile: tessa-cole · cost estimated unless marked (act). Recorded per the tokens-to-value deliverable.
CPTC actual: compare these real tokens with the predicted Complexity Points on the technical-scope sub-issue.
RRA MERGE-LANE DISPOSITION: HOLD — RETURN TO TRC (head drift)PR #9 (HEL-3113) is NOT merged. The TRC GO-WITH-CHANGES verdict (comment 5129666040) was issued at exact head Independently re-verified at merge time (2026-07-30, RRA)
Risk note driving the returnThe main-side delta merged into this branch ( Disposition
— Rhea Ramos · credentials: helios-agent-rra (RRA) · agent: rhea-ramos 🪙 Token usage (from Hermes state.db — real per-session data)
profile: rhea-ramos · cost estimated unless marked (act). Recorded per the tokens-to-value deliverable. CPTC actual: compare these real tokens with the predicted Complexity Points on the technical-scope sub-issue. |
TRC RE-REVIEW VERDICT: GO-WITH-CHANGES — PR #9 (HEL-3113) at exact head
|
| session | model | in | out | reasoning | est cost |
|---|---|---|---|---|---|
20260730_040633_636d3a |
anthropic/claude-fable-5 | 1,809,165 | 34,360 | 0 | $17.3173 (est) |
| TOTAL | — | 1,809,165 | 34,360 | $17.3173 |
profile: tessa-cole · cost estimated unless marked (act). Recorded per the tokens-to-value deliverable.
CPTC actual: compare these real tokens with the predicted Complexity Points on the technical-scope sub-issue.
RRA merge lane round 2 — PR #9 MERGED at exact TRC-verified headMerged: SSC-ENG/hermes-agent#9 — squash merge Governed merge evidence
Linear stateHEL-3113 remains Ready for Production — correct post-merge/pre-deploy state per team precedent (HEL-3114). Merged is not deployed; deployment/LIVE verification is a separate step. Non-gating follow-ups (unchanged)
— Rhea Ramos · credentials: helios-agent-rra (RRA) · agent: rhea-ramos 🪙 Token usage (from Hermes state.db — real per-session data)
profile: rhea-ramos · cost estimated unless marked (act). Recorded per the tokens-to-value deliverable. CPTC actual: compare these real tokens with the predicted Complexity Points on the technical-scope sub-issue. |
|
Follow-up filed and delivered: #14 (fix/HEL-3113-per-board-review-isolation) implements per-board corrupt-DB isolation in run_scheduled_reviews per the GO-WITH-CHANGES verdict above. Exact-head CI green (55fc153c9). |
HEL-3113 follow-up from TRC GO-WITH-CHANGES on PR #9: run_scheduled_reviews() had no per-board exception isolation. In a multi-board install, one corrupt board's DatabaseError raised out of the loop into the dispatcher watcher's generic logger.exception, re-introducing traceback noise and aborting reviews for every board later in iteration order. - Add hermes_cli.kanban_db.is_corrupt_db_error() as the single source of truth for corrupt-board-file classification (matches KanbanDbCorruptError plus the two raw sqlite3 message shapes). gateway._is_corrupt_board_db_error() now delegates to it so the dispatcher and telemetry cadence agree. - run_scheduled_reviews() wraps each board's review in its own try/except: a classified corrupt board logs one error line (no traceback) and continues to the next board; a genuinely unexpected error still gets logger.exception, scoped to that board only. - run_scheduled_reviews(skip_slugs=...) lets callers skip boards already quarantined this tick by dispatch without a redundant connect attempt; the gateway watcher wires disabled_corrupt_boards through as skip_slugs. - Tests: two-board fixture (healthy default + corrupt broken) asserts the healthy board's artifact is written/persisted, the corrupt board is skipped without a traceback, and skip_slugs bypasses the open attempt entirely.
…14) HEL-3113 follow-up from TRC GO-WITH-CHANGES on PR #9: run_scheduled_reviews() had no per-board exception isolation. In a multi-board install, one corrupt board's DatabaseError raised out of the loop into the dispatcher watcher's generic logger.exception, re-introducing traceback noise and aborting reviews for every board later in iteration order. - Add hermes_cli.kanban_db.is_corrupt_db_error() as the single source of truth for corrupt-board-file classification (matches KanbanDbCorruptError plus the two raw sqlite3 message shapes). gateway._is_corrupt_board_db_error() now delegates to it so the dispatcher and telemetry cadence agree. - run_scheduled_reviews() wraps each board's review in its own try/except: a classified corrupt board logs one error line (no traceback) and continues to the next board; a genuinely unexpected error still gets logger.exception, scoped to that board only. - run_scheduled_reviews(skip_slugs=...) lets callers skip boards already quarantined this tick by dispatch without a redundant connect attempt; the gateway watcher wires disabled_corrupt_boards through as skip_slugs. - Tests: two-board fixture (healthy default + corrupt broken) asserts the healthy board's artifact is written/persisted, the corrupt board is skipped without a traceback, and skip_slugs bypasses the open attempt entirely. Co-authored-by: SSC-ENG <225143396+SSC-ENG@users.noreply.github.com>
Linear: HEL-3113\nGitHub issue: SSC-Engineering/HELIOS-AGENTIC-OS#725\n\nSummary:\n- run the existing deterministic rolling 48-hour hole reader once per nominal 00:15/12:15 America/Phoenix boundary from the supervised gateway dispatcher\n- persist durable JSON and derived Markdown under each board review directory, with on-demand replay using the same runner\n- emit CRITICAL REVIEW.MISSED_RUN when a prior cadence boundary is absent\n- keep workflow-stage and idle-capacity states UNKNOWN and HAA/Linear-estimate states UNINSTRUMENTED until HEL-3110/observer writers land\n\nTests:\n- python -m pytest -q tests/hermes_cli/test_kanban_telemetry.py tests/hermes_cli/test_kanban_cli_dispatch_passthrough.py tests/gateway/test_kanban_watchers_mixin.py tests/gateway/test_kanban_notifier_watcher_dispatch_gate.py (18 passed)\n- isolated regression tests for rate-limit and WAL fallback (2 passed)\n- python -m ruff check on all changed Python files (passed)\n- isolated CLI behavioral run produced and persisted JSON+Markdown artifacts against a copied board DB\n\nDeployment impact: no production deploy or live schedule mutation. The cadence becomes active when this code is released and the gateway dispatcher is running.\nRollback: revert commit 0bbfc84.