feat(observability): add bounded residual runtime diagnostics

[Skywind5487]

Closes #114.

Completion claim

Reconstructs the Phase-1 runtime-observability capability as a bounded residual diagnostic extension on current fork main, without replaying the historical telemetry stack.

Intent-readable history

The branch no longer carries a single mega-squash. The final tree is split into two intent commits (code + tests + docs travel together per intent):

1. feat(observability): process-lifetime bounded delivery boundary (07581832b)
   • MonitoringEmitter.emit_buffered() — an already-opted-in producer whose observation lifetime predates its exporter enqueues into the existing bounded queue without enabling unrelated producers or creating a second store.
   • subscribe() never starts dispatch on a non-empty queue — the buffered residual survives the gateway's two-sink assembly (span streamer first, diagnostic-log streamer second) and is drained by the ordinary post-assembly snapshot emit.
   • Dispatcher leaves buffered events queued when the subscriber set is empty.
   • hermes_cli installs the dormant observation boundary for the process lifetime (fail-open, no config/network/disk I/O).
2. feat(observability): residual runtime fact vocabulary (8a69bebad)
   • STABLE_FACTS closed residual subset (stream / SQLite-persistence / delivery); lifecycle and tool remain upstream-owned (Relay).
   • Dormant static-log normalizer over exact code-owned templates; never fuzzy-matches or stringifies args.
   • SQLite persistence exports only the frozen coarse bucket; producer-side redaction contract.
   • Single enablement gate; telemetry stays fail-open.

Review fix (blocking, folded in)

PR #122 review found subscribe() could start dispatch on the first (span-filtered) subscriber and lose a pre-buffered gateway_diagnostic before the diagnostic-log streamer attached. Fixed: subscribe only enables the plane; the initial gateway-health snapshot starts dispatch after the full fan-out is assembled. Regression test_startup_fact_survives_multi_sink_subscribe_order exercises the real order: buffered residual → span subscriber → diagnostic subscriber → initial snapshot, asserting the residual reaches only its owning sink. Over-engineering cleanup folded in: dropped runtime-unused DOMAIN_OWNERSHIP + its exact-snapshot test (YAGNI; Relay behavioral test already proves non-duplication) and the duplicate enabled() gate.

Verification

• tests/hermes_cli/test_runtime_observability.py — fact vocabulary, redaction, fail-open, template anchoring, multi-sink ordering.
• tests/monitoring/test_emitter.py — subscribe-never-starts-dispatch, hot path, unsubscribe.
• tests/monitoring/test_otlp_exporter.py — sink/filter compatibility.
• Local: 20 passed, 1 skipped (otlp extra).

[github-actions]

૮ >ﻌ< ა ci review

_{ran on 5dbee48 — feat(observability): residual runtime fact vocabulary}

⚠️ Warnings

CI timings · View report · View job

Wall time 4m28s vs 3m25s (+30.7%). 12 job(s) slower, 9 faster, 3 unchanged.

• Python tests / Run tests slice 8/12: -35.0s
• Python tests / Run tests slice 3/12: -11.0s
• OSV scan / Scan lockfiles / osv-scan: +8.0s
• Python tests / Run tests slice 5/12: +7.0s
• Python lints / ruff enforcement (blocking): +7.0s

---

OSV vulnerability scan · View job

5 known vulnerabilities found in pinned dependencies.

• CVE-2026-67213 in package-lock.json
• CVE-2026-67213 in website/package-lock.json
• CVE-2026-70608 in package-lock.json
• CVE-2026-56876 in package-lock.json
• CVE-2026-70606 in package-lock.json

How to fix:

Review the findings in the Security tab. Update the affected dependencies if a patched version is available.

[Skywind5487]

Review follow-up for the #114 lifetime finding:

• Removed on_session_start as the residual-observability bootstrap entirely.
• hermes_cli package import now installs the adapter as a dormant, idempotent process-lifetime boundary before SessionDB/gateway runtime construction; install does no config/network/disk work.
• Fixed the second half of the same lifetime issue: an enabled startup residual fact can predate the OTLP subscriber. MonitoringEmitter.emit_buffered() queues only that explicitly opted-in residual event in the existing bounded queue. It does not set normal emitter enablement, so unrelated monitoring producers remain disabled.
• Preserved upstream lazy dispatcher start: an empty subscribe does not start the thread; a pre-buffered event or later first event does.
• flush() with zero subscribers is a no-op, so no-exporter deployments do not gain a shutdown timeout.
• Refreshed all seven exact log templates against current upstream main@5dd15872a6878a19b9b5478b6968b38f48dd311f; no wording drift today.
• Addressed the ponytail maintenance warning with a source-anchor regression: the focused test parses the four real emit-site files with Python AST and asserts every mapped literal still exists at a logger call. A wording-only upstream change now fails CI instead of silently removing a structured event.

Final branch shape after refresh: current fork main@243352e7b8bddc9f33eba1b6506810f8dd88beaa → one squashed feature commit. PR remains draft; CI on the final head is the execution authority.

[Skywind5487]

🔍 Summary

HOLD — one blocking startup-order race breaks the PR's core process-lifetime guarantee. The prior-art split, privacy boundary, and CI are otherwise in good shape.

Ponytail / over-engineering pass

• hermes_cli/observability/runtime_observability.py — DOMAIN_OWNERSHIP is runtime-unused ownership metadata duplicated by the docs and an exact snapshot test. The behavioral Relay-ownership test already proves the important non-duplication invariant. yagni: delete the constant + exact dict assertion.
• _RuntimeDiagnosticLogHandler.emit() / _emit_fact() — enabled state is checked twice for every mapped record, so the same config gate is read twice. shrink: keep one gate.

net: ~-18 lines possible (over-engineering pass only).

🔴 Critical

• agent/monitoring/emitter.py: MonitoringEmitter.subscribe() — when a residual event is pre-buffered, the first subscribe() sees a non-empty queue and immediately starts _run(). But the real gateway startup registers the span streamer first (otlp_exporter.start_streaming(..., event_filter=_gateway_health_event)) and the diagnostic-log streamer second. _gateway_health_event() accepts only gateway_health / cron_execution, so the dispatcher can dequeue the early gateway_diagnostic during that gap and the first subscriber filters it out before the diagnostic subscriber exists. This directly contradicts the PR's stated multi-sink contract ("does not start dispatch from subscribe()") and can lose the exact startup diagnostic this revision is meant to preserve.

  Suggestion: do not start dispatch from subscribe() just because the queue is non-empty. In the current gateway path, the ordinary initial snapshot emitted after both subscribers are attached already starts the dispatcher and drains the buffered residual through the complete fan-out. Add a deterministic regression for the actual order: buffered residual → span subscriber → diagnostic subscriber → initial snapshot → verify residual reaches diagnostic sink and snapshot reaches span sink.

⚠️ Warnings

• tests/hermes_cli/test_runtime_observability.py:test_startup_fact_survives_until_first_monitoring_subscriber — this test uses only one delivered.extend subscriber, so it does not reproduce the multi-sink startup order claimed in the PR body and cannot catch the race above. AGENTS.md explicitly asks for real-path/E2E validation where mocks or partial wiring can hide integration bugs. Exercise the two real subscriber semantics (or start_gateway_health_export() with a deterministic fake SDK/sink) instead of stopping at the emitter-only case.

💡 Suggestions

• Keep the exact-template source-anchor invariant; unlike a catalog snapshot, it checks a real relationship the adapter depends on: every configured exact template must still exist at its owned emit site.

✅ Looks Good

• Prior-art disposition checks out: upstream NousResearch#64536, NousResearch#73120, NousResearch#68882, NousResearch#73544 are merged; NousResearch#48871, NousResearch#51714, NousResearch#80359 remain open/unmerged; NousResearch#9596, NousResearch#22095, NousResearch#65077 are closed/unmerged. NousResearch#73120 is specifically the restored Relay landing after the earlier revert cycle.
• Upstream main has advanced from the PR's checked 5dd15872… to cb0fd836…, but the four intervening commits touch Desktop/package files only; none touch the relevant monitoring/stream/SQLite/delivery seams, so the preflight conclusion remains current.
• Closed residual vocabulary, no raw rendered log export, frozen SQLite buckets, fail-open boundaries, and no duplicate lifecycle/tool plane all match #114's intended scope.
• CI and Docker workflow runs are green on cfb8045; the blocker is an untested interleaving, not a currently failing check.

Standards

1 finding. The integration-sensitive startup lifetime contract is not validated through the actual two-subscriber assembly path; the focused test proves only single-subscriber buffering. Worst issue: real gateway fan-out ordering is untested and currently races.

Spec

1 blocking finding. #114 requires a tested, complete optional observability subsystem. The current subscribe() behavior can silently discard a pre-exporter residual diagnostic before its owning diagnostic sink attaches, so the startup-lifetime completion claim is not yet satisfied.

Verdict: HOLD / fix then re-review.

[Skywind5487]

Review fix applied — addressing the HOLD on the startup-order race.

New head: e7bdce0aa (fork/runtime-observability).

• MonitoringEmitter.subscribe() no longer starts dispatch when the queue is non-empty. This closes the race where a pre-buffered residual gateway_diagnostic could be dequeued while only the span streamer (filter gateway_health/cron_execution) was attached, and lost before the diagnostic streamer subscribed. The first ordinary emit() after the full subscriber set attaches starts dispatch and drains buffered events through the complete fan-out.
• Removed the runtime-unused DOMAIN_OWNERSHIP constant + snapshot test (yagni), and collapsed the double enabled() read to a single gate.
• Regression coverage now exercises the actual two-sink order: buffered residual → span subscriber → diagnostic subscriber → initial snapshot → residual reaches diagnostic sink, snapshot reaches span sink.
• Added emitter-level test_subscribe_never_starts_dispatch_on_buffered_events.

Validation: hermetic runner 20/20 focused tests; Ruff + compileall clean. Requesting re-review.

[Skywind5487]

Second review round — flush() residual + history cleanup applied.

New head: 5dbee4847 (forced update; branch history rewritten to 2 behavior intents, no review chronology).

flush() residual fixed

MonitoringEmitter.flush() now returns immediately when the dispatcher has not started:

if timeout <= 0 or not self._subscribers or not self._started:
    return

With a subscriber attached but _started == False (partial startup: buffered residual → subscriber → initial snapshot never emitted), nothing can drain the queue, so waiting on queue.join() would only add an artificial shutdown delay — fail-open now holds on that path too.

Regression added: test_flush_is_noop_before_dispatcher_starts (buffer → subscribe → flush, asserts an instant return and that the buffered event stays queued). 21 focused tests pass under the hermetic runner; ruff + compileall clean.

Docs de-staled

Removed the sentence "the first subscriber starts normal dispatch and drains it" (now opposite to the code) and rewrote the lifetime paragraph to state: subscribe() never starts dispatch; the ordinary post-assembly snapshot emit drains the buffered residual through the full fan-out; flush is a no-op whenever nothing can drain the queue (no subscriber, or subscriber attached before the dispatcher started).

History

Folded the fix into the two behavior-intent commits via autosquash:

1. de5f0ea03 — process-lifetime bounded delivery boundary (emitter emit_buffered, subscribe-never-starts-dispatch, flush no-op before start, dormant process-lifetime install boundary).
2. 5dbee4847 — residual runtime fact vocabulary (closed stable subset, static-log normalizer, redaction, fail-open) + docs.

Cumulative diff vs fork main: 6 files, +1066/-15. Requesting re-review.