feat(telemetry): local-first telemetry & observability by jquesnelle · Pull Request #51714 · NousResearch/hermes-agent

jquesnelle · 2026-06-24T06:24:58Z

Add a built-in telemetry system that records what the agent does — workflows, model calls, tool calls, errors — to the local machine, powers /insights, and can export to an operator-chosen destination. Default-on locally; nothing leaves the machine unless the user exports it or opts into the aggregate plane.

Three planes with a hard wall between them:

local: full-fidelity observability (real model/provider/tool names), on by default, never leaves the machine.
aggregate: opt-in metadata, default off. No uploader ships — consent is recorded via telemetry.consent_state, and preview shows what would be produced, computed locally.
trajectories: full message content, opt-in, exported only to the operator's own destination.

Mechanism:

Bundled telemetry plugin registers observational lifecycle hooks (on_session_start / post_api_request / post_tool_call / on_session_finalize). No core call sites are edited; hooks already carry the data.
Fire-and-forget emitter: emit() returns in microseconds, never blocks or raises into a model/tool call. A daemon thread writes events to an append-only JSONL log and the tel_* tables in state.db (its own sqlite connection, separate from SessionDB).
tel_runs / tel_model_calls / tel_tool_calls live in the declarative SCHEMA_SQL and are reconciled automatically; SCHEMA_VERSION 16 -> 17.
metrics derives rollups for /usage and /insights; rollup builds per-run summaries for hermes telemetry preview.

Consent is config, not a parallel command surface. The config file is the root of trust: set telemetry.consent_state with hermes config set, or pin any telemetry.* key (including allow_aggregate) via managed scope, which overrides the user's value per key. hermes telemetry exposes only what config cannot: status (report), preview (query), and export.

Export:

exporter_bulk writes telemetry (and, when the trajectories plane is enabled, session content) to ndjson/json.
otlp_exporter streams spans to a configured OpenTelemetry Collector over OTLP/HTTP. The SDK is an optional extra (hermes-agent[otlp]), lazily installed via tools.lazy_deps on first use.
Secrets are always redacted on every export path (redact_sensitive_text(force=True)); content export is gated by the trajectories plane, and PII scrubbing follows telemetry.content_redaction. OTLP auth headers reference environment variable names, never inline values.

No outbound emission to Nous. The aggregate uploader is intentionally not built.

Add a built-in telemetry system that records what the agent does — workflows, model calls, tool calls, errors — to the local machine, powers `/insights`, and can export to an operator-chosen destination. Default-on locally; nothing leaves the machine unless the user exports it or opts into the aggregate plane. Three planes with a hard wall between them: - local: full-fidelity observability (real model/provider/tool names), on by default, never leaves the machine. - aggregate: opt-in metadata, default off. No uploader ships — consent is recorded via telemetry.consent_state, and `preview` shows what would be produced, computed locally. - trajectories: full message content, opt-in, exported only to the operator's own destination. Mechanism: - Bundled `telemetry` plugin registers observational lifecycle hooks (on_session_start / post_api_request / post_tool_call / on_session_finalize). No core call sites are edited; hooks already carry the data. - Fire-and-forget emitter: emit() returns in microseconds, never blocks or raises into a model/tool call. A daemon thread writes events to an append-only JSONL log and the tel_* tables in state.db (its own sqlite connection, separate from SessionDB). - tel_runs / tel_model_calls / tel_tool_calls live in the declarative SCHEMA_SQL and are reconciled automatically; SCHEMA_VERSION 16 -> 17. - metrics derives rollups for /usage and /insights; rollup builds per-run summaries for `hermes telemetry preview`. Consent is config, not a parallel command surface. The config file is the root of trust: set telemetry.consent_state with `hermes config set`, or pin any telemetry.* key (including allow_aggregate) via managed scope, which overrides the user's value per key. `hermes telemetry` exposes only what config cannot: status (report), preview (query), and export. Export: - exporter_bulk writes telemetry (and, when the trajectories plane is enabled, session content) to ndjson/json. - otlp_exporter streams spans to a configured OpenTelemetry Collector over OTLP/HTTP. The SDK is an optional extra (hermes-agent[otlp]), lazily installed via tools.lazy_deps on first use. - Secrets are always redacted on every export path (redact_sensitive_text(force=True)); content export is gated by the trajectories plane, and PII scrubbing follows telemetry.content_redaction. OTLP auth headers reference environment variable names, never inline values. No outbound emission to Nous. The aggregate uploader is intentionally not built.

github-actions · 2026-06-24T06:26:08Z

🔎 Lint report: `feat/telemetry-observability` vs `origin/main`

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 11724 on HEAD, 11698 on base (🆕 +26)

🆕 New issues (55):

Rule	Count
`unsupported-operator`	23
`unresolved-import`	15
`unresolved-attribute`	10
`invalid-argument-type`	7

First entries

tests/telemetry/test_governance.py:13: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
tests/hermes_cli/test_aux_config.py:54: [unresolved-attribute] unresolved-attribute: Attribute `keys` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tests/tools/test_web_providers.py:218: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["search_backend"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tests/telemetry/test_otlp_exporter.py:27: [unresolved-import] unresolved-import: Cannot resolve imported module `opentelemetry.sdk.trace`
tests/hermes_cli/test_aux_config.py:37: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["title_generation"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tests/telemetry/test_export_redaction.py:20: [unsupported-operator] unsupported-operator: Operator `not in` is not supported between objects of type `Literal["SECRETKEY123"]` and `str | None`
tests/tools/test_web_providers.py:217: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["backend"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tests/hermes_cli/test_destructive_slash_confirm_gate.py:32: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tests/tools/test_web_providers.py:219: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["extract_backend"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tests/cron/test_suggestions.py:213: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["monitor"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
hermes_cli/main.py:12371: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `None` in union `Any | None | dict[Unknown, Unknown]`
agent/telemetry/otlp_exporter.py:68: [unresolved-import] unresolved-import: Cannot resolve imported module `opentelemetry.sdk.trace.export`
tests/cli/test_resume_display.py:716: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["resume_display"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tests/telemetry/test_export_redaction.py:43: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["[email]"]` and `str | None`
hermes_cli/config.py:5287: [unresolved-attribute] unresolved-attribute: Attribute `items` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
agent/telemetry/otlp_exporter.py:67: [unresolved-import] unresolved-import: Cannot resolve imported module `opentelemetry.sdk.trace`
hermes_cli/mcp_startup.py:66: [invalid-argument-type] invalid-argument-type: Argument to constructor `float.__new__` is incorrect: Expected `str | Buffer | SupportsFloat | SupportsIndex`, found `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tests/cli/test_fast_command.py:484: [invalid-argument-type] invalid-argument-type: Argument to bound method `TestCase.assertIn` is incorrect: Expected `Iterable[Any] | Container[Any]`, found `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tests/tools/test_refresh_agent_mcp_tools.py:257: [invalid-argument-type] invalid-argument-type: Argument to constructor `float.__new__` is incorrect: Expected `str | Buffer | SupportsFloat | SupportsIndex`, found `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tests/hermes_cli/test_aux_config.py:47: [unsupported-operator] unsupported-operator: Operator `not in` is not supported between objects of type `Literal["session_search"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tests/agent/test_auxiliary_config_bridge.py:285: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["model"]` and `Unknown | int | str | ... omitted 18 union elements`
tools/browser_tool.py:1321: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tests/telemetry/test_spans_trace.py:19: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
tests/run_agent/test_in_place_compaction.py:257: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tests/telemetry/test_export_redaction.py:42: [unsupported-operator] unsupported-operator: Operator `not in` is not supported between objects of type `Literal["alice@example.com"]` and `str | None`
... and 30 more

✅ Fixed issues (31):

Rule	Count
`unsupported-operator`	15
`unresolved-attribute`	9
`invalid-argument-type`	7

First entries

tests/cron/test_suggestions.py:213: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["monitor"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/tools/test_web_providers.py:219: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["extract_backend"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tools/browser_tool.py:1321: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/run_agent/test_in_place_compaction.py:257: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/cli/test_reasoning_command.py:552: [invalid-argument-type] invalid-argument-type: Argument to bound method `TestCase.assertIn` is incorrect: Expected `Iterable[Any] | Container[Any]`, found `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/tools/test_web_providers.py:217: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["backend"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/tools/test_browser_console.py:341: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["record_sessions"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/cli/test_resume_display.py:716: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["resume_display"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/agent/test_curator.py:1213: [unsupported-operator] unsupported-operator: Operator `>` is not supported between objects of type `Unknown | int | str | ... omitted 6 union elements` and `Literal[0]`
tests/tools/test_browser_lightpanda.py:242: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["engine"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/hermes_cli/test_kanban_core_functionality.py:3431: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
hermes_cli/config.py:5252: [unresolved-attribute] unresolved-attribute: Attribute `items` is not defined on `int`, `str`, `list[Unknown]`, `float`, `None` in union `Unknown | int | str | ... omitted 17 union elements`
hermes_cli/config.py:5243: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/tools/test_web_providers.py:218: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["search_backend"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
hermes_cli/mcp_startup.py:66: [invalid-argument-type] invalid-argument-type: Argument to constructor `float.__new__` is incorrect: Expected `str | Buffer | SupportsFloat | SupportsIndex`, found `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/tools/test_refresh_agent_mcp_tools.py:257: [invalid-argument-type] invalid-argument-type: Argument to constructor `float.__new__` is incorrect: Expected `str | Buffer | SupportsFloat | SupportsIndex`, found `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/cli/test_fast_command.py:484: [invalid-argument-type] invalid-argument-type: Argument to bound method `TestCase.assertIn` is incorrect: Expected `Iterable[Any] | Container[Any]`, found `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/hermes_cli/test_mcp_reload_confirm_gate.py:33: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/hermes_cli/test_aux_config.py:54: [unresolved-attribute] unresolved-attribute: Attribute `keys` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/agent/test_auxiliary_config_bridge.py:285: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["model"]` and `Unknown | int | str | ... omitted 17 union elements`
gateway/restart.py:16: [invalid-argument-type] invalid-argument-type: Argument to constructor `float.__new__` is incorrect: Expected `str | Buffer | SupportsFloat | SupportsIndex`, found `Unknown | int | str | ... omitted 17 union elements`
hermes_cli/config.py:5233: [unresolved-attribute] unresolved-attribute: Attribute `items` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/gateway/test_whatsapp_reply_prefix.py:119: [unsupported-operator] unsupported-operator: Operator `>=` is not supported between objects of type `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements` and `int`
tests/agent/test_curator.py:1208: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["curator"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/hermes_cli/test_aux_config.py:37: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["title_generation"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
... and 6 more

Unchanged: 6111 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

policy.resolve() / TelemetryDecision was a read-only projection used only by `hermes telemetry status` for display. The actual behavior gates already read telemetry.* straight from config: the emitter (whether to write) and the plugin loader (whether to auto-load) each call .get("local", True) on the loaded config, never through policy. Make config the single chokepoint the status command reads too: it now resolves local/allow_aggregate/consent_state inline from the loaded config, the same way the other gates do. policy.py keeps only what config can't express on its own — the consent constants, ensure_install_id(), and may_upload_aggregate(config) as a pure function (the gate a future uploader must consult). resolve() and the TelemetryDecision dataclass are removed; policy.py drops 107 -> 70 lines. No behavior change: status renders identically, and the default-on local plane is still defaulted in DEFAULT_CONFIG plus a fail-safe .get(..., True) at each gate.

Rename the telemetry tiers away from the borrowed control-plane/data-plane jargon to plain language, across code, CLI output, config, and docs: - "local plane" -> "local telemetry" - "aggregate plane" -> "aggregate metrics" - "trajectories plane" -> "trajectories" / "telemetry.trajectories" - "three planes with a hard wall" -> "three settings, isolated from each other" User-facing `hermes telemetry status` now reads "Local telemetry: on" / "Aggregate metrics: off" / "Content export: off (trajectories disabled)". The OTLP resource attribute key telemetry.plane is renamed to telemetry.scope (wire-level identifier; nothing consumes it yet). No behavior change — wording only. Status renders identically apart from the labels; tests updated to match the new strings.

OutThisLife · 2026-06-26T18:15:54Z

Took a close read of the code (events, spans, schema, emitter, plugin, otlp_exporter) with an eye toward driving a desktop trace-waterfall off this store. Capture/storage/consent layer is genuinely strong; flagging that the span/trace layer is declared but largely unwired, which is the thing that matters if anything wants to reconstruct a trace from tel_*.

What's great

Emitter hot-path invariant is textbook (emitter.py): non-blocking put_nowait, ring-buffer drops oldest, never raises, daemon writer, JSONL source-of-truth + SQLite index on a separate connection.
Pure lifecycle-hook plugin, every callback @_safe fail-open, zero core edits.
Consent-as-config, redaction, OTLP SDK as a lazy optional extra, declarative SCHEMA_VERSION 16→17.
The tel_spans (timing+parent) + tel_model_calls/tel_tool_calls (detail keyed by span_id) star-schema is the right shape.

Gaps in the trace/span layer

tel_spans is never written. The emitter's _TABLE_COLUMNS only maps run/model_call/tool_call/error; there's no span event or writer. The one table carrying start_ns/end_ns/parent_span_id/kind stays empty.
No timestamp on calls. ModelCallEvent/ToolCallEvent only carry latency_ms/duration_ms. emit() stamps ts_ns, but _TABLE_COLUMNS for those events doesn't include it, so tel_model_calls/tel_tool_calls have no time column at all — calls can't be ordered or placed on a timeline from the tables.
No parent/lineage. spans.py defines a parent_span_id contextvar but the plugin never sets/advances it, and the event dataclasses don't carry it. subagent_start/stop are no-ops, and since each on_session_start mints a fresh trace_id/run_id, a delegated subagent's run isn't linked to its parent.
OTLP output isn't traces. export_batch does tracer.start_span(name).end() per event → zero-duration spans stamped "now", not attached to their trace_id, no parent. In a backend these land as disconnected instantaneous root spans. (The exporter docstring says events carry trace_id/run_id/span_id/parent_span_id, but trace_id/parent_span_id aren't on the call events.)

Net: as-is this is an excellent metrics system (counts, tokens, cost, durations, error classes per run → /insights, /usage), but not yet a trace system — the two meet only at the run level.

If wiring real span capture is in scope, it'd make this a store a desktop waterfall (and any OTel backend) could render directly: populate tel_spans with start/end/parent, set the parent_span_id contextvar around each model/tool call, and capture subagent lineage in the subagent_* hooks (or a parent run id). Happy to help connect that to the trace-viewer side if useful.

The existing hook tests call the plugin's _on_* callbacks directly, which passes even if the bundled plugin stops auto-loading or a hook name drifts from what core fires — real runs would go dark while the suite stays green. Add test_plugin_e2e.py, which drives the real dispatch chain through public entry points only (discover_plugins -> invoke_hook -> registered callback -> emitter -> tel_* tables), exactly as core does: - one completed turn produces tel_runs / tel_model_calls / tel_tool_calls rows with real provider/model/tool values and correct counts; - telemetry.local=false means the plugin does not load and nothing is written. Verified robust against test ordering (singleton resets for the plugin manager and the emitter in the fixture).

Aggregate metrics are derived from the local tel_* tables — they're a coarsened view of local data, not an independent capture path. With telemetry.local=false nothing is written, so an aggregate opt-in had nothing to aggregate, yet may_upload_aggregate() returned True and `status` showed "Aggregate metrics: on". The config could claim a state it couldn't fulfill. Gate aggregate on local being enabled: - may_upload_aggregate() now requires local_enabled AND allow_aggregate AND consent_state == aggregate. - `telemetry status` computes aggregate_enabled the same way and, when consent is aggregate but local is off, prints "inert: local telemetry is off — nothing to aggregate" instead of the opt-in hint. Happy path is unchanged (local on + consent aggregate -> on). Adds policy and CLI tests for the inert combo.

The subagent_start/stop hooks are registered but no-op. The prior comment implied subagents need no handling because they inherit via contextvars — misleading, since a delegated child runs on a separate thread with its own session id and trace. Clarify the real situation: a subagent's model/tool calls are already captured as their own tel_runs row via the child's run_conversation, so nothing is lost. These hooks are reserved for recording parent->child lineage (needs a tel_runs.parent_run_id column), deferred until a consumer needs the delegation tree. Comment-only.

Addresses the review on #51714: the trace/span layer was declared but unwired — tel_spans was never written, call rows had no timestamp, and nothing set parent lineage, so the store was metrics-only and couldn't reconstruct a trace. Wire the span layer (keeping the praised star-schema shape): - New SpanEvent (span_id/trace_id/run_id/parent_span_id/name/kind/start_ns/end_ns) mapped into tel_spans via the emitter's _TABLE_COLUMNS. - The plugin mints a root span per run and, on each model/tool call, emits a SpanEvent (timing + parent = the run's root) keyed by the SAME span_id as the detail row, so tel_model_calls / tel_tool_calls JOIN to their span. - Call hooks fire on completion, so end_ns = now and start_ns is reconstructed from the measured latency/duration. The run's root span is emitted at finalize with the true run start/end. Result: tel_spans is a connected, single-trace_id, run -> calls tree a desktop waterfall (or any reader) can render directly, ordered by start_ns. Existing metrics rows (tel_runs/model_calls/tool_calls) are unchanged. OTLP: spans now flow to the exporter with their trace/parent/timing attributes. The exporter still emits one OTel span per event rather than reconstructing OTel SpanContexts into a connected trace tree; that projection is left for a follow-up and the module docstring now says so plainly instead of over-claiming. Adds test_spans_trace.py (connected-tree + detail-row JOIN) over the real dispatch path. Accurate (pre-hook) start times, real OTLP SpanContexts, and subagent cross-run lineage remain follow-ups.

…itten Self-review after the #51714 feedback found the reviewer's dead-table finding was not isolated — the schema advertised far more than the code populates, and our own tests hid it by hand-feeding fields production never sends. Make the surface honest by subtraction. Schema (10 tel_* tables -> 5): - Delete tel_gateway_events, tel_cron_events, tel_skill_events, tel_memory_events, tel_feedback_events — declared, never written, never read. - Drop columns nothing populates: tel_runs.{profile_id,estimated_cost_usd, cost_status}; tel_model_calls.{ttft_ms,estimated_cost_usd,cost_status, cost_source,end_reason,retry_count}; tel_tool_calls.{backend,retry_count, approval}; tel_spans.attrs_json. Cost duplicated the existing sessions billing columns and was always NULL here. - events.py / emitter _TABLE_COLUMNS / OTLP _span_attrs / rollup / preview display all trimmed to match. Correctness: - end_reason no longer hardcodes "completed". Production finalize callers pass `reason` (shutdown/session_expired/session_reset); _coarse_end_reason now reads it and maps accordingly. - Fix a latent bug the trim exposed: the model_call hook passed end_reason= to ModelCallEvent, which the @_safe wrapper was silently swallowing — so tel_model_calls dropped every row in real runs. Now writes correctly. Tests: - Stop hand-feeding estimated_cost_usd / turn_exit_reason that no production call site sends. Finalize is now driven with the real `reason` kwarg, and assertions cover only fields that are actually populated. This is what let the model_call drop hide — the suite graded on a fictional contract. Net: a smaller system that does what it says. Verified end-to-end over the real dispatch path (runs + connected span tree + model/tool rows populate; dead tables gone). 160 telemetry/state/insights tests green.

Match the docs to the code after the dead-schema cut and span layer: - List the actual tel_* tables (runs, spans, model_calls, tool_calls, error_events) instead of a vague "indexed tel_* tables". - Add a "Traces and spans" section: a run = one session, each call is a child span under the run root in tel_spans keyed by span_id, reconstructable as a connected run -> calls tree. Note subagent cross-run lineage isn't recorded. - Fix stale "tool failure rates by category" -> "by tool" (categories were removed; insights groups by raw tool name). - OTLP: state plainly that events export as per-event spans and the tel_spans parent/timing linkage isn't reconstructed into connected SpanContexts yet, matching the exporter's own docstring. - README: "telemetry plane" -> "telemetry system" (stale rename miss); mention spans. Config reference verified to match DEFAULT_CONFIG exactly (9 keys).

…ability # Conflicts: # hermes_cli/plugins.py # hermes_state.py

benbarclay · 2026-07-06T07:06:56Z

This is nice, most otel implementation just drop data that cannot be shipped off the box. Letting users have a copy is a nice touch. Might have to keep an eye on how quickly the table grows, otel data can grow quickly.

benbarclay · 2026-07-06T07:21:19Z

+
+
+def new_id() -> str:
+    return uuid.uuid4().hex


Probably doesn't matter, but a span ID should only be 8 bytes long - https://opentelemetry.io/docs/specs/otel/overview/#spancontext

benbarclay · 2026-07-06T07:25:05Z

+
+
+# ── event -> span attribute mapping (real values) ───────────────────────────
+def _span_attrs(ev: Dict[str, Any]) -> Dict[str, Any]:


There's a bunch of semantic conventions we may want to align with: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md

The upside of aligning with those is that anything else consuming this data and correlating it against other systems may be able to get richer data. Not sure if that actually pans out in practice, but the theory is nice.

Addresses the review on NousResearch#51714: the trace/span layer was declared but unwired — tel_spans was never written, call rows had no timestamp, and nothing set parent lineage, so the store was metrics-only and couldn't reconstruct a trace. Wire the span layer (keeping the praised star-schema shape): - New SpanEvent (span_id/trace_id/run_id/parent_span_id/name/kind/start_ns/end_ns) mapped into tel_spans via the emitter's _TABLE_COLUMNS. - The plugin mints a root span per run and, on each model/tool call, emits a SpanEvent (timing + parent = the run's root) keyed by the SAME span_id as the detail row, so tel_model_calls / tel_tool_calls JOIN to their span. - Call hooks fire on completion, so end_ns = now and start_ns is reconstructed from the measured latency/duration. The run's root span is emitted at finalize with the true run start/end. Result: tel_spans is a connected, single-trace_id, run -> calls tree a desktop waterfall (or any reader) can render directly, ordered by start_ns. Existing metrics rows (tel_runs/model_calls/tool_calls) are unchanged. OTLP: spans now flow to the exporter with their trace/parent/timing attributes. The exporter still emits one OTel span per event rather than reconstructing OTel SpanContexts into a connected trace tree; that projection is left for a follow-up and the module docstring now says so plainly instead of over-claiming. Adds test_spans_trace.py (connected-tree + detail-row JOIN) over the real dispatch path. Accurate (pre-hook) start times, real OTLP SpanContexts, and subagent cross-run lineage remain follow-ups.

…itten Self-review after the NousResearch#51714 feedback found the reviewer's dead-table finding was not isolated — the schema advertised far more than the code populates, and our own tests hid it by hand-feeding fields production never sends. Make the surface honest by subtraction. Schema (10 tel_* tables -> 5): - Delete tel_gateway_events, tel_cron_events, tel_skill_events, tel_memory_events, tel_feedback_events — declared, never written, never read. - Drop columns nothing populates: tel_runs.{profile_id,estimated_cost_usd, cost_status}; tel_model_calls.{ttft_ms,estimated_cost_usd,cost_status, cost_source,end_reason,retry_count}; tel_tool_calls.{backend,retry_count, approval}; tel_spans.attrs_json. Cost duplicated the existing sessions billing columns and was always NULL here. - events.py / emitter _TABLE_COLUMNS / OTLP _span_attrs / rollup / preview display all trimmed to match. Correctness: - end_reason no longer hardcodes "completed". Production finalize callers pass `reason` (shutdown/session_expired/session_reset); _coarse_end_reason now reads it and maps accordingly. - Fix a latent bug the trim exposed: the model_call hook passed end_reason= to ModelCallEvent, which the @_safe wrapper was silently swallowing — so tel_model_calls dropped every row in real runs. Now writes correctly. Tests: - Stop hand-feeding estimated_cost_usd / turn_exit_reason that no production call site sends. Finalize is now driven with the real `reason` kwarg, and assertions cover only fields that are actually populated. This is what let the model_call drop hide — the suite graded on a fictional contract. Net: a smaller system that does what it says. Verified end-to-end over the real dispatch path (runs + connected span tree + model/tool rows populate; dead tables gone). 160 telemetry/state/insights tests green.