feat(telemetry): export gateway health diagnostics

[victor-kyriazakos]

Summary

• adds a Gateway Health and Diagnostics Export on top of PR feat(telemetry): local-first telemetry & observability #51714's telemetry spine
• emits content-free gateway health metrics plus redacted gateway diagnostic events to an operator-configured OTLP endpoint
• wires fail-open startup/shutdown into the gateway without sidecar/watchdog scope
• documents DataDog/OpenTelemetry Collector setup and config knobs
• includes a no-Docker local OTLP capture collector and probe for tmux smoke testing

Taxonomy alignment

This is Service Health Monitoring plus Operational Diagnostics only. It does not export prompts, messages, tool args/results, session history, product usage analytics, governance/audit logs, quality reports, or trajectories.

Scope additions

• immediate gateway lifecycle events from write_runtime_status() transitions
• startup-failed and gateway-exit events
• platform state-change and platform-fatal events
• OTLP metrics for gateway/platform health
• OTLP traces/events for gateway health/lifecycle events
• OTLP logs for redacted gateway diagnostics on /v1/logs
• gateway-only OTLP filters so generic run/model/tool telemetry is not exported by this plane
• local tmux smoke scripts under scripts/observability/

Review fixes

Addressed independent review blockers:

• gateway export now passes an event filter into the OTLP streamer so this plane does not stream run/model/tool telemetry
• metric startup failure does not start the diagnostic streamer and cleans up partial runtime state
• OTLPStreamer.shutdown() unsubscribes itself from the emitter
• diagnostic log handler is fully fail-open for malformed log records
• metric error codes are bounded through the classifier; gateway logger allowlist uses gateway / gateway.* only

Verification

• python -m pytest tests/telemetry/test_gateway_health_export.py -q -o 'addopts=' -> 12 passed
• python -m pytest tests/telemetry -q -o 'addopts=' -> 88 passed
• python -m pytest tests/gateway/test_status.py tests/gateway/test_status_command.py tests/gateway/test_version_command.py tests/cli/test_version_command.py -q -o 'addopts=' -> 112 passed
• python -m py_compile agent/telemetry/gateway_health.py agent/telemetry/gateway_health_export.py agent/telemetry/events.py agent/telemetry/otlp_exporter.py gateway/status.py gateway/run.py hermes_cli/config.py hermes_cli/main.py scripts/observability/otel_capture_collector.py scripts/observability/gateway_health_export_probe.py
• git diff --check
• docs code-fence sanity check passed
• security grep scan clean
• tmux local OTLP smoke passed: /v1/logs, /v1/metrics, and /v1/traces captured

Stacked on #51714; should merge after or into that telemetry PR.

CI triage note

Compared #61252 with base PR #51714. The broad failing test/build jobs were inherited from #51714: hermes-agent[otlp] pinned OpenTelemetry 1.30.0, while mistralai==2.4.8 requires opentelemetry-api>=1.33.1. This branch now bumps the OTLP extra and tools.lazy_deps to OpenTelemetry 1.39.1, matching the existing uv.lock package set, and reran uv lock.

The only #61252-specific failure found was ruff unspecified-encoding in the local OTLP probe script. Fixed by adding explicit UTF-8 decoding.

[victor-kyriazakos]

Local smoke test: call flow and reproduction

Verified end to end on this branch: the in-process emitter sent real OTLP protobuf to a local capture collector on :4318 and all three routes arrived. Decoded payloads showed 8 gauge series on /v1/metrics (including hermes.platform.up=0 / hermes.platform.degraded=1 for a fatal Slack connector with error_code=auth_failed), a starting -> running lifecycle event plus health snapshots on /v1/traces, and 5 diagnostic logs on /v1/logs with token/e-mail values already rewritten to [redacted] / [email] before egress.

Who calls who, when, with what

sequenceDiagram
    autonumber
    participant Probe as gateway_health_export_probe.py
    participant Runtime as start_gateway_health_export()<br/>(gateway_health_export.py)
    participant Status as write_runtime_status()<br/>(gateway/status.py)
    participant Emitter as telemetry emitter<br/>(agent/telemetry/emitter.py)
    participant SDK as OTel SDK 1.39.1<br/>(MeterProvider / OTLPStreamer / LoggerProvider)
    participant Collector as capture collector :4318<br/>(otel_capture_collector.py)

    Note over Collector: started first in tmux, writes each POST to JSONL

    Probe->>Runtime: start (config: enabled=true,<br/>endpoint=http://127.0.0.1:4318/v1/traces)
    Runtime->>SDK: load SDK, build MeterProvider (observable gauges),<br/>span streamer (event_filter=gateway_health),<br/>log streamer (gateway_diagnostic -> /v1/logs)
    Runtime->>Emitter: subscribe streamers to gateway events

    Probe->>Status: write_runtime_status(starting -> running,<br/>agents=2, busy, drainable)
    Status->>Emitter: emit_runtime_status_transition(prev, curr)<br/>(fail-open, content-free)
    Probe->>Status: write_runtime_status(slack platform -> fatal,<br/>error=auth_failed)
    Status->>Emitter: platform.state_change + platform.fatal events
    Probe->>Emitter: gateway logger warning<br/>(raw line contains token + e-mail)
    Note over Emitter: redaction rewrites secrets to [redacted]/[email]<br/>BEFORE anything leaves the process

    Emitter->>SDK: gateway_health events -> span projection
    Emitter->>SDK: gateway_diagnostic events -> log records<br/>(INVALID_TRACE_ID/INVALID_SPAN_ID, uncorrelated)
    SDK->>Collector: POST /v1/metrics (periodic reader, gauges)
    SDK->>Collector: POST /v1/traces (batch span processor)
    SDK->>Collector: POST /v1/logs (batch log processor)

    Probe->>Runtime: shutdown() (flush + unsubscribe)
    Probe->>Collector: read JSONL, assert all three paths present
    Note over Probe: exit 0 = /v1/metrics, /v1/traces, /v1/logs all received

Loading

The collector is a dev/test receiver only. In production the same emitter points at a customer-owned OTEL Collector (receivers: [otlp] -> exporters: [datadog]); recipe in docs/observability/telemetry.md.

Reproduce (no Docker)

# branch feat/gateway-health-otel, needs the [otlp] extra installed

# terminal 1: local capture collector
tmux new-session -d -s otel-smoke \
  "python scripts/observability/otel_capture_collector.py \
     --host 127.0.0.1 --port 4318 --log /tmp/hermes_otel_capture.jsonl"

# terminal 2: drive the real exporter through lifecycle transitions,
# a fatal slack platform, and a redacted warning log, then flush
python scripts/observability/gateway_health_export_probe.py \
  --endpoint http://127.0.0.1:4318/v1/traces \
  --log /tmp/hermes_otel_capture.jsonl --wait 8

# exit 0 prints: {"requests": 6, "paths": ["/v1/logs", "/v1/metrics", "/v1/traces"]}

[teknium1]

Superseded by PR #64536, which landed on main (merge commit 7de33cc) as the standalone successor to this work — same gateway health & diagnostics OTLP export, extended with cron scheduler/execution health, background-work signals, and the durable execution ledger. All of the design here carried forward into the merged version. Thanks @victor-kyriazakos!