feat(agent): OTel observability integration

[karthikkrishnaswamysr]

What does this PR do?

Adds OpenTelemetry observability integration in Hermes Agent, including runtime instrumentation hooks and Grafana/OTel config assets.

This enables trace and metrics visibility in Grafana/Tempo/Prometheus when OTel is enabled, while preserving default behavior when observability is not configured.

Related Issue

N/A

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• Added OTel shim implementation: agent/otel_shim.py
• Integrated OTel lifecycle in gateway flow: gateway/run.py
• Integrated OTel lifecycle in CLI flow: cli.py
• Added OTel callback bridge in runtime execution path: run_agent.py
• Added dashboard asset: otel/grafana/hermes_observability_dashboard.json
• Added otel-lgtm collector config: otel/otel-lgtm/otelcol-config.yaml
• Updated observability docs and datasource verification guidance: README.md

How to Test

1. Configure environment variables (via .env and also in Environment section of systemctl service):
   • OTEL_ENABLED=true
   • OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
   • OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
2. Start observability backend (Grafana + Tempo + Prometheus + OTLP collector).
3. Run Hermes Agent (hermes and/or gateway), execute a prompt with tool activity, and verify:
   • traces visible in Tempo datasource
   • metrics visible in Prometheus datasource
   • dashboard imports from otel/grafana/hermes_observability_dashboard.json

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits
• I searched for existing PRs to ensure this isn't a duplicate
• My PR contains only related changes
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes
• I've tested on my platform

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — or N/A
• I've updated cli-config.yaml.example if needed — or N/A
• I've updated CONTRIBUTING.md/AGENTS.md if needed — or N/A
• I've considered cross-platform impact — or N/A
• I've updated tool descriptions/schemas if needed — or N/A

Screenshots / Logs

Logs when OTEL is enabled

Grafana Dashboard Screens

Drill down into the Traces

[karthikkrishnaswamysr]

Demo

Short walkthrough video of OTel tracing + stuck-run telemetry:
https://www.youtube.com/watch?v=iDXlCsaHjtI

[SpartanDavie]

Are you still working on this? It looks great

[karthikkrishnaswamysr]

Are you still working on this? It looks great

This was done for 0.8.0 version, and we are atleast 4 to 5 releases ahead, i did some testing long back. Currently I see there is going to be a plugin with langfuse. So not sure this will be helpful. I am anyway open to suggestions on how to take this forward.

[TeddyAlbina]

Amazing, i was searching the doc yesterday about OTEL

[nujovich]

Really excited to see this landing in core! 🙌

This would be a great complement to hermes-telemetry (https://github.com/nujovich/hermes-telemetry), a zero-infra observability plugin I built for the Hermes Challenge. Right now it captures tokens, cost, latency and budget enforcement via hooks, storing everything locally in SQLite — no extra services needed.

If OTel lands in core, the natural next step would be wiring hermes-telemetry as an OTel exporter, so users can choose: lightweight slash-command visibility with /stats and /budget out of the box, OR full Grafana/Tempo traces if they have the infra. Best of both worlds.

Would love to coordinate once this merges. Great work! 🔥

[teknium1]

Thank you for the substantial tracing, metrics, dashboard, and operational setup work. This is an automated hermes-sweeper review.

• This is a new in-tree integration for external observability infrastructure: the PR adds the OTel shim (agent/otel_shim.py:80), Grafana dashboard assets, and Promtail/Grafana setup documentation (README.md, otel/).
• Hermes' standing policy is that observability/metrics backends and analytics dashboards do not land in the core tree; they should be standalone plugins to avoid taking on maintenance for external products (AGENTS.md:126-135; CONTRIBUTING.md:88-101).
• Current main already exposes backend-neutral lifecycle hooks intended for trace/export consumers (docs/observability/README.md:19-27, :118-129). A standalone OpenTelemetry plugin can register against those hooks without modifying run_agent.py, cli.py, gateway/run.py, or delegation internals.
• The PR also makes runtime behavior configurable through new non-secret OTEL_* environment variables (agent/otel_shim.py:80-88); a standalone plugin can instead provide its own setup/config flow.

Please consider publishing the exporter and Grafana/collector assets as a standalone plugin repo, installable through ~/.hermes/plugins/ or a pip entry point, and share it in #plugins-skills-and-skins.

---

Closed as not-planned per standing maintainer policy (in-tree-provider-integration). This is a design-direction decision, not a code-quality judgment — see the Contribution Rubric in AGENTS.md for what the project is looking for. If you believe this policy was misapplied to your change, comment here and a maintainer will take a look.