Skip to content

feat: add privacy-preserving Hermes telemetry - #50894

Closed
visualfox-ch wants to merge 8 commits into
NousResearch:mainfrom
visualfox-ch:fix/delegation-api-mode-resolution
Closed

visualfox-ch wants to merge 8 commits into
NousResearch:mainfrom
visualfox-ch:fix/delegation-api-mode-resolution

Conversation

@visualfox-ch

@visualfox-ch visualfox-ch commented Jun 22, 2026

Copy link
Copy Markdown

Summary

  • add privacy-preserving cron_run telemetry events for completed cron runs
  • add privacy-preserving gateway_action telemetry events for normal gateway agent turns
  • add privacy-preserving tool_call telemetry around tool dispatch
  • add privacy-preserving model_call telemetry around provider calls
  • ensure hermes status --all still redacts provider/API secrets instead of printing raw key material

Privacy / Safety

  • telemetry is local JSONL under $HERMES_HOME/ops/events/YYYY-MM-DD.jsonl
  • telemetry is best-effort: sink/write failures do not break cron, gateway, tools, or model calls
  • raw prompts, model responses, chat IDs, user IDs, tool arguments, shell commands, file paths, tool results, provider names, model names, response IDs, and secrets are not written
  • useful identifiers are hashed (*_hash) where correlation is needed
  • errors are grouped via error type + fingerprint instead of raw exception text
  • usage/count metadata is kept only as coarse numeric fields where available
  • --all status mode now means all sections, not raw secrets

Verification

Local verification on macOS with the repo venv:

./venv/bin/python -m pytest \
  tests/test_model_call_telemetry.py \
  tests/test_tool_call_telemetry.py \
  tests/test_transform_tool_result_hook.py \
  tests/test_hermes_telemetry.py \
  tests/tools/test_delegate.py \
  tests/cron/test_cron_telemetry.py \
  tests/test_gateway_telemetry.py \
  tests/hermes_cli/test_status.py -q

./venv/bin/python -m ruff check \
  agent/chat_completion_helpers.py \
  model_tools.py \
  tests/test_model_call_telemetry.py \
  tests/test_tool_call_telemetry.py

Result:

  • 194 passed, 3 third-party deprecation warnings
  • ruff: all checks passed
  • py_compile: passed for touched modules/tests

Live smoke

Also verified in a live Hermes setup:

  • cron telemetry emitted cron_run/ok events
  • gateway telemetry emitted a gateway_action/ok event after a Telegram gateway turn
  • CLI smoke emitted model_call/ok events and a tool_call/ok event
  • local JSONL inspection showed only privacy-preserving event metadata

Additional gateway smoke fix

Live Telegram smoke after the gateway restart exposed a config-shape bug in the Telegram adapter: telegram.allow_from can be stored as a YAML scalar when set via hermes config set, while the adapter expected an iterable. Added scalar/CSV handling plus focused regression coverage.

Test isolation follow-up

The live hardened Telegram profile exports allowlist settings. tests/gateway/test_telegram_auth_check.py now explicitly clears allowed_chats in its adapter fixture so group-auth unit tests do not depend on the operator's local gateway allowlist. Focused auth and telemetry tests pass locally.

@alt-glitch alt-glitch added type/bug Something isn't working tool/delegate Subagent delegation P3 Low — cosmetic, nice to have labels Jun 22, 2026
@visualfox-ch

Copy link
Copy Markdown
Author

Status update from visualfox-ch: the delegation fix is still current on the fork branch and the targeted delegate test suite passes locally: python3 -m pytest tests/tools/test_delegate.py -q -o 'addopts=' → 136 passed. No further local action pending; this PR appears to be waiting on upstream maintainer/CI gating.

@visualfox-ch visualfox-ch changed the title fix(delegation): honor configured api mode feat: add privacy-preserving Hermes telemetry Jul 8, 2026
Michael Bohl added 4 commits July 8, 2026 21:12
Apply delegation.api_mode from config before provider defaults so provider-based subagents can use chat_completions when the selected model does not support responses API.

Assisted-by: Hermes:gpt-5.5
@visualfox-ch
visualfox-ch force-pushed the fix/delegation-api-mode-resolution branch from d90791b to e018d5e Compare July 8, 2026 19:24
@visualfox-ch

Copy link
Copy Markdown
Author

Superseded by current-main, reviewable split PRs: #64470 (delegation API mode), #64472 (privacy-preserving telemetry), and #64473 (Telegram scalar allowlist). The replacement branches were rebuilt on current upstream and verified independently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P3 Low — cosmetic, nice to have tool/delegate Subagent delegation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants