Skip to content

feat: add privacy-preserving runtime telemetry - #64472

Open
visualfox-ch wants to merge 5 commits into
NousResearch:mainfrom
visualfox-ch:feat/privacy-preserving-telemetry-v2
Open

feat: add privacy-preserving runtime telemetry#64472
visualfox-ch wants to merge 5 commits into
NousResearch:mainfrom
visualfox-ch:feat/privacy-preserving-telemetry-v2

Conversation

@visualfox-ch

Copy link
Copy Markdown

Summary

  • add privacy-preserving local telemetry primitives
  • instrument cron, gateway, tool, and model calls without recording prompts, responses, secrets, or raw identifiers
  • redact secret values from the expanded status output
  • add focused telemetry and redaction regression tests

Privacy properties

  • no prompt or response bodies
  • no API keys, tokens, or credential values
  • bounded categorical metadata and hashed identifiers only
  • local telemetry storage remains optional and failure-isolated

Test plan

  • python -m pytest tests/test_hermes_telemetry.py tests/cron/test_cron_telemetry.py tests/test_gateway_telemetry.py tests/test_tool_call_telemetry.py tests/test_model_call_telemetry.py tests/hermes_cli/test_status.py -q -o 'addopts='
  • ruff check hermes_telemetry.py cron/scheduler.py gateway/run.py model_tools.py agent/chat_completion_helpers.py hermes_cli/status.py

This is the telemetry-only replacement for #50894; delegation and Telegram changes are submitted separately.

@alt-glitch alt-glitch added type/feature New feature or request comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery comp/cron Cron scheduler and job management telemetry Touches outbound telemetry, usage attribution, or analytics — needs opt-in gating before merge P3 Low — cosmetic, nice to have labels Jul 14, 2026

@tonydwb tonydwb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Verdict: Approved

Privacy-preserving runtime telemetry adding model call and cron completion events. Uses stable_hash for provider/model identifiers rather than raw values. Well-structured with best-effort error handling (telemetry failures do not break model calls). 1116 additions, 3 deletions. The hermes_telemetry module provides the actual privacy guarantees.


Reviewed by Hermes Agent

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the privacy-focused telemetry work. The feature is not already present on current main, but this version needs changes before it is safe to salvage.

Problems

  • cron/scheduler.py:620-682 reintroduces per-job profile routing. Maintainer commit 7d8d000b1 explicitly removed this exact behavior, and current cron/jobs.py:54-64 treats profile isolation as a security boundary. Please remove this non-telemetry behavior from the PR.
  • hermes_telemetry.py:79-80 redacts every key containing token; this converts gateway/run.py:216-217's input_tokens/output_tokens and agent/chat_completion_helpers.py:169's estimated_context_tokens into [REDACTED] in persisted JSONL. The gateway test only checks the pre-serialization payload (tests/test_gateway_telemetry.py:42-59).
  • hermes_telemetry.py:126-128 writes local JSONL unconditionally, but the PR contains no config or docs changes. This conflicts with the stated optional-storage behavior and leaves website/docs/reference/faq.md:54 inaccurate.

Suggested changes

  • Keep cron instrumentation limited to the existing execution path; do not restore profile routing.
  • Preserve safe numeric usage metrics and add serialized-event regression tests.
  • Add a profile-safe user configuration gate and update the privacy documentation.

Automated hermes-sweeper review.

Comment thread cron/scheduler.py


@contextmanager
def _job_profile_context(job_id: str, profile: Optional[str]):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove the per-job profile context from this telemetry PR. Commit 7d8d000b1 explicitly reverted this wrapper, and current cron/jobs.py:54-64 documents per-profile cron isolation as a security boundary; this reintroduces behavior unrelated to telemetry without restoring its public schema or CLI surface.

Comment thread hermes_telemetry.py
for key, item in value.items():
key_text = str(key)
key_lc = key_text.lower()
if any(fragment in key_lc for fragment in _SENSITIVE_KEY_FRAGMENTS):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This catches numeric metric names such as input_tokens, output_tokens, and estimated_context_tokens, so their persisted values become [REDACTED]. Keep secret-bearing values redacted, but allow these numeric usage counters and add a serialized JSONL assertion.

Comment thread hermes_telemetry.py
"""
event = build_event(event_type, payload, status=status, source=source, ts=ts)
path = event_path(ts, hermes_home)
path.parent.mkdir(parents=True, exist_ok=True)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This creates the telemetry directory on every emission with no configuration gate. The PR describes local storage as optional, but it changes no config or docs files; please add a profile-safe opt-in/disable mechanism before creating persistent event files.

@teknium1 teknium1 added sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit labels Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state telemetry Touches outbound telemetry, usage attribution, or analytics — needs opt-in gating before merge type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants