Skip to content

feat: add opt-in execution receipts - #49371

Draft
mistakeknot wants to merge 1 commit into
NousResearch:mainfrom
mistakeknot:feat/execution-receipts
Draft

feat: add opt-in execution receipts#49371
mistakeknot wants to merge 1 commit into
NousResearch:mainfrom
mistakeknot:feat/execution-receipts

Conversation

@mistakeknot

@mistakeknot mistakeknot commented Jun 20, 2026

Copy link
Copy Markdown

Why this is useful

Hermes can already run long, multi-tool, delegated sessions across CLI, desktop, and gateways, but today there is no small built-in evidence stream that answers basic post-run questions without storing raw tool payloads:

  • Which tools actually ran in this agent turn?
  • Did each tool complete, error, get blocked by a plugin, or get cancelled/skipped?
  • How long did it take, and which session/task/tool call did it belong to?
  • Were there explicit evidence gaps that a UI or later audit feature should surface?

This PR adds that minimal substrate as an opt-in, local-only observability primitive. It is intentionally not an analytics/exporter feature and not a signing/provenance system yet. The value is giving Hermes plugins, local dashboards, and future audit/provenance work a stable redacted event shape to build on without asking them to persist raw post_tool_call args/results.

Refs #6642 and #5041.

Summary

Adds an opt-in local execution-receipts substrate for agent-loop tool outcomes:

  • new execution_receipt plugin hook with a cheap no-op path when no listener is registered
  • metadata-only receipt builder for tool outcomes (ok, error, blocked, cancelled)
  • bundled disabled-by-default observability/execution_receipts plugin that writes owner-only local JSONL under $HERMES_HOME/execution-receipts/receipts.jsonl
  • /receipts status, /receipts tail [N], and /receipts gaps helper commands
  • docs and focused coverage for redaction, plugin enablement, disabled/default behavior, skipped/cancelled paths, and end-to-end enabled-plugin writing

Alternatives / existing overlaps

Safety / privacy defaults

This is intentionally a small P1:

  • disabled by default
  • local-only; no external network calls/exporter
  • fail-open hook/plugin behavior
  • no raw terminal output, file contents, prompts, environment variables, tokens, child summaries, raw error messages, or full tool args/results
  • plugin sink whitelists the redacted args/result metadata shape before writing, so a buggy caller cannot smuggle raw payload fields by setting redacted: true
  • no new user-facing HERMES_* env vars or .env configuration

Non-goals in this PR

  • no cryptographic signing / HMAC / hash chaining
  • no policy enforcement or routing
  • no remote telemetry backend
  • no OpenTelemetry/Langfuse exporter
  • no durable provenance claim beyond best-effort local JSONL

Design question for maintainers

I used a dedicated execution_receipt hook instead of extending post_tool_call because receipts are redacted durable evidence records, while post_tool_call remains useful for observers that intentionally receive raw tool args/results. Happy to collapse this into the existing hook surface if maintainers prefer fewer hook names.

Test plan

Final local validation after rebasing onto current origin/main:

/Users/sma/.local/bin/uv run --extra dev pytest \
  tests/run_agent/test_execution_receipts.py \
  tests/plugins/test_execution_receipts_plugin.py \
  tests/hermes_cli/test_plugins.py::test_execution_receipt_is_valid_hook \
  -q -o 'addopts='
# 20 passed

/Users/sma/.local/bin/uv run --extra dev ruff check \
  agent/execution_receipts.py \
  agent/tool_executor.py \
  hermes_cli/plugins.py \
  plugins/observability/execution_receipts/__init__.py \
  tests/run_agent/test_execution_receipts.py \
  tests/plugins/test_execution_receipts_plugin.py \
  tests/hermes_cli/test_plugins.py
# All checks passed!

/Users/sma/.local/bin/uv run --extra dev pytest \
  tests/hermes_cli/test_plugins.py \
  tests/plugins/test_execution_receipts_plugin.py \
  tests/run_agent/test_execution_receipts.py \
  tests/run_agent/test_run_agent.py \
  -q -o 'addopts='
# 490 passed

@alt-glitch alt-glitch added type/feature New feature or request comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have labels Jun 20, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the privacy-focused local observability proposal. Current main still exposes raw post_tool_call observations (model_tools.py:974-1022) but has no redacted durable receipt stream, so the feature remains distinct.

Problems

  • agent/execution_receipts.py:157 persists arbitrary mapping keys as field_names. Mapping keys are caller-controlled data, so this can store sensitive argument content despite the documented no-raw-arguments guarantee.
  • The executor has moved substantially since the PR base. Current agent/tool_executor.py:360-375 models malformed arguments as a non-executed outcome, and current tests enforce aligned concurrent/sequential post-hook ownership at tests/run_agent/test_run_agent.py:3389-3510. Receipt wiring needs to cover those current paths explicitly.

Suggested changes

  • Use schema-derived safe field names, or omit field names; add a test with a secret-bearing key.
  • Integrate against the current executor and test malformed-argument receipt status plus concurrent/sequential parity.

This is an automated hermes-sweeper review.

"size_bytes": _payload_size_bytes(payload),
}
if isinstance(payload, Mapping):
metadata["field_names"] = sorted(str(key) for key in payload.keys())[:50]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

field_names is persisted metadata, but mapping keys are caller-controlled data. Please derive names from a trusted tool schema or omit them, and add a test where a key itself contains a secret; otherwise this violates the stated no-raw-arguments privacy contract.

@teknium1 teknium1 added sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users labels Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants