Skip to content

docs(observability): document the audit and receipt-chain integration seam - #74717

Open
pofky wants to merge 1 commit into
NousResearch:mainfrom
pofky:docs/audit-receipt-chain-integration-seam
Open

docs(observability): document the audit and receipt-chain integration seam#74717
pofky wants to merge 1 commit into
NousResearch:mainfrom
pofky:docs/audit-receipt-chain-integration-seam

Conversation

@pofky

@pofky pofky commented Jul 30, 2026

Copy link
Copy Markdown

What does this PR do?

Adds docs/observability/audit-receipt-chains.md, documenting the integration seam for
plugins that build signed, tamper-evident records of tool activity, and cross-links it from
the observer hooks reference.

This is the documentation @teknium1 scoped when closing #487: a page describing the
three-hook pattern, pointing at the external projects and the cross-implementation spec
work, so the receipt-chain use case is discoverable without any core changes.

The page covers:

  • The three-hook pattern (pre_tool_call for authorization, post_tool_call for outcome,
    transform_tool_result for attaching a proof reference), with the payload fields a
    receipt actually needs.
  • Four rules that matter more for receipts than for tracing: accept **kwargs; never return
    a value from pre_tool_call unless you mean to block; stay fail-open rather than leaning
    on Hermes catching your exception; do not silently drop entries, because an audit trail
    with invisible gaps is worse than none.
  • The property a self-verified hash chain does not give you, which is the substance of
    @fernandosmither's objection in Feature: Cryptographic Audit Trail — SHA-256 Hash-Chained Action Log for Tamper-Proof Agent Accountability (inspired by OpenFang) #487: an operator with write access can fork and rewrite
    the chain and internal verification still passes. The page states the three things that
    close that gap (signatures, an external anchor, deterministic canonicalization) as
    evaluation criteria rather than as a pitch for any one project.
  • The external implementations, alphabetically, with an explicit note that Hermes endorses
    none of them.
  • The limit that applies to all of them: a hook cannot prove completeness, because an agent
    that never dispatches through it is never seen.

Related Issue

Refs #20034

Type of Change

  • 📝 Documentation update

Changes Made

  • docs/observability/audit-receipt-chains.md (new)
  • docs/observability/README.md: one paragraph in "Existing Consumers" pointing at it

Notes on duplicates and accuracy

Not a duplicate of #26808. That PR edited website/docs/user-guide/features/hooks.md to
expand the post_tool_call use-case list, and its author closed it as stale for lack of
maintainer follow-up rather than on merit. This PR targets the observability docs instead,
and covers the parts #20034 asked for that #26808 did not: the external projects, the
canonicalization requirement, and the anchoring gap. I deliberately did not re-do the
user-guide edit, so the two do not conflict if #26808 is ever revived.

Consistent with #49371. The draft opt-in execution receipts PR describes itself as
local-only and explicitly not a signing or proving feature, so the page frames it as the
substrate these plugins compose with rather than something they replace. Happy to reword if
that PR's scope shifts.

One inconsistency surfaced while writing this. The observer contract in
docs/observability/README.md documents status, error_type, error_message,
session_id, and tool_call_id on post_tool_call, while the callback signature in
website/docs/user-guide/features/hooks.md lists only tool_name, args, result,
task_id, and duration_ms. Rather than pick a winner in a docs PR, the page notes both
and tells plugin authors to read every field with kwargs.get(...). If you tell me which is
authoritative I will send a follow-up reconciling them.

Verification. Every claim is taken from this repo rather than assumed: the hook names
and return-value behavior from docs/observability/README.md, the payload shapes from
_DEFAULT_PAYLOADS in hermes_cli/hooks.py, the status vocabulary (ok, error,
blocked, cancelled) from the observer contract, and the registration pattern from
plugins/disk-cleanup/__init__.py. All relative links were checked to resolve.

Disclosure. I maintain Provenrail, one of the five listed implementations. I wrote the
page to be useful with that entry removed, listed everything alphabetically, described the
others from their own stated descriptions, and put the evaluation criteria in the reader's
hands rather than ranking anyone. Drop or reword the Provenrail line if you would prefer the
list came from someone with no stake in it. This PR was drafted with Claude Code and
reviewed by me before submission.

… seam

Adds docs/observability/audit-receipt-chains.md covering the three-hook pattern
(pre_tool_call, post_tool_call, transform_tool_result) for plugins that build
signed, tamper-evident records, the rules that differ from tracing, the external
anchoring property a self-verified hash chain does not provide, and the external
implementations. Cross-linked from the observer hooks reference.

Refs NousResearch#20034.
@alt-glitch alt-glitch added type/docs Documentation improvements comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have labels Jul 30, 2026
@x7peeps

x7peeps commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

🔍 Quality Review: This PR needs regression tests to meet our quality standards. Please add focused tests covering the bug fix before merging. Per our contribution rubric, tests should assert behavior contracts, not just snapshot values.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for turning the #487 outcome into focused plugin documentation. The hook-based direction matches current main, but two documentation details need correction.

Problems

  • docs/observability/audit-receipt-chains.md:65 describes block as the only intentional non-None pre_tool_call result. Current Hermes also supports {"action": "approve", ...} and escalates it to the human-approval gate (hermes_cli/plugins.py:2130-2193; website/docs/developer-guide/plugins/index.md:606-620).
  • The #487 maintainer closure requested a cross-link from the plugin development guide. This diff links only from docs/observability/README.md; the developer hook reference at website/docs/developer-guide/plugins/index.md:600-620 still does not expose the new page.

Suggested changes

  • Document both intentional pre_tool_call directives, while keeping None as the recorder behavior.
  • Add the requested developer-guide cross-link.

This is an automated hermes-sweeper review.

positional parameters will break on the next field Hermes adds.

**Never return a value from `pre_tool_call` unless you mean to block.** Returning
`{"action": "block", "message": "..."}` blocks the tool. A recorder must return `None`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please include the other intentional behavior-changing return: {"action": "approve", "message": "..."} escalates to Hermes's human-approval gate. Current pre_tool_call handling accepts both block and approve directives (hermes_cli/plugins.py:2130-2193), so “unless you mean to block” is incomplete.

@teknium1 teknium1 added the sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users label Jul 30, 2026
@pofky

pofky commented Aug 4, 2026

Copy link
Copy Markdown
Author

Happy to add tests if there is anything to assert here, but this PR is documentation only: it adds docs/observability/audit-receipt-chains.md and one cross-link from docs/observability/README.md. No source file, no behavior, and no bug fix is touched, so there is no contract for a regression test to pin.

Two things I can do instead if either would help:

  1. Add a doc example that is executed in CI (the three-hook pattern as a runnable snippet) so the payload fields the page documents cannot silently drift from pre_tool_call / post_tool_call / transform_tool_result.
  2. Add an assertion that the hook payload keys named on the page still exist, which would catch exactly the drift the page would otherwise be wrong about.

Say which you prefer and I will push it. Otherwise this is ready: it is the docs contribution @teknium1 scoped when closing #487, and it addresses the gap tracked in #20034.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users type/docs Documentation improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants