docs(observability): document the audit and receipt-chain integration seam - #74717
docs(observability): document the audit and receipt-chain integration seam#74717pofky wants to merge 1 commit into
Conversation
… seam Adds docs/observability/audit-receipt-chains.md covering the three-hook pattern (pre_tool_call, post_tool_call, transform_tool_result) for plugins that build signed, tamper-evident records, the rules that differ from tracing, the external anchoring property a self-verified hash chain does not provide, and the external implementations. Cross-linked from the observer hooks reference. Refs NousResearch#20034.
|
🔍 Quality Review: This PR needs regression tests to meet our quality standards. Please add focused tests covering the bug fix before merging. Per our contribution rubric, tests should assert behavior contracts, not just snapshot values. |
teknium1
left a comment
There was a problem hiding this comment.
Thanks for turning the #487 outcome into focused plugin documentation. The hook-based direction matches current main, but two documentation details need correction.
Problems
docs/observability/audit-receipt-chains.md:65describesblockas the only intentional non-Nonepre_tool_callresult. Current Hermes also supports{"action": "approve", ...}and escalates it to the human-approval gate (hermes_cli/plugins.py:2130-2193;website/docs/developer-guide/plugins/index.md:606-620).- The #487 maintainer closure requested a cross-link from the plugin development guide. This diff links only from
docs/observability/README.md; the developer hook reference atwebsite/docs/developer-guide/plugins/index.md:600-620still does not expose the new page.
Suggested changes
- Document both intentional
pre_tool_calldirectives, while keepingNoneas the recorder behavior. - Add the requested developer-guide cross-link.
This is an automated hermes-sweeper review.
| positional parameters will break on the next field Hermes adds. | ||
|
|
||
| **Never return a value from `pre_tool_call` unless you mean to block.** Returning | ||
| `{"action": "block", "message": "..."}` blocks the tool. A recorder must return `None`. |
There was a problem hiding this comment.
Please include the other intentional behavior-changing return: {"action": "approve", "message": "..."} escalates to Hermes's human-approval gate. Current pre_tool_call handling accepts both block and approve directives (hermes_cli/plugins.py:2130-2193), so “unless you mean to block” is incomplete.
|
Happy to add tests if there is anything to assert here, but this PR is documentation only: it adds Two things I can do instead if either would help:
Say which you prefer and I will push it. Otherwise this is ready: it is the docs contribution @teknium1 scoped when closing #487, and it addresses the gap tracked in #20034. |
What does this PR do?
Adds
docs/observability/audit-receipt-chains.md, documenting the integration seam forplugins that build signed, tamper-evident records of tool activity, and cross-links it from
the observer hooks reference.
This is the documentation @teknium1 scoped when closing #487: a page describing the
three-hook pattern, pointing at the external projects and the cross-implementation spec
work, so the receipt-chain use case is discoverable without any core changes.
The page covers:
pre_tool_callfor authorization,post_tool_callfor outcome,transform_tool_resultfor attaching a proof reference), with the payload fields areceipt actually needs.
**kwargs; never returna value from
pre_tool_callunless you mean to block; stay fail-open rather than leaningon Hermes catching your exception; do not silently drop entries, because an audit trail
with invisible gaps is worse than none.
@fernandosmither's objection in Feature: Cryptographic Audit Trail — SHA-256 Hash-Chained Action Log for Tamper-Proof Agent Accountability (inspired by OpenFang) #487: an operator with write access can fork and rewrite
the chain and internal verification still passes. The page states the three things that
close that gap (signatures, an external anchor, deterministic canonicalization) as
evaluation criteria rather than as a pitch for any one project.
none of them.
that never dispatches through it is never seen.
Related Issue
Refs #20034
Type of Change
Changes Made
docs/observability/audit-receipt-chains.md(new)docs/observability/README.md: one paragraph in "Existing Consumers" pointing at itNotes on duplicates and accuracy
Not a duplicate of #26808. That PR edited
website/docs/user-guide/features/hooks.mdtoexpand the
post_tool_calluse-case list, and its author closed it as stale for lack ofmaintainer follow-up rather than on merit. This PR targets the observability docs instead,
and covers the parts #20034 asked for that #26808 did not: the external projects, the
canonicalization requirement, and the anchoring gap. I deliberately did not re-do the
user-guide edit, so the two do not conflict if #26808 is ever revived.
Consistent with #49371. The draft opt-in execution receipts PR describes itself as
local-only and explicitly not a signing or proving feature, so the page frames it as the
substrate these plugins compose with rather than something they replace. Happy to reword if
that PR's scope shifts.
One inconsistency surfaced while writing this. The observer contract in
docs/observability/README.mddocumentsstatus,error_type,error_message,session_id, andtool_call_idonpost_tool_call, while the callback signature inwebsite/docs/user-guide/features/hooks.mdlists onlytool_name,args,result,task_id, andduration_ms. Rather than pick a winner in a docs PR, the page notes bothand tells plugin authors to read every field with
kwargs.get(...). If you tell me which isauthoritative I will send a follow-up reconciling them.
Verification. Every claim is taken from this repo rather than assumed: the hook names
and return-value behavior from
docs/observability/README.md, the payload shapes from_DEFAULT_PAYLOADSinhermes_cli/hooks.py, thestatusvocabulary (ok,error,blocked,cancelled) from the observer contract, and the registration pattern fromplugins/disk-cleanup/__init__.py. All relative links were checked to resolve.Disclosure. I maintain Provenrail, one of the five listed implementations. I wrote the
page to be useful with that entry removed, listed everything alphabetically, described the
others from their own stated descriptions, and put the evaluation criteria in the reader's
hands rather than ranking anyone. Drop or reword the Provenrail line if you would prefer the
list came from someone with no stake in it. This PR was drafted with Claude Code and
reviewed by me before submission.